microYOLO: Towards Single-Shot Object Detection on Microcontrollers

In plain words

Object detectors find things in a picture and draw a box around each one. The usual fast detectors need a sizable graphics chip. microYOLO asks whether a tiny computer chip with no graphics hardware, the kind found in small cameras and sensors, can run such a detector at all.

To make it fit, the authors shrink the picture to a small square, build a much smaller network, and then trim and compress the trained model so it fits in the chip's memory. The result is a slow but working detector: a few frames per second. The paper is a work in progress. Its accuracy depends strongly on the task, and it is clearest on a simple task with few objects in view.

TL;DR

Single-shot detection (YOLO style) on a Cortex-M7 microcontroller, with no hardware accelerator. The model takes 128x128 RGB images, is pruned during the last part of training, and is quantized to 8 bits afterwards. The authors report about 3.5 FPS, under 800 KB Flash and under 350 KB RAM on an OpenMV H7 R2. The accuracy is modest and task-dependent, and this is a work in progress §1 §3.1.

Problem

Single-shot detectors are efficient, but deployments so far target larger embedded boards such as the Jetson Xavier. The authors target microcontrollers with roughly 1 to 2 MB of Flash and under 1 MB of RAM, which they treat as typical limits §1.

Assumptions: input is lowered to 128x128, the backbone is small, and the platform is a Cortex-M7 at 480 MHz §1 §2 §3.3.

Method

The network follows YOLO's layout: a CNN backbone, then a detection head that predicts on an S x S grid, with B boxes per cell and N class scores per cell §2. Changes relative to the original YOLO:

1. Input: 3 x 128 x 128 RGB tab. 1 §2.
2. Backbone: one standard convolution (3 to 64 channels, kernel 4, stride 2), then seven depthwise-separable convolutions, with max-pooling after the first convolution and after the second and third groups tab. 1 §2. Depthwise-separable convolutions are used to keep the parameter count low §2.
3. Head: two linear layers (1024 to 1024, then 1024 to S x S x N + B x 5), with batch normalization instead of dropout before the last linear layer tab. 1 §2.
4. Grid and boxes: S = 5 and B = 2 were chosen as the best trade-off for the target platform and resolution §2.
5. Pre-training: the backbone is pre-trained on Caltech-256 classification (top-1 about 38%, top-5 about 61%) §2.
6. Pruning: all convolutional and linear layers are pruned with an iterative gradual schedule and an L1-norm criterion §2. In the detection runs it is applied during the last 100 epochs, every 20 epochs §3.
7. Quantization: 8-bit post-training quantization. Quantization-aware training was tried, not used, because the authors saw no benefit for their application §2.
8. Deployment: an existing compression and C-code pipeline turns the trained model into code for the microcontroller §2 §3.3.

The three detection tasks are: humans (1 class), vehicles (5 classes: truck, bus, car, bicycle, motorcycle) derived from COCO, and mini-fridge items (4 classes: water, milk, chocolate milk, orange juice) on a self-recorded dataset §3 fig. 3.

# Detection head forward pass  [§2] [tab. 1]
# Input:  images (B, 3, 128, 128)
# Output: raw predictions (B, S*S*N + 5*B)   # see Pitfalls

def forward(images):
    # 1. Backbone: one convolution, then seven depthwise-separable  [tab. 1]
    features = backbone(images)                 # (B, 64, 4, 4)
    # 2. Flatten the feature map for the linear layers  [tab. 1]
    flat = flatten(features)                    # (B, 1024)
    # 3. First linear layer with ReLU  [§2] [tab. 1]
    hidden = relu(linear_1024(flat))            # (B, 1024)
    # 4. Batch norm replaces dropout before the last layer  [§2]
    hidden = batch_norm(hidden)                 # (B, 1024)
    # 5. Output layer  [tab. 1]
    return linear_out(hidden)                   # (B, S*S*N + 5*B)

# Training recipe  [§3] [§2]
# Input:  training set with per-image (class, box) lists
# Output: pruned model, then 8-bit model

def train(model, loader):
    # 1. Optimizer: SGD, lr 0.001, momentum 0.9, weight decay 0.005  [§3]
    optimizer = sgd(lr=0.001, momentum=0.9, weight_decay=0.005)
    for epoch in range(400):                    # 400 epochs  [§3]
        for images, targets in loader:
            # 2. Affine, brightness, saturation, hue augmentation  [§3]
            images = augment(images)
            # 3. Drop boxes smaller than 12x12 px  [§3]
            targets = drop_small_boxes(targets, min_px=12)
            loss = detection_loss(model(images), targets)  # not stated
            update(model, loss, optimizer)
        # 4. Iterative L1 pruning: last 100 epochs, every 20  [§2] [§3]
        if epoch >= 300 and (epoch - 300) % 20 == 0:
            prune_l1(model)                      # sparsity target not stated
    return model

# 5. After training: 8-bit post-training quantization, no QAT  [§2]

Hyperparameters

Variant for larger inputs in the resolution study: first-conv kernel 7, and padding 2 in the next two depthwise-separable convolutions. For smaller inputs, fewer neurons in the linear layers §3.1.

Results

Author-reported. Setup: validation sets of three tasks, 400 epochs, 3 seeds, deployed on OpenMV H7 R2 §3 §3.1 §3.3.

The fridge task loses mAP50 during the pruning epochs, but quantization does not cost much §3.1. The paper's error analysis reports false negatives as the most common error type, most pronounced on the unrestricted vehicles task §3.2 fig. 4.

Paper vs code

No official code. The paper text, abstract and bibliography source contain no code link. Searches on GitHub for "microYOLO", "muYOLO", "microyolo microcontroller object detection" and "yolo openmv deutel" found one unrelated third-party repository, which this card does not use. The first author's GitHub account contains no related repository. The deployment pipeline cited §2 was not searched, because it is a separate work.

Pitfalls

• Output-size formula is ambiguous. The paper writes the output as "S x S x N + B x 5" §2 tab. 1. Read literally that is 25N + 10 for S = 5 and B = 2, which is not the usual YOLO layout of S x S x (N + 5B). With no code, an implementer should not assume one reading; state which one is used before coding.
• Class count per task is taken from the task lists §3 and the figure legends fig. 3. Output sizes differ across tasks, which the authors name as the reason for the small Flash difference in tab. 2 §3.3.
• Pooling stride is not stated. Stride 2 is consistent with the linear-layer input of 1024 = 64 x 4 x 4 tab. 1. This is derived, not stated.
• Pruning schedule is partly ambiguous. The text says pruning occurs during the last 100 epochs, every 20 §3. Whether that means epochs 300, 320, ... (0-indexed) or 301, 321, ... is not stated. Target sparsity and the pruning step size are not stated §2.
• Loss, batch size, LR schedule, NMS and the confidence threshold for inference are not stated. The 50% value is used for visualisation and error analysis only §3.2 fig. 3.
• Resolution-dependent architecture. Larger inputs need a different first kernel and padding; smaller inputs need fewer linear neurons §3.1. The main deployment numbers use 128 x 128 tab. 2.
• Quantization and pruning cost. Pruning costs accuracy on the fridge task during training §3.1. Aggressive pruning gives up to 8 FPS, but with significantly lower mAP50 §3.3.
• Small-object drop. Boxes smaller than 12 x 12 px are dropped from the ground truth §3. Objects below that size are invisible to training.
• Results are work in progress. Three seeds per task, no baseline detector reported, and different datasets per task. The simplified vehicles task restricts ground truth to the three largest boxes per image §3.1, so its numbers are not comparable to the unrestricted task.
• Abstract and table values differ slightly. The abstract and conclusion say "about 3.5 FPS" §1 §4; tab. 2 lists 3.45 to 3.49. Flash is "less than 800 KB" §1; tab. 2 lists 706 to 771 KB.
• Figure and table numbers. Numbers in this card follow the order of the LaTeX labels in the source, because no compiled PDF was in the source archive. Check them against the published PDF before citing.

Lineage

• Builds on: YOLO [redmon2016yolo]; depthwise-separable convolutions [chollet2017xception]; iterative gradual pruning with L1 norm [zhu2017prune]; the compression and C-code deployment pipeline [deutel2022deployment] §2.
• Compared against: no baseline detector is reported. Earlier object detectors are discussed in the introduction only §1.
• Superseded by: not known from content in this wiki.