Split CNN Inference on Networked Microcontrollers

In plain words

Microcontrollers have only tens to hundreds of kilobytes of working memory, and the middle layers of a neural network can hold far more than that at once. This paper stops treating each layer as one indivisible job. The model is cut into small pieces, down to single output positions of a convolution and single columns of the final linear layer. Each small board stores only its share of the weights and computes only its share of the outputs. A PC acts as the conductor: it sends each board the inputs it needs and collects the results. Boards with more capacity get larger shares. The price is network traffic. With more boards, each one computes less, but more data moves back and forth, so the total time grows as boards are added.

TL;DR

Splits one CNN across several Teensy 4.1 boards, down to single output positions and weight columns, so that no board has to hold a whole layer. Work is shared by a capability rating built from clock speed, communication delay and bandwidth eq. 5; that rating formula does not appear in the released code. Author-reported: MobileNetV2 at 112×112 input runs on 3 boards, and total latency grows from 42.97 s to 56.89 s from 3 to 8 boards, mostly communication. No accuracy numbers are reported.

Problem

• On a single microcontroller, peak RAM used by intermediate activations and buffers often blocks inference even when the weights fit in flash. Typical MCUs have tens to hundreds of KB of RAM §1.
• Layer-boundary splitting does not help when one layer alone exceeds a board's RAM §1, Challenge 1.
• Existing partitioning schemes for GPUs and phones (COTS, MoDNN) assume that each worker can hold its sub-matrix or all channels of its feature maps. For example, MoDNN's workers must hold all kernels, and the paper estimates about 34 KB of kernel weights for MobileNetV2 layers with 960 channels §2.
• Single-board MobileNetV2 is infeasible on the Teensy 4.1 testbed §7.1. The paper gives two different RAM figures for that board, see Pitfalls.

Method

Setting. A PC is the coordinator; up to eight MCUs are workers. The coordinator does global orchestration, neuron mapping, activation routing and synchronisation. Workers store model fragments and compute locally §3. The pipeline has three phases: offline preprocessing (the model is reinterpreted and split into weight fragments), deployment initialisation (each worker receives its fragments in flash and reports its parameters), and layer-by-layer split execution §3.

1. Model reinterpretation. The pre-trained model is traced and turned into an internal representation with explicit dependencies. For each convolution output, the receptive field is derived; for a linear layer, the mapping is recorded directly. The metadata and weights are serialised for deployment §4.1, fig. 3.

2. Splitting within a layer. Partitioning is done per output neuron, so that both weights and activations can be distributed §4.2.

• Convolution: the output positions are cut into contiguous runs, one run per worker, with run lengths proportional to the ratings. Each run is then turned into per-kernel work §4.2, alg. 1, fig. 4. A kernel's weights are sent to a worker when first needed there; later uses only increase a reference count alg. 1.
• Linear: the weight matrix is cut into column runs, one per worker, again proportional to the ratings. Each column is one output neuron [§4.2, alg. 2, fig. 5].

# Assign output positions of a conv layer to workers  [alg. 1]
# Input: output shape (c, h, w), ratings R, number of workers N
def split_conv(c, h, w, ratings, n_workers):
    start = 0
    for r in range(n_workers):
        # Quota of output positions for worker r  [§4.2]
        quota = ratings[r] / sum(ratings) * c * h * w
        i = start
        while i - start < quota:
            # Kernel (output channel) that position i belongs to
            kernel = i // (h * w)
            if kernel not yet on worker r:
                send kernel weights to worker r
            else:
                increment usage count of kernel on worker r
            i = i + 1
        start = i

# Assign weight columns of a linear layer to workers  [alg. 2]
# W has shape (h, w); one column per output neuron
def split_linear(w, ratings, n_workers):
    start = 0
    for r in range(n_workers):
        # Number of columns for worker r  [§4.2]
        quota = ratings[r] / sum(ratings) * w
        i = start
        while i - start < quota:
            assign column W[:, i] to worker r
            i = i + 1
        start = i

3. Cross-layer activation mapping. The coordinator builds two maps for each pair of consecutive layers §4.3, alg. 3, fig. 6.

• The assignment map records, for each input activation of layer i, which workers of layer i+1 need it. Stage 1 walks the output positions of layer i+1 in the same rating-proportional way, traces each one's receptive field, and marks the inputs it needs alg. 3, lines 4–20 of the algorithm.
• The routing map is derived from the assignment map: for each activation of layer i, it lists which worker produces it and which downstream workers need it alg. 3, lines 21–34. For a linear layer, every input counts as needed by every worker §4.3.

4. Execution. Layer by layer, the coordinator uses the routing map to send each worker its inputs. Workers compute their assigned neurons in parallel and return partial outputs. The coordinator assembles them into the next layer's input §4.4, alg. 4. The paper's current system always routes through the coordinator; direct MCU-to-MCU transfer is left for future work §6.2. Tasks are scheduled round-robin to overlap communication with computation §6.2. The paper states that no worker ever holds the full activations or weights of a layer §4.4.

5. Capability ratings. Each worker has a clock frequency f (MHz), a communication delay per KB d, a bandwidth B (KB/s) and a communication coefficient K_c that counts the data sent per unit of work. A constant K_1 converts million cycles to KB of output §5.1. The time for a worker with workload W (million cycles) is

t = \frac{W}{f} + \left(d + \frac{1}{B}\right) f(W) eq. 1

f(W) = K_1 K_c W eq. 2

Setting t = 1 s and solving gives the data volume a worker can process per second, which defines the rating §5.2, eq. 3–4:

R_i = \frac{f_i K_1}{\left(d_i + \frac{1}{B_i}\right) f_i K_1 K_{c_i} + 1} eq. 5

K_c depends on the CNN and the split and "can be estimated through profiling or simulation" §5.2.

6. Rating-based allocation. A worker's weight fragment is proportional to its rating, S_i = R_i S_m / Σ R_j, where S_m is the total model size [§5.3, eq. 6]. If a fragment exceeds the worker's storage limit S_it, an overflow rating R_io = (S_i − S_it) Σ R_j / S_m is computed §5.3, eq. 7. The overflow is spread evenly over workers with spare storage, and this repeats until every fragment fits §5.3.

# Fragment share of one worker  [eq. 6]
def fragment_size(rating, ratings, total_size):
    return rating * total_size / sum(ratings)


# Rating to move off a worker whose fragment exceeds its storage  [eq. 7]
def overflow_rating(size, storage, ratings, total_size):
    # Not stated: how this rating is reassigned in each pass; the text
    # says only that the adjustment repeats until every fragment fits [§5.3]
    return (size - storage) * sum(ratings) / total_size

7. Other optimisations. Convolution, batch normalisation and ReLU are fused into one operation by folding the BatchNorm parameters into the convolution §5.4. All parameters and activations are quantised from 32-bit float to 8-bit integers §5.4.

Implementation. Teensy 4.1 boards (ARM Cortex-M7, 600 MHz, 1 MB RAM, 8 MB flash) are connected to a gigabit switch. Fragments are stored as JSON on LittleFS. Communication uses TCP with explicit acknowledgements and packets of up to 1400 bytes §6.1–6.2. A deployment tool flashes boards and distributes fragments §6.3.

Hyperparameters

K_1 by setting (author-reported, tab. 1):

Results

Author-reported. Setup: Teensy 4.1 testbed (3 to 8 boards), MobileNetV2 at 112 × 112 × 3 §7.1. The baselines are an even split and a frequency-only split; a single board is infeasible and no number is given for it §7.2.1.

Rating-based allocation, one inference on 3 boards tab. 2, §7.2.2:

Latency split into computation and communication, one inference [§7.2.3, fig. 9]:

Other reported results:

• Per-layer peak RAM on 3 boards stays within the per-board budget. The budget value is not given in the text §7.2.1, fig. 8.
• Communication time per layer grows with the number of boards and is concentrated in the early layers §7.3.1, fig. 10.
• Simulated per-board peak memory falls several-fold as boards are added. The gains flatten beyond roughly 10 to 20 boards; the simulation covers up to 120 workers §7.4, fig. 12. No exact numbers are in the text.
• The total data moved in one pass is about 4.21 MB across three workers, with about 480 KB per worker in the most communication-heavy layer §6.2.

Paper vs code

Code: https://github.com/shashsuresh/split-inference-on-MCUs at a87238bd1904183142a967fd89763e8d8b6ea276. The repository's README describes it as code for the first author's master's thesis README.md. It has no LICENSE file.

• The capability rating is not in the code. The paper's rating (eq. 5), K_1, K_c, delay and bandwidth §5.1–5.2 do not appear in the Rust or MCU sources. The simulation sets every worker's portion to 1, so work is split evenly main.rs. The split uses integer portions operations.rs. The "Optimized" column of Table 2 therefore cannot be reproduced from this code alone.
• Quota rounding. Paper alg. 1 uses an unrounded quota n = R/ΣR · c·h·w §4.2. The code rounds the per-unit size up: per_1 = ceil(output_count / Σ portions) operations.rs.
• Linear split. The code splits linear layers by output column, as the paper says operations.rs.
• Input size. The paper uses 112 × 112 × 3 §7.1. The code crops images to 224 × 224 util.rs, and the simulation uses (3, 224, 224) main.rs. I did not confirm the 112 × 112 setting in the MCU code.
• Quantisation. Weights use a per-layer min/max range with scale = range/255 quant.rs. Activation scales are smoothed with an exponential moving average, 0.99 old and 0.01 new quant.rs. The paper does not describe this calibration.
• BatchNorm folding. The code folds BatchNorm into the convolution with a variance epsilon of 1e-5 merge.rs. The paper gives no epsilon.
• ReLU6. The repository's to-do list includes replacing ReLU6 with ReLU and naming post-training quantisation todolist. The paper does not mention ReLU6.
• Transport. The MCU side uses an Ethernet client and a 1400-byte message size [code: MCU_code/PlatformIO_code/worker_code/include/communication.h#L9, #L41], consistent with §6.2. I did not check the acknowledgement logic.
• Testbed config. The testbed.json in the repository lists four entries with ids 0 to 3 testbed.json. The paper reports eight boards.
• Accuracy. The conclusion says the optimisations preserve accuracy §9. The paper reports no accuracy measurement for the quantised, split model.

No discrepancies were found for the layer-to-layer routing logic. I did not review it line by line.

Pitfalls

• The rating is not reproducible from the code. To implement eq. 5 to 7, take the formulas from the paper. Then choose values for K_c, d and B yourself, since the paper gives none. Table 2's "Optimized" numbers cannot be checked against this repository.
• K_1 is a measured constant that varies with setting. It ranges from 0.127 to 0.228 KB/MCycle across workloads and frequencies tab. 1. Use the value for your frequency and workload, not one global constant.
• Overflow handling is underspecified. The paper says the overflow rating is spread "evenly" and the step repeats §5.3. It does not say whether that rating is subtracted from the board that overflowed, nor when the loop stops.
• Communication dominates at scale. Computation falls from 15.37 s to 7.07 s, but communication rises from 27.60 s to 49.82 s from 3 to 8 boards §7.2.3. Every intermediate result goes through the coordinator §6.2, so each activation crosses the network at least twice (worker to coordinator and back out). The paper does not report the cost of that extra hop separately.
• Frequency-only can lose to an even split. In case 5 of Table 2, the frequency-only split takes 33.01 s, slower than the even split at 32.81 s tab. 2. Case 2 also favours frequency-only over the rating (12.40 s vs 12.52 s).
• Injected delays are emulated, not measured. The Table 2 delays are inserted before send and receive operations §7.2.2. They do not reproduce a real wireless link.
• The RAM budget is inconsistent. The Teensy 4.1 is said to have 1 MB of on-chip RAM §6.1. The evaluation calls "512 KB on-chip RAM of a single MCU" the typical budget §7.2.1. Which budget the Figure 8 check uses is not stated.
• Peak RAM is sampled, not continuously measured. The probe samples every 1 ms and at layer boundaries §7.1. A peak shorter than 1 ms between samples would be missed, except at layer boundaries.
• Simulation is not hardware. The 120-worker results come from a thread simulator that "preserves the same partitioning and communication logic" §7.4. Treat them as a model, not a measurement.

Lineage

• Builds on: MobileNetV2, the evaluation model §6. The quantisation scheme follows arXiv:1712.05877, as cited in the repository README README.md. BatchNorm folding follows a public write-up linked in the code merge.rs.
• Compared against: COTS and MoDNN for GPU and smartphone partitioning §2; Neurosurgeon and CoAct for edge-cloud splitting §8.
• Superseded by: not known.