Model partitioning

Overview

Model partitioning means running one trained network on several devices at once, with each device holding only part of the weights and computing only part of the outputs. The reason to do it is memory. A single device may have enough flash for the weights but not enough working RAM for the intermediate activations of one layer 2605.09357 §1. Cutting between layers does not help when one layer alone is larger than a device's RAM 2605.09357 §1, Challenge 1.

The one source in this set cuts finer than a layer. It splits each convolution by output position and each linear layer by output column 2605.09357 §4.2. Workers receive their weight fragments once at deployment. A coordinator then sends each worker the inputs it needs for every layer and collects the partial outputs [arxiv:2605.09357 §3, §4.4]. No worker holds the full activations or weights of a layer 2605.09357 §4.4.

The cost is communication. Every intermediate result passes through the coordinator, so each activation crosses the network at least twice 2605.09357 §6.2. On the testbed, computation time fell from 15.37 s on three boards to 7.07 s on eight, while communication time rose from 27.60 s to 49.82 s 2605.09357 §7.2.3.

Variants

Rating-weighted split across microcontrollers

Each worker gets a share of every layer that is proportional to a capability rating. The rating combines clock frequency, per-kilobyte communication delay, bandwidth and a communication coefficient 2605.09357 §5.1, eq. 5. Convolution output positions are cut into contiguous runs, and linear-layer columns into contiguous runs, one run per worker 2605.09357 §4.2, alg. 1, alg. 2. Weight fragments are sized the same way, with an overflow rule for workers whose storage is too small 2605.09357 §5.3, eq. 6–7. Activations are routed between consecutive layers by a map that records which inputs each worker needs 2605.09357 §4.3, alg. 3. The paper deploys this on eight Teensy 4.1 boards, each with a 600 MHz Cortex-M7 [arxiv:2605.09357 §6.1]. Card: Split CNN Inference on Networked Microcontrollers.

Other partitioning schemes named in the paper for GPUs and phones assume each worker can hold its sub-matrix or all channels of a feature map 2605.09357 §2. This variant is aimed at the case where it cannot. The paper gives no number for a single board, since one board cannot hold the model 2605.09357 §7.1, §7.2.1.

Even split as a baseline

The same paper uses an even split, where every worker gets the same share regardless of speed, and a frequency-only split as baselines 2605.09357 §7.2.1. These are the simplest versions of the variant above, and the paper's comparison in tab. 2 is the reference for how much the rating adds 2605.09357 tab. 2.

When to use which

This set has one paper, so only the rating-weighted variant is covered. The comparison with an even split rests on that paper's numbers alone. A reader should not expect the same gains on other boards or links.

Implementation sketch

The common core is to assign each worker a contiguous run of output positions, sized by its rating 2605.09357 alg. 1. The linear case is the same with columns in place of positions 2605.09357 alg. 2.

# Assign output positions of a conv layer to workers  [alg. 1]
# Input: output shape (c, h, w), ratings R, number of workers N
def split_conv(c, h, w, ratings, n_workers):
    start = 0
    for r in range(n_workers):
        # Quota of positions for worker r, proportional to its rating
        quota = ratings[r] / sum(ratings) * c * h * w
        i = start
        while i - start < quota:
            # Output channel that position i belongs to
            kernel = i // (h * w)
            if kernel not yet on worker r:
                send kernel weights to worker r
            else:
                increment usage count of kernel on worker r
            i = i + 1
        start = i

Notes on where the variant differs from the core:

• The released code rounds the per-unit size up, while alg. 1 uses the unrounded quota 2605.09357 §4.2, code: pc_code/Algorithms/src/lib/operations.rs#L8.
• The rating in eq. 5 needs values for the communication coefficient, delay and bandwidth that the paper does not give 2605.09357 §5.2.

Contradictions

• RAM budget of one board. The paper gives the Teensy 4.1 as having 1 MB of on-chip RAM 2605.09357 §6.1. Its evaluation calls 512 KB the typical single-MCU RAM 2605.09357 §7.2.1. The text does not say which budget the per-layer RAM check uses.
• Optimized versus frequency-only. In case 2 the frequency-only split is faster (12.40 s) than the rating-weighted one (12.52 s), while in case 5 the frequency-only split is slower than an even split 2605.09357 tab. 2. The paper's text presents the rating as the better method, so these rows show it is not better in every case.
• Rating formula in the released code. The paper's rating does not appear in the released source. The simulation gives every worker an equal portion 2605.09357 §5.2, code: pc_code/Simulation/src/main.rs#L20. Table 2's "Optimized" column therefore cannot be checked from the code alone.
• Input size. The paper evaluates 112 × 112 × 3 inputs 2605.09357 §7.1. The released code crops to 224 × 224 util.rs.

Papers

• Split CNN Inference on Networked Microcontrollers: splits MobileNetV2 across up to eight Teensy 4.1 boards by output position and weight column, with rating-weighted shares. Reports latency and RAM on the testbed, with no accuracy numbers.