Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks

In plain words

Designers of "fast" neural networks usually count arithmetic: fewer multiply-adds (FLOPs) should mean a faster model. This paper points out that wall-clock time is FLOPs divided by how many operations per second the hardware actually achieves (FLOPS). Many popular lightweight building blocks, especially the depthwise convolution, do very little arithmetic per byte they move. The chip spends its time waiting on memory, so cutting FLOPs often barely cuts latency.

The fix is a block called partial convolution. Instead of filtering every channel of a feature map, it runs an ordinary convolution on only a quarter of the channels and leaves the other three quarters untouched. A cheap 1x1 convolution afterwards mixes information across all channels. Think of it as sending only a few scouts to look around while everyone else waits, then holding a meeting where everyone shares notes. The idea relies on different channels carrying a lot of the same information.

Stacking these blocks gives FasterNet, a family of plain convolutional networks. In the authors' measurements, FasterNet runs faster than mobile networks, vision transformers and MLP models of similar accuracy on a GPU, a desktop CPU and an ARM chip. The smallest variant beats MobileViT-XXS in both speed and accuracy, and the largest matches Swin-B's accuracy with higher GPU throughput. The lesson for engineers: measure achieved speed on the target device, not just operation counts.

TL;DR

Latency = FLOPs / FLOPS eq. 1, and depthwise/group convolutions get low FLOPS because they are memory-bound §3.1, tab. 1. PConv applies a regular k\times k conv to the first (or last) c_p = c/4 channels and copies the rest through, giving 1/16 the FLOPs and about 1/4 the memory access of a full conv §3.2, eq. 4, eq. 5. A FasterNet block is PConv → 1×1 conv (expand ×2) → BN → activation → 1×1 conv, with a residual shortcut and no norm/activation elsewhere in the block §3.4, tab. 8. Six variants (T0, T1, T2, S, M, L) span 0.34 to 15.5 GFLOPs and 71.9% to 83.5% ImageNet-1k top-1 tab. 3. Read it if you design or deploy CNN backbones where measured latency, not FLOPs, is the target.

Problem

Fast-network work such as MobileNets, ShuffleNets and GhostNet lowers FLOPs mainly through depthwise (DWConv) or group convolution (GConv), plus extra data manipulation (concatenation, shuffling, pooling) §1. Mobile ViTs combine DWConv with modified attention and costly normalization and activation layers §1. These reductions often do not turn into lower latency because the achieved FLOPS drops. For example, CycleMLP-B1 has half the FLOPs of ResNet50 yet runs slower on CPU, 116.1 ms vs 73.0 ms §1, tab. 3. On an Intel CPU, many such networks reach lower FLOPS than ResNet50 §1, fig. 2.

The authors trace the low FLOPS to frequent memory access §1, §3.1. DWConv has only h \times w \times k^2 \times c FLOPs, but to recover accuracy networks widen it to c' > c channels (×6 in MobileNetV2 inverted residuals), which raises memory access above that of a regular conv §3.1, eq. 2, eq. 3.

Method

Notation. Input and output \mathbf{I}, \mathbf{O} \in \mathbb{R}^{c \times h \times w}, kernel size k, partial channels c_p, partial ratio r = c_p / c §3.1, §3.2. FLOPs here means the number of multiply-adds §1, footnote.

Why DWConv is slow §3.1. Memory access for a widened DWConv and for a regular Conv:

h \times w \times 2c' + k^2 \times c' \approx h \times w \times 2c' eq. 2

h \times w \times 2c + k^2 \times c^2 \approx h \times w \times 2c eq. 3

The h \times w \times 2c' term is input/output I/O, which the authors consider already minimal and hard to reduce §3.1.

Partial convolution (PConv) §3.2, fig. 1. Feature maps are highly redundant across channels (visualized for an intermediate ResNet50 layer) §3.2, fig. 3. PConv applies a regular Conv to a contiguous block of c_p channels, the first or the last ones, chosen for contiguous memory access. The remaining c - c_p channels are kept unchanged, not dropped, because the following pointwise conv needs them §3.2. Input and output channel counts are equal §3.2.

\text{FLOPs}_{\text{PConv}} = h \times w \times k^2 \times c_p^2 eq. 4

\text{MemAccess}_{\text{PConv}} = h \times w \times 2c_p + k^2 \times c_p^2 \approx h \times w \times 2c_p eq. 5

With r = 1/4 that is 1/16 of a regular Conv's FLOPs and 1/4 of its memory access §3.2.

PConv + PWConv ≈ T-shaped Conv §3.3, fig. 5. A PConv followed by a 1×1 pointwise conv (PWConv) has an effective receptive field shaped like a "T": a full k\times k window on c_p channels plus the centre position on all channels. To justify the extra weight on the centre, the authors compute the per-position Frobenius norm of each 3\times3 filter \mathbf{F} \in \mathbb{R}^{k^2 \times c}, \|\mathbf{F}_i\| = \sqrt{\sum_{j=1}^{c} |f_{ij}|^2}, take the argmax position as "salient", and histogram it over a pre-trained ResNet18. The centre (position 5) is most often salient in all four stages §3.3, fig. 6. Implementing the T-shape as PConv + PWConv is cheaper than a direct T-shaped conv:

\text{FLOPs}_{\text{T-shaped}} = h \times w \times \left(k^2 \times c_p \times c + c \times (c - c_p)\right) eq. 6

\text{FLOPs}_{\text{PConv+PWConv}} = h \times w \times \left(k^2 \times c_p^2 + c^2\right) eq. 7

Eq. 6 exceeds eq. 7 when (k^2 - 1)c > k^2 c_p, e.g. c_p = c/4, k = 3 §3.3.

FasterNet architecture §3.4, fig. 4, tab. 8.

Algorithm: FasterNet forward (classification)   [§3.4, fig. 4, tab. 8]
Input: image x (3 × h × w); variant gives width c, depths (b1..b4), activation Acti
1. Embedding: Conv 4×4, stride 4, c output channels; BN           -> c   × h/4  × w/4
2. Stage 1: b1 FasterNet blocks at width c
3. Merging: Conv 2×2, stride 2, 2c output channels; BN            -> 2c  × h/8  × w/8
4. Stage 2: b2 blocks at width 2c
5. Merging: Conv 2×2, stride 2, 4c; BN                            -> 4c  × h/16 × w/16
6. Stage 3: b3 blocks at width 4c
7. Merging: Conv 2×2, stride 2, 8c; BN                            -> 8c  × h/32 × w/32
8. Stage 4: b4 blocks at width 8c
9. Classifier: global average pool -> Conv 1×1 to 1280 -> Acti -> FC to 1000
Output: logits

FasterNet block (width d)   [§3.4, tab. 8]
  y = PConv 3×3, stride 1, r = 1/4 on x     # no norm / activation
  y = Conv 1×1, d -> 2d
  y = BN(y); y = Acti(y)                    # the block's only norm + activation
  y = Conv 1×1, 2d -> d
  return x + y                              # shortcut reuses the input

Design choices §3.4:

• More blocks go into stages 3 and 4, because blocks there use less memory access and reach higher FLOPS §3.4, tab. 1.
• Norm and activation appear only after the middle PWConv. The stated reasons are feature diversity and lower latency §3.4. The appendix adds that removing activations between PConv and PWConv keeps the pair closer to a regular Conv app. E.
• BatchNorm is used instead of LayerNorm, GroupNorm or InstanceNorm because BN can be folded into the adjacent conv at inference §3.4. All benchmarks fold BN where applicable §4.
• Activation: GELU for T0/T1, ReLU for T2/S/M/L, chosen empirically §3.4, §4.5, tab. 8.
• Downstream detection drops the classifier app. D.

Variant configurations tab. 8 (FLOPs at 224×224):

PConv approximation experiment §4.2. Four datasets are built by running ImageNet-1k val images through a pre-trained ResNet50 and recording the feature maps before and after the first 3×3 conv of each stage. Each is split 70/10/20 into train/val/test. A PConv + PWConv network is trained with MSE loss to map input to output, and DWConv + PWConv and GConv + PWConv are trained the same way for comparison §4.2. Optimizer and epochs for this experiment: not stated.

Hyperparameters

ImageNet-1k training §4.3, tab. 6:

COCO detection / instance segmentation with Mask R-CNN §4.4, app. B, tab. 7. Settings follow PoolFormer without further tuning §4.4.

Benchmark protocol §4: GPU = RTX 2080Ti; CPU = Intel i9-9900X, single thread; ARM = Cortex-A72, single thread. Latency uses batch size 1 and throughput batch size 32. BN is folded at inference. Software stack (framework, runtime, precision): not stated.

Results

All numbers are author-reported.

Operator FLOPS §4.1, tab. 1. 10 stacked layers of each operator; FLOPs are the total for the 10 layers. Only the averages over the four feature-map sizes are shown here.

Full per-size table tab. 1 (GPU throughput fps / CPU latency ms / ARM latency ms):

PConv reaches 10.5×, 6.2× and 22.8× the FLOPS of DWConv on GPU, CPU and ARM §4.1.

Approximating a regular 3×3 conv (test MSE on ResNet50 feature maps) §4.2, tab. 2:

ImageNet-1k §4.3, tab. 3. 224×224 except MobileViT and EdgeNeXt at 256×256; OOM = out of memory.

Headline claims §4.3: against MobileViT-XXS, FasterNet-T0 gains 2.9 points of accuracy and a speedup of 2.8× (GPU), 3.3× (CPU) and 2.4× (ARM). FasterNet-L matches Swin-B at 83.5%, with 36% higher GPU throughput and 37% less CPU time than Swin-B. Against ConvNeXt-B (83.8%) the margins are 28% and 15%. Full accuracy–throughput/latency curves are in fig. 7 and fig. 8.

Baseline caveat app. A: ShuffleNetV2, MobileNetV2 and GhostNet were trained without the advanced training techniques FasterNet uses, so the authors call the comparison against them "not totally fair".

COCO, Mask R-CNN 1× §4.4, tab. 4. FLOPs at 1280×800.

FasterNet-L cuts PVT-Large's GPU latency by 38% (152.2 → 93.8 ms) with +1.1 box AP and +0.4 mask AP §4.4. Note that FasterNet-S trails PVT-Small and PoolFormer-S24 in AP, and wins only on latency tab. 4.

Ablation (T0 base; T0* = T0 with modified width and depth, not specified) §4.5, tab. 5:

Paper vs code

Not checked yet. The abstract links the official repository p. 1, and its owner matches the first author. It was not cloned when this card was compiled, so code_commit and code_license are not recorded and nothing on this card has been verified against the code.

Pitfalls

• FLOPs = multiply-adds §1, footnote. FLOPS in tab. 1 appears to be FLOPs × throughput, so it is also in multiply-adds per second and not directly comparable with vendor peak FLOP/s figures that count a multiply-add as 2 ops.
• The "1/4 memory access" claim assumes an in-place write. Eq. 5 counts only the c_p channels. An implementation that splits the tensor and concatenates it back (needed for autograd if you overwrite in place) also copies the c - c_p untouched channels, so the measured saving depends on the implementation. The paper does not say which form produced its numbers.
• Stride must be 1. The untouched channels keep their resolution, so PConv cannot downsample. Downsampling comes from the separate 2×2 stride-2 merging convs app. F.
• Channel choice matters for memory layout. Use a contiguous block, the first or last c_p channels, not a scattered subset §3.2. In NCHW that is a contiguous slice. In channels-last layouts it is not, and the paper does not discuss this.
• No norm or activation after PConv or after the last PWConv. Adding BN/activation after every conv, as in ResNet or MobileNet habits, contradicts the design §3.4, app. E.
• Activation depends on variant: GELU for T0/T1, ReLU for T2 and up tab. 8. The classifier's activation after the 1280-d 1×1 conv uses the same function tab. 8. Whether that head conv has BN: tab. 8 lists none.
• Speed numbers need BN folded into the adjacent convs §4. Unfolded BN, or LayerNorm, is slower: on T0, LN costs 19% GPU throughput at the same accuracy tab. 5.
• Where stochastic depth is applied (per-block residual branch, linear vs uniform rate across depth) is not stated tab. 6.
• "# of forward pass 188k" is listed for every variant tab. 6. With about 1.3M training images §4.3 and 300 epochs, that matches batch 2048 (M/L). At batch 4096 (T0–S) the same epochs give roughly half as many steps, so the entry may be wrong for those variants. Trust epochs and batch size.
• Progressive resolution: train at 192 for 280 epochs, then 224 for the last 20, and test at 224 with crop ratio 0.9 §4.3, tab. 6. Training only at 224 will not reproduce the stated cost and may change accuracy.
• Small numeric inconsistencies: FLOPs and params differ slightly between tab. 3 and tab. 8 (T2 1.91 vs 1.90 G; S 4.56 vs 4.55; M 8.74 vs 8.72; L 15.52 vs 15.49 G and 93.5 vs 93.4 M params). The text says FasterNet-S gains +2.4 mask AP over ResNet50 §4.4, but tab. 4 gives 36.9 − 34.4 = 2.5.
• Benchmarks are single-threaded on CPU and ARM §4, and FasterNet-L and other large models run out of memory on the ARM board tab. 3. Multi-threaded or accelerator (NPU, TensorRT int8) speedups are not measured.
• Unspecified ablation models: T0* variants change width and depth in unstated ways tab. 5, so the r ablation is not a clean single-variable comparison.
• Limited receptive field: the authors list the purely convolutional, local receptive field as a limitation app. F.

Lineage

• Builds on: depthwise separable and group convolutions (MobileNets, ShuffleNets, GhostNet), which PConv aims to replace §1. The macro design resembles ConvNeXt with DWConv swapped for PConv, but the activations between PConv and PWConv are removed on purpose app. E. PConv equals a group conv that processes one group and leaves the others untouched app. E.
• Compared against: ShuffleNetV2, MobileNetV2, GhostNet, MobileViT, EdgeNeXt, PVT, CycleMLP, PoolFormer, ResNet50, ConvNeXt, Swin on ImageNet tab. 3; ResNet, ResNeXt, PoolFormer, PVT on COCO tab. 4.
• Not to be confused with the "partial convolution" used for image inpainting (masked pixels), which applies filters to partial pixels, not partial channels app. E.