Mixed-precision quantization

Overview

Mixed-precision quantization stores or computes different parts of one model in different widths, instead of one width everywhere. The aim is to keep the few sensitive tensors wide and shrink the rest. The motivation is concrete: on a microcontroller, reducing every tensor to 8 bits loses too much accuracy for some models, notably an RNNPool-based face detector 2210.16556 §1 2210.16556 §6.3. Choosing each tensor's width by hand, or by search, is the hard part. Each tensor adds a binary choice, so N tensors give 2^N combinations, and an exhaustive search is not practical 2210.16556 §1 2210.16556 §4.1.

Only two cards in this wiki cover the topic, and they differ in kind. The first is a compiler paper that searches per tensor and measures the result against a RAM limit 2210.16556 §4.1 2210.16556 §6.7. The second is a survey that recommends a fixed split by layer type and states it without an experiment 2601.03290 §III-A 2601.03290 §VIII-B. The evidence base is thin, so treat the choices below as hypotheses to check on your own model.

Variants

Per-tensor search under a memory limit

Each tensor gets either a low or a high bitwidth. In the MinUn compiler these are 8 and 16 bits, in posit or fixed-point form 2210.16556 §4.1 2210.16556 §5. The search has three stages:

1. Run the model once with every promotable tensor at the high width and once at the low width, and log the tensor values in each run 2210.16556 §4.1.2 2210.16556 alg. 2.
2. Score each tensor by the 95th-percentile of its absolute difference between the two runs, divided by its element count 2210.16556 §4.1.2 2210.16556 alg. 3.
3. Promote tensors in score order while estimated memory stays under a soft limit. Tensors that cross the limit are tried alone and then together, each time with a measured accuracy, and the best candidate is kept 2210.16556 §4.1.3 2210.16556 alg. 4 2210.16556 alg. 5.

The method needs an accuracy measurement on representative inputs and a compile-time budget. The paper caps memory management at 2 hours per run 2210.16556 §5. Only two bitwidths are searched; the authors report no measurable gain from more options for posits 2210.16556 §6.8.2.

The paper's headline comparison is SqueezeNet on 48,000 images. The float32 model gets 55.02% at 4.42 MB of peak RAM. MinUn with posits and exact-cover placement gets 54.81% at 1.16 MB. Shiftry gets 48.33% at 2.21 MB, and TensorFlow Lite 8-bit zero-skew gets 45.73% at 1.11 MB 2210.16556 §6.7. So the mixed-width result is more accurate than the 8-bit baseline, but it uses slightly more RAM than that baseline in this table. Card: MinUn: Accurate ML Inference on Microcontrollers.

When the search is run at Shiftry's RAM limit, the paper reports that Shiftry's own fixed-point assignment drops 23.3% on the RNNPool face model. The two MinUn search strategies stay at -0.3% on that model 2210.16556 §6.5. The other classifiers in the same table show drops below 1% 2210.16556 §6.5.

Fixed split by layer type

The survey recommends FP16 for layer norms and for the first and last layers, and INT8 for dense layers 2601.03290 §III-A 2601.03290 §VIII-B. The rule needs no calibration data and no search. The paper states it without an ablation, so its benefit is not measured in the card 2601.03290 Hyperparameters table. Its pipeline lists mixed FP16/INT8 quantization as a step that gives a further 50% size reduction 2601.03290 §VIII-H. The card's arithmetic shows that the stated per-step reductions do not reach the paper's 8-12x overall figure 2601.03290 Pitfalls.

The survey's only figure for this split is read by eye from a plot. FP16 alone is shown at 50% of size with 99.5% relative accuracy, and INT8 at 25% with 98.7% 2601.03290 fig. 6. The plot does not separate a per-layer mix from uniform FP16 or INT8. Card: Lightweight Transformer Architectures for Edge Devices in Real-Time Applications.

When to use which

Implementation sketch

A common core runs a sensitivity score, a greedy promotion under a budget, and an accuracy check. The variants differ in the score and in what happens at the limit. The sketch is simplified: it stops at the first tensor that exceeds the budget, while the paper keeps a list of overshooting tensors and tries them with measured accuracy 2210.16556 §4.1.3.

# Assign a bitwidth per tensor under a memory budget  [arxiv:2210.16556 §4.1]
def assign_bitwidths(tensors, ram_limit, low, high, accuracy):
    # Run all-high and all-low, log each tensor's values  [§4.1.2]
    hi, lo = log_values(tensors, high), log_values(tensors, low)
    # Sensitivity: 95th-percentile deviation per element  [§4.1.2] [alg. 3]
    score = {t: p95(abs(hi[t] - lo[t])) / t.numel for t in tensors}
    # Start all low, then promote most sensitive first  [§4.1.3] [alg. 4]
    rho = {t: low for t in tensors}
    for t in sorted(tensors, key=score.get, reverse=True):
        rho[t] = high
        if ram_used(rho) > ram_limit:
            rho[t] = low  # would overshoot, keep it low
    # Measure the candidate on representative inputs  [§4.1.3] [alg. 5]
    return rho, accuracy(rho)

Layer-type variants replace the loop with a fixed map from layer kind to width. The score step is then unnecessary 2601.03290 §VIII-B.

Contradictions

The two papers give different INT8 accuracy-loss figures, and the survey does not say which applies to which setting. Its introduction gives 1 to 5% without calibration 2601.03290 §I-C. Its own analysis gives 0.5 to 1.2% from FP32 to INT8 2601.03290 §VIII-B. A separate section says below 1% for calibrated TensorFlow Lite models 2601.03290 §IV-B.

The two papers also choose widths by different rules. MinUn picks each tensor's width from measured sensitivity 2210.16556 §4.1. The survey picks by layer type without a measurement 2601.03290 §III-A. These do not conflict in their numbers, because neither tests the other's rule. The wiki has no result that compares them.

Papers

• MinUn: Accurate ML Inference on Microcontrollers: per-tensor 8- or 16-bit search under a RAM limit, with exact-cover memory placement, measured on SqueezeNet and tiny models.
• Lightweight Transformer Architectures for Edge Devices in Real-Time Applications: survey that recommends an FP16/INT8 split by layer type and a six-step pipeline, with no measured ablation.