--- kind: concept slug: mixed-precision-quantization title: Mixed-precision quantization summary: >- Giving different tensors or layers different bitwidths or number formats instead of one width for the whole model. A compiler for microcontrollers picks 8- or 16-bit per tensor under a RAM limit, with a measured result of 54.81% against 55.02% float. A survey recommends FP16 for norms and end layers and INT8 for dense layers, but reports no ablation for that split. tags: [quantization, mixed-precision, microcontroller, edge-deployment, memory] papers: ["2210.16556", "2601.03290"] related: [post-training-quantization, quantization-aware-training, weight-quantization, low-precision-arithmetic, adaptive-rounding, network-pruning, knowledge-distillation] status: draft updated_at: 2026-10-25 --- ## Overview Mixed-precision quantization stores or computes different parts of one model in different widths, instead of one width everywhere. The aim is to keep the few sensitive tensors wide and shrink the rest. The motivation is concrete: on a microcontroller, reducing every tensor to 8 bits loses too much accuracy for some models, notably an RNNPool-based face detector [arxiv:2210.16556 §1] [arxiv:2210.16556 §6.3]. Choosing each tensor's width by hand, or by search, is the hard part. Each tensor adds a binary choice, so N tensors give 2^N combinations, and an exhaustive search is not practical [arxiv:2210.16556 §1] [arxiv:2210.16556 §4.1]. Only two cards in this wiki cover the topic, and they differ in kind. The first is a compiler paper that searches per tensor and measures the result against a RAM limit [arxiv:2210.16556 §4.1] [arxiv:2210.16556 §6.7]. The second is a survey that recommends a fixed split by layer type and states it without an experiment [arxiv:2601.03290 §III-A] [arxiv:2601.03290 §VIII-B]. The evidence base is thin, so treat the choices below as hypotheses to check on your own model. ## Variants ### Per-tensor search under a memory limit Each tensor gets either a low or a high bitwidth. In the MinUn compiler these are 8 and 16 bits, in posit or fixed-point form [arxiv:2210.16556 §4.1] [arxiv:2210.16556 §5]. The search has three stages: 1. Run the model once with every promotable tensor at the high width and once at the low width, and log the tensor values in each run [arxiv:2210.16556 §4.1.2] [arxiv:2210.16556 alg. 2]. 2. Score each tensor by the 95th-percentile of its absolute difference between the two runs, divided by its element count [arxiv:2210.16556 §4.1.2] [arxiv:2210.16556 alg. 3]. 3. Promote tensors in score order while estimated memory stays under a soft limit. Tensors that cross the limit are tried alone and then together, each time with a measured accuracy, and the best candidate is kept [arxiv:2210.16556 §4.1.3] [arxiv:2210.16556 alg. 4] [arxiv:2210.16556 alg. 5]. The method needs an accuracy measurement on representative inputs and a compile-time budget. The paper caps memory management at 2 hours per run [arxiv:2210.16556 §5]. Only two bitwidths are searched; the authors report no measurable gain from more options for posits [arxiv:2210.16556 §6.8.2]. The paper's headline comparison is SqueezeNet on 48,000 images. The float32 model gets 55.02% at 4.42 MB of peak RAM. MinUn with posits and exact-cover placement gets 54.81% at 1.16 MB. Shiftry gets 48.33% at 2.21 MB, and TensorFlow Lite 8-bit zero-skew gets 45.73% at 1.11 MB [arxiv:2210.16556 §6.7]. So the mixed-width result is more accurate than the 8-bit baseline, but it uses slightly more RAM than that baseline in this table. Card: [MinUn: Accurate ML Inference on Microcontrollers](/papers/2210.16556). When the search is run at Shiftry's RAM limit, the paper reports that Shiftry's own fixed-point assignment drops 23.3% on the RNNPool face model. The two MinUn search strategies stay at -0.3% on that model [arxiv:2210.16556 §6.5]. The other classifiers in the same table show drops below 1% [arxiv:2210.16556 §6.5]. ### Fixed split by layer type The survey recommends FP16 for layer norms and for the first and last layers, and INT8 for dense layers [arxiv:2601.03290 §III-A] [arxiv:2601.03290 §VIII-B]. The rule needs no calibration data and no search. The paper states it without an ablation, so its benefit is not measured in the card [arxiv:2601.03290 Hyperparameters table]. Its pipeline lists mixed FP16/INT8 quantization as a step that gives a further 50% size reduction [arxiv:2601.03290 §VIII-H]. The card's arithmetic shows that the stated per-step reductions do not reach the paper's 8-12x overall figure [arxiv:2601.03290 Pitfalls]. The survey's only figure for this split is read by eye from a plot. FP16 alone is shown at 50% of size with 99.5% relative accuracy, and INT8 at 25% with 98.7% [arxiv:2601.03290 fig. 6]. The plot does not separate a per-layer mix from uniform FP16 or INT8. Card: [Lightweight Transformer Architectures for Edge Devices in Real-Time Applications](/papers/2601.03290). ## When to use which | Situation | Use | Why | |---|---|---| | Hard RAM ceiling on a microcontroller, you control the compiler, and you can run accuracy checks on representative inputs | Per-tensor search | The only measured result here. SqueezeNet reaches 54.81% against 55.02% float at 1.16 MB [arxiv:2210.16556 §6.7] | | Uniform 8-bit loses too much accuracy for one model | Per-tensor search, keeping sensitive tensors at 16 bits | The paper gives this as the motivating case [arxiv:2210.16556 §1] [arxiv:2210.16556 §6.3] | | Minimum RAM matters more than accuracy | Uniform 8-bit | The 8-bit TensorFlow Lite baseline used 1.11 MB against 1.16 MB for MinUn, at 45.73% accuracy [arxiv:2210.16556 §6.7] | | No calibration data and no time for a search | Fixed layer-type split | Needs no data [arxiv:2601.03290 §III-A]. Its benefit is not measured, so test it on your model first | ## Implementation sketch A common core runs a sensitivity score, a greedy promotion under a budget, and an accuracy check. The variants differ in the score and in what happens at the limit. The sketch is simplified: it stops at the first tensor that exceeds the budget, while the paper keeps a list of overshooting tensors and tries them with measured accuracy [arxiv:2210.16556 §4.1.3]. ```pseudocode # Assign a bitwidth per tensor under a memory budget [arxiv:2210.16556 §4.1] def assign_bitwidths(tensors, ram_limit, low, high, accuracy): # Run all-high and all-low, log each tensor's values [§4.1.2] hi, lo = log_values(tensors, high), log_values(tensors, low) # Sensitivity: 95th-percentile deviation per element [§4.1.2] [alg. 3] score = {t: p95(abs(hi[t] - lo[t])) / t.numel for t in tensors} # Start all low, then promote most sensitive first [§4.1.3] [alg. 4] rho = {t: low for t in tensors} for t in sorted(tensors, key=score.get, reverse=True): rho[t] = high if ram_used(rho) > ram_limit: rho[t] = low # would overshoot, keep it low # Measure the candidate on representative inputs [§4.1.3] [alg. 5] return rho, accuracy(rho) ``` Layer-type variants replace the loop with a fixed map from layer kind to width. The score step is then unnecessary [arxiv:2601.03290 §VIII-B]. ## Contradictions The two papers give different INT8 accuracy-loss figures, and the survey does not say which applies to which setting. Its introduction gives 1 to 5% without calibration [arxiv:2601.03290 §I-C]. Its own analysis gives 0.5 to 1.2% from FP32 to INT8 [arxiv:2601.03290 §VIII-B]. A separate section says below 1% for calibrated TensorFlow Lite models [arxiv:2601.03290 §IV-B]. The two papers also choose widths by different rules. MinUn picks each tensor's width from measured sensitivity [arxiv:2210.16556 §4.1]. The survey picks by layer type without a measurement [arxiv:2601.03290 §III-A]. These do not conflict in their numbers, because neither tests the other's rule. The wiki has no result that compares them. ## Papers - [MinUn: Accurate ML Inference on Microcontrollers](/papers/2210.16556): per-tensor 8- or 16-bit search under a RAM limit, with exact-cover memory placement, measured on SqueezeNet and tiny models. - [Lightweight Transformer Architectures for Edge Devices in Real-Time Applications](/papers/2601.03290): survey that recommends an FP16/INT8 split by layer type and a six-step pipeline, with no measured ablation.