Adaptive rounding

Overview

Rounding a trained weight to the nearest value on a fixed grid is the default in post-training quantization. The problem adaptive rounding addresses is that the nearest value is not always the best one: when many weights are rounded together, their errors add up through the layer's inputs, so the choice for one weight depends on the others 2004.10568 §1 2004.10568 §2.

The invariant all variants share is that the scale and the grid are fixed, and only the per-weight floor-or-ceil decision is optimised. The network is not retrained 2004.10568 §3.1 2004.10568 §5.

Only one paper with a card covers this concept, so this page describes one method. Other rounding-optimisation methods are not covered here.

Variants

Layer-wise adaptive rounding (AdaRound)

Each layer is optimised on its own. For every weight the method chooses between the floor and the ceiling of W / s. The choice is first relaxed to a continuous value h in [0, 1] using a rectified sigmoid with stretch parameters 1.1 and -0.1 2004.10568 §3.3. A regulariser then pushes each h towards 0 or 1, with its exponent annealed from large to small 2004.10568 §3.3 2004.10568 fig. 2. After optimisation each h is taken as its nearest end point 2004.10568 §3.3 [arxiv:2004.10568 fig. 3]. The paper derives the per-layer loss from a second-order expansion of the task loss, with a diagonal Hessian and a mean-squared error on pre-activations 2004.10568 §3.2 2004.10568 app. B. Card: Up or Down? Adaptive Rounding for Post-Training Quantization.

The reconstruction target is the output of the float layer after its activation, and the input is the output of the already-quantized earlier layers. This is the "asymmetric" part of the objective. It is about which inputs each side sees, not about an asymmetric quantization grid 2004.10568 §3.3.

When to use which

The gain depends on the grid. With a min-max grid, nearest rounding gives 0.23% top-1 on Resnet18 and adaptive rounding gives 61.96%. With a weight-MSE grid, the same comparison is 23.99% and 68.60% 2004.10568 tab. 6. Choose the grid before measuring the gain.

Activations are kept in FP32 in most results. The paper also reports 8-bit activation rows with min/max ranges, which it says rarely hurt 2004.10568 §5.2 2004.10568 tab. 7.

Implementation sketch

Language-agnostic pseudocode of the layer-wise method. The regulariser weight, the beta range and any warm-up are not stated in the paper. The card lists the values the reference implementation uses, and its "Paper vs code" section lists the differences.

# Adaptive rounding for one linear or convolutional layer
# Input:  weight W, fixed scale s, unlabelled calibration inputs X
# Output: a 0/1 choice per weight, giving the quantized weights
def adaround_layer(W, s, X, steps):
    # Floor of each weight on the grid  [arxiv:2004.10568 §3.1]
    lo = floor(W / s)
    # Soft variable per weight, initialised from the fractional part  [§3.3]
    V = init_from_fraction(W / s - lo)
    for t in range(steps):
        # Mini-batch of calibration inputs, batch size 32  [arxiv:2004.10568 §5]
        x = sample_batch(X, 32)
        # Rectified sigmoid, zeta - gamma = 1.2 and gamma = -0.1  [§3.3]
        h = clip(1.2 * sigmoid(V) - 0.1, 0, 1)
        # Quantized weights using the relaxed choice  [§3.3]
        soft_W = s * clip(lo + h, qmin, qmax)
        # Output of the float layer and of the soft-quantized layer, fed by
        # quantized earlier layers, after the activation  [§3.3]
        target = act(layer_fp(x))
        pred = act(layer(x, soft_W))
        # Push h towards 0 or 1, with beta annealed over the steps  [§3.3]
        reg = sum(1 - abs(2 * h - 1) ** beta(t, steps))
        loss = mse(pred, target) + lam * reg
        V = optimizer_step(V, loss)
    # Hard decision: round up where h ends above 0.5  [§3.3] [fig. 3]
    return lo + (h >= 0.5)

The regulariser weight lam, the bounds qmin and qmax, and the annealing schedule beta are not given in the paper. The card gives the reference values.

Contradictions

Only one card covers this concept, so no disagreement between papers is known.

Within the paper, the abstract and conclusion say that 4-bit Resnet18 and Resnet50 weights are within 1% of FP32 2004.10568 §6. The main Resnet18 row (68.60) is 1.08 points below FP32 (69.68). The literature-table row (68.71), which uses 2048 images and 20k iterations, is 0.97 below 2004.10568 tab. 7. Quote the number from the row you use.

Papers

• Up or Down? Adaptive Rounding for Post-Training Quantization: the layer-wise method with a rectified-sigmoid relaxation and an asymmetric reconstruction loss.