LSQ+: Improving low-bit quantization through learnable offsets and better initialization

In plain words

Neural networks can run on small devices if their numbers are stored with fewer bits. Older methods store each value on an evenly spaced grid that starts at zero. That works for activations that are never negative, but newer activation functions such as Swish dip slightly below zero and then rise without limit. On this kind of data, the method either wastes half of its grid on negative values or throws the negative values away, and accuracy drops, especially at very low bit widths.

LSQ+ gives the grid a second adjustable knob. Besides the spacing, the grid can also slide, so its lowest level lines up with the most negative values that actually occur in a layer. The grid spacing and the slide are learned during training. The paper also changes how the starting values are chosen. Starting points based on simple statistics of the weights, and on a fit that minimises squared error for the activations, make training more repeatable from run to run. The sliding grid costs almost nothing at inference, because its offset can be folded into a bias term.

TL;DR

LSQ+ replaces the symmetric quantizer of LSQ on activations with an asymmetric one that learns both a scale s and an offset β §3.1 eq. 2. Weights stay symmetric and signed §3.2.1. Initial values come from weight statistics (three standard deviations) and from an MSE fit on activations §3.2 eq. 8. Author-reported: EfficientNet-B0 at W4A4 reaches 73.8% against 71.9% for LSQ and 76.1% full precision on ImageNet tab. 2. At W2A2 the MSE initialisation narrows the run-to-run spread from ±4.7 to ±1.9 points compared with min-max tab. 5.

Problem

• Swish-like activations are skewed: the negative part is bounded near −0.278 while the positive part is unbounded §3.1 [fn. 2 in §3.1].
• Unsigned learned quantization clamps every negative activation to zero, which loses information §3.1.
• Signed quantization avoids that loss but spends one bit on a sign, which is costly at 2, 3 or 4 bits §1 §3.1.
• Gradient-based learned quantizers are sensitive to initialisation. Poor starting values give high variance in final accuracy, especially with depthwise separable convolutions §3.2.
• The reported EfficientNet-B0 loss under LSQ at W4A4 is 4.1% §1. The paper's own table gives 76.1 − 71.9 = 4.2 points tab. 2; see Pitfalls.

Method

Notation. Bit-width is b. Activations use either the unsigned range n = 0, p = 2^b − 1, or the signed range n = −2^(b−1), p = 2^(b−1) − 1 §3.1. Weights are quantized with symmetric signed quantization §3.2.1.

Symmetric baseline (LSQ). A value x is divided by a learned scale s, clamped to [n, p], rounded, and multiplied back by s §3.1 eq. 1.

Core change: learned asymmetric quantization. The offset β is subtracted before scaling and added back after rounding §3.1 eq. 2:

\bar{x} = \operatorname{round}\big(\operatorname{clamp}((x-\beta)/s,\, n,\, p)\big) \qquad [\text{eq. 2}]

\hat{x} = \bar{x}\,s + \beta \qquad [\text{eq. 2}]

Gradients. Both s and β are learned. The round operation is replaced by a straight-through estimator §3.1 eq. 3 eq. 4.

• For s, inside the range the gradient is −(x − β)/s + round((x − β)/s); outside the range it is n or p eq. 3.
• For β, the gradient is 0 inside the range and 1 outside eq. 4.

Inference cost. For weights with symmetric quantization and activations with offset β, the product ŵx̂ splits into a quantized integer product plus a term β·s_w·w̄ that depends only on the weights. That term can be precomputed and folded into the bias at compile time §3.1 eq. 5. The paper concludes that asymmetric activation quantization has zero runtime overhead §6.

Four configurations. Table 1 lists four parametrisations. Config 1 is unsigned and symmetric (LSQ). Config 2 is signed and symmetric. Config 3 is signed and asymmetric. Config 4 is unsigned and asymmetric tab. 1. Configs 1 and 2 have no offset §3.1. Configs 3 and 4 learn β for activations §3.1.

Initialisation (§3.2)

The authors use the same asymmetric configuration for both weights and activations as the scheme above, but initialise the parameters differently from LSQ §3.2.

Weight scale §3.2.1. The paper assumes each layer's weights are roughly Gaussian §3.2.1. With mean μ and standard deviation σ of the layer's weights, the initial scale is the larger of |μ − 3σ| and |μ + 3σ|, divided by 2^(b−1) §3.2.1.

Activation scale and offset, min-max form. Let x_min and x_max be the lowest and highest activation values. The min-max solution sends x_min to n and x_max to p §3.2.2 eq. 6. It gives s_init = (x_max − x_min)/(p − n) and β_init = x_min − n·s_init §3.2.2 eq. 7.

Activation scale and offset, MSE form (the proposed method). The paper chooses s and β to minimise the squared Frobenius error between the quantized and original activations §3.2.2 eq. 8. No closed form exists, so the authors optimise with autograd through eq. 3 and eq. 4 over a few batches of data §3.2.2.

# Asymmetric activation quantizer  [§3.1] [eq. 2]
# Input:  x (activations); scale s > 0; offset beta; integer range n, p
# Output: x_hat, same shape as x
def quantize_activation(x, s, beta, n, p):
    # Shift by the offset and divide by the scale  [eq. 2]
    scaled = (x - beta) / s
    # Clamp to the integer range before rounding  [eq. 2]
    clamped = clamp(scaled, n, p)
    # Round to the nearest integer code  [eq. 2]
    codes = round(clamped)
    # Map codes back to real values  [eq. 2]
    x_hat = codes * s + beta
    return x_hat

# Scale init for symmetric signed weight quantization  [§3.2.1]
# Input:  weights of one layer; bit-width b
# Output: s_init
def init_weight_scale(weights, b):
    # Mean and standard deviation of the layer's weights  [§3.2.1]
    mu = mean(weights)
    sigma = std(weights)
    # Largest magnitude within three standard deviations  [§3.2.1]
    span = max(abs(mu - 3 * sigma), abs(mu + 3 * sigma))
    # Divide by 2^(b-1), as written in the paper  [§3.2.1]
    return span / 2 ** (b - 1)

# Min-max initialisation for activations  [§3.2.2] [eq. 7]
# Input:  x_min, x_max (observed range); integer range n, p
# Output: s_init, beta_init
def init_minmax(x_min, x_max, n, p):
    # Stretch the observed range over the integer range  [eq. 7]
    s_init = (x_max - x_min) / (p - n)
    # Place x_min at the lower bound n  [eq. 7]
    beta_init = x_min - n * s_init
    return s_init, beta_init

# MSE initialisation of s and beta for one activation quantizer  [§3.2.2]
# Input:  calibration batches; integer range n, p; starting s, beta
# Output: s_init, beta_init
def init_mse(batches, n, p, s, beta):
    # Optimiser, learning rate and step count are not stated  [§3.2.2]
    for batch in batches:
        # Quantize the batch with the current parameters  [eq. 2]
        x_hat = quantize_activation(batch, s, beta, n, p)
        # Squared Frobenius error, minimised over s and beta  [eq. 8]
        loss = sum((x_hat - batch) ** 2)
        # Gradients use the straight-through rules of eq. 3 and eq. 4
        s, beta = gradient_step(s, beta, loss)
    return s, beta

Experimental setup (as stated). Weights start from pretrained floating-point weights in every experiment §4. Experiments run on ImageNet §4. For the initialisation study, EfficientNet-B0 uses Config 4 for activations and symmetric signed weights, trained for 5 runs per initialisation §4.3 tab. 5. Min-max statistics come from the first batch of input images §4.3. The results in tables 2 to 4 use the proposed initialisation §4. The notation W×A× means weights and output activations of all layers are quantized to × bits §1, footnote.

Scope of quantization. Quantized layers are described as "all layers" §1, footnote. Squeeze-and-excite layers use no offset because their sigmoid has no negative part §5.1. Bottleneck outputs that have no activation are quantized with symmetric signed activations §5.1.

Hyperparameters

Results

Author-reported. Setup: ImageNet, pretrained floating-point start, and the best accuracy over the grid search, except tab. 5 which reports the mean over 5 runs tab. 2 tab. 5 caption.

EfficientNet-B0, full precision 76.1% tab. 2:

MixNet-S, full precision 75.9% tab. 3:

ResNet18, full precision 70.1% tab. 4:

Initialisation study, EfficientNet-B0, Config 4, mean ± Δacc over 5 runs tab. 5:

Fixed versus learned offset, EfficientNet-B0 W4A4, Config 4 tab. 6:

Other reported facts. LSQ+ matches LSQ on ResNet18, with no gap between Configs 1, 3 and 4 §4.2. Config 1 on ResNet18 at W4A4 reaches 70.7%, above the 70.1% full-precision accuracy, which the authors read as a sanity check of their LSQ implementation §4.2.

Learned offsets. Most learned β values are negative, and none is below the Swish minimum §5.1 fig. 2. The paper's explanation is eq. 4: once β < x_min, the β gradient is zero §5.1 eq. 4.

Figures. fig. 1 compares the weight scale of one EfficientNet-B0 layer after fine-tuning, with LSQ and with LSQ+ initialisation. LSQ starts further from its converged value fig. 1 §4.3. fig. 2 plots layer-wise β values after convergence for W4A4 EfficientNet-B0 fig. 2 §5.1.

Paper vs code

No official code. Searches: the paper, its arXiv metadata and the LaTeX source contain no code or project link. GitHub repository search for "lsq plus", "LSQ quantization learnable offsets", "lsq plus quantization" and "lsq" within the Qualcomm-AI-research organisation returned no repositories from the authors. The only hit, Sergasgr/CodeAlign-Distill (MIT), is a third-party project that lists an LSQ quantizer among its features; it is not used here.

Pitfalls

• Gradient outside the range. Eq. 3 says the scale gradient is "n or p" when the value is clipped, but does not say which applies to which side. The natural reading is n on the lower clip and p on the upper clip eq. 2. This is an inference from eq. 2, not a stated rule.
• Gradient of β can stop early. Once β drops below x_min, every value is inside the range, and the β gradient is zero by eq. 4. Learned β values in the paper never fell below x_min §5.1.
• Weight init denominator. The formula divides by 2^(b−1) §3.2.1, while the signed range has p − n = 2^b − 1. The paper does not say whether this is intended.
• Mean definition. The text says μ is "the mean (same as ⟨|w|⟩)" §3.2.1. The mean of signed weights and the mean absolute value differ, and the Gaussian model is defined for the signed mean. Choose one and state it.
• LSQ constant. Sec. 3.2.1 gives the LSQ weight initialisation as ⟨|w|⟩/√2 for 4 bits, which uses 2^(b−1) = 8 under the square root. Sec. 4.3 defines p as the range's upper bound, which is 7 for signed 4-bit. The two choices give 0.707 and 0.756 times ⟨|w|⟩, so reproduce the one you use.
• MSE initialisation is under-specified. Optimiser, learning rate, step count, batch count and whether it fits s and β jointly are not stated. The initial s must stay positive, and the paper does not say how this is enforced.
• Unbounded activations. The x_min and x_max for the positive Swish range come from "a few forward passes" §3.2.2. The pass count, data, and whether it is a true min and max or a percentile are not stated.
• Mixed reporting. Tables 2 to 4 report the best accuracy after a grid search, according to the caption of tab. 5. The grid is not reported. Table 5 reports means over 5 runs. The 73.8% in tab. 2 and 73.0% in tab. 5 are not comparable.
• Rounded headline numbers. The abstract and text give a 1.8% W4A4 gain. Table 2 gives 73.5 − 71.9 = 1.6 for Config 3 and 73.8 − 71.9 = 1.9 for Config 4. The text gives "1.6–1.8%" §4.1. W2A2 gives 49.1 − 43.5 = 5.6 for Config 3, matching "up to 5.6%". The 4.1% LSQ loss in §1 does not match 76.1 − 71.9 = 4.2 tab. 2.
• Typo in Table 5. The row label is "Mix-max", while the text says min-max §4.3.
• Fixed-offset baseline. The "Fixed β = 0 (LSQ)" row in Table 6 is 71.9%, identical to Table 2's Config 1. The paper does not say whether it was run separately.
• Zero overhead holds for the bias fold only. Eq. 5 shows the offset term becoming a per-output bias. It does not cover layers that do not accept a bias term, or non-linear operations placed between the quantizer and the next layer.
• Dependence on pretrained weights. Every experiment starts from pretrained floating-point weights §4. The paper does not test training from scratch.
• Weight statistics are a model assumption. The weight init assumes a Gaussian distribution per layer §3.2.1. The paper's only check is the one layer in fig. 1.

Lineage

• Builds on: LSQ, learned step size quantization, arXiv:1902.08153 §3 [bib: lsq]. LSQ+ adds the offset β and changes the initialisation of s §3.1 §3.2.
• Compared against: PACT, DSQ and QIL on ResNet18 tab. 4. Min-max and LSQ initialisation on EfficientNet-B0 tab. 5. The paper also cites TQT §2 and the straight-through estimator for its gradients §3.1.
• Related: asymmetric configurations are said to be usable with other learned quantizers §2.