WRPN: Wide Reduced-Precision Networks

In plain words

Computers store each number of a neural network with a fixed number of bits. Fewer bits means less memory, less data moved and cheaper arithmetic. Most earlier work shrank only the weights, the numbers the model learns. The authors observed that during training, and when a server processes a batch of images at once, the intermediate values the network produces (its activations) take far more memory than the weights do.

So they shrink both, down to 4 bits for activations and 2 bits for weights, and train the network from scratch. Shrinking hurts accuracy, so they compensate by making every layer wider, with more channels, which keeps the total arithmetic close to the original. In their ImageNet tests this matched the full-precision baseline, and the gains only show up on hardware that handles low-precision arithmetic well.

TL;DR

WRPN trains convolutional networks from scratch with reduced-precision weights and activations, using one simple clip-and-round quantizer and straight-through gradients. Each layer's filter count is multiplied by a factor above one to recover the lost capacity. The paper is author-reported on AlexNet, ResNet-34 and batch-normalised Inception on ILSVRC-12. Relevant if you need low-bit convolutional inference or training and can accept an unstated training recipe.

Problem

• Activation maps dominate memory when the mini-batch is large. They take more than 98% of training memory at batch sizes 128 and above §2 fig. 2. Reducing only weight precision mainly helps inference at small batch sizes §1.
• Lowering activation precision has been reported to hurt accuracy. The authors report that prior binary-activation networks lose about 12% (XNOR-Net) and 8% (DoReFa) top-1 on AlexNet §6.
• Assumption: the target hardware runs low-precision arithmetic natively. The paper states that GPUs gain at most about 4x from low precision, while FPGAs and ASICs gain much more §1 §5.1 fig. 3.

Method

1. Quantizer for k > 1 bits. Weights are first clipped to [-1, 1]. Activations are clipped to [0, 1]. Each is then rounded to a uniform grid. Weights use one sign bit, so they have 2^(k-1) - 1 positive levels. Activations are unsigned and have 2^k - 1 steps §5 eq. 2. For k = 2 the weight grid is {-1, 0, 1} (ternary) and the activation grid is {0, 1/3, 2/3, 1}, which is derived from eq. 2 and not stated in the text.

2. Quantizer for k = 1. Binary weights use the sign of each weight, scaled by the mean absolute value, following BWN §5. Binary activations use the same formula as the k > 1 case with k = 1, which reduces to rounding after clipping to [0, 1] §5 eq. 2. The scope of the mean (per layer or per channel) is not stated.

3. Training. The rounding step is treated with a straight-through estimator §5. The paper says gradients are not quantized and stay 32-bit, and the weights are kept in reduced-precision format §5. Whether a float master copy of the weights is kept for updates is not stated.

4. Widening. The depth and the other design choices of the baseline network are kept. Only the number of filter maps in each layer is multiplied by a factor. The factors studied are 1.1x, 1.3x, 2x and 3x §3 §4.1 tab. 4. The paper does not say whether the first and last layers are widened.

5. Layer computation. The forward pass multiplies k-bit signed weights by k-bit unsigned activations, then scales the result by a floating-point constant. The backward pass multiplies 32-bit gradients by k-bit weights §5. The scaling constant's value is not stated.

6. Scope. The first and last layers are kept at full precision in the AlexNet study, to match prior work §3. For ResNet-34 the first and last layers are kept at full precision §4.1. For batch-normalised Inception the paper does not state which layers are quantized §4.2.

Layer computation, in pseudocode:

# WRPN convolution in training and inference  [§5] [eq. 2]
# Input:  activations (B, C, H, W), real-valued
# Output: (B, C_out, H', W'), real-valued after rescaling

def wrpn_conv(activations, weights, k_act, k_weight):
    # Clip and round activations onto the unsigned k-bit grid  [§5]
    acts_q = quantize_unsigned(activations, k_act)
    # Clip and round weights onto the signed k-bit grid  [§5]
    weights_q = quantize_signed(weights, k_weight)
    # Convolve the quantized operands  [§5]
    out = convolve(acts_q, weights_q)
    # Scale by a float constant; its value is not stated  [§5]
    return out * scale

Quantizers, in pseudocode:

# Signed k-bit weight quantizer, k > 1  [§5] [eq. 2]
# Input:  w (any shape); k (bits, one of them the sign)
# Output: values on the grid j / (2^(k-1) - 1) inside [-1, 1]

def quantize_signed(w, k):
    # Hard clip to [-1, 1]  [§5]
    w = clip(w, -1.0, 1.0)
    # Number of positive levels; one bit is the sign  [eq. 2]
    levels = 2 ** (k - 1) - 1
    # Round to the nearest level and map back to [-1, 1]  [eq. 2]
    return round(levels * w) / levels

# Unsigned k-bit activation quantizer  [§5] [eq. 2]
# Input:  a (any shape); k (bits)
# Output: values on the grid j / (2^k - 1) inside [0, 1]

def quantize_unsigned(a, k):
    # Hard clip to [0, 1]  [§5]
    a = clip(a, 0.0, 1.0)
    # Number of steps on the unsigned grid  [eq. 2]
    levels = 2 ** k - 1
    # Round to the nearest level  [eq. 2]
    return round(levels * a) / levels

# 1-bit weights, BWN-style  [§5]
# Input:  w (any shape)
# Output: sign(w) scaled by a mean magnitude

def binarize_weights(w):
    # Scale is the mean absolute value; its scope is not stated  [§5]
    scale = mean(abs(w))
    return scale * sign(w)

Training step with straight-through gradients:

# One training step  [§5]
# Input:  images, labels from ILSVRC-12  [§4]
# Output: updated weights

def train_step(images, labels, weights):
    # Forward pass with quantized activations and weights  [§5]
    logits = network_forward(images, weights)
    # Loss function is not stated in the paper
    loss = task_loss(logits, labels)
    # Rounding is passed through as identity; the exact gradient
    # through the clip is not stated  [§5]
    grads = straight_through_backward(loss)
    # Gradients stay 32-bit float; optimizer and learning rate not stated  [§5]
    return optimizer_update(weights, grads)

Hyperparameters

Results

Author-reported. The setup is ILSVRC-12, trained from scratch, with the baseline and quantized runs sharing one recipe §3 §4. A dash means the configuration was not run tab. 1. Values are top-1 validation accuracy in percent.

AlexNet, 1x wide tab. 1. Rows are weight precision W, columns activation precision A. The 2-bit weight, 32-bit activation cell is from TTQ, not the authors' scheme §3.

AlexNet, 2x wide tab. 2. Same layout as tab. 1.

AlexNet compute cost, 2x wide vs 1x wide tab. 3. Ratio of compute cost §3, footnote. The cost is the product of FMA count and summed operand widths. The raw FMA count of 2x AlexNet is 3.9x that of the baseline §3.

ResNet-34 tab. 4. Pre-activation variant. The 1x-wide 1-bit row is listed with a cost of 0.03x.

Batch-normalised Inception, 2x wide tab. 5. The 1x full-precision row is the only 1x entry.

Headline numbers, as stated in the text. AlexNet 2x wide with 4-bit activations and 2-bit weights is at par with the full-precision network §3. ResNet-34 2x wide with 1-bit weights and activations reaches 69.85%, which is 3.7 points below the baseline §4.1. The introduction also quotes 48.04% for a 1.3x-wide binary AlexNet and 69.85% for ResNet-34, but the 1.3x result is in no table §1.

Hardware. Figure 3 has six panels fig. 3. Panel (b) shows first-order bit-count efficiency estimates. Panel (c) shows the GPU speed-up over FP32, which the text places at up to about 4x. Panel (d) shows the FPGA speed-up. Panels (e) to (g) show FPGA and ASIC performance and energy per watt. The text reports FPGA and ASIC gains of 6.5x to 100x over FP32 §1, and 2 to 3 orders of magnitude for the ASIC §5.1.

Paper vs code

No official code.

The arXiv abstract page has no comment or code link. The paper's footnote says the authors "will open-source" their implementations §4, and no repository was linked. Searches run: the method name alone (wrpn) on GitHub returned unrelated or unlicensed repositories by individuals; wide reduced precision networks and wrpn restricted to the intel organisation returned no results. None were used.

Pitfalls

• Baseline mismatch in the 2x AlexNet text. The text says 4-bit weights and 4-bit activations "surpass the baseline accuracy by 1.44%" §3. The number 58.6 in tab. 2 is 1.4 above the 57.2 1x baseline, but 1.9 below the 60.5 that tab. 2 itself lists for the 2x full-precision network. Compare quantized 2x runs to the 2x full-precision run.
• Cell order. The text says "4-bits weights and 2-bits activations" §3. The highlighted cell in tab. 2 is the 2-bit weight, 4-bit activation configuration (57.2). The 4-bit weight, 2-bit activation cell (57.3) is also near the baseline, so both readings fit the numbers, but the order is inconsistent with the table's row = W, column = A layout.
• Rounding in the text. ResNet 2-bit/2-bit is said to degrade accuracy "by only 0.2%" §4.1. tab. 4 gives 73.32 against 73.59, a 0.27-point drop. BN-Inception's 4-bit/2-bit loss is said to be 0.02% §4.2, but tab. 5 gives 71.61 against 71.64, a 0.03-point drop.
• Missing training recipe. The optimizer, learning rate, batch size, epoch count, weight decay and augmentation are not stated for AlexNet or BN-Inception. The text only says the runs share the baseline's hyperparameters and epochs §3. The ResNet recipe is also not stated, beyond "similar data augmentation" to prior work §4.1. Reproducing the tables needs a recipe that is not in the paper.
• Which layers are widened. The paper widens "each layer's filter maps" but does not say whether the first and last layers are widened, while keeping them at full precision §3 §4.1.
• The rounding grid has no scale. Quantized values lie in [-1, 1] or [0, 1] with no per-tensor or per-channel scale for k > 1 eq. 2. The paper says the real range is restored by affine transforms in hardware §5, but the transform is not written out. Activations are clipped to [0, 1], so negative inputs are mapped to 0; the paper does not say where the quantizer sits relative to batch norm and the activation function.
• Gradient through the clip. The straight-through estimator is cited §5, but the paper does not give the exact backward function, including whether gradients are zeroed outside the clip range.
• 1-bit weight scale. The BWN scale is the mean absolute weight §5. Whether it is per layer or per output channel is not stated, and the paper contrasts its choice with XNOR-Net's per-channel scale §5.
• Widening is not free. The 2x AlexNet raw operation count is 3.9x the baseline §3 tab. 3. The paper argues that low-bit operations keep the total cost below baseline, but only where the hardware exploits low precision. On GPU, the paper finds only about 4x speed-up §5.1.
• Cost numbers are model-specific. The compute-cost ratio of tab. 3 is defined in the footnote §3. The text quotes 49% for 4-bit/4-bit 2x AlexNet, while tab. 3 lists 0.5x. The ResNet and BN-Inception cost columns are given without a formula.
• Figures are not in the text. The memory-footprint curves fig. 1 and the hardware gains fig. 3 appear only as plots. Read the exact values from the figures, not from the prose.
• Hardware claims are partly prior work. Some GPU and FPGA numbers come from an earlier comparison §5.1. The ASIC numbers are from a synthesized processing element, not a full chip §5.1.

Lineage

• Builds on: the straight-through estimator for rounding §5; the DoReFa k-bit weight quantizer with its tanh normalisation §5 eq. 1. WRPN drops the tanh and the max normalisation in favour of clip-and-round §5. The BWN scaling for 1-bit weights §5. Widening follows Wide ResNet, but keeps the depth and the operand precision fixed §3.
• Compared against: BNN, DoReFa, XNOR-Net, BWN, TWN, TTQ, INQ, fine-grained ternary quantization and a logarithmic low-precision method §1 §6.
• Superseded by: not stated in this card.