Towards Accurate Binary Convolutional Neural Network

In plain words

Neural networks for images store their settings as decimal numbers and do many multiplications. A binary network keeps every setting and every signal as either +1 or −1. Those operations can be done with cheap logic gates, which makes the network fast and power-friendly on dedicated chips. The catch is that a crude binary network usually loses a lot of accuracy.

This paper tries to close that gap. Instead of forcing each setting to a single +1 or −1, it writes each setting as a weighted sum of several such values, so a few cheap binary pieces together stand in for one precise number. The signals entering each layer get the same treatment, with several thresholds that each keep one bit of information. The result is trained with the usual gradient method, using a trick that passes gradients straight through the rounding step.

The authors report that with enough pieces the network gets close to its full-precision version on the ImageNet image benchmark. They argue that run-time work reduces to bitwise operations plus a few scale factors, and they discuss the hardware benefits, but they do not measure any speed.

TL;DR

• Weights of each conv layer are written as a weighted sum of M binary filters; the binary filters come from sign functions of shifted, centred weights §3.1.1 eq. 2 eq. 1.
• Activations are binarized at N thresholds with trainable shifts and scale factors, so the binary inputs carry more than one bit per value §3.2 eq. 11.
• A convolution then becomes M×N bitwise convolutions with scale factors eq. 12.
• Author-reported: on ImageNet with ResNet-18, M=N=5 gives 65.0% Top-1 versus 69.3% full precision and 51.2% for XNOR-Net tab. 3. Binary weights with full-precision activations give 68.3% tab. 1.
• No official code was found. Several details are not stated (see Pitfalls), and the paper reports no measured speed.

Problem

Binary networks with binary weights and activations make multiplications into cheap bitwise operations, but previous binarization schemes lose a lot of accuracy on large classification tasks such as ImageNet. The paper cites Resnet-18 Top-1 dropping from 69.3% to 60.8% with binary weights (BWN) and to 51.2% with binary weights and activations (XNOR-Net) §1. Fixed-point quantization to a few bits is an alternative. The authors argue that a K-bit multiplier needs more hardware than K one-bit operations, so K binary pieces are preferable §5.2.

Method

Notation. A conv layer's weight tensor has shape (w, h, c_in, c_out). Training keeps real-valued weights and updates them every step; at test time only binary weights are used §3.

1. Weight approximation (as a whole, the main variant). §3.1.1

• Write the real filter W as a sum of M binary filters: W ≈ α_1 B_1 + … + α_M B_M, with B_m in {−1, +1} eq. 1 §3.1.1.
• Fix each basis by taking the sign of the centred weights shifted by u_m times the standard deviation of W eq. 2.
• The shifts u_m are evenly spaced over [−std(W), std(W)], or can be trained §3.1.1.
• With the bases fixed, solve a linear least-squares problem for α eq. 3. The paper does not state the solver.
• Forward pass: compute the bases and α, then convolve with each basis and sum weighted by α eq. 4 eq. 5 eq. 6.
• Backward pass: the straight-through estimator (STE) passes the gradient through the sign function, so ∂c/∂W equals Σ_m α_m ∂c/∂B_m eq. 7 §3.1.1.

2. Weight approximation (channel-wise, a variant). §3.1.2

Each output channel gets its own set of M bases and α. This adds no inference cost but costs more training computation. The authors leave it as future work and do not use it in the experiments. With M=1 it reduces to BWN §3.1.2.

3. Binary activations with shifts. §3.2

• Pass the real input R through a clipped function h_v(x) = clip(x + v, 0, 1) eq. 8.
• Binarize with H_v(R) = 2·[h_v(R) ≥ 0.5] − 1, which is +1 when R ≥ 0.5 − v and −1 otherwise eq. 9 §5.3.
• For gradients, use the STE with a mask on the input eq. 10.
• Put batch normalization before the activation, so the input has roughly zero mean and unit variance §3.2.
• Approximate R with N binary activations, R ≈ Σ_n β_n A_n with A_n = H_{v_n}(R) eq. 11. Both β_n and v_n are trainable; they are fixed at test time §3.2.

4. Combined bitwise convolution. §3.2 fig. 1

The convolution becomes Σ_m Σ_n α_m β_n Conv(B_m, A_n) eq. 12. This is M×N binary convolutions, which can run in parallel fig. 1.

# Approximate one layer's real weights with M binary bases  [§3.1.1]
# Input:  weights (w, h, c_in, c_out); shifts (M,)
# Output: bases (M, w, h, c_in, c_out); alpha (M,)

def binary_bases(weights, shifts):
    # Centre the weights and take the spread of the whole tensor  [eq. 2]
    centred = weights - mean(weights)
    spread = std(weights)
    bases = []
    for shift in shifts:
        # Sign of the shifted centred weights; sign(0) not stated  [eq. 2]
        bases.append(sign(centred + shift * spread))
    # Scale factors by least squares on the fixed bases  [eq. 3]
    alpha = least_squares(bases, weights)
    return bases, alpha

# Binary convolution at test time  [§3.2] [eq. 11] [eq. 12]
# Input:  real activations; bases (M,); alpha (M,); act_shifts (N,); betas (N,)
# Output: approximated convolution output

def abc_conv(activations, bases, alpha, act_shifts, betas):
    # Binarize the input at N thresholds, each with its own shift  [eq. 9]
    binary_acts = []
    for shift in act_shifts:
        binary_acts.append(binarize(activations, shift))
    output = 0
    for m, basis in enumerate(bases):
        for n, binary_act in enumerate(binary_acts):
            # One bitwise convolution per (basis, activation) pair  [eq. 12]
            output = output + alpha[m] * betas[n] * conv(basis, binary_act)
    return output

5. Training. §3.3 alg. 1

• Each training step: forward through conv, optional max-pool, batch norm, and the N binary activations for every layer except the last; then backward using full-precision gradients; then update W, β and v alg. 1.
• Max-pooling is placed before batch norm and the activation, because max-pool on binary inputs was observed to push most outputs to +1 §3.3.
• Initialize from a full-precision pretrained model, then fine-tune on the new binary structure §3.3.
• The update rule is named as ADAM or momentum SGD in the algorithm; the experiments use SGD with momentum and an initial learning rate of 0.01 alg. 1 [supp. Parameter settings].

6. Run-time simplifications. §5.3

• The shift in H_v can be folded into a comparator threshold, which avoids extra adds §5.3.
• A batch-norm affine map a·R + b can be folded into the thresholds, giving the threshold (0.5 − v − b)/a §5.3.

Hyperparameters

Per-configuration shift values (from [supp. Parameter settings], which is the source of record):

Results

Author-reported. ImageNet ILSVRC12 classification; Top-1 and Top-5 accuracy in percent §4.1. The text calls the 50k-image set the validation set; the caption of tab. 3 says test accuracy.

Weight approximation only, full-precision activations, ResNet-18 tab. 1 §4.1.1

Configuration grid, weight bases M and activation bases N tab. 2 §4.1.2. Gaps are to the full-precision model of the same network, as in the source.

Comparison with other ResNet-18 schemes tab. 3 §4.1.3. W and A are weight and activation bit widths.

Forest trails (supplementary), VGG-like network, M = N = 5, three classes [supp. Application on visual perception of forest trails] tab. S5

The 0.3-point difference comes from one run per model; no variance is reported.

Paper vs code

No official code.

Searched: the arXiv abstract page lists no code, data or project link. The LaTeX source contains no code or project URL; its only contact details are author email addresses §4. GitHub repository searches for "ABC-Net" returned third-party reimplementations (for example zhang-xuan1314/ABC-Net and zhuyinheng/ABC-Net-pytorch); these are not from the authors and were not used. A search for the title restricted to a DJI organisation returned nothing, and a search for "binary weight bases" restricted to a second DJI organisation name failed with an error.

Pitfalls

• The STE window in eq. 10 does not match the forward threshold. The forward clip in eq. 8 is clip(R + v, 0, 1), so the threshold is at R = 0.5 − v (eq. 9, and the same threshold in §5.3). The STE mask in eq. 10 is I[0 ≤ R − v ≤ 1], which is centred on R = v + 0.5, not on the threshold. For a consistent gradient, the mask would be I[−v ≤ R ≤ 1 − v]. The paper does not discuss this. This is an inference from the equations; implement the forward pass as in eqs. 8–9 and decide the mask deliberately.
• Shift formula is undefined for M = 1. u_i = −1 + (i−1)·2/(M−1) divides by zero when M = 1. The supplementary table uses u = 0 for M = 1 [supp. Parameter settings].
• Sign of zero and the least-squares solver are not stated. The sign function's value at 0 and the solver for α (closed form or iterative) are not given §3.1.1 eq. 3.
• The gradient through α is not discussed. Eq. 7 keeps only the paths through the bases. Whether ∂α/∂W is ignored is implied by the STE step, not stated.
• Large M needs ℓ2 regularisation, with no strength given. The bases become correlated for large M, which can make eq. 3 rank-deficient §3.3.
• Learning-rate decay in Algorithm 1 is applied inside the layer loop. The line η ← λη sits inside the loop over layers alg. 1. Whether the rate decays once per step or once per layer is therefore unclear. The decay value λ is not given.
• Algorithm 1 indexes the activation parameters with M. The update loop for β and v runs over m = 1..M, but there are N activations. The intended index is probably n = 1..N. This is an inference from the text (§3.2 defines N activations with their own β_n and v_n).
• First and last layers, and shortcut convolutions, are not stated to be binarized. Algorithm 1 binarizes activations for layers l < L alg. 1. Whether the first conv, the final classifier, and ResNet's downsampling shortcuts use the same scheme is not stated.
• Batch size, epochs, schedule and weight decay are not stated. Only SGD with momentum and an initial learning rate of 0.01 are given [supp. Parameter settings].
• Shift values in the grid look irregular. The M = 3, N = 5 row lists five v values as −3.5, −2.5, −1.5, 0.0, 2.5, which breaks the spacing used in other rows. The M = 5, N = 3 row prints "−0.9, 0.0 0.9" with a missing comma, read here as −0.9, 0.0, 0.9. The forest-trails row prints u = −1, 0.5, 0.0, 0.5, 1, with a repeated 0.5 and unsorted order. Check these against the authors' values before relying on them.
• Large shifts move thresholds far into the tail. A shift of v = −3.5 puts the threshold at R ≥ 4.0 in input units eq. 9. This is sensitive to the batch-norm output scale. The paper does not say how often such thresholds fire.
• Batch-norm folding assumes a positive scale. The folded threshold (0.5 − v − b)/a §5.3 reverses the comparison when a < 0. The paper does not address this case.
• The paper's gap claim and its table disagree. Text says the Top-1 gap to full precision at M = 5 (full-precision activations) is 0.9 points §4.1.1. Table 1 gives 69.3 − 68.3 = 1.0 points.
• The "comparable accuracy" claim applies only to one variant. The introduction claims comparable accuracy to the full-precision model §1. In the table, that holds only for ABC-Net with full-precision activations (68.3 vs 69.3). With binary activations at M = N = 5 the gap is 4.3 points for ResNet-18 tab. 2 tab. 3.
• BNN's ImageNet row is the authors' own run. The paper says it implemented BNN on ImageNet §4.1.3, footnote. Do not treat 42.2% as BNN's published figure.
• Speed and energy are not measured. The paper argues that bitwise operations are faster and cheaper than fixed-point multiplies §5.2, and describes M×N binary convolutions per layer eq. 12. It reports no timings, hardware results or energy figures. The count of M×N binary convolutions also means the work per layer grows with M and N, and the net speedup is not quantified.
• Training cost is higher than the test-time cost. The training forward pass computes all bases and binary activations alg. 1. The channel-wise variant needs more training computation §3.1.2.
• Forest-trails result has no variance. The 78.0% vs 77.7% comparison comes from one run per model, with no error bars tab. S5. The dataset is described as 8 hours of 1920 × 1080, 30 fps video, with three labels (turn right, go straight, turn left) [supp. Application on visual perception of forest trails].
• Numbering of equations and figures is derived, not read from the PDF. Equation numbers were counted in the source order (eqs. 1–12 in the main text). Figure and table numbers in the supplement use the S prefix. Check against the compiled PDF before quoting.

Lineage

• Builds on: Binary-weight networks and XNOR-Net (Rastegari et al. 2016) for binary weights with a scaling factor, which is the M = 1 case of this paper's weight approximation §2 §3.1.2. Binarized Neural Networks (Courbariaux et al. 2016, arXiv:1602.02830) for binary weights and activations §2. DoReFa-Net (Zhou et al. 2016, arXiv:1606.06160) for the clipped activation h_v §3.2. The straight-through estimator (Bengio et al. 2013, arXiv:1308.3432) §3.1.1.
• Compared against: BWN and XNOR-Net (Rastegari et al. 2016) tab. 3; DoReFa-Net (Zhou et al. 2016) tab. 3; BNN (Courbariaux et al. 2016) tab. 3. Fixed-point quantization schemes are discussed but not tested §5.2.
• Superseded by: not stated in this card's sources.