Binary neural network

Overview

A binary network keeps its weights and activations as values of +1 or −1, so a multiply becomes a cheap bitwise operation. With binary values, a dot product reduces to XNOR and popcount operations 2112.06126 §6.1 eq. 14. The gap to full precision is the problem: a single-bit scheme loses much accuracy on large tasks. ABC-Net Towards Accurate Binary Convolutional Neural Network closes part of that gap by writing each layer's real weights as a weighted sum of M binary filters, and its activations as a weighted sum of N binary activations 1711.11294 §3.1.1, §3.2. The convolution then becomes M×N binary convolutions, each scaled by trained factors 1711.11294 eq. 12. The network is trained with full-precision gradients through a straight-through estimator, starting from a full-precision pretrained model [arxiv:1711.11294 §3.1.1, §3.3].

The invariant across the variants is the test-time form: binary bases and binary activations, combined with a small number of real scale factors and shifts 1711.11294 §3. Two surveys add the earlier single-bit schemes and an FPGA case study. 2112.06126 §6.1 covers the single-bit weight and sign schemes, and 2505.08992 §V.A describes one binarised design on an FPGA. Both surveys copy their numbers from the works they cite, so their results are secondhand. Channel-wise weight bases are described but not tested 1711.11294 §3.1.2.

Variants

Per-layer weight bases with binary activations

M weight bases shared by the whole layer, and N binary activations with trainable shifts 1711.11294 §3.1.1, §3.2. This is the setting in which the paper reports its headline number: ResNet-18 reaches 65.0% Top-1 at M = N = 5, against 69.3% full precision 1711.11294 tab. 2. Card: Towards Accurate Binary Convolutional Neural Network.

Binary weights with full-precision activations

The same weight approximation, but the activations are left at full precision. At M = 5 on ResNet-18 this reaches 68.3% Top-1 1711.11294 tab. 1. The activations then stay real-valued, so the bitwise argument in the paper applies to the weight side only 1711.11294 §5.2. The paper reports no speed measurement for either variant.

Single basis and scale (BWN and XNOR-Net)

With M = 1, each layer keeps one binary basis and one real scale, which the ABC-Net paper says reduces to the BWN scheme 1711.11294 §3.1.2. The survey gives the scale as the mean absolute weight of the matrix 2112.06126 eq. 15 2112.06126 §6.1. On ImageNet with ResNet-18, the survey lists BWN at 60.8% Top-1 with full-precision activations and XNOR-Net at 51.2% with binary activations 2112.06126 tab. 4. ABC-Net's own table lists the same two numbers 1711.11294 tab. 3. Its M = 1 run with full-precision activations reaches 62.8% 1711.11294 tab. 1, which is not the same row (see Contradictions).

Sign and stochastic binarization without scales (BinaryConnect, BinaryNet)

BinaryConnect draws each weight as +1 with a probability set by a hard sigmoid of the real weight. BinaryNet takes the sign of the weight instead 2112.06126 §6.1 eq. 11, eq. 12, eq. 13. The survey notes that BinaryConnect keeps real-valued master weights for the gradient updates [arxiv:2112.06126 §6.1]. The survey's CIFAR-10 table lists BinaryConnect at 91.73% with binary weights and float32 activations, and BNN at 88.6% with 1-bit weights and activations, on a CNN 2112.06126 tab. 3. These are CIFAR-10 numbers and cannot be compared with the ImageNet rows above.

Binarised streaming pipelines on an FPGA (FINN)

FINN is a case study in the FPGA survey. It runs a binarised network as a layer-pipelined stream, with weights in LUT arrays and 1-bit activations passed between stages 2505.08992 §V.A. The survey quotes FINN on a Zynq-7020 at 1-bit weights and activations, about 5 TOPS counted in binary operations, around 50k LUTs (90% of the device) and no DSP use. Energy efficiency is not reported 2505.08992 tab. 2. Binary operations are not comparable with the TOPS figures of other designs in the same table. This case uses 1-bit weights and activations, so it does not run the multi-basis form of ABC-Net.

When to use which

The paper's accuracy numbers are from ImageNet with ResNet-18, ResNet-34 and ResNet-50 1711.11294 tab. 2. A VGG-like network on a forest-trail task is reported separately, with a single run per model [arxiv:1711.11294 tab. S5]. The survey's CIFAR-10 results cover different networks and are not ranked against ImageNet results 2112.06126 tab. 3. Results for other tasks or architectures are not claimed.

Implementation sketch

Python-flavoured pseudocode of the test-time form. The shifts, the number of bases and the scales are trained; the sign of zero and the least-squares solver are not stated in the ABC-Net paper 1711.11294 §3.1.1, eq. 3. The BinaryNet sign maps zero to +1 2112.06126 eq. 13. That is a different scheme and does not settle the ABC-Net choice.

# Step 1: M binary weight bases from shifted, centred weights  [eq. 2]
def weight_bases(W, shifts):
    centred = W - mean(W)
    spread = std(W)
    # sign(0) is not stated for ABC-Net; BinaryNet maps zero to +1  [eq. 13]
    return [sign(centred + u * spread) for u in shifts]

# Step 2: scales alpha by least squares on the fixed bases  [eq. 3]
# Solver not stated in the paper
alpha = least_squares(bases, W)

# Step 3: N binary activations with trainable shifts v_n  [eq. 8, eq. 9]
def binary_acts(R, v):
    # Threshold sits at R = 0.5 - v_n
    return [2 * (clip(R + v_n, 0, 1) >= 0.5) - 1 for v_n in v]

# Step 4: convolution as M x N bitwise convolutions  [eq. 11, eq. 12]
out = sum(alpha[m] * beta[n] * conv(bases[m], acts[n])
          for m in range(M) for n in range(N))

Training uses a straight-through gradient through the sign and rounding steps 1711.11294 eq. 7, eq. 10. The mask given for the activations does not match the forward threshold, see Contradictions.

Contradictions

• Gap to full precision, same paper. The text gives a Top-1 gap of 0.9 points for M = 5 with full-precision activations 1711.11294 §4.1.1. Table 1 gives 69.3 − 68.3 = 1.0 points 1711.11294 tab. 1.
• "Comparable accuracy" claim. The introduction says the accuracy is comparable to the full-precision model 1711.11294 §1. The tables show that for binary weights with full-precision activations only (68.3 vs 69.3), and show a 4.3-point gap for binary weights and activations at M = N = 5 1711.11294 tab. 2, tab. 3.
• Straight-through mask versus threshold. The forward threshold is at R = 0.5 − v 1711.11294 eq. 9. The gradient mask is written as 0 ≤ R − v ≤ 1, which is centred on a different point 1711.11294 eq. 10. The paper does not discuss the difference.
• BNN row in the comparison table. The table lists BNN at 42.2% Top-1; the paper states this is its own run on ImageNet, not a published figure 1711.11294 tab. 3, §4.1.3. The survey's BNN figure of 88.6% is a CIFAR-10 result 2112.06126 tab. 3. The two figures are for different datasets and are not a disagreement.
• BWN at M = 1. ABC-Net says its M = 1 case reduces to BWN [arxiv:1711.11294 §3.1.2]. Its own table reports M = 1 with full-precision activations at 62.8% Top-1 1711.11294 tab. 1. The survey lists BWN at 60.8% for the same network and setting 2112.06126 tab. 4. The paper does not explain the gap between the two numbers.

Papers

• Towards Accurate Binary Convolutional Neural Network: weight and activation approximation by M binary weight bases and N binary activations, with ResNet-18 results on ImageNet.
• Neural Network Quantization for Efficient Inference: A Survey: survey of quantization schemes, including the binary weight, sign and stochastic schemes and their CIFAR-10 and ImageNet numbers.
• Edge-AI FPGA dataflow and tiling review: survey of FPGA dataflow and tiling, with the FINN binarised streaming pipeline as one case study.