Error feedback

Overview

A compressed model (for example a sparse one) is cheaper to run, but the compression throws information away. Error feedback keeps the discarded part in a dense copy of the parameters. Each step is computed at the compressed point, and the update goes to the dense copy, so the difference between the two (the error) is carried forward rather than lost 2006.07253 §3.

The paper links this to the error-feedback literature (Stich et al., 2018; Karimireddy et al., 2019), which it cites as the source of the idea 2006.07253 §3. Those papers have no cards here, so this page says nothing about their content.

Variants

Dense copy with a magnitude mask (DPF)

The sparse model is m ⊙ x, where m is a binary mask recomputed from the dense weights x by global magnitude every 16 iterations. The gradient is taken at the sparse model and applied to the dense weights, so a weight that was cut can grow back into the mask 2006.07253 §3 2006.07253 alg. 1. The paper states the error as e_t = x̂_t − x_t and writes the update as a gradient at the dense point plus that error 2006.07253 §3.

Card: Dynamic Model Pruning with Feedback.

Reported results, all author-reported and from scratch without extra fine-tuning unless noted: WideResNet-28-2 on CIFAR-10 at 99% sparsity reaches 88.92 ± 0.29 top-1 against 83.45 ± 0.38 for SNIP 2006.07253 tab. 2. On ImageNet, ResNet-50 at 80% sparsity loses 0.47 points top-1 against the dense model 2006.07253 tab. 4. With 60 extra fine-tuning epochs, the paper reports 91.76 for ResNet-20 at 90% sparsity 2006.07253 tab. 3.

Not covered here

The paper also uses the same scheme for structured (filter) pruning, but only on CIFAR-10 and in a preliminary form 2006.07253 §6 2006.07253 tab. 7. Quantised or distributed uses of error feedback are not covered by any card yet.

When to use which

Only one card exists, so this table compares the pattern with the simpler choices the paper itself uses as baselines.

The mask-update period is a design choice. The paper uses 16 iterations for all experiments and calls it the best value in its tested range 2006.07253 app. A.3.3.

Implementation sketch

Core loop of the dense-copy pattern, as in the paper's Algorithm 1. Sparsity schedule and mask scope are in the card's hyperparameters.

# Keep dense weights x; recompute the mask from x every p steps  [alg. 1]
def train_with_error_feedback(x, steps, p, sparsity_at):
    for t in range(steps):
        if t % p == 0:
            # Global magnitude mask over the prunable weights  [§5.1]
            mask = magnitude_mask(x, sparsity_at(t))
        # Compressed model used for the forward and backward pass  [§3]
        x_hat = mask * x
        grad = gradient_at(x_hat)
        # The update goes to the dense copy, not the compressed one  [§3]
        x = optimizer_step(x, grad)
    return x, mask * x  # dense copy, compressed model for inference

The timing of the schedule (iterations or epochs) is ambiguous in the paper and is discussed on the card 2006.07253 app. A.2.

Contradictions

None known. Only one card lists this concept.

Papers

• Dynamic Model Pruning with Feedback: applies the dense-copy pattern to dynamic magnitude pruning and reports CIFAR-10 and ImageNet results.