---
kind: concept
slug: error-feedback
title: Error feedback
summary: >-
  Error feedback keeps a dense copy of the parameters and computes each step at a compressed copy, then applies the update to the dense copy so the compression error is not lost. In the one card so far, it lets pruned weights return to a sparse network trained in a single run. Only one paper with a card covers it.
tags: [sparsity, model-compression, optimization]
papers: ["2006.07253"]
related: [network-pruning, dynamic-sparse-training, quantization-aware-training]
status: draft
updated_at: 2026-11-06
---

## Overview

A compressed model (for example a sparse one) is cheaper to run, but the
compression throws information away. Error feedback keeps the discarded part
in a dense copy of the parameters. Each step is computed at the compressed
point, and the update goes to the dense copy, so the difference between the
two (the error) is carried forward rather than lost [arxiv:2006.07253 §3].

The paper links this to the error-feedback literature (Stich et al., 2018;
Karimireddy et al., 2019), which it cites as the source of the idea
[arxiv:2006.07253 §3]. Those papers have no cards here, so this page says
nothing about their content.

## Variants

### Dense copy with a magnitude mask (DPF)

The sparse model is `m ⊙ x`, where `m` is a binary mask recomputed from the
dense weights `x` by global magnitude every 16 iterations. The gradient is
taken at the sparse model and applied to the dense weights, so a weight that
was cut can grow back into the mask [arxiv:2006.07253 §3] [arxiv:2006.07253 alg. 1].
The paper states the error as `e_t = x̂_t − x_t` and writes the update as a
gradient at the dense point plus that error [arxiv:2006.07253 §3].

Card: [Dynamic Model Pruning with Feedback](/papers/2006.07253).

Reported results, all author-reported and from scratch without extra
fine-tuning unless noted: WideResNet-28-2 on CIFAR-10 at 99% sparsity reaches
88.92 ± 0.29 top-1 against 83.45 ± 0.38 for SNIP [arxiv:2006.07253 tab. 2].
On ImageNet, ResNet-50 at 80% sparsity loses 0.47 points top-1 against the
dense model [arxiv:2006.07253 tab. 4]. With 60 extra fine-tuning epochs, the
paper reports 91.76 for ResNet-20 at 90% sparsity [arxiv:2006.07253 tab. 3].

### Not covered here

The paper also uses the same scheme for structured (filter) pruning, but only
on CIFAR-10 and in a preliminary form [arxiv:2006.07253 §6] [arxiv:2006.07253 tab. 7].
Quantised or distributed uses of error feedback are not covered by any card
yet.

## When to use which

Only one card exists, so this table compares the pattern with the simpler
choices the paper itself uses as baselines.

| Situation | Use | Why |
|---|---|---|
| You need sparse weights from one training run and cannot afford a second stage | Dense copy with a magnitude mask (DPF) | The paper reports one-run training with no separate repair step [arxiv:2006.07253 §1] [arxiv:2006.07253 §5.2] |
| You can prune a trained model and fine-tune it | One-shot pruning with fine-tuning (baseline in the paper) | The paper's comparison uses this as a baseline, and it commits to one mask [arxiv:2006.07253 tab. 1] [arxiv:2006.07253 §3] |
| You need a fixed mask for a fresh run | Do not reuse DPF's final mask as a fixed ticket | Retraining the final mask from scratch did not recover the accuracy [arxiv:2006.07253 §6] [arxiv:2006.07253 fig. 5] |
| Structured (filter) pruning at scale | Not established by this card | The structured results are CIFAR-10 only and preliminary [arxiv:2006.07253 §6] |

The mask-update period is a design choice. The paper uses 16 iterations for
all experiments and calls it the best value in its tested range
[arxiv:2006.07253 app. A.3.3].

## Implementation sketch

Core loop of the dense-copy pattern, as in the paper's Algorithm 1. Sparsity
schedule and mask scope are in the card's hyperparameters.

```pseudocode
# Keep dense weights x; recompute the mask from x every p steps  [alg. 1]
def train_with_error_feedback(x, steps, p, sparsity_at):
    for t in range(steps):
        if t % p == 0:
            # Global magnitude mask over the prunable weights  [§5.1]
            mask = magnitude_mask(x, sparsity_at(t))
        # Compressed model used for the forward and backward pass  [§3]
        x_hat = mask * x
        grad = gradient_at(x_hat)
        # The update goes to the dense copy, not the compressed one  [§3]
        x = optimizer_step(x, grad)
    return x, mask * x  # dense copy, compressed model for inference
```

The timing of the schedule (iterations or epochs) is ambiguous in the paper
and is discussed on the card [arxiv:2006.07253 app. A.2].

## Contradictions

None known. Only one card lists this concept.

## Papers

- [Dynamic Model Pruning with Feedback](/papers/2006.07253): applies the dense-copy pattern to dynamic magnitude pruning and reports CIFAR-10 and ImageNet results.