---
kind: concept
slug: gradient-leakage
title: Gradient leakage
summary: >-
  Recovering training inputs and labels from gradients that federated parties share. The one variant covered here, CAFE, needs the server to know each batch's row indices in vertical federated learning. Batch size did not block it in the tested range, and the proposed fake-gradient defense has no formal privacy guarantee.
tags: [privacy, federated-learning]
papers: ["2110.15122"]
related: [federated-learning, vertical-federated-learning, differential-privacy]
status: draft
updated_at: 2026-11-09
---

## Overview

In federated training, parties send gradients rather than raw records. A common
belief is that larger batches make the gradients harder to invert, because each
gradient mixes more records [arxiv:2110.15122 §1]. Gradient leakage is the
attack that tests this belief: the attacker starts from random dummy inputs and
labels, then changes them until the gradients they produce match the gradients
it observed.

The invariant shared by the attacks is that the observed gradients are a fixed
set of numbers, while the unknown inputs number K times the input size, where K
is the batch size. Recovery therefore depends on extra structure. The attack in
[CAFE](/papers/2110.15122) gets that structure from the batch indices: in
vertical federated learning, the server sends each batch's row ids to every
party, so the server knows which rows each gradient came from [arxiv:2110.15122
§3.1] [arxiv:2110.15122 §3.3].

## Variants

### Index-aligned inversion in vertical FL (CAFE)

The attack runs in three stages, each a fitting problem [arxiv:2110.15122 §3.3].

1. **Gradient at the first-layer outputs.** A matrix V is fitted so that its
   sums over each known batch match the first-layer bias gradient
   [arxiv:2110.15122 eq. 6] [arxiv:2110.15122 alg. 1]. The authors prove this
   objective is strongly convex when K < N, for fixed model parameters
   [arxiv:2110.15122 §3.3] [arxiv:2110.15122 app. B].
2. **First-layer inputs.** A matrix of estimated inputs is fitted so that its
   product with V matches the first-layer weight gradient [arxiv:2110.15122
   eq. 8] [arxiv:2110.15122 alg. 2].
3. **Data and labels.** Dummy rows and dummy labels are optimized against three
   terms: gradient matching over all parameters, a truncated total-variation
   term on the dummy inputs, and a term that pulls the dummy rows' first-layer
   inputs toward the stage-2 estimate [arxiv:2110.15122 eq. 9]
   [arxiv:2110.15122 alg. 4].

The two schedules are a nested loop, which runs stages 1 and 2 to a fixed
budget before stage 3, and a single loop, which does one update of each stage
per iteration [arxiv:2110.15122 alg. 3] [arxiv:2110.15122 alg. 4]. The text
says the single loop needs fewer iterations [arxiv:2110.15122 §4.2]. Table 5
shows a mixed picture: the single loop took 7300, 6600 and 12400 iterations on
CIFAR-10, MNIST and Linnaeus 5, while the nested row for Step III took 1635 and
2468 on CIFAR-10 and MNIST, and the nested Step II and III rows hit the 20000
cap on Linnaeus 5 [arxiv:2110.15122 tab. 5].

Author-reported results at batch size 40 with four workers: PSNR 31.83 on
CIFAR-10, 43.15 on MNIST and 33.22 on Linnaeus 5, against 3.86 to 18.94 for five
baselines the authors implemented [arxiv:2110.15122 tab. 2]. Removing the
internal-representation term hurts most in the authors' ablation
[arxiv:2110.15122 tab. 4]. The paper reports a maximum of 800 leaked images at
batch size 100, 4 workers, on its own hardware [arxiv:2110.15122 §4].

### Earlier attacks (named only)

The paper compares against DLG, iDLG, inverting gradients, SAPAG, R-GAP, a
theory-oriented attack, GradInversion and a leakage evaluation framework
[arxiv:2110.15122 §2] [arxiv:2110.15122 tab. 1]. Those papers have no card
here, so this page describes none of their methods. The DLG objective is the
baseline that stage 3 generalizes [arxiv:2110.15122 §3.3]. Links for reading:
[DLG on arXiv](https://arxiv.org/abs/1906.08935) and
[GradInversion on arXiv](https://arxiv.org/abs/2104.07586).

## When to use which

The CAFE attack applies when its conditions hold. Check these against your
setup before deciding how much weight the paper's results carry.

| Situation | Implication | Why |
|---|---|---|
| Vertical FL, and batch row ids go to every party | The attack's key precondition is met | The server knows the rows in each batch, which the attack needs [arxiv:2110.15122 §3.3] |
| Batch size is the only defense you planned | Not supported by this paper's range | PSNR stayed above 30 (CIFAR-10), 32 (MNIST) and 28 (Linnaeus 5) for K from 10 to 100 with N = 800 [arxiv:2110.15122 tab. 3] |
| Only fully connected layers are learned | Steps 1 and 2 may recover the data alone | The paper notes this for such models [arxiv:2110.15122 §3.3] |
| You need a formal privacy guarantee | Fake gradients do not provide one | The defense is not differential privacy [arxiv:2110.15122 app. F.3] |
| You need a leakage-blocking defense with small utility loss | Fake gradients are the paper's proposal | Authors report PSNR 7.67 with fake gradients against 28.68 with true gradients on CIFAR-10 at unchanged parameters [arxiv:2110.15122 §4.4]; the training-loss gap is larger than the text suggests (see Contradictions) |
| Training while being attacked | Higher model learning rates lowered PSNR in one experiment | The paper calls this an effective defense, but no theory covers it and the settings are defined only by learning rate [arxiv:2110.15122 §4.3] [arxiv:2110.15122 tab. 7] |

**Defense comparison.** The paper compares fake gradients with gradient
clipping to norm 3 under differential privacy at several epsilon values. At
8000 iterations the training loss is 2.69 under every DP setting, 1.15 with fake
gradients and 0.46 with true gradients [arxiv:2110.15122 tab. 11]. The authors
say the noise needed to block leakage also stops the loss from falling
[arxiv:2110.15122 app. F.3].

**Implementation caveats.** Steps 1 and 2 are proven only for fixed model
parameters [arxiv:2110.15122 §3.3]. The paper's worked example has layer width
1024 with N = 800, which fits the Theorem 2 condition N < d2; the released code
uses width 256 and the card's Pitfalls section explains why the guarantee then
does not cover it. The released code also differs from the paper on learning
rates, the TV gate and the PSNR definition, and covers MNIST only. Treat
reproduction numbers as needing re-measurement.

## Implementation sketch

The shared core is a loop over batches that fits the three stages in turn. The
single-loop form is shown; the nested form runs the first two stages to
convergence before each stage-3 update [arxiv:2110.15122 alg. 3].

```pseudocode
# Known to the server: batch row ids s, observed aggregated gradients [§3.1]
# Stage 1: fit V so its batch sums match the first-layer bias gradient [eq. 6-7]
# Stage 2: fit H_hat so H_hat^T V matches the first-layer weight gradient [eq. 8]
# Stage 3: fit dummy inputs x_hat and labels y_hat to the observed gradients,
#          a truncated TV term, and H_hat [eq. 9]
def attack_iteration(s, grads, V, H_hat, x_hat, y_hat):
    V = update_stage1(V, s, grads.first_bias)
    H_hat = update_stage2(H_hat, V, s, grads.first_weight)
    x_hat, y_hat = update_stage3(x_hat, y_hat, s, grads, H_hat)
    return V, H_hat, x_hat, y_hat
```

Optimizer details are not fixed by the paper for stages 2 and 3 [arxiv:2110.15122
alg. 2] [arxiv:2110.15122 alg. 4]. The per-dataset learning rates and loss
weights are in [arxiv:2110.15122 tab. 9].

## Contradictions

None known across papers: this page cites one card. Within that card the
authors call fake gradients similar in training trajectory to true gradients
[arxiv:2110.15122 §4.4], while the same paper's Table 11 gives a training loss
of 1.15 against 0.46 at 8000 iterations [arxiv:2110.15122 tab. 11]. The
batch-size results are not monotonic on CIFAR-10: K = 40 (PSNR 31.83) scores
below K = 20 (35.70), and the paper gives no explanation [arxiv:2110.15122 tab. 3].
The text's claim that the single loop needs fewer iterations [arxiv:2110.15122
§4.2] is not borne out for the nested Step III row in Table 5, which needed fewer
iterations on CIFAR-10 and MNIST [arxiv:2110.15122 tab. 5].

## Papers

- [CAFE: Catastrophic Data Leakage in Vertical Federated Learning](/papers/2110.15122): index-aligned three-stage inversion for vertical FL, with a fake-gradient defense.