---
kind: concept
slug: jacobian-regularization
title: Jacobian regularization
summary: >-
  Penalises a network's Jacobian during training, so its response to small input changes stays bounded or low-rank. Covered: a top-singular-value penalty on a random sketch of a diffusion denoiser's input Jacobian, used against memorisation. The trade-off: each step costs about 3.8 times more, the penalty weight is not stated, and the one source is a diffusion paper with author-reported gains.
tags: [regularization, jacobian, diffusion-models, memorization]
papers: ["2611.01775"]
related: [diffusion-models, training-data-extraction, classifier-free-guidance, spectral-normalization, gradient-penalty, lipschitz-constraint, nuclear-norm-regularization]
status: draft
updated_at: 2026-11-21
---

## Overview

A regulariser of this kind penalises the Jacobian of a learned map, the matrix
of partial derivatives of its outputs with respect to its inputs. A norm or
singular value of that matrix measures how strongly the map responds in its
most sensitive direction, and the penalty pushes that response down. The part
all variants share is that the penalty is computed from matrix-vector products
or a small sketch, never from the full matrix.

The only source here applies this to diffusion denoisers. At intermediate noise
levels, a memorised sample's input Jacobian shows one dominant singular
direction before the spectrum relaxes again [arxiv:2611.01775 §3.2]. The paper
treats that dominant direction as the signature of a trajectory committing to a
training copy, and penalises it during training [arxiv:2611.01775 §4.1].

## Variants

### Top singular value of a random sketch

The penalty is the squared top singular value of W J, where J is the input
Jacobian of the denoiser and W has k orthonormal rows [arxiv:2611.01775 §4.1]
[arxiv:2611.01775 app. D.2]. The top singular value is the largest eigenvalue
of a k by k Gram matrix, and the sketch is built with vector-Jacobian products
[arxiv:2611.01775 app. D.1]. The penalty is applied to only m states per
update. They are chosen by the input-gradient norm of the base loss, from a
timestep window, so the base backward pass supplies the ranking
[arxiv:2611.01775 §4.1] [arxiv:2611.01775 app. D.1]. CelebA defaults are batch
512, m = 32, k = 8 and Adam with learning rate 1e-4 [arxiv:2611.01775 tab. 3].
The CelebA runs also keep a slow and a fast weight copy, merged every 500
updates [arxiv:2611.01775 app. D.3] [arxiv:2611.01775 tab. 3]. Card:
[Spectral signatures, memorization basins](/papers/2611.01775).

The authors report that this lowers memorisation on synthetic sets and on
CelebA32, and that FID rises [arxiv:2611.01775 tab. 1]. No official code was
found at compile time, so these numbers are the authors' own and unverified
[arxiv:2611.01775 §5].

### Not a training penalty: detection and sampling-time reflection

The same Jacobian quantities also appear at inference, and two of the paper's
uses are not regularisers. The JER score estimates the effective rank of the
Jacobian from one low-noise query, using random probes, and is thresholded to
flag copied outputs [arxiv:2611.01775 §4.2] [arxiv:2611.01775 app. B.2]. The
GeoSteer operator reflects the model's denoising displacement along the top
right singular vector during sampling, leaving the weights unchanged
[arxiv:2611.01775 §4.3] [arxiv:2611.01775 app. C.3]. They are listed here so
that they are not mistaken for training-time regularisers.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Training a diffusion model where copying matters more than FID, and about 3.8 times the step time is acceptable | Top-singular-value sketch penalty | Lowest memorisation in every column of the authors' table, CelebA32 1.56 against 8.80 for standard training, but FID rises from 54.89 to 59.77 [arxiv:2611.01775 tab. 1] [arxiv:2611.01775 tab. 4] |
| Training where FID matters more than the last drop in memorisation | Tikhonov score regularisation, or the reduced-learning-rate control | In the same table Tikhonov has the best CelebA32 FID (49.03) with memorisation 6.50, and reduced learning rate has 6.80 [arxiv:2611.01775 tab. 1] |
| Model already trained, no access to training | JER detection or GeoSteer, not this penalty | Both work on a finished model, at sampling or after generation [arxiv:2611.01775 §4.2] [arxiv:2611.01775 §4.3] |
| Need an exact Jacobian rather than a sketch | Only as a reference point, not for training | In the steering runs, a full Jacobian reached 3.0% DDIM copy rate against 18.0% at k = 8 [arxiv:2611.01775 app. F.8] |

Cost: a training step with the penalty took about 1046 ms against 272 ms for a
plain update at batch 512 on the authors' hardware, and peak memory rose from
16.95 to 19.63 GiB [arxiv:2611.01775 tab. 4]. Most of the extra time comes
from building and differentiating the sketch [arxiv:2611.01775 app. D].

## Implementation sketch

The penalty weight is the main missing input: the paper states it for the
synthetic runs only, as a scale that makes the penalty gradient about a tenth
of the base gradient [arxiv:2611.01775 app. E.1]. The CelebA value is not
given [arxiv:2611.01775 app. D.1].

```pseudocode
# One training step with a sketched top-singular-value penalty
# [arxiv:2611.01775 §4.1] [arxiv:2611.01775 alg. 2]
# Inputs: noised batch; fixed sketch W (k rows); m states; weight lam
# Output: scalar loss

def penalised_loss(model, batch, W, m, lam):
    # Base diffusion or flow loss, averaged over the batch [app. D.1]
    base = mean(per_example_loss(model, batch))
    # Input gradients come from the same backward pass [alg. 2]
    grads = input_gradients(base, batch.states)
    # Keep the m states with the largest gradient norm in the window [app. D.1]
    chosen = top_m(norm(grads, axis=1), batch.timesteps, m)
    # Detach the selected states before sketching [alg. 2]
    states = detach(batch.states[chosen])
    # Sketch each input Jacobian as W J [app. D.2]
    sketched = sketch_jacobians(model, states, W)
    # Squared top singular value of W J, from a k by k Gram matrix [app. D.1]
    penalty = mean(max_eigenvalue(sketched @ transpose(sketched)))
    return base + lam * penalty
```

## Contradictions

There is only one source card, so no disagreement between papers. Within that
paper, the text says the geometry improves with the sketch size k [arxiv:2611.01775 §5.1],
but the steering copy rates for DDIM are 18.0%, 19.7%, 19.0% and 10.0% for
k = 8, 16, 32 and 64 [arxiv:2611.01775 app. F.8]. The authors' own figures are
steering results, not training results, so the trend does not settle the
training choice of k.

## Papers

- [Spectral signatures, memorization basins](/papers/2611.01775): a top-singular-value penalty on a sketched denoiser Jacobian for training (GeoReg-DM), a single-query detector (JER) and a sampling-time reflection (GeoSteer-DM), with author-reported reductions in memorisation.
