Concept · 1 paper · updated 2026-11-21
Jacobian regularization
Penalises a network's Jacobian during training, so its response to small input changes stays bounded or low-rank. Covered: a top-singular-value penalty on a random sketch of a diffusion denoiser's input Jacobian, used against memorisation. The trade-off: each step costs about 3.8 times more, the penalty weight is not stated, and the one source is a diffusion paper with author-reported gains.
regularizationjacobiandiffusion-modelsmemorization
Markdown4 min read
AI-compiled overview, updated 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
A regulariser of this kind penalises the Jacobian of a learned map, the matrix of partial derivatives of its outputs with respect to its inputs. A norm or singular value of that matrix measures how strongly the map responds in its most sensitive direction, and the penalty pushes that response down. The part all variants share is that the penalty is computed from matrix-vector products or a small sketch, never from the full matrix.
The only source here applies this to diffusion denoisers. At intermediate noise levels, a memorised sample's input Jacobian shows one dominant singular direction before the spectrum relaxes again 2611.01775 §3.2. The paper treats that dominant direction as the signature of a trajectory committing to a training copy, and penalises it during training 2611.01775 §4.1.
Variants
Top singular value of a random sketch
The penalty is the squared top singular value of W J, where J is the input Jacobian of the denoiser and W has k orthonormal rows 2611.01775 §4.1 2611.01775 app. D.2. The top singular value is the largest eigenvalue of a k by k Gram matrix, and the sketch is built with vector-Jacobian products 2611.01775 app. D.1. The penalty is applied to only m states per update. They are chosen by the input-gradient norm of the base loss, from a timestep window, so the base backward pass supplies the ranking 2611.01775 §4.1 2611.01775 app. D.1. CelebA defaults are batch 512, m = 32, k = 8 and Adam with learning rate 1e-4 2611.01775 tab. 3. The CelebA runs also keep a slow and a fast weight copy, merged every 500 updates 2611.01775 app. D.3 2611.01775 tab. 3. Card: Spectral signatures, memorization basins.
The authors report that this lowers memorisation on synthetic sets and on CelebA32, and that FID rises 2611.01775 tab. 1. No official code was found at compile time, so these numbers are the authors' own and unverified 2611.01775 §5.
Not a training penalty: detection and sampling-time reflection
The same Jacobian quantities also appear at inference, and two of the paper's uses are not regularisers. The JER score estimates the effective rank of the Jacobian from one low-noise query, using random probes, and is thresholded to flag copied outputs 2611.01775 §4.2 2611.01775 app. B.2. The GeoSteer operator reflects the model's denoising displacement along the top right singular vector during sampling, leaving the weights unchanged 2611.01775 §4.3 2611.01775 app. C.3. They are listed here so that they are not mistaken for training-time regularisers.
When to use which
| Situation | Use | Why |
|---|---|---|
| Training a diffusion model where copying matters more than FID, and about 3.8 times the step time is acceptable | Top-singular-value sketch penalty | Lowest memorisation in every column of the authors' table, CelebA32 1.56 against 8.80 for standard training, but FID rises from 54.89 to 59.77 2611.01775 tab. 1 2611.01775 tab. 4 |
| Training where FID matters more than the last drop in memorisation | Tikhonov score regularisation, or the reduced-learning-rate control | In the same table Tikhonov has the best CelebA32 FID (49.03) with memorisation 6.50, and reduced learning rate has 6.80 2611.01775 tab. 1 |
| Model already trained, no access to training | JER detection or GeoSteer, not this penalty | Both work on a finished model, at sampling or after generation 2611.01775 §4.2 2611.01775 §4.3 |
| Need an exact Jacobian rather than a sketch | Only as a reference point, not for training | In the steering runs, a full Jacobian reached 3.0% DDIM copy rate against 18.0% at k = 8 2611.01775 app. F.8 |
Cost: a training step with the penalty took about 1046 ms against 272 ms for a plain update at batch 512 on the authors' hardware, and peak memory rose from 16.95 to 19.63 GiB 2611.01775 tab. 4. Most of the extra time comes from building and differentiating the sketch 2611.01775 app. D.
Implementation sketch
The penalty weight is the main missing input: the paper states it for the synthetic runs only, as a scale that makes the penalty gradient about a tenth of the base gradient 2611.01775 app. E.1. The CelebA value is not given 2611.01775 app. D.1.
# One training step with a sketched top-singular-value penalty
# [arxiv:2611.01775 §4.1] [arxiv:2611.01775 alg. 2]
# Inputs: noised batch; fixed sketch W (k rows); m states; weight lam
# Output: scalar loss
def penalised_loss(model, batch, W, m, lam):
# Base diffusion or flow loss, averaged over the batch [app. D.1]
base = mean(per_example_loss(model, batch))
# Input gradients come from the same backward pass [alg. 2]
grads = input_gradients(base, batch.states)
# Keep the m states with the largest gradient norm in the window [app. D.1]
chosen = top_m(norm(grads, axis=1), batch.timesteps, m)
# Detach the selected states before sketching [alg. 2]
states = detach(batch.states[chosen])
# Sketch each input Jacobian as W J [app. D.2]
sketched = sketch_jacobians(model, states, W)
# Squared top singular value of W J, from a k by k Gram matrix [app. D.1]
penalty = mean(max_eigenvalue(sketched @ transpose(sketched)))
return base + lam * penaltyContradictions
There is only one source card, so no disagreement between papers. Within that paper, the text says the geometry improves with the sketch size k 2611.01775 §5.1, but the steering copy rates for DDIM are 18.0%, 19.7%, 19.0% and 10.0% for k = 8, 16, 32 and 64 2611.01775 app. F.8. The authors' own figures are steering results, not training results, so the trend does not settle the training choice of k.
Papers
- Spectral signatures, memorization basins: a top-singular-value penalty on a sketched denoiser Jacobian for training (GeoReg-DM), a single-query detector (JER) and a sampling-time reflection (GeoSteer-DM), with author-reported reductions in memorisation.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.