---
kind: concept
slug: feature-caching-across-timesteps
title: Feature caching across timesteps
summary: >-
  Speeds up diffusion sampling by running the full network only every N steps and reusing cached high-level features in between. Training-free, for U-Net samplers with skip connections. Author-reported 2.15x on Stable Diffusion v1.5 and 4.12x on ImageNet LDM-4-G, both with quality loss.
tags: [inference, latency, diffusion]
papers: ["2312.00858"]
related: [diffusion-models, latent-diffusion]
status: draft
updated_at: 2026-10-24
---

## Overview

Diffusion sampling runs the denoiser once per step, so inference cost grows
with the step count [arxiv:2312.00858 §1]. Feature caching exploits the fact
that the coarse, high-level features of a U-Net change little between adjacent
steps. Some steps run the full network and store an intermediate feature; the
steps in between reuse it and recompute only the shallow part of the network
[arxiv:2312.00858 §3.2] [arxiv:2312.00858 §3.3].

The invariant: the sampler's update rule is unchanged, and the weights are not
retrained. Only the noise prediction is approximated on the cached steps.

## Variants

The only variant with a card so far is the one below. Other caching methods
would be added here once their papers have cards.

### Uniform and non-uniform 1:N caching (DeepCache)

A branch index m picks one skip connection. A full pass at step t caches the
feature that enters up block m from the main branch. Each later step in the
block runs only down blocks 1 to m and reuses the cached feature
[arxiv:2312.00858 §3.3] [arxiv:2312.00858 eq. 4] [arxiv:2312.00858 eq. 5].
The uniform schedule puts full passes every N steps, so there are ceil(T / N)
of them [arxiv:2312.00858 eq. 6]. The non-uniform schedule spaces the full passes
more densely where the authors observed feature similarity to drop, using a
power p and a centre c [arxiv:2312.00858 §3.3] [arxiv:2312.00858 eq. 7].
Card: [DeepCache: Accelerating Diffusion Models for Free](/papers/2312.00858).

Reported settings: N between 2 and 20 on the LDM, CIFAR-10 and LSUN models,
and N=5 as the Stable Diffusion setting in the main table
[arxiv:2312.00858 tab. 1] [arxiv:2312.00858 tab. 2] [arxiv:2312.00858 tab. 3].
The sampler is DDIM or PLMS, and the authors state other fast samplers also work
[arxiv:2312.00858 §4.1] [arxiv:2312.00858 app. A]. The method was not tested with
other samplers in the paper.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Pretrained U-Net sampler, no budget for retraining | Feature caching with a fixed N | Training-free; no weight change [arxiv:2312.00858 §1] |
| Model is not a U-Net with skip connections (for example a transformer denoiser) | Not covered by this card | The method relies on the skip-connection structure [arxiv:2312.00858 §3.1] |
| Quality matters more than speed | N=2 or N=3 | On LDM-4-G, N=2 gives FID 3.39 and N=3 gives 3.44, against 3.37 for the regenerated baseline [arxiv:2312.00858 tab. 1] |
| Need about 4x speed on LDM-4-G | N=5 | 4.12x at FID 3.59 [arxiv:2312.00858 tab. 1] |
| Need 7x or more | N=10 or higher, with a clear quality check | FID rises to 4.41 at N=10 and 8.23 at N=20 [arxiv:2312.00858 tab. 1] |
| Text-to-image on Stable Diffusion v1.5 | N=5, checking CLIP | 2.15x with CLIP 29.46 against 29.51 on PartiPrompts [arxiv:2312.00858 tab. 3] |
| Image quality is sensitive to the dataset | Test on your data | On LSUN-Bedroom, N=5 raises FID from 6.62 to 9.49 [arxiv:2312.00858 tab. 2] |
| Fewer sampling steps are an alternative | Compare at equal throughput | On ImageNet with LDM-4-G, the paper reports DeepCache roughly level with DDIM using fewer steps [arxiv:2312.00858 tab. 4] |
| Caching is applied, but the cache is replaced with zeros | Not an option | Zeroing the cache raises FID sharply, so the cached feature carries the signal [arxiv:2312.00858 tab. 5] |

## Implementation sketch

The core is a loop over blocks of N steps. The first step of each block runs
the full network and keeps the cached feature. The others run the shallow part
and reuse it. The schedule function decides where the full passes fall.

```pseudocode
# Full passes at the start of each N-step block, uniform 1:N [§3.3] [eq. 6]
# Input:  T total steps; interval N
# Output: sorted list of steps that run the full network
def full_steps_uniform(T, N):
    return list(range(0, T, N))

# One block of N steps: a full pass, then N - 1 partial passes [alg. 1]
# Input:  latent x, step t, interval N, skip branch m
# Output: the N updated latents
def cached_block(x, t, N, m):
    # Full pass: run every block and keep the feature entering up block m [eq. 4]
    cached, x_next = full_forward_and_cache(x, t, m)
    latents = [x_next]
    # Partial passes: run down blocks 1..m on the newest latent [eq. 5]
    for n in range(2, N + 1):
        # Recompute the shallow path only
        eps = partial_forward(latents[-1], t - n + 1, m, cached)
        # Apply the sampler's own update with this step's timestep
        latents.append(sampler_step(latents[-1], eps, t - n + 1))
    return latents
```

Where variants differ: the skip branch m, the schedule that produces the full
pass steps, and the sampler. The paper's own code does not match its
non-uniform LDM setup in every respect; see the card below.

## Contradictions

- **Stable Diffusion headline.** The abstract's 2.3x speed-up with a 0.05 CLIP
  drop matches the non-uniform N=6 row on PartiPrompts [arxiv:2312.00858 app. F]
  [arxiv:2312.00858 tab. 13]. The main table's N=5 row gives 2.15x with the same
  CLIP score of 29.46 [arxiv:2312.00858 tab. 3]. The paper does not list the
  N=6 row in the main table.
- **LDM FID drop.** The abstract gives a drop of 0.22 against the regenerated
  baseline of 3.37 [arxiv:2312.00858 tab. 1]. The text gives the change as
  "from 3.39 to 3.59". The 3.39 is the uniform N=2 result, not the baseline.
- **Non-uniform LDM with N=20.** The main table gives FID 7.11
  [arxiv:2312.00858 tab. 1]. The appendix comparison gives 7.36
  [arxiv:2312.00858 app. C] [arxiv:2312.00858 tab. 11]. The selected-value table
  gives centre 120 and power 1.4, which the sweep gives as FID 7.13
  [arxiv:2312.00858 app. B] [arxiv:2312.00858 tab. 8]. No single configuration
  reproduces the main value.
- **Small mismatches.** PLMS-50 on COCO shows CLIP 30.30 in the main table and
  30.24 in the appendix [arxiv:2312.00858 tab. 3] [arxiv:2312.00858 app. F].
  The sFID value 50.6 for DDIM-91 in the sampling-steps comparison is far from the
  other rows and is probably a typo; the paper does not confirm it
  [arxiv:2312.00858 tab. 4].
- **Paper and public code.** The released LDM code fixes the centre at 0, but the
  paper's LDM table uses centres of 110 and 120 [arxiv:2312.00858 app. B]. The
  full list of code differences is on the card:
  [DeepCache card](/papers/2312.00858).

## Papers

- [DeepCache: Accelerating Diffusion Models for Free](/papers/2312.00858): training-free
  1:N feature caching for U-Net diffusion samplers, with uniform and non-uniform schedules.