DeepCache: Accelerating Diffusion Models for Free

In plain words

Making an image with a diffusion model means running a large neural network many times in a row, each time removing a little noise. Every one of those runs is expensive, and most of the work repeats: the coarse, high-level picture of the image changes very little from one run to the next, while the fine detail changes more.

DeepCache takes advantage of that. Every so often it runs the whole network and keeps the coarse part of the result, like a cache in a computer that holds a value which rarely changes. For the runs in between, it only executes the small outer layers that handle fine detail, and it reuses the stored coarse part instead of recomputing it.

The method needs no retraining. It is applied while the image is being generated, to a model that already exists, so it can be tried on pretrained models that would be costly to retrain. The price is that the less often the full run happens, the more the image can drift from what the full model would have made, so the interval between full runs is a dial between speed and quality.

TL;DR

• Training-free acceleration for pretrained U-Net diffusion samplers, with no change to the weights §1.
• A full network pass runs every N steps. The steps in between reuse cached high-level U-Net features and recompute only the shallow blocks §3.3.
• Author-reported: 2.15x throughput on Stable Diffusion v1.5 with a 0.05 CLIP-score drop, and 4.12x on ImageNet LDM-4-G at FID 3.59 tab. 3 tab. 1.
• The headline numbers are not from a single configuration; check the Pitfalls before quoting them.

Problem

Diffusion sampling is sequential: each denoising step needs its own forward pass through the network, so inference is slow §1. Earlier speedups either reduce the number of sampling steps (DDIM and related solvers) or shrink the network at each step through pruning, distillation or quantization §1. Pruning and distillation methods reduce cost per step but normally need retraining, which is costly for large pretrained models such as Stable Diffusion §1 §2.

The paper's assumption is that, in a U-Net sampler, the high-level features are similar across adjacent denoising steps, so recomputing them is wasteful §3.2 fig. 2. The authors report that for several models (Stable Diffusion, LDM and DDPM), at least 10% of the adjacent steps have a feature similarity above 0.95 to a given step, and that for DDPM on LSUN some timesteps are highly similar to up to 80% of the other steps §3.2. The method assumes a U-Net with skip connections, whose outer branches are cheap and whose inner path is expensive §3.1 §3.3. The limitations section says the gain is smaller when the shallowest skip branch holds a large share of the compute §5.

Method

Preliminary. The sampler follows the usual reverse diffusion process, in which a network predicts the noise at each timestep §3.1 eq. 1 eq. 2.

Observation. Adjacent steps produce similar high-level U-Net features §3.2 fig. 2.

Cache and partial pass. For a chosen skip branch m, which links the down block D_m to the up block U_m, the feature that enters up block U_m from the main branch is cached after a full pass at step t §3.3 eq. 4. For the next step, only the down blocks D_1 to D_m are run on the fresh latent. The cached feature replaces the expensive main branch, and the up blocks U_m to U_1 are run to get the noise prediction §3.3 eq. 5. The mechanism is shown in fig. 3.

1:N schedule. One full pass is followed by N minus 1 partial passes. The steps that get a full pass are the multiples of N, so the number of full passes is k equal to ceil(T / N) §3.3 eq. 6.

Non-uniform 1:N. The authors observe that similarity drops in some parts of the trajectory (for example around 40% of the LDM steps), so they place the full passes more densely where similarity falls §3.3. The step set is generated from evenly spaced values in a power-transformed domain, centred on a step c, with a power p. The paper calls this one of several possible schedules §3.3 eq. 7.

Sampler. The algorithm is written for the DDPM update alg. 1. The experiments use DDIM (for DDPM, LDM and the CIFAR and LSUN models) and PLMS (for Stable Diffusion) §4.1. The paper states that the method works with other fast samplers app. A.

Algorithm (one block of N steps).

# One DeepCache block: full pass at step t, then N - 1 partial passes [alg. 1]
# Input:  x_t (B, C, H, W); timestep t; interval N; branch index m
# Output: latents x_{t-1}, ..., x_{t-N}
# sampler_step(x, eps, t) is the sampler's reverse update (DDPM in alg. 1)
def deepcache_block(x_t, t, N, m):
    # Full pass: keep every down-block output, with skips[0] = x_t  [alg. 1]
    skips = [x_t]
    for i in range(1, d + 1):
        skips.append(down_block[i](skips[i - 1]))
    u = middle_block(skips[d])
    # Store the main-branch feature entering up block m  [alg. 1] [eq. 4]
    for i in range(d, 0, -1):
        if i == m:
            cached = u
        u = up_block[i](concat(u, skips[i]))
    latents = [sampler_step(x_t, u, t)]  # x_{t-1}  [alg. 1]
    # Partial passes: run down blocks 1..m only, reuse the cached feature
    # [alg. 1] [eq. 5]
    x_cur = latents[0]
    for n in range(2, N + 1):
        skips = [x_cur]
        for i in range(1, m + 1):
            skips.append(down_block[i](skips[i - 1]))
        u = cached
        for i in range(m, 0, -1):
            u = up_block[i](concat(u, skips[i]))
        # Use this step's own latent and timestep; the paper's line
        # writes x_t and alpha_t here, see Pitfalls  [alg. 1]
        x_cur = sampler_step(x_cur, u, t - n + 1)
        latents.append(x_cur)
    return latents

Full-pass steps, uniform. Taken from the paper's definitions §3.3 eq. 6.

# Steps that run the full U-Net, uniform 1:N  [§3.3] [eq. 6]
# Input:  T total steps; interval N
# Output: list of steps that run the full U-Net
def uniform_full_steps(T, N):
    return list(range(0, T, N))

Full-pass steps, non-uniform. Taken from the paper's definitions §3.3 eq. 7.

# Steps that run the full U-Net, non-uniform 1:N  [§3.3] [eq. 7]
# Input:  T total steps; interval N; centre c; power p
# Output: sorted list of integer steps that run the full U-Net
def nonuniform_full_steps(T, N, c, p):
    # Same number of full passes as the uniform schedule  [§3.3]
    k = ceil(T / N)
    # k values evenly spaced between the two p-th roots, end excluded  [eq. 7]
    roots = linspace_excl((-c) ** (1 / p), (T - c) ** (1 / p), k)
    # Raise to the power p, shift by c, round, and keep unique values  [§3.3]
    steps = unique_int(roots ** p + c)
    return steps

Key equations:

\mathcal{I} = \{\, iN \mid 0 \le i < k \,\}, \qquad k = \lceil T / N \rceil eq. 6

\mathcal{L} = \{\, l_i \mid l_i \in \operatorname{linear\_space}\big((-c)^{1/p},\, (T-c)^{1/p},\, k\big) \,\} eq. 7

\mathcal{I} = \operatorname{unique\_int}\big(\{\, l^{p} + c \mid l \in \mathcal{L} \,\}\big) §3.3

F^{t}_{\text{cache}} \leftarrow U^{t}_{m+1}(\cdot) eq. 4

\operatorname{Concat}\big(D^{t-1}_{m}(\cdot),\, F^{t}_{\text{cache}}\big) eq. 5

Hyperparameters

Results

Author-reported. Setup: class-conditional ImageNet at 256 px with LDM-4-G, 250 DDIM steps; CIFAR-10 at 32 px and LSUN-Bedroom and -Churches at 256 px, with 100 DDIM steps; Stable Diffusion v1.5 at 512 px with 50 PLMS steps on PartiPrompts and COCO2017. Baselines are the unchanged sampler, plus Diff-Pruning, Spectral Diffusion and BK-SDM where the paper includes them §4.1 §4.3. Throughput is measured on one RTX 2080 GPU, and MACs are averaged over all steps §4.2.

Stable Diffusion v1.5 (text-to-image), CLIP score ViT-g/14 tab. 3

ImageNet 256, class-conditional, LDM-4-G, 250 DDIM steps tab. 1

CIFAR-10 (32 px) and LSUN (256 px), 100 DDIM steps tab. 2

Comparison with fewer DDIM steps at equal throughput tab. 4. On ImageNet with LDM-4-G, DeepCache matches DDIM with 59 steps at throughput 0.733 against 0.727 (FID 3.59 for both), and at throughput 0.471 it reaches FID 3.44 against 3.46 for DDIM with 91 steps. The DeepCache rows appear to be the N=5 and N=3 configurations of tab. 1; the table does not name them.

Ablations tab. 5 tab. 6. Replacing the cached features with a zero matrix, under otherwise identical settings, raises FID from 9.74 to 192.98 on CIFAR-10 and from 7.36 to 312.12 on LDM-4-G tab. 5. Adding the shallow partial passes improves DDIM at equal full-pass count: 50 steps 4.67 to 4.35, 20 steps 6.84 to 5.73, and 10 steps 13.36 to 10.38 tab. 6. tab. 5 names its models but not N; tab. 6 names neither the model nor N.

Paper vs code

Code: horseee/DeepCache at 1691a3ab22308fe06685e4ff151a9ab175be2dc4, Apache-2.0 at the root. The LDM experiment folder is a separate MIT-licensed copy. The repository contains diffusers-based pipelines for SD and SDXL, DDPM, and LDM experiment scripts.

• Uniform schedule matches. The DDPM code uses the same full-pass steps as the uniform formula deepcache_denoising.py.
• LDM non-uniform drops the centre. The LDM sampling code takes only the power p, so the centre is fixed at 0 ddpm.py ddpm.py sample_ddp_deepcache.py. The paper's LDM table uses centres of 110 and 120 app. B. The LDM script can't reproduce those values.
• The power is searched, not fixed. The SD and DDPM code lowers p by 0.02 until the step count matches the request pipeline_stable_diffusion.py deepcache_denoising.py. The paper does not describe this. The SD version raises an error if p reaches 1. The DDPM version returns the last result without checking the count.
• Partial pass uses the current latent. The DDPM code feeds the most recent latent and that step's own timestep into each update deepcache_denoising.py. The paper's algorithm writes x_t and alpha_t in the partial-pass update alg. 1. The code follows the intent of the algorithm; the paper's line looks like a typo (not checked with the authors).
• CIFAR-10 "quad" flag is not the cache schedule. The README says to add --skip_type quad for CIFAR-10 README.md. In the code, that flag sets the DDIM timestep spacing to quadratic deepcache.py. The cache schedule is set separately by --non_uniform ddim.py. The paper states only "100 DDIM steps" for CIFAR-10 tab. 2 and does not name a spacing.
• SD defaults differ from the headline setting. The SD README example uses N=2 with the uniform schedule README.md. The generate script defaults to block 0 and layer 0, with no update interval set, and passes no guidance override generate.py. The headline N=5 setting is not a default.
• Guidance scales are not in the paper. The SD pipeline defaults to 7.5 and the LDM script to 1.5 pipeline_stable_diffusion.py sample_ddp_deepcache.py.
• Eta. The DDPM script defaults to eta 0.0 ddim.py. The paper does not state eta.

Pitfalls

• The headline numbers come from different configurations. The abstract's 2.3x speedup with a 0.05 CLIP drop on Stable Diffusion matches the non-uniform N=6 row on PartiPrompts (speed 2.30, CLIP 29.46) app. F, tab. 13. The main table's DeepCache row is N=5, with speed 2.15x and the same CLIP 29.46 tab. 3. The N=5 rows in app. F show CLIP 29.41 (uniform) and 29.50 (non-uniform), so the 29.46 in the main table matches neither N=5 entry.
• The LDM FID drop in the abstract uses a different baseline than the text. The abstract's 0.22 FID drop is 3.59 minus 3.37, the regenerated LDM-4* baseline tab. 1. The text says the FID changes "from 3.39 to 3.59". The 3.39 is the N=2 uniform result, not the baseline.
• The non-uniform N=20 LDM result has three values. Main tab. 1 gives FID 7.11, which matches the centre-120, power-1.2 row of the sweep app. B, tab. 7. The appendix comparison gives 7.36 app. C, tab. 11, which matches the centre-80 row. The selected-hyperparameter table gives centre 120 and power 1.4 app. B, and power 1.4 at centre 120 gives FID 7.13 in the sweep app. B, tab. 8. No single configuration reproduces the main value. Trust the main-table value only with a check against the code.
• Other small mismatches. PLMS-50 on COCO has CLIP 30.30 in tab. 3 but 30.24 in app. F. PLMS-25 on PartiPrompts has 29.33 against 29.32.
• Table 4 sFID looks wrong. DDIM-91 shows sFID 50.6, far from the others at about 5. The cell is probably a typo. This was not verified against any other source.
• Ablation tables are under-specified. tab. 5 does not state N, and tab. 6 states neither N nor the dataset. The numbers match other tables (9.74 is the CIFAR-10 N=10 row, 7.36 is the LDM non-uniform N=20 row), so the settings look inferred rather than stated.
• Non-uniform hyperparameters are tuned on the reported evaluation. The paper reports that the best values differ a little by dataset and cluster near step 15 with power near 1.4 app. B. It does not describe a held-out split for choosing them.
• Throughput definition is thin. The paper says throughput is samples per second on one RTX 2080 §4.2. Batch size, and whether the text encoder and VAE are included, are not stated. Speed ratios are the throughput ratios as reported.
• Non-uniform centre with a negative base (inferred, not run). The schedule takes (-c) raised to 1/p, with c > 0. In Python this gives a complex number, not an error, and the code then depends on integer casting to discard the imaginary part. Check the output for a small case before relying on it.
• The SD non-uniform path needs explicit arguments (inferred, not run). The script defaults centre and power to None and only sets the uniform flag on the command line generate.py. Running non-uniform without --center and --pow will probably fail with an arithmetic error on None.
• Quality falls with N. The authors note degradation at N=20 and say the speedup is limited when the shallowest skip branch holds about half of the model §5. On LSUN-Bedroom, N=5 raises FID from 6.62 to 9.49 tab. 2, so the cost is not small for every dataset.
• Mapping branch indices across models is not stated. The DDPM index is clear (code index = paper branch minus 1). The LDM and SD indices are not, because the paper gives only a number and the SD code uses a block and a layer.

Lineage

• Builds on: DDIM (arXiv:2010.02502), which it uses as a sampler §4.1; PLMS (arXiv:2202.09778), used for Stable Diffusion §4.1; and the U-Net skip-connection structure of DDPM, LDM and Stable Diffusion §3.1.
• Compared against: Diff-Pruning (Fang et al., "Structural pruning for diffusion models", NeurIPS 2023) §4.3; Spectral Diffusion (Yang et al., "Diffusion probabilistic model made slim", CVPR 2023) §4.3; BK-SDM (arXiv:2305.15798) §4.3; DDIM and PLMS at matched throughput §4.4.
• Superseded by: not known from content already in this wiki.