---
kind: concept
slug: latent-diffusion
title: Latent diffusion
summary: >-
  Diffusion on a learned, smaller code: an autoencoder compresses samples, a denoiser works on codes, and a decoder maps them back. 30 cards cover motion, 3D, editing, video, animation, driving, restoration, depth, safety, watermarks and robot control. Latent size, condition entry and guidance matter most. Results are author-reported, not comparable.
tags: [diffusion, latent-space, motion-generation, 3d-generation, image-generation, image-editing, inpainting, video-generation, talking-face, text-to-image, super-resolution, transformer, controllable-generation, safety, monocular-depth, correspondence, watermarking, robot-manipulation, grasp-synthesis, subgoal-generation, human-animation, autonomous-driving]
papers: ["2212.04048", "2212.04493", "2212.04489", "2302.08113", "2304.06025", "2305.07015", "2305.14720", "2310.07204", "2312.03594", "2312.06662", "2402.17485", "2404.10667", "2409.10141", "2412.07772", "2502.12048", "2503.23377", "2212.09748", "2302.08453", "2307.01952", "2211.05105", "2303.15435", "2306.03881", "2312.02145", "2512.18007", "2312.11243", "2410.11013", "2411.14786", "2311.16498", "2311.16518", "2311.17918"]
related: [diffusion-models, classifier-free-guidance, variational-autoencoder, attention-mechanism, knowledge-distillation, kv-cache-management, layout-to-image-generation, text-to-image-personalization, diffusion-inpainting, text-guided-image-editing, image-to-3d-generation, image-super-resolution, novel-view-synthesis, vision-transformer, text-to-image-generation, parameter-efficient-fine-tuning, safety-filter, image-watermarking, dense-correspondence, monocular-depth-estimation, ensembles, noise-schedule, self-supervised-learning, vision-language-action-model, flow-matching, optical-flow, robotic-grasp-generation, imitation-learning, video-generation, world-model]
status: draft
updated_at: 2026-10-31
---

## Overview

Latent diffusion trains a diffusion model on a compressed representation of
the data. An autoencoder maps each sample to a smaller latent and back. A
denoiser learns to remove noise from latents, and the decoder runs once on the
final latent [arxiv:2212.04048 §3]. The iterative part therefore works on a
small tensor, not on the full sample. That is the speed argument the generative
cards rely on [arxiv:2212.04048 §1] [arxiv:2212.04048 §3.2] [arxiv:2302.08113 §5].

The invariant shared by the generative cards: the denoiser works on a learned
code, not on raw samples, and a decoder returns the final code to data space
[arxiv:2212.04048 §3.2] [arxiv:2212.04493 §3.2] [arxiv:2312.06662 §4.1]
[arxiv:2412.07772 §4.1] [arxiv:2212.04489 §3.1] [arxiv:2503.23377 app. C.1].
What differs is the latent (a 3D VQ code, a continuous video latent, a Stable
Diffusion latent, a motion code), the parameterisation of the target, and how
conditions enter. VASA-1 is the furthest from the pattern: its diffusion
transformer works on head-pose and facial-dynamics codes from a separate
encoder, and its frame renderer is not a diffusion model [arxiv:2404.10667 §3].
The robot-policy card also uses a frozen VAE latent, but as a second output of
a control model, not as the generated sample [arxiv:2512.18007 §3.2] [app. A.2].

Three robot cards keep the same split between an autoencoder and a denoiser,
but change what the code stands for. In GraspLDM the code is a 6-DoF grasp,
produced by a conditional VAE [arxiv:2312.11243 §3.1]. In FastGrasp it is a
human hand pose, and the denoiser is conditioned on object points
[arxiv:2411.14786 §3.2]. In TaKSIE the diffusion model draws the next subgoal
image [arxiv:2410.11013 §3.4.1].

Most image cards use a Stable Diffusion checkpoint as a fixed or fine-tuned
base [arxiv:2302.08113 §5] [arxiv:2305.14720 §4.1] [arxiv:2212.04489 §4]
[arxiv:2312.03594 §3.1] [arxiv:2304.06025 §4.3] [arxiv:2302.08453 §3.1]
[arxiv:2305.07015 §3.1] [arxiv:2409.10141 §4] [arxiv:2211.05105 §3] [arxiv:2311.16518 §4.1], so image
claims here are about those systems, not about latent diffusion in general.
Two cards depart from that base. DiT reuses only a Stable Diffusion
autoencoder and trains its own denoiser [arxiv:2212.09748 §4], and SDXL
trains a new, larger UNet [arxiv:2307.01952 §2.1]. For the general
noise-prediction setup, see [Diffusion models](/concepts/diffusion-models).

Two of the newer cards extend an image model in time. MagicAnimate and Drive-WM
insert temporal layers into a UNet, and Drive-WM also adds layers across camera
views [arxiv:2311.16498 §3.1] [arxiv:2311.17918 §3.1]. The image models they
start from are a Stable Diffusion 1.5 UNet for MagicAnimate
[code: configs/prompts/animation.yaml#L1-L2] and Stable Diffusion checkpoints
of an unstated version for Drive-WM [arxiv:2311.17918 §5.1].

Three of the newer cards put the latent space to other uses. The Stable
Signature changes only the decoder and leaves the denoiser alone [arxiv:2303.15435
§4.2]. DIFT runs one denoiser pass and generates nothing [arxiv:2306.03881 §4.2].
Marigold trains the denoiser to output depth latents, not images [arxiv:2312.02145
§3.1].

The two surveys in the list ([arxiv:2310.07204] and [arxiv:2502.12048]) run no
experiments. They are used for orientation only, and their numbers are not
reported here.

## Variants

### Motion latent diffusion with a two-stage training

Stage one trains a transformer autoencoder with a reconstruction term and a KL
term, so that a whole motion clip becomes a latent of size n × d
[arxiv:2212.04048 §3.1] [app. H.1]. Stage two freezes the encoder and trains a
transformer denoiser on the latents, with the condition as an extra input
[arxiv:2212.04048 §3.2] [arxiv:2212.04048 §3.3]. The paper's best setting uses
one token of width 256 [arxiv:2212.04048 §5] [tab. 5]. Card:
[Executing your Commands via Motion Diffusion in Latent Space](/papers/2212.04048).

### Latent size in the motion model

The paper compares one to ten tokens of width 256 [arxiv:2212.04048 tab. 5].
The single-token latent gives the best FID of those tested (0.473). Two tokens
give 0.585, five give 1.554, seven give 1.011 and ten give 1.716
[arxiv:2212.04048 tab. 5]. Reconstruction improves with more tokens, though:
the seven-token model with skip connections has the lowest reconstruction
error in the table [arxiv:2212.04048 tab. 4]. So a better autoencoder did not
produce a better generative model here.

### Condition entry in the motion model

The paper concatenates the condition embedding with the latent. It compares
this with cross-attention and reports concatenation as the better choice
[arxiv:2212.04048 §3.3] [tab. 5]. Cross-attention reaches R-Precision Top 3 of
0.592 and FID of 1.922, against 0.772 and 0.473 for concatenation
[arxiv:2212.04048 tab. 5]. Cross-attention has a higher multimodality score
(3.768 against 2.413), so it produces more varied outputs but matches the text
less well [arxiv:2212.04048 tab. 5]. The attention block itself is described on
[Attention mechanism](/concepts/attention-mechanism).

### Condition encoders in the motion model

Text conditions use a frozen CLIP text encoder [arxiv:2212.04048 §4.2]. Action
labels use a learnable embedding per class [arxiv:2212.04048 §3.3]. Both are
trained with the same conditional objective [arxiv:2212.04048 eq. 3].

### Guidance scale and dropout in the motion model

The condition is dropped for 10% of training samples, and at sampling time the
conditional and unconditional predictions are combined with scale s
[arxiv:2212.04048 §3.3] [eq. 4]. The paper's default is s = 7.5 with dropout
0.1 [arxiv:2212.04048 app. E] [tab. 11]. Raising s increases text match: R-
Precision Top 3 rises from 0.648 at s = 1.5 to 0.772 at s = 7.5. FID is lower
at intermediate values, with 0.350 at s = 2.5 against 0.473 at s = 7.5
[arxiv:2212.04048 tab. 11]. Dropout 0.1 is not the FID-best setting in the
sweep: dropout 0.2 gives 0.279 at s = 7.5 [arxiv:2212.04048 tab. 11].

### Sampler and step count in the motion model

The paper samples with DDIM and no added noise (eta = 0), using 50 steps
after training with 1000 [arxiv:2212.04048 §4.2] [app. C]. Running the same
model with 1000 DDPM steps gives FID 0.568 against 0.473 with 50 DDIM steps,
and takes 148.97 s against 10.24 s for 2048 clips on one V100
[arxiv:2212.04048 tab. 10]. More steps did not help FID in this evaluation.

### Per-input weights for 3D shapes

SDFusion compresses truncated signed-distance grids with a 3D VQ-VAE, then
trains a diffusion model on the codes [arxiv:2212.04493 §3.1] [§3.2]. Partial
shapes, images and text each have an encoder. Their features are concatenated
and fed to the denoiser through cross-attention [arxiv:2212.04493 §3.3]
[eq. 3]. At sampling time each input gets its own weight, so the output follows
the heavier inputs more closely [arxiv:2212.04493 §3.3] [eq. 4]. The authors
report lower UHD and higher TMD than MPC and AutoSDF on shape completion
[arxiv:2212.04493 tab. 1], and the best Pix3D F-score of 0.432 against 0.415 for
AutoSDF [arxiv:2212.04493 tab. 2]. Card:
[SDFusion](/papers/2212.04493).

### Multi-view body diffusion, then remeshing

PSHuman fine-tunes Stable Diffusion 2.1-unclip to draw six body views and one
face view, as color and normal maps, from one full-body photo [arxiv:2409.10141
§3.1] [§4]. Renders of a body model (SMPL-X) fitted to the photo are encoded
by a pretrained VAE and concatenated with the noise latent as extra input
[arxiv:2409.10141 §3.1.2]. A face branch joins the body views through shared
self-attention [arxiv:2409.10141 §3.1.1]. A second stage starts from an SMPL-X
mesh and deforms and remeshes it until it fits the generated normal and
silhouette maps [arxiv:2409.10141 §3.2] [eq. 7] [eq. 8]. The authors report a
Chamfer distance of 0.9688 against 1.5511 for ICON on CAPE-NFP, when the
ground-truth body model is given [arxiv:2409.10141 tab. 1]. The released code
does not texture the mesh the way the paper describes, so take the texture step
from the card's Paper vs code notes, not from the paper alone
[arxiv:2409.10141 §3.2]. Card:
[PSHuman](/papers/2409.10141).

### Training-free composition for panoramas and regions

MultiDiffusion trains nothing. A pretrained Stable Diffusion v2 model is run on
overlapping crops of a larger canvas, or on masked regions, at every denoising
step. The outputs are averaged per pixel into one next latent
[arxiv:2302.08113 §3] [alg. 1] [eq. 5]. For a panorama this gives a plain
average over the crops, and for layouts the masks act as weights [arxiv:2302.08113
§4.1] [§4.2]. The authors report a panorama FID of 10.3, against 18.4 for
Blended Latent Diffusion and 45.5 for Stable Inpainting [arxiv:2302.08113
tab. 1], and a region IoU of 0.26, against 0.17 and 0.16 [arxiv:2302.08113
tab. 2]. The cost is one reference call per view and per step
[arxiv:2302.08113 app. B.1]. Card:
[MultiDiffusion](/papers/2302.08113).

### Subject tokens from a vision-language encoder

BLIP-Diffusion puts a frozen image encoder and a trainable multimodal encoder
in front of Stable Diffusion v1-5 [arxiv:2305.14720 §4.1]. Sixteen query
outputs are projected into the text-embedding space and inserted into the
prompt tokens, so the U-Net sees them as part of the text condition
[arxiv:2305.14720 §3.2]. The authors report DINO 0.594 zero-shot and 0.670
after fine-tuning, against 0.668 for DreamBooth [arxiv:2305.14720 tab. 1]. A
fine-tune takes 40 to 120 steps per subject [arxiv:2305.14720 §3.3] [tab. 3].
The second pre-training stage is not in the released code, so the base model
cannot be reproduced from that repository [arxiv:2305.14720 §3.2]. Card:
[BLIP-Diffusion](/papers/2305.14720).

### Single-image editing with a fine-tuned copy and a guidance switch

SINE fine-tunes a Stable Diffusion model on one source image, then edits that
image with a text prompt [arxiv:2212.04489 §3.1] [§4]. The fine-tuned model is
mixed into classifier-free guidance only while the sampler is in its early
steps (t > K). Late steps use the pretrained model alone, so the fine-tuned
model fixes only the coarse structure [arxiv:2212.04489 §3.2] [eq. 3]. A patch
fine-tune, with a positional code for each crop, lets the edit run at
resolutions other than the training size [arxiv:2212.04489 §3.3]. The paper
sets K = 400 and v = 0.7 and does not state w [arxiv:2212.04489 §4] [§4.3].
Its evidence is qualitative. The text gives no headline number, and the sweeps
for K and v appear only as plots [arxiv:2212.04489 fig. 8]. Card:
[SINE](/papers/2212.04489).

### Task prompts for one inpainting model

PowerPaint fine-tunes a Stable Diffusion inpainting model with three learned
task prompts: one for context filling, one for text-guided object insertion,
and one for shape-guided insertion [arxiv:2312.03594 §3.2] [fig. 2]. Object
removal reuses the context prompt as the positive prompt and the object prompt
as the negative prompt in classifier-free guidance [arxiv:2312.03594 §3.2]
[eq. 5]. Shape control blends the two text embeddings with a fitting degree
alpha [arxiv:2312.03594 §3.2] [eq. 7]. The authors report the lowest FID on
OpenImages and MSCOCO object insertion, 4.38 and 5.12 against 4.76 and 5.65 for
SD-Inpainting [arxiv:2312.03594 tab. 1], and the lowest FID on Places2
context filling, 2.59 against 3.03 [arxiv:2312.03594 tab. 3]. The training
mix samples 80% inpainting and 20% text-to-image [arxiv:2312.03594 §3.3]. Card:
[A Task is Worth One Word](/papers/2312.03594).

### Frame-by-frame video from one photo and a pose sequence

DreamPose conditions a Stable Diffusion UNet on one photo through a CLIP and
VAE adapter, and on five consecutive DensePose frames through extra input
channels [arxiv:2304.06025 §4.2]. Each output frame is sampled on its own, so
temporal smoothness comes from the conditions, not from a video model
[arxiv:2304.06025 §4.1]. Sampling uses separate guidance weights for the image
and the pose [arxiv:2304.06025 §4.4] [eq. 4]. Per-subject tuning is the costly
step: the authors report about 10 minutes for the UNet and 20 minutes for the
VAE decoder, and 18 s per rendered frame [arxiv:2304.06025 §6]. The UBC Fashion
table favours DreamPose on every metric, with FID 13.04 against 22.87 for TPSMM
[arxiv:2304.06025 tab. 1]. The paper's training frame size differs from its
code, so check the card's Paper vs code section before building. Card:
[DreamPose](/papers/2304.06025).

### Temporal layers and reference attention for a whole clip

MagicAnimate extends a Stable Diffusion 1.5 UNet along time. Temporal attention
layers let the frames of one clip agree with each other [arxiv:2311.16498 §3.1].
A trainable copy of the UNet reads the reference photo at each denoising step,
and its normalised attention states join the keys and values of the generator's
self-attention [arxiv:2311.16498 §3.2] [eq. 2]. Poses enter as DensePose maps
through a ControlNet [arxiv:2311.16498 §3.3]. Long videos are cut into
overlapping windows, and their noise predictions are averaged at each step
[arxiv:2311.16498 §3.3]. The released config uses 16-frame windows, 4 frames of
overlap, 25 DDIM steps and a guidance scale of 7.5
[code: configs/prompts/animation.yaml#L14-L15]
[code: magicanimate/pipelines/pipeline_animation.py#L546]. Author-reported on TikTok,
FVD is 179.07 against 292.80 for DisCo, and FID-VID is 21.75 against 59.90
[arxiv:2311.16498 tab. 1a]. Removing the temporal layers raises FVD to 247.30
[arxiv:2311.16498 tab. 2a]. Training settings are not stated, and the
supplement in the card's source was empty [arxiv:2311.16498 §3.4] [§4]. The
paper's reference-branch equation takes the noisy latent, but the released
pipeline feeds the clean reference latent [arxiv:2311.16498 eq. 1]. Card:
[MagicAnimate](/papers/2311.16498).

### Audio-driven portrait video with a reference network

EMO starts from the Stable Diffusion 1.5 UNet with its text cross-attention
replaced by reference attention to one portrait [arxiv:2402.17485 §3.1]
[§3.2.1]. Audio features enter through audio layers, and a face-region mask and
a head-speed value act as weak conditions [arxiv:2402.17485 §3.2]. Clips are
12 frames, and each clip reuses 4 motion frames from the previous one
[arxiv:2402.17485 §4.1] [§3.2.4]. The sampler is DDIM with 40 steps
[arxiv:2402.17485 §4.1]. On HDTF, the authors report FID 8.76, the best in
their table, and FVD 67.66 [arxiv:2402.17485 tab. 1]. The paper gives no
guidance scale, and no code is released [arxiv:2402.17485 §4.1]. Card:
[EMO](/papers/2402.17485).

### Diffusion over motion codes, not image latents

VASA-1 learns four separate codes for each face: a canonical 3D appearance
volume, identity, head pose and facial dynamics [arxiv:2404.10667 §3.1]. A
diffusion transformer generates the pose and dynamics codes from audio, and a
decoder turns them into frames [arxiv:2404.10667 §3] [fig. 2]. The diffusion
therefore runs in a compact motion space, and the frame renderer is not a
latent-image diffusion model. The generator works in overlapping chunks, so
output can be streamed [arxiv:2404.10667 §3.2]. The default is 50 sampling
steps, and 10 steps give about a fivefold speed-up in latent generation
[arxiv:2404.10667 §4.3] [tab. 4]. On a VoxCeleb2 subset the authors report
SyncNet confidence 8.84 against 5.84 for SadTalker [arxiv:2404.10667 tab. 1].
Card: [VASA-1](/papers/2404.10667).

### Joint audio and video latents

JavisDiT is a diffusion transformer with one branch for video and one for
audio, linked by bidirectional cross-attention [arxiv:2503.23377 §3.1]. The
video latent comes from a frozen OpenSora VAE, and the audio is a mel
spectrogram encoded by a frozen AudioLDM2 VAE [arxiv:2503.23377 app. C.1]
[app. C.2.2]. A learned prior estimator maps the prompt to spatial and
temporal tokens that enter both branches [arxiv:2503.23377 §3.2]. Sampling uses
rectified flow [arxiv:2503.23377 app. C.1]. The authors report JavisScore 0.154
against 0.151 for FoleyCrafter, but UniVerse-1 is better on FVD (194.2 against
204.1), KVD and TV-IB [arxiv:2503.23377 tab. 1]. Card:
[JavisDiT](/papers/2503.23377).

### One latent for images and video

W.A.L.T uses a causal autoencoder without a codebook. It maps images and video
into one continuous latent space, so a single transformer trains on both
[arxiv:2312.06662 §3] [§4.1]. The denoiser alternates two window types: spatial
windows within one latent frame, and spatiotemporal windows across frames
[arxiv:2312.06662 §4.2]. The default latent has 8 channels [arxiv:2312.06662
§5.2] [tab. 3]. Reported results include FID 2.56 on ImageNet 256 without
classifier-free guidance [arxiv:2312.06662 tab. 2], and K600 frame-prediction
FVD of 3.3 against 4.3 for MAGVIT-v2 [arxiv:2312.06662 tab. 1]. The denoiser is
trained with v-prediction [arxiv:2312.06662 §3]. Card:
[Photorealistic Video Generation with Diffusion Models](/papers/2312.06662).

### Multiview driving video with layout and action tokens

Drive-WM first trains a single-view, conditional image latent diffusion model.
It then adds temporal layers after each spatial block and self-attention across
the six camera views. The spatial weights stay frozen, and only the temporal and
multiview weights are trained [arxiv:2311.17918 §3.1] [app. B.1]. Conditions are
mapped to tokens and concatenated: image tokens from context frames and
reference views, layout tokens for 3D boxes, the HD map and BEV segmentation,
text tokens, and ego-action tokens [arxiv:2311.17918 §3.3] [eq. 5]. To keep
overlapping cameras consistent, three reference views are generated first, and
the other three are generated from their neighbours and their own previous frame
[arxiv:2311.17918 §3.2] [app. B.2]. Author-reported on nuScenes, FID is 15.8 and
FVD is 122.7, against 52.6 and 452.0 for DriveDreamer [arxiv:2311.17918 tab. 1].
Factorized generation raises the keypoint-matching score from 45.8% to 94.4%,
and costs FID (16.4 against 15.8) [arxiv:2311.17918 tab. 2]. For planning, the
model renders the future of each candidate trajectory from VAD, and perception
models score the result [§4.1] [§4.2]. Open-loop average L2 is 0.80 m and
collision is 0.26%, against 1.02 m and 0.93% for VAD with a random command
[arxiv:2311.17918 tab. 3]. No model weights or method code are released, so the
numbers are author statements only [code: README.md#L11-L15]. Card:
[Driving into the Future](/papers/2311.17918).

### Distilled, streaming video latents

CausVid turns a bidirectional latent video transformer into a causal one that
generates chunk by chunk with a key-value cache [arxiv:2412.07772 §4.1]
[§4.4]. A few-step student is distilled from a slower bidirectional teacher
with distribution matching [arxiv:2412.07772 §4.2]. The authors report 1.3 s
latency and 9.4 FPS on one H100, against 219.2 s for the bidirectional teacher
on a 120-frame clip [arxiv:2412.07772 tab. 3]. The card records the released
code as non-commercial, so the code cannot be reused commercially, although
the paper itself is CC BY 4.0. Card:
[From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](/papers/2412.07772).

### Transformer denoiser on latent patches

DiT replaces the U-Net denoiser with a transformer. A frozen autoencoder
encodes each image, the latent is cut into patches, and each patch becomes a
token [arxiv:2212.09748 §3.1] [§3.2]. Timestep and class condition enter
through adaLN-Zero: they regress a scale, shift and gate for each block, and
the gate starts at zero, so every block begins as the identity [arxiv:2212.09748
§3.2]. The paper compared this with in-context, cross-attention and plain
adaLN conditioning, and adaLN-Zero beat the first two at every training stage
in its block study [arxiv:2212.09748 §3.2] [fig. 5]. Halving the patch size
quadruples the tokens and at least quadruples the transformer cost. Across its
twelve configurations, FID falls as forward-pass Gflops rise, though the paper
gives no fitted coefficient [arxiv:2212.09748 §3.2] [§5] [tab. 4]. The
author-reported class-conditional ImageNet FID with guidance is 2.27 at 256 by
256 and 3.04 at 512 by 512, against 3.60 for LDM-4 with guidance at 256
[arxiv:2212.09748 tab. 2] [tab. 3]. The released code is licensed for
non-commercial use, so reimplement from the paper rather than reuse that code.
Card: [Scalable Diffusion Models with Transformers](/papers/2212.09748).

### Frozen base with a control adapter

T2I-Adapter does not change the latent diffusion model. It trains small
adapter networks that read a control map, such as a sketch, a segmentation,
a pose or a depth map, and add multi-scale features to the encoder of a frozen
Stable Diffusion UNet [arxiv:2302.08453 §3.3] [eq. 4]. Several adapters can be
combined as a weighted sum without retraining, but the weights are set by hand
[arxiv:2302.08453 §3.3] [eq. 5] [§5]. Adapters are trained with the base model
frozen, using a cubic timestep rule that favours high-noise steps
[arxiv:2302.08453 §3.4]. The author-reported COCO validation FID is 16.78 with
text and segmentation, and 17.36 with text and sketch, against 24.68 for
Stable Diffusion with text only [arxiv:2302.08453 tab. 1]. The rows differ in
their input, so this does not isolate the adapter from the control map alone.
The paper's model is SD 1.4, but the released main branch targets SDXL, and
its injection points and training data differ from the paper's. Card:
[T2I-Adapter](/papers/2302.08453).

### Restoration with a frozen generator and a trained encoder

StableSR keeps Stable Diffusion 2.1 frozen and trains only a time-aware encoder
and scale-and-shift layers that modulate the U-Net features [arxiv:2305.07015
§3.1] [eq. 1]. The encoder reads the low-resolution latent and the diffusion
step, so its guidance changes along the sampling trajectory. The authors place
the strongest guidance near a signal-to-noise ratio of 5e-2 [arxiv:2305.07015
§3.1] [fig. 3]. An optional fidelity module adds encoder features into the VAE
decoder, scaled by a user weight w; w = 0.5 is the default
[arxiv:2305.07015 §3.2] [tab. 4]. Inputs of any size are handled by denoising
overlapping latent patches and blending their noise predictions with Gaussian
weights [arxiv:2305.07015 §3.3] [alg. 1]. The authors report DIV2K-Valid FID of
24.44 against 26.47 for LDM, which they fine-tuned with the same settings
[arxiv:2305.07015 tab. 1]. PSNR, SSIM and LPIPS are lower than several
baselines [arxiv:2305.07015 §4.3]. Sampling is the cost: 200 steps take 15.16 s
for a 512 px output on one V100, against 0.83 s for the 4-step SD-Turbo variant
[arxiv:2305.07015 tab. 5]. The released code is licensed for non-commercial
use, so reimplement from the paper rather than reuse that code. Card:
[StableSR](/papers/2305.07015).

### Semantic prompts and a noised low-resolution start

SeeSR keeps the Stable Diffusion 2-base UNet, its text encoder and its VAE
frozen [arxiv:2311.16518 §3.3] [§4.1]. The trained parts are a ControlNet cloned
from the UNet encoder, an image encoder that feeds it, and representation
cross-attention modules placed after the text cross-attention [§3.3]. The
semantic input comes from DAPE, a RAM tagger fine-tuned with LoRA on degraded
images. DAPE gives predicted tags, which enter as text, and representation
embeddings, which enter as soft prompts [arxiv:2311.16518 §3.2]. At inference
the low-resolution latent is noised to training step 999 and used as the start
of sampling, instead of Gaussian noise [§3.4]. The authors say this removes
false sky texture [fig. 3]. Author-reported on DIV2K-Val, FID is 31.93 against
36.57 for StableSR, and SeeSR is best on LPIPS, DISTS, FID, MUSIQ and CLIPIQA
[arxiv:2311.16518 tab. 2]. On the three paired sets it is below the best
baseline on PSNR and SSIM [tab. 2]. A supplementary ablation shows the start
helps PSNR and LPIPS on DIV2K-Val but lowers CLIPIQA, from 0.7314 without it to
0.6834 with it [arxiv:2311.16518 suppl. LRE table]. The model has 2283.7M
parameters and takes about 7.24 s per 512 px output with 50 steps on one V100
[arxiv:2311.16518 suppl. complexity table]. Card: [SeeSR](/papers/2311.16518).

### Larger two-encoder base with size conditioning

SDXL uses a UNet about three times the size of SD 1.x and 2.x, with 2.6B
parameters, and reads the prompt with two text encoders whose outputs are
concatenated [arxiv:2307.01952 §2.1]. Training uses the original image size and
the crop offset as extra conditions, which keeps small or oddly shaped images
in the data [arxiv:2307.01952 §2.2] [alg. 1]. At inference the authors set the
crop offset to zero [arxiv:2307.01952 §2.2]. Multi-aspect buckets, with the
target size as a further condition, cover landscape and portrait shapes
[arxiv:2307.01952 §2.3]. A separate refiner, run on the first 200 of 1000 noise
levels, is optional [arxiv:2307.01952 §2.5]. The authors report 48.44% of
human votes for refined SDXL against 7.91% for SD 1.5 and 6.71% for SD 2.1, in
a four-way study [arxiv:2307.01952 §2.5]. Size conditioning also improves
class-conditional ImageNet FID-5k at 512 by 512 from 39.76 with no
conditioning to 36.53 [arxiv:2307.01952 §2.2]. The two text encoders total 817M
parameters, and the refiner is a second full model, so memory is a constraint
[arxiv:2307.01952 §2.1] [§3]. The weights carry a licence separate from the
MIT code, and that licence was not checked. Card: [SDXL](/papers/2307.01952).

### Safety term added to guidance during sampling

SLD changes only the sampler of a Stable Diffusion model. It adds a third noise
estimate, conditioned on a text description of unwanted content, and pushes the
prompt estimate away from it [arxiv:2211.05105 §3] [eq. 3] [eq. 4]. The push is
element-wise. It is zero where the prompt estimate already exceeds the concept
estimate by a threshold, and otherwise scales with a concept factor
[arxiv:2211.05105 eq. 5] [eq. 6]. A warm-up delays the term, and a momentum term
carries it across steps [arxiv:2211.05105 §3] [alg. 1]. The default concept is
a list of about twenty categories, such as violence, nudity and weapons
[arxiv:2211.05105 §4]. No training or external classifier is used, and the
method applies to any latent diffusion model with classifier-free guidance
[arxiv:2211.05105 app. D]. The authors report that the probability of an
inappropriate image on the I2P benchmark falls from 0.39 for the baseline to
0.09 at the strongest preset, and from 0.35 to 0.06 on the sexual subset
[arxiv:2211.05105 tab. 2]. Negative prompting reaches 0.18 in the same table,
so it is close to the middle presets and behind the strongest one. The cost is
measured on image quality: COCO FID-30k rises from 14.43 to 18.28 at the
strongest preset and 18.76 at the maximum [arxiv:2211.05105 tab. 3]. The
classifier that labels the outputs is described as conservative, so part of
the measured rate reflects false positives [arxiv:2211.05105 tab. 2]. The
paper's maximum preset uses a threshold outside its recommended range
[arxiv:2211.05105 tab. 1] [§4]. The authors ask that any deployment state which
content is suppressed, because the same mechanism can serve censorship
[arxiv:2211.05105 §7]. Card:
[Safe Latent Diffusion](/papers/2211.05105).

### Watermark in the decoder, read by a separate extractor

The Stable Signature fine-tunes only a copy of the autoencoder's decoder, so
that every output carries one fixed 48-bit key. The diffusion process is not
touched [arxiv:2303.15435 §4.2] [fig. 1]. The key is read by an extractor that
was pre-trained with simulated edits and then whitened, so its bits on ordinary
images are close to fair coin flips [arxiv:2303.15435 §4.1] [app. A.1]. The
decoder fine-tune runs about 100 steps on fewer than 500 images
[arxiv:2303.15435 §4.2]. Detection flags an image when enough bits match the
key, and identification picks the best of N user keys [arxiv:2303.15435 §3].
The authors report 0.99 bit accuracy on unmodified output and 0.92 after a
combined crop, brightness and JPEG edit, with FID moving by at most 0.6
[arxiv:2303.15435 tab. 1]. At N = 10^5 users, identification is 98% accurate on
unmodified images and 40% after the combined edit [arxiv:2303.15435 §5.2]. The
headline text-to-image numbers come from an internal model, not a public one
[arxiv:2303.15435 §5]. Card: [The Stable Signature](/papers/2303.15435).

### Denoiser features for matching points, with no training

DIFT takes a real image, noises it to a chosen level t, runs the denoiser once,
and keeps the activations of one upsampling block, averaged over several noise
draws [arxiv:2306.03881 §4.2] [app. C]. Points are matched by nearest neighbour
under cosine distance [arxiv:2306.03881 §3] [eq. 1]. No parameter is trained
and no correspondence label is used [arxiv:2306.03881 §1]. The time step and
block index are grid-searched per task. The two backbones are Stable Diffusion
2-1 and an ADM model [arxiv:2306.03881 §5.1] [app. C]. On SPair-71k the authors
report 52.9 PCK per image for the Stable Diffusion features, against 33.9 for
DINO and 38.4 for OpenCLIP [arxiv:2306.03881 tab. 1]. The supervised SCorrSAN
reaches 55.3 on the same table. The paper describes noising the image, but the
card's Paper vs code notes record that the released Stable Diffusion code noises
the VAE latent instead [arxiv:2306.03881 §4.2]. Card:
[Emergent Correspondence from Image Diffusion](/papers/2306.03881).

### Depth as the denoised target, with an ensemble

Marigold fine-tunes only the denoising U-Net of Stable Diffusion v2 to output
depth, with text conditioning disabled [arxiv:2312.02145 §3.2] [§4.1]. The
frozen VAE encodes both the image and the depth map, and their latents are
concatenated at the U-Net input [arxiv:2312.02145 §3.2]. Training uses only
synthetic depth from Hypersim and Virtual KITTI, normalised per image to a fixed
percentile range, so the target is affine-invariant [arxiv:2312.02145 §3.3]
[eq. 3]. At inference, ten runs start from different noise. Each run is aligned
by a scale and a shift, and the pixel-wise median is taken [arxiv:2312.02145
§3.4] [eq. 4]. The authors report AbsRel 5.5 on NYUv2 for the ten-run ensemble,
against 6.0 for a single run [arxiv:2312.02145 tab. 1]. Going from one run to
ten cuts NYUv2 AbsRel by about 8% [arxiv:2312.02145 §4.3] [fig. 6]. On DIODE the
method is weaker than DPT, with AbsRel 30.8 against 18.2 [arxiv:2312.02145
tab. 1]. The output is not in metres, so scale must come from elsewhere
[arxiv:2312.02145 §1]. Card: [Repurposing Diffusion-Based Image Generators for
Monocular Depth Estimation](/papers/2312.02145).

### Motion-image latents as an auxiliary target for a robot policy

A pi-series vision-language-action model gets a second head: a diffusion
transformer that denoises latents of a predicted motion image, trained jointly
with the action head [arxiv:2512.18007 §3.2]. The target is an optical-flow
picture between the frame at time t and a frame k steps later, encoded by a
frozen SDXL VAE to a 4 × 28 × 28 latent [arxiv:2512.18007 §3.3] [app. A.1]
[app. A.2]. The head has six blocks, each with self-attention over the latent
tokens and cross-attention to the policy's prefix [app. A.2]. Training has two
stages. Warm-up trains only the motion head, on flow derived from DROID, and the
joint stage then unfreezes everything except the VAE [arxiv:2512.18007 §3.3].
At inference the motion head is not run, so the action path is unchanged
[arxiv:2512.18007 §1] [§4.3]. The VAE latent is the same kind of object as in
the image cards, but the denoised quantity is a motion code for a policy, not
a picture for a user. Card:
[Robotic VLA Benefits from Joint Learning with Motion Image Diffusion](/papers/2512.18007).

### Diffusion over grasp codes from a conditional VAE

GraspLDM trains a conditional VAE on 6-DoF grasps first. A point-cloud encoder
and a grasp encoder and decoder give each grasp a latent, and the VAE is then
frozen [arxiv:2312.11243 §3.1] [§3.3]. A 1000-step DDPM is trained on those
latents, with the shape latent as a condition, so the denoiser works on a short
code and not on the grasp pose itself [arxiv:2312.11243 §3.2] [§3.3]. The
authors motivate this by the gap between the VAE prior and its posterior, and
they report that decoding random prior samples alone gives poor grasps
[arxiv:2312.11243 §1] [§3]. Sampling 100 grasps takes 0.75 s with 100 DDIM
steps and 7.39 s with 1000 DDPM steps, on one RTX 3080 Ti. The median success
in simulation is 0.756 and 0.792, against 0.660 for the VAE alone
[arxiv:2312.11243 tab. 1]. A new task needs only a task-conditioned denoiser
trained after the VAE [arxiv:2312.11243 §4.2]. Card:
[GraspLDM](/papers/2312.11243).

### Hand-pose latent with an object-conditioned denoiser and an adaptation MLP

FastGrasp encodes a hand as a 768-dimensional latent with an autoencoder, and
trains a latent diffusion model over that latent. The denoiser is conditioned
on 3000 points sampled from the object surface [arxiv:2411.14786 §3.2] [eq. 8].
The latent is read as 256 tokens of three values each [arxiv:2411.14786 §3.2].
After the diffusion model is frozen, a small MLP is trained to add an offset to
the sampled latent. The decoder then maps the sum to 61 MANO parameters
[arxiv:2411.14786 §3.3] [eq. 9] [eq. 10]. The authors report that training the
diffusion model directly on the physical constraints oscillated and did not
converge, which is why the work is split in two [arxiv:2411.14786 §3.3]. On
out-of-domain HO-3D, with training on GRAB, they report a penetration volume of
5.23 against 7.37 for GraspTTA, and 0.14 s per pose on an NVIDIA A40
[arxiv:2411.14786 tab. 2] [§4.2]. The timing does not state the sampler, and the
paper does not state a step count [arxiv:2411.14786 §3.4]. Card:
[FastGrasp](/papers/2411.14786).

### Next-subgoal image generation from a progress memory

TaKSIE uses a diffusion image generator to propose the next subgoal for a
language-conditioned robot. The generator is built on a text-to-image latent
diffusion model [arxiv:2410.11013 §4.2.2]. Its inputs are the current view, an
embedding of the instruction and a progress embedding from a GRU over the
keyframes so far [arxiv:2410.11013 §3.4.1] [§3.3.2]. The paper accepts
ControlNet or InstructPix2Pix as the conditioned form and reports both
[arxiv:2410.11013 §3.4.1] [app. D]. Training subgoals are picked from
demonstrations by a progress signal, not at fixed intervals [arxiv:2410.11013
§3.3.1]. At run time a progress check moves on to the next subgoal at cosine
similarity 0.96, or after 20 steps [arxiv:2410.11013 §3.3.3] [§4.2.3]. The
authors report a CALVIN average length of 3.18, against 2.80 for SuSIE, which
uses fixed intervals [arxiv:2410.11013 tab. 2]. Card:
[TaKSIE](/papers/2410.11013). The released code differs from the paper in
several settings, so read the card's Paper vs code section before building.

### Surveys, not methods

Two cards are surveys. The state-of-the-art report on diffusion for visual
computing explains how Stable Diffusion runs the noise removal on a compressed
image, and it summarises the 2021 to 2023 literature [arxiv:2310.07204 §1]
[§3.2]. It gives almost no hyperparameters, so take training recipes from the
original papers it cites. The EEG survey names latent diffusion as one of the
decoder families used for EEG-to-image work, next to GANs and VAEs
[arxiv:2502.12048 §4.5]. Its numbers come from different setups and should not
be compared. Cards: [State of the Art on Diffusion Models for Visual
Computing](/papers/2310.07204), [A Survey on Bridging EEG Signals and Generative
AI](/papers/2502.12048).

## Guidance scale across cards

The values differ by card and by task, and the cards do not compare them
directly.

| Card | Scale in the paper | Notes |
|---|---|---|
| Motion, [arxiv:2212.04048 app. E] | 7.5 | Swept from 1.5 to 7.5 [tab. 11] |
| SDFusion, [arxiv:2212.04493 §3.3] | Per-input weights s_i | Released code uses different values per task; see its Paper vs code section |
| MultiDiffusion, [arxiv:2302.08113 §5] | Not stated | Released code uses 7.5 with a negative prompt |
| BLIP-Diffusion, [arxiv:2305.14720 app. A.5] | 7.5 | |
| SINE, [arxiv:2212.04489 §4] | Not stated; v = 0.7 on the pretrained term | Guidance switch at K = 400 [§4.3] |
| PowerPaint, [arxiv:2312.03594 §3.2] | Not stated | The card records a code default of 7.5 |
| DreamPose, [arxiv:2304.06025 §4.4] | Separate image and pose weights, not stated in the text | Card records code defaults of 0.5 and 0.5 |
| EMO, [arxiv:2402.17485 §4.1] | Not stated | |
| VASA-1, [arxiv:2404.10667 §4] | Audio 0.5 and gaze 1.0 by default | Audio scales 0.0, 1.0 and 2.0 tested [tab. 4] |
| W.A.L.T, [arxiv:2312.06662 tab. 8] | Not stated for class-conditional runs | Guidance listed only for text-to-video stages |
| CausVid, [arxiv:2412.07772 §5] | 3.5 for the teacher | A 2.0 setting appears in the released 4-step config |
| DiT, [arxiv:2212.09748 §3.1] | Not stated; tables use 1.25 and 1.50 | Guidance is applied to the first three latent channels only [arxiv:2212.09748 app. A] |
| T2I-Adapter, [arxiv:2302.08453 §4.2] | Not stated | The released main branch samples with scale 7.5 |
| StableSR, [arxiv:2305.07015 §5.1] | Negative-prompt scale s; default s = 1 leaves guidance unchanged | Tested 0 to 7.5 [arxiv:2305.07015 tab. 6]. The card records a code default of 7.0 when enabled |
| PSHuman, [arxiv:2409.10141 §4] | Not stated in the paper | The card records a code default of 3.0 at inference |
| SDXL, [arxiv:2307.01952 §2.2] | 8.0 in comparison figures; demo default 5.0 | The demo's 5.0 corresponds to w = 4.0 in the paper's notation [arxiv:2307.01952 app. Diffusion Models] |
| SLD, [arxiv:2211.05105 §3] | Not stated for the I2P and quality runs; ablations use 15 [arxiv:2211.05105 app. I] | The safety concept scale is separate, with presets from 200 to 5000 [arxiv:2211.05105 tab. 1] |
| Stable Signature, [arxiv:2303.15435 app. A.3] | 3.0 for the text-to-image runs, 50 steps | Set for the reported generation; the decoder change does not alter guidance [arxiv:2303.15435 §4.2] |
| DIFT, [arxiv:2306.03881 app. B] | Not applicable | No sampling loop; one denoiser pass per noise copy |
| Marigold, [arxiv:2312.02145 §3.2] | Not applicable | Text conditioning is disabled, so there is no guidance term |
| Motion-image head, [arxiv:2512.18007 §3.2] | Not stated | The head is trained with the flow-matching loss, and the card records no guidance term |
| GraspLDM, [arxiv:2312.11243 §3.2] | Not applicable | The method has no guidance term; the denoiser is conditioned on the shape latent only |
| FastGrasp, [arxiv:2411.14786 §3.4] | Not stated in the paper | The released code uses w = 2 for HO-3D and 0.5 for GRAB and OakInk, so the paper's silence is not a default |
| TaKSIE, [arxiv:2410.11013 §4.2.3] | Text 2.5 and image 2.5 | Two separate scales, with a negative prompt for the text term [arxiv:2410.11013 §3.4.1] |
| MagicAnimate, [code: configs/prompts/animation.yaml#L15] | Not in the card's hyperparameter table; released config uses 7.5 | Guidance is on when the scale exceeds 1 [code: magicanimate/pipelines/pipeline_animation.py#L586] |
| SeeSR, [arxiv:2311.16518 §4.1] | Not stated in the paper | Code default 5.5 [code: test_seesr.py#L251] |
| Drive-WM, [arxiv:2311.17918 app. B.1] | 5.0 | Sampled with DDIM, 50 steps, eta 1.0 [arxiv:2311.17918 app. B.1] |

## When to use which

| Situation | Use | Why |
|---|---|---|
| Generation must be fast and the data is long and redundant, such as motion clips | Latent diffusion instead of diffusion on raw sequences | Author-reported: 10.24 s against 225.28 s for MDM at 50 steps, 2048 clips, one V100 [arxiv:2212.04048 tab. 10] |
| Choosing the motion latent size | One token of width 256 | Lowest FID among the sizes tested [arxiv:2212.04048 tab. 5]. Five denoiser layers gave a lower FID (0.314) than the chosen nine, but the authors kept nine [arxiv:2212.04048 tab. 5] |
| Motion condition is a sentence and the model is small | Concatenate the embedding | Better R-Precision and FID than cross-attention in that model [arxiv:2212.04048 tab. 5] |
| Motion text match matters more than the lowest FID | Higher guidance scale, up to about 7.5 | R-Precision rises with s [arxiv:2212.04048 tab. 11] |
| Motion sampling budget is tight | 50 DDIM steps | Fewer steps than the 1000 used in training, with lower FID than 1000 DDPM steps [arxiv:2212.04048 tab. 10] |
| Output is 3D geometry and the inputs are partial, text or images | Latent diffusion over a 3D VQ code, with per-input weights | Author-reported UHD and TMD gains over MPC and AutoSDF on completion [arxiv:2212.04493 tab. 1] |
| Need a subject to appear with little or no per-subject training | Pretrained subject encoder, then an optional short fine-tune | Author-reported DINO 0.594 zero-shot against 0.668 for DreamBooth [arxiv:2305.14720 tab. 1] |
| Need a larger canvas or a region layout, and cannot train | Averaging a fixed model over views | Author-reported panorama FID and region IoU gains [arxiv:2302.08113 tab. 1] [tab. 2] |
| Need to edit one real photo from a text prompt, and per-image tuning is affordable | Fine-tune on the photo, then guide with the pretrained model after step K | Author-reported qualitatively only; no numbers in text [arxiv:2212.04489 §4.2] |
| Need one model for object insertion, removal and context filling | Learned task prompts on an inpainting model | Author-reported lowest FID on the object and context tables [arxiv:2312.03594 tab. 1] [tab. 3]. LaMa has the lowest LPIPS on Places2 [arxiv:2312.03594 tab. 3] |
| Need an animated clip of one known person from a pose sequence | Image and pose conditioned UNet, with per-subject tuning | Author-reported FID and FVD gains over MRAA and TPSMM [arxiv:2304.06025 tab. 1]. Tuning takes about 30 minutes per subject [arxiv:2304.06025 §6] |
| Need a temporally consistent clip from one photo and dense poses, with inference code you can run | Temporal layers and reference attention on a Stable Diffusion 1.5 UNet, with a DensePose ControlNet | Author-reported TikTok FVD 179.07 against 292.80 for DisCo [arxiv:2311.16498 tab. 1a]. Released code is inference only, and training settings are not stated [arxiv:2311.16498 §3.4] |
| Need a talking portrait from audio, with no 3D model or landmarks | Reference attention and audio layers on an image UNet | Author-reported best FID on HDTF [arxiv:2402.17485 tab. 1]. No code released [arxiv:2402.17485 §4.1] |
| Need real-time or streamed talking faces | Diffusion over face motion codes, rendered by a decoder | Author-reported SyncNet and pose gains; 50 default steps, 10 tested [arxiv:2404.10667 tab. 1] [§4.3] |
| Need images and video from one model | A shared image and video latent with window attention | Author-reported FVD and FID results, with trade-offs [arxiv:2312.06662 tab. 1] [tab. 2] |
| Need multi-camera driving video with box, map and ego-action controls, for testing a planner | Latent video diffusion with layout tokens, multiview layers and factorized view generation | Author-reported nuScenes FVD 122.7 against 452.0 for DriveDreamer [arxiv:2311.17918 tab. 1]. No weights or method code are released, so this is a paper-only option [code: README.md#L11-L15] |
| Need synchronised sound and video from a prompt | Two diffusion branches on frozen video and audio VAEs | Author-reported JavisScore gain over FoleyCrafter; UniVerse-1 is better on FVD [arxiv:2503.23377 tab. 1] |
| Need streaming, interactive video | Causal chunks with a cache, plus few-step distillation | Author-reported 1.3 s latency against 219.2 s for the teacher [arxiv:2412.07772 tab. 3]. The paper notes the student is less diverse than the teacher [arxiv:2412.07772 §6] |
| Want a transformer denoiser on latents, and the compute budget can grow | Transformer over latent patches with adaLN-Zero conditioning | Author-reported FID 2.27 against 3.60 for LDM-4 with guidance at 256 [arxiv:2212.09748 tab. 2]. The released code is non-commercial [arxiv:2212.09748 §3.2] |
| Have a frozen Stable Diffusion model and need spatial control, without retraining it | Trained control adapters, composed by a hand-set weighted sum | Author-reported FID 16.78 against 24.68 for text only, though the rows differ in input [arxiv:2302.08453 tab. 1] |
| Need to restore a real low-resolution photo with a generative prior, and can afford hundreds of sampler steps | Frozen Stable Diffusion with a trained time-aware encoder and a fidelity dial | Author-reported DIV2K-Valid FID 24.44 against 26.47 for LDM; lower PSNR and SSIM than several baselines [arxiv:2305.07015 tab. 1] [§4.3]. Use the 4-step variant when latency matters [arxiv:2305.07015 tab. 5] |
| Need restoration of real photos where the content is ambiguous, and a model of about 2.3B parameters at roughly 7 s per 512 px image on a V100 is affordable | Frozen Stable Diffusion 2-base with tag and tagger-embedding prompts, and a noised low-resolution start | Author-reported DIV2K-Val FID 31.93 against 36.57 for StableSR [arxiv:2311.16518 tab. 2]. Lower PSNR and SSIM than the best baseline on the paired sets [tab. 2]. Protocol differs from StableSR's own DIV2K-Valid test (see Contradictions) |
| Need a textured 3D human from one full-body photo, with no per-subject optimisation | Multi-view body diffusion on a Stable Diffusion unclip model, then SMPL-X-initialised remeshing | Author-reported Chamfer gains over ICON and SIFU on CAPE and THuman2.1 [arxiv:2409.10141 tab. 1]. Body-model ground truth is used in the geometry comparison, which removes pose error [arxiv:2409.10141 §4] |
| Need a higher-quality base image model, and memory is not the limit | Larger UNet with two text encoders, size and crop conditions, optional refiner | Author-reported 48.44% of votes for refined SDXL against 7.91% for SD 1.5 [arxiv:2307.01952 §2.5]. Two large models hurt accessibility and speed [arxiv:2307.01952 §3] |
| Need fewer inappropriate images from a Stable Diffusion model, with no retraining, and can accept some quality loss | Safety term in classifier-free guidance during sampling | Author-reported I2P inappropriate probability 0.39 to 0.09 at the strongest preset, with COCO FID rising from 14.43 to 18.76 [arxiv:2211.05105 tab. 2] [tab. 3]. Measurement depends on two classifiers [arxiv:2211.05105 app. D] |
| Need to show that an image came from your own model, or to trace it to one customer, and you can change the decoder | Decoder fine-tune on a fixed key, with a pre-trained extractor and a match threshold | Author-reported 0.99 bit accuracy unmodified and 0.92 after a combined edit [arxiv:2303.15435 tab. 1]. At 10^5 users, 98% identification unmodified and 40% after the combined edit [arxiv:2303.15435 §5.2]. Released code is non-commercial, so reimplement from the paper |
| Need point matches between two images with no labels, and one feature pass per image per noise draw fits the budget | Denoiser features at one noise level, nearest-neighbour matching | Author-reported 52.9 PCK per image against 33.9 for DINO on SPair-71k; the supervised SCorrSAN reaches 55.3 [arxiv:2306.03881 tab. 1] |
| Robot policy must also model how the scene will move, and inference must stay as fast as the base policy | Auxiliary motion-image latent head, trained jointly and dropped at inference | Author-reported LIBERO average 97.5% against 96.9% for pi0.5 [arxiv:2512.18007 tab. 1]; motion image 97.5% against 95.7% for a future-image target on pi0.5 [tab. 3]. The project page lists code as coming soon, and no official repository was found |
| Need many grasps per object, sampled fast, and a retrain per new task should touch only a small model | Conditional VAE grasp latent with a DDPM prior, sampled with 100 DDIM steps | Author-reported median success 0.756 at 100 DDIM steps against 0.792 at 1000 DDPM steps, and 0.75 s against 7.39 s for 100 grasps [arxiv:2312.11243 tab. 1]. Real-robot success 80% [tab. 2] |
| Need a hand pose for a given object in one pass, with few penetrations | Hand-pose latent diffusion, then an object-conditioned adaptation MLP | Author-reported penetration volume 5.23 against 7.37 for GraspTTA on out-of-domain HO-3D [arxiv:2411.14786 tab. 2]. The paper does not state the step count [arxiv:2411.14786 §3.4], and the timing sampler is not stated [§4.2] |
| Need a language-conditioned robot to pick visual subgoals by task progress, not by fixed intervals | Subgoal image diffusion with a progress GRU and a progress check | Author-reported CALVIN average length 3.18 against 2.80 for SuSIE [arxiv:2410.11013 tab. 2]. Real-robot success 71.9% against 56.3% [tab. 3]. The paper's best row uses R3M, while the released inference config sets CLIP [arxiv:2410.11013 tab. 2] [code: cfg/cfg.yaml#L15] |
| Need relative depth from one photo of an unfamiliar scene, with no metric scale, and about 500 denoiser passes per image is affordable | Denoiser fine-tuned on synthetic depth, with a median ensemble of ten runs | Author-reported best average rank and NYUv2 AbsRel 5.5 [arxiv:2312.02145 tab. 1]. Weaker than DPT on DIODE [arxiv:2312.02145 tab. 1]. Cost is about 500 U-Net passes [arxiv:2312.02145 §4.1]. Model weights carry a separate licence |

The speed comparison with MDM is not at matched quality. MDM's FID is 7.334
at 50 DDIM steps and 0.544 at 1000 DDPM steps [arxiv:2212.04048 tab. 10].
Compare at a fixed quality before choosing on speed alone. The same caution
applies to the video cards: their speed and quality numbers come from different
models and setups [arxiv:2412.07772 tab. 3] [arxiv:2312.06662 tab. 1]. The
talking-face and audio-video cards also use different metrics and test sets, so
their tables cannot be ranked against each other.

## Implementation sketch

Common core of the two stages and of sampling, with the motion paper's settings
as comments. Variants differ in the encoder, the denoiser architecture and the
condition embedding.

```pseudocode
# Stage 1: train the autoencoder once [arxiv:2212.04048 §3.1]
# z holds n tokens of width d; the loss is reconstruction plus KL [app. H.1]
z = encode(x)
x_hat = decode(z)

# Stage 2: freeze the encoder and train the denoiser [arxiv:2212.04048 §3.2]
# Frozen encoder, so no gradient flows into it
z0 = encode(x)
# Condition embedding from a frozen text encoder or a class embedding [§3.3]
c = embed(condition)
# Empty condition for 10% of samples, so one model learns both branches [§3.3]
c = drop(c, p=0.1)
# Training uses 1000 noise levels [arxiv:2212.04048 §4.2]
t = random_step(0, 1000)
eps = random_normal_like(z0)
# Forward noising of the latent [arxiv:2212.04048 eq. 1]
z_t = add_noise(z0, eps, t)
# Noise-prediction MSE with the condition [arxiv:2212.04048 eq. 3]
loss = mse(eps, denoiser(z_t, t, c))

# Sampling with classifier-free guidance [arxiv:2212.04048 eq. 4] [app. E]
# Start from noise in latent space
z = random_normal(n, d)
# 50 DDIM steps with eta = 0 [arxiv:2212.04048 §4.2] [app. C]
for t in ddim_steps(50):
    # Conditional and empty-condition predictions
    eps_c = denoiser(z, t, c)
    eps_u = denoiser(z, t, empty)
    # Guided prediction; s = 7.5 is the paper's default [app. E]
    eps = s * eps_c + (1 - s) * eps_u
    z = ddim_step(z, eps, t)
# Decode the final latent once [arxiv:2212.04048 §3.2]
return decode(z)
```

## Contradictions

- **Claimed versus tabulated accuracy.** The text says the paper's model is
  best on FID, R-Precision and MM Distance on both datasets [arxiv:2212.04048
  §4.3]. The tables show MotionDiffuse with the best R-Precision and MM
  Distance on both datasets, and the paper's model second on those metrics
  [arxiv:2212.04048 tab. 1] [arxiv:2212.04048 tab. 2]. Only FID is clearly best
  for the paper's model in the tables.
- **Speed.** The abstract and [arxiv:2212.04048 §5] describe two orders of
  magnitude over diffusion on raw motion. The timing table gives about 22 times
  at 50 DDIM steps and about 30 times at 1000 DDPM steps [arxiv:2212.04048
  tab. 10].
- **Layer count.** The text calls the layer counts similar [arxiv:2212.04048
  §5]. The ablation gives FID 0.314 for five layers and 0.473 for nine
  [arxiv:2212.04048 tab. 5].
- **Latent size in two video and motion ablations.** The motion paper finds one
  token gives the best FID of the sizes tested [arxiv:2212.04048 tab. 5]. W.A.L.T
  calls 8 channels a sweet spot, but its table shows 16 channels with lower FVD
  (67.0 against 75.4) at slightly lower IS [arxiv:2312.06662 tab. 3]. The two
  ablations measure different tasks and metrics, so neither result settles a
  rule for latent size.
- **W.A.L.T ablation baseline.** The default configuration scores 60.7 FVD, but
  the self-conditioning row at p_sc = 0.9 scores 61.4 [arxiv:2312.06662 tab. 3].
  The paper does not say which row is the baseline.
- **FID against human preference for SDXL.** On 10k COCO pairs, SDXL has the
  worst FID of the three models compared, but slightly higher CLIP. Its human
  study favours it [arxiv:2307.01952 §2.5]. The paper's own discussion says FID
  and CLIP do not track these results, so the two measures disagree on the
  winner.
- **Consistent hard-mode gains.** The robot-policy paper says its method
  "consistently improves" success in both RoboTwin modes [arxiv:2512.18007
  §4.3]. Its table shows lower hard-mode scores than pi0 on Move Can Pot, Move
  Playingcard and Stack Blocks Two, and a hard-mode average gain of 0.4 points
  [arxiv:2512.18007 tab. 2].
- **Best FID versus best LPIPS on Places2.** PowerPaint's summary says it has the
  best FID on Places2 context filling [arxiv:2312.03594 tab. 3]. The same table
  shows LaMa with the lowest LPIPS, 0.1193 against 0.1263 on all samples
  [arxiv:2312.03594 tab. 3]. The two metrics disagree on the best method, so a
  single winner depends on which metric matters.
- **TikTok video and frame metrics.** MagicAnimate has the best FVD and FID-VID
  in its TikTok table, but its FID of 32.09 is higher than DisCo's 30.75 and
  than its own ablation without temporal layers, 27.54 [arxiv:2311.16498 tab. 1a]
  [tab. 2a]. The video and frame metrics pick different winners.
- **StableSR's DIV2K score.** SeeSR reports StableSR at FID 36.57 on DIV2K-Val
  [arxiv:2311.16518 tab. 2]. StableSR reports FID 24.44 on DIV2K-Valid
  [arxiv:2305.07015 tab. 1]. The test sets differ: SeeSR's are 3K random 512 px
  crops degraded by its training pipeline, and StableSR's are 3K pairs with
  128 px LR inputs. The numbers are not comparable.
- **Combined driving reward.** The text says the combined Drive-WM reward beats
  each sub-reward [arxiv:2311.17918 §5.4]. Table 4 shows the object reward alone
  gives the same average L2 of 0.80 m. The combined reward is better only on
  collision, 0.26% against 0.27% [arxiv:2311.17918 tab. 4].
- **Single-view or multi-view row.** The text calls the 12.99 FID row
  single-view [arxiv:2311.17918 §5.2], but Table 1 marks it multi-view
  [tab. 1]. The card takes the table reading as the more likely one.

## Papers

- [Executing your Commands via Motion Diffusion in Latent Space](/papers/2212.04048): text- and action-conditioned human motion generation with a transformer autoencoder and a latent denoiser.
- [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](/papers/2212.04493): a diffusion model over a 3D VQ-VAE latent, conditioned on partial shapes, images and text with per-input weights.
- [SINE: SINgle Image Editing with Text-to-Image Diffusion Models](/papers/2212.04489): fine-tuning a Stable Diffusion model on one image, with a guidance switch and patch-based training for editing.
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](/papers/2302.08113): averaging a fixed Stable Diffusion model over crops or masked regions, with no training.
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](/papers/2305.14720): subject tokens from a vision-language encoder, spliced into the prompt of a Stable Diffusion model.
- [DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion](/papers/2304.06025): a frame-by-frame video generator from one photo and a pose sequence, with dual guidance.
- [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](/papers/2312.03594): one inpainting model with learned task prompts for insertion, removal and context filling.
- [Photorealistic Video Generation with Diffusion Models](/papers/2312.06662): one transformer over a shared image and video latent, with window attention.
- [EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](/papers/2402.17485): a portrait video generator from audio, with reference attention on a Stable Diffusion 1.5 UNet.
- [VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time](/papers/2404.10667): diffusion over learned face motion codes, decoded to frames, for real-time talking faces.
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](/papers/2412.07772): a causal, chunked video latent generator distilled to four steps, with a key-value cache.
- [JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization](/papers/2503.23377): two diffusion branches for video and audio on frozen VAEs, with shared spatio-temporal priors.
- [State of the Art on Diffusion Models for Visual Computing](/papers/2310.07204): a 2023 survey of diffusion models, including Stable Diffusion's latent space.
- [A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond](/papers/2502.12048): a survey of EEG-to-media generation, where latent diffusion is one decoder family.
- [Scalable Diffusion Models with Transformers](/papers/2212.09748): a transformer denoiser over latent patches with adaLN-Zero conditioning, scaled in depth, width and tokens.
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](/papers/2302.08453): small control adapters on a frozen Stable Diffusion UNet, composed by weighted sum.
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](/papers/2307.01952): a larger two-encoder latent UNet with size and crop conditioning and an optional refiner.
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](/papers/2211.05105): a safety term added to classifier-free guidance during sampling, with no retraining.
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](/papers/2305.07015): a frozen Stable Diffusion 2.1 steered by a trained time-aware encoder, with a fidelity dial and patch-blended sampling.
- [PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing](/papers/2409.10141): six-view body diffusion on a Stable Diffusion unclip model, then SMPL-X-initialised remeshing.
- [The Stable Signature: Rooting Watermarks in Latent Diffusion Models](/papers/2303.15435): a decoder fine-tune that embeds a fixed 48-bit key in every output, read by a separate extractor.
- [Emergent Correspondence from Image Diffusion](/papers/2306.03881): point matching from one denoiser pass's features, with no training.
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](/papers/2312.02145): a denoiser fine-tuned on synthetic depth, with a median ensemble of ten runs.
- [Robotic VLA Benefits from Joint Learning with Motion Image Diffusion](/papers/2512.18007): a motion-image latent diffusion head on a robot policy, trained jointly and unused at inference.
- [GraspLDM: Generative 6-DoF Grasp Synthesis using Latent Diffusion Models](/papers/2312.11243): a DDPM over the latents of a conditional VAE for grasp poses, with a small denoiser per task.
- [FastGrasp: Efficient Grasp Synthesis with Diffusion](/papers/2411.14786): latent diffusion over a hand-pose autoencoder, then an adaptation MLP trained for contact and penetration.
- [Incorporating Task Progress Knowledge for Subgoal Generation in Robotic Manipulation through Image Edits](/papers/2410.11013): a diffusion image generator for the next subgoal, with a progress memory and a progress check.
- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](/papers/2311.16498): a Stable Diffusion UNet with temporal layers, a reference-attention copy and DensePose control, run over overlapping windows.
- [SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution](/papers/2311.16518): Stable Diffusion x4 restoration with tag and tagger-embedding prompts, sampled from a noised low-resolution latent.
- [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](/papers/2311.17918): a latent video diffusion world model for six driving cameras, with layout and action tokens and tree planning.