Latent diffusion

Overview

Latent diffusion trains a diffusion model on a compressed representation of the data. An autoencoder maps each sample to a smaller latent and back. A denoiser learns to remove noise from latents, and the decoder runs once on the final latent 2212.04048 §3. The iterative part therefore works on a small tensor, not on the full sample. That is the speed argument the generative cards rely on 2212.04048 §1 2212.04048 §3.2 2302.08113 §5.

The invariant shared by the generative cards: the denoiser works on a learned code, not on raw samples, and a decoder returns the final code to data space 2212.04048 §3.2 2212.04493 §3.2 2312.06662 §4.1 2412.07772 §4.1 2212.04489 §3.1 2503.23377 app. C.1. What differs is the latent (a 3D VQ code, a continuous video latent, a Stable Diffusion latent, a motion code), the parameterisation of the target, and how conditions enter. VASA-1 is the furthest from the pattern: its diffusion transformer works on head-pose and facial-dynamics codes from a separate encoder, and its frame renderer is not a diffusion model 2404.10667 §3. The robot-policy card also uses a frozen VAE latent, but as a second output of a control model, not as the generated sample 2512.18007 §3.2 app. A.2.

Three robot cards keep the same split between an autoencoder and a denoiser, but change what the code stands for. In GraspLDM the code is a 6-DoF grasp, produced by a conditional VAE 2312.11243 §3.1. In FastGrasp it is a human hand pose, and the denoiser is conditioned on object points 2411.14786 §3.2. In TaKSIE the diffusion model draws the next subgoal image 2410.11013 §3.4.1.

Most image cards use a Stable Diffusion checkpoint as a fixed or fine-tuned base 2302.08113 §5 2305.14720 §4.1 2212.04489 §4 2312.03594 §3.1 2304.06025 §4.3 2302.08453 §3.1 2305.07015 §3.1 2409.10141 §4 2211.05105 §3 2311.16518 §4.1, so image claims here are about those systems, not about latent diffusion in general. Two cards depart from that base. DiT reuses only a Stable Diffusion autoencoder and trains its own denoiser 2212.09748 §4, and SDXL trains a new, larger UNet 2307.01952 §2.1. For the general noise-prediction setup, see Diffusion models.

Two of the newer cards extend an image model in time. MagicAnimate and Drive-WM insert temporal layers into a UNet, and Drive-WM also adds layers across camera views 2311.16498 §3.1 2311.17918 §3.1. The image models they start from are a Stable Diffusion 1.5 UNet for MagicAnimate animation.yaml and Stable Diffusion checkpoints of an unstated version for Drive-WM 2311.17918 §5.1.

Three of the newer cards put the latent space to other uses. The Stable Signature changes only the decoder and leaves the denoiser alone [arxiv:2303.15435 §4.2]. DIFT runs one denoiser pass and generates nothing 2306.03881 §4.2. Marigold trains the denoiser to output depth latents, not images [arxiv:2312.02145 §3.1].

The two surveys in the list (2310.07204 and 2502.12048) run no experiments. They are used for orientation only, and their numbers are not reported here.

Variants

Motion latent diffusion with a two-stage training

Stage one trains a transformer autoencoder with a reconstruction term and a KL term, so that a whole motion clip becomes a latent of size n × d 2212.04048 §3.1 app. H.1. Stage two freezes the encoder and trains a transformer denoiser on the latents, with the condition as an extra input 2212.04048 §3.2 2212.04048 §3.3. The paper's best setting uses one token of width 256 2212.04048 §5 tab. 5. Card: Executing your Commands via Motion Diffusion in Latent Space.

Latent size in the motion model

The paper compares one to ten tokens of width 256 2212.04048 tab. 5. The single-token latent gives the best FID of those tested (0.473). Two tokens give 0.585, five give 1.554, seven give 1.011 and ten give 1.716 2212.04048 tab. 5. Reconstruction improves with more tokens, though: the seven-token model with skip connections has the lowest reconstruction error in the table 2212.04048 tab. 4. So a better autoencoder did not produce a better generative model here.

Condition entry in the motion model

The paper concatenates the condition embedding with the latent. It compares this with cross-attention and reports concatenation as the better choice 2212.04048 §3.3 tab. 5. Cross-attention reaches R-Precision Top 3 of 0.592 and FID of 1.922, against 0.772 and 0.473 for concatenation 2212.04048 tab. 5. Cross-attention has a higher multimodality score (3.768 against 2.413), so it produces more varied outputs but matches the text less well 2212.04048 tab. 5. The attention block itself is described on Attention mechanism.

Condition encoders in the motion model

Text conditions use a frozen CLIP text encoder 2212.04048 §4.2. Action labels use a learnable embedding per class 2212.04048 §3.3. Both are trained with the same conditional objective 2212.04048 eq. 3.

Guidance scale and dropout in the motion model

The condition is dropped for 10% of training samples, and at sampling time the conditional and unconditional predictions are combined with scale s 2212.04048 §3.3 eq. 4. The paper's default is s = 7.5 with dropout 0.1 2212.04048 app. E tab. 11. Raising s increases text match: R- Precision Top 3 rises from 0.648 at s = 1.5 to 0.772 at s = 7.5. FID is lower at intermediate values, with 0.350 at s = 2.5 against 0.473 at s = 7.5 2212.04048 tab. 11. Dropout 0.1 is not the FID-best setting in the sweep: dropout 0.2 gives 0.279 at s = 7.5 2212.04048 tab. 11.

Sampler and step count in the motion model

The paper samples with DDIM and no added noise (eta = 0), using 50 steps after training with 1000 2212.04048 §4.2 app. C. Running the same model with 1000 DDPM steps gives FID 0.568 against 0.473 with 50 DDIM steps, and takes 148.97 s against 10.24 s for 2048 clips on one V100 2212.04048 tab. 10. More steps did not help FID in this evaluation.

Per-input weights for 3D shapes

SDFusion compresses truncated signed-distance grids with a 3D VQ-VAE, then trains a diffusion model on the codes 2212.04493 §3.1 §3.2. Partial shapes, images and text each have an encoder. Their features are concatenated and fed to the denoiser through cross-attention 2212.04493 §3.3 eq. 3. At sampling time each input gets its own weight, so the output follows the heavier inputs more closely 2212.04493 §3.3 eq. 4. The authors report lower UHD and higher TMD than MPC and AutoSDF on shape completion 2212.04493 tab. 1, and the best Pix3D F-score of 0.432 against 0.415 for AutoSDF 2212.04493 tab. 2. Card: SDFusion.

Multi-view body diffusion, then remeshing

PSHuman fine-tunes Stable Diffusion 2.1-unclip to draw six body views and one face view, as color and normal maps, from one full-body photo [arxiv:2409.10141 §3.1] §4. Renders of a body model (SMPL-X) fitted to the photo are encoded by a pretrained VAE and concatenated with the noise latent as extra input 2409.10141 §3.1.2. A face branch joins the body views through shared self-attention 2409.10141 §3.1.1. A second stage starts from an SMPL-X mesh and deforms and remeshes it until it fits the generated normal and silhouette maps 2409.10141 §3.2 eq. 7 eq. 8. The authors report a Chamfer distance of 0.9688 against 1.5511 for ICON on CAPE-NFP, when the ground-truth body model is given 2409.10141 tab. 1. The released code does not texture the mesh the way the paper describes, so take the texture step from the card's Paper vs code notes, not from the paper alone 2409.10141 §3.2. Card: PSHuman.

Training-free composition for panoramas and regions

MultiDiffusion trains nothing. A pretrained Stable Diffusion v2 model is run on overlapping crops of a larger canvas, or on masked regions, at every denoising step. The outputs are averaged per pixel into one next latent 2302.08113 §3 alg. 1 eq. 5. For a panorama this gives a plain average over the crops, and for layouts the masks act as weights [arxiv:2302.08113 §4.1] §4.2. The authors report a panorama FID of 10.3, against 18.4 for Blended Latent Diffusion and 45.5 for Stable Inpainting [arxiv:2302.08113 tab. 1], and a region IoU of 0.26, against 0.17 and 0.16 [arxiv:2302.08113 tab. 2]. The cost is one reference call per view and per step 2302.08113 app. B.1. Card: MultiDiffusion.

Subject tokens from a vision-language encoder

BLIP-Diffusion puts a frozen image encoder and a trainable multimodal encoder in front of Stable Diffusion v1-5 2305.14720 §4.1. Sixteen query outputs are projected into the text-embedding space and inserted into the prompt tokens, so the U-Net sees them as part of the text condition 2305.14720 §3.2. The authors report DINO 0.594 zero-shot and 0.670 after fine-tuning, against 0.668 for DreamBooth 2305.14720 tab. 1. A fine-tune takes 40 to 120 steps per subject 2305.14720 §3.3 tab. 3. The second pre-training stage is not in the released code, so the base model cannot be reproduced from that repository 2305.14720 §3.2. Card: BLIP-Diffusion.

Single-image editing with a fine-tuned copy and a guidance switch

SINE fine-tunes a Stable Diffusion model on one source image, then edits that image with a text prompt 2212.04489 §3.1 §4. The fine-tuned model is mixed into classifier-free guidance only while the sampler is in its early steps (t > K). Late steps use the pretrained model alone, so the fine-tuned model fixes only the coarse structure 2212.04489 §3.2 eq. 3. A patch fine-tune, with a positional code for each crop, lets the edit run at resolutions other than the training size 2212.04489 §3.3. The paper sets K = 400 and v = 0.7 and does not state w 2212.04489 §4 §4.3. Its evidence is qualitative. The text gives no headline number, and the sweeps for K and v appear only as plots 2212.04489 fig. 8. Card: SINE.

Task prompts for one inpainting model

PowerPaint fine-tunes a Stable Diffusion inpainting model with three learned task prompts: one for context filling, one for text-guided object insertion, and one for shape-guided insertion 2312.03594 §3.2 fig. 2. Object removal reuses the context prompt as the positive prompt and the object prompt as the negative prompt in classifier-free guidance 2312.03594 §3.2 eq. 5. Shape control blends the two text embeddings with a fitting degree alpha 2312.03594 §3.2 eq. 7. The authors report the lowest FID on OpenImages and MSCOCO object insertion, 4.38 and 5.12 against 4.76 and 5.65 for SD-Inpainting 2312.03594 tab. 1, and the lowest FID on Places2 context filling, 2.59 against 3.03 2312.03594 tab. 3. The training mix samples 80% inpainting and 20% text-to-image 2312.03594 §3.3. Card: A Task is Worth One Word.

Frame-by-frame video from one photo and a pose sequence

DreamPose conditions a Stable Diffusion UNet on one photo through a CLIP and VAE adapter, and on five consecutive DensePose frames through extra input channels 2304.06025 §4.2. Each output frame is sampled on its own, so temporal smoothness comes from the conditions, not from a video model 2304.06025 §4.1. Sampling uses separate guidance weights for the image and the pose 2304.06025 §4.4 eq. 4. Per-subject tuning is the costly step: the authors report about 10 minutes for the UNet and 20 minutes for the VAE decoder, and 18 s per rendered frame 2304.06025 §6. The UBC Fashion table favours DreamPose on every metric, with FID 13.04 against 22.87 for TPSMM 2304.06025 tab. 1. The paper's training frame size differs from its code, so check the card's Paper vs code section before building. Card: DreamPose.

Temporal layers and reference attention for a whole clip

MagicAnimate extends a Stable Diffusion 1.5 UNet along time. Temporal attention layers let the frames of one clip agree with each other 2311.16498 §3.1. A trainable copy of the UNet reads the reference photo at each denoising step, and its normalised attention states join the keys and values of the generator's self-attention 2311.16498 §3.2 eq. 2. Poses enter as DensePose maps through a ControlNet 2311.16498 §3.3. Long videos are cut into overlapping windows, and their noise predictions are averaged at each step 2311.16498 §3.3. The released config uses 16-frame windows, 4 frames of overlap, 25 DDIM steps and a guidance scale of 7.5 animation.yaml pipeline_animation.py. Author-reported on TikTok, FVD is 179.07 against 292.80 for DisCo, and FID-VID is 21.75 against 59.90 2311.16498 tab. 1a. Removing the temporal layers raises FVD to 247.30 2311.16498 tab. 2a. Training settings are not stated, and the supplement in the card's source was empty 2311.16498 §3.4 §4. The paper's reference-branch equation takes the noisy latent, but the released pipeline feeds the clean reference latent 2311.16498 eq. 1. Card: MagicAnimate.

Audio-driven portrait video with a reference network

EMO starts from the Stable Diffusion 1.5 UNet with its text cross-attention replaced by reference attention to one portrait 2402.17485 §3.1 §3.2.1. Audio features enter through audio layers, and a face-region mask and a head-speed value act as weak conditions 2402.17485 §3.2. Clips are 12 frames, and each clip reuses 4 motion frames from the previous one 2402.17485 §4.1 §3.2.4. The sampler is DDIM with 40 steps 2402.17485 §4.1. On HDTF, the authors report FID 8.76, the best in their table, and FVD 67.66 2402.17485 tab. 1. The paper gives no guidance scale, and no code is released 2402.17485 §4.1. Card: EMO.

Diffusion over motion codes, not image latents

VASA-1 learns four separate codes for each face: a canonical 3D appearance volume, identity, head pose and facial dynamics 2404.10667 §3.1. A diffusion transformer generates the pose and dynamics codes from audio, and a decoder turns them into frames 2404.10667 §3 fig. 2. The diffusion therefore runs in a compact motion space, and the frame renderer is not a latent-image diffusion model. The generator works in overlapping chunks, so output can be streamed 2404.10667 §3.2. The default is 50 sampling steps, and 10 steps give about a fivefold speed-up in latent generation 2404.10667 §4.3 tab. 4. On a VoxCeleb2 subset the authors report SyncNet confidence 8.84 against 5.84 for SadTalker 2404.10667 tab. 1. Card: VASA-1.

Joint audio and video latents

JavisDiT is a diffusion transformer with one branch for video and one for audio, linked by bidirectional cross-attention 2503.23377 §3.1. The video latent comes from a frozen OpenSora VAE, and the audio is a mel spectrogram encoded by a frozen AudioLDM2 VAE 2503.23377 app. C.1 app. C.2.2. A learned prior estimator maps the prompt to spatial and temporal tokens that enter both branches 2503.23377 §3.2. Sampling uses rectified flow 2503.23377 app. C.1. The authors report JavisScore 0.154 against 0.151 for FoleyCrafter, but UniVerse-1 is better on FVD (194.2 against 204.1), KVD and TV-IB 2503.23377 tab. 1. Card: JavisDiT.

One latent for images and video

W.A.L.T uses a causal autoencoder without a codebook. It maps images and video into one continuous latent space, so a single transformer trains on both 2312.06662 §3 §4.1. The denoiser alternates two window types: spatial windows within one latent frame, and spatiotemporal windows across frames 2312.06662 §4.2. The default latent has 8 channels [arxiv:2312.06662 §5.2] tab. 3. Reported results include FID 2.56 on ImageNet 256 without classifier-free guidance 2312.06662 tab. 2, and K600 frame-prediction FVD of 3.3 against 4.3 for MAGVIT-v2 2312.06662 tab. 1. The denoiser is trained with v-prediction 2312.06662 §3. Card: Photorealistic Video Generation with Diffusion Models.

Multiview driving video with layout and action tokens

Drive-WM first trains a single-view, conditional image latent diffusion model. It then adds temporal layers after each spatial block and self-attention across the six camera views. The spatial weights stay frozen, and only the temporal and multiview weights are trained 2311.17918 §3.1 app. B.1. Conditions are mapped to tokens and concatenated: image tokens from context frames and reference views, layout tokens for 3D boxes, the HD map and BEV segmentation, text tokens, and ego-action tokens 2311.17918 §3.3 eq. 5. To keep overlapping cameras consistent, three reference views are generated first, and the other three are generated from their neighbours and their own previous frame 2311.17918 §3.2 app. B.2. Author-reported on nuScenes, FID is 15.8 and FVD is 122.7, against 52.6 and 452.0 for DriveDreamer 2311.17918 tab. 1. Factorized generation raises the keypoint-matching score from 45.8% to 94.4%, and costs FID (16.4 against 15.8) 2311.17918 tab. 2. For planning, the model renders the future of each candidate trajectory from VAD, and perception models score the result §4.1 §4.2. Open-loop average L2 is 0.80 m and collision is 0.26%, against 1.02 m and 0.93% for VAD with a random command 2311.17918 tab. 3. No model weights or method code are released, so the numbers are author statements only README.md. Card: Driving into the Future.

Distilled, streaming video latents

CausVid turns a bidirectional latent video transformer into a causal one that generates chunk by chunk with a key-value cache 2412.07772 §4.1 §4.4. A few-step student is distilled from a slower bidirectional teacher with distribution matching 2412.07772 §4.2. The authors report 1.3 s latency and 9.4 FPS on one H100, against 219.2 s for the bidirectional teacher on a 120-frame clip 2412.07772 tab. 3. The card records the released code as non-commercial, so the code cannot be reused commercially, although the paper itself is CC BY 4.0. Card: From Slow Bidirectional to Fast Autoregressive Video Diffusion Models.

Transformer denoiser on latent patches

DiT replaces the U-Net denoiser with a transformer. A frozen autoencoder encodes each image, the latent is cut into patches, and each patch becomes a token 2212.09748 §3.1 §3.2. Timestep and class condition enter through adaLN-Zero: they regress a scale, shift and gate for each block, and the gate starts at zero, so every block begins as the identity [arxiv:2212.09748 §3.2]. The paper compared this with in-context, cross-attention and plain adaLN conditioning, and adaLN-Zero beat the first two at every training stage in its block study 2212.09748 §3.2 fig. 5. Halving the patch size quadruples the tokens and at least quadruples the transformer cost. Across its twelve configurations, FID falls as forward-pass Gflops rise, though the paper gives no fitted coefficient 2212.09748 §3.2 §5 tab. 4. The author-reported class-conditional ImageNet FID with guidance is 2.27 at 256 by 256 and 3.04 at 512 by 512, against 3.60 for LDM-4 with guidance at 256 2212.09748 tab. 2 tab. 3. The released code is licensed for non-commercial use, so reimplement from the paper rather than reuse that code. Card: Scalable Diffusion Models with Transformers.

Frozen base with a control adapter

T2I-Adapter does not change the latent diffusion model. It trains small adapter networks that read a control map, such as a sketch, a segmentation, a pose or a depth map, and add multi-scale features to the encoder of a frozen Stable Diffusion UNet 2302.08453 §3.3 eq. 4. Several adapters can be combined as a weighted sum without retraining, but the weights are set by hand 2302.08453 §3.3 eq. 5 §5. Adapters are trained with the base model frozen, using a cubic timestep rule that favours high-noise steps 2302.08453 §3.4. The author-reported COCO validation FID is 16.78 with text and segmentation, and 17.36 with text and sketch, against 24.68 for Stable Diffusion with text only 2302.08453 tab. 1. The rows differ in their input, so this does not isolate the adapter from the control map alone. The paper's model is SD 1.4, but the released main branch targets SDXL, and its injection points and training data differ from the paper's. Card: T2I-Adapter.

Restoration with a frozen generator and a trained encoder

StableSR keeps Stable Diffusion 2.1 frozen and trains only a time-aware encoder and scale-and-shift layers that modulate the U-Net features [arxiv:2305.07015 §3.1] eq. 1. The encoder reads the low-resolution latent and the diffusion step, so its guidance changes along the sampling trajectory. The authors place the strongest guidance near a signal-to-noise ratio of 5e-2 [arxiv:2305.07015 §3.1] fig. 3. An optional fidelity module adds encoder features into the VAE decoder, scaled by a user weight w; w = 0.5 is the default 2305.07015 §3.2 tab. 4. Inputs of any size are handled by denoising overlapping latent patches and blending their noise predictions with Gaussian weights 2305.07015 §3.3 alg. 1. The authors report DIV2K-Valid FID of 24.44 against 26.47 for LDM, which they fine-tuned with the same settings 2305.07015 tab. 1. PSNR, SSIM and LPIPS are lower than several baselines 2305.07015 §4.3. Sampling is the cost: 200 steps take 15.16 s for a 512 px output on one V100, against 0.83 s for the 4-step SD-Turbo variant 2305.07015 tab. 5. The released code is licensed for non-commercial use, so reimplement from the paper rather than reuse that code. Card: StableSR.

Semantic prompts and a noised low-resolution start

SeeSR keeps the Stable Diffusion 2-base UNet, its text encoder and its VAE frozen 2311.16518 §3.3 §4.1. The trained parts are a ControlNet cloned from the UNet encoder, an image encoder that feeds it, and representation cross-attention modules placed after the text cross-attention §3.3. The semantic input comes from DAPE, a RAM tagger fine-tuned with LoRA on degraded images. DAPE gives predicted tags, which enter as text, and representation embeddings, which enter as soft prompts 2311.16518 §3.2. At inference the low-resolution latent is noised to training step 999 and used as the start of sampling, instead of Gaussian noise §3.4. The authors say this removes false sky texture fig. 3. Author-reported on DIV2K-Val, FID is 31.93 against 36.57 for StableSR, and SeeSR is best on LPIPS, DISTS, FID, MUSIQ and CLIPIQA 2311.16518 tab. 2. On the three paired sets it is below the best baseline on PSNR and SSIM tab. 2. A supplementary ablation shows the start helps PSNR and LPIPS on DIV2K-Val but lowers CLIPIQA, from 0.7314 without it to 0.6834 with it 2311.16518 suppl. LRE table. The model has 2283.7M parameters and takes about 7.24 s per 512 px output with 50 steps on one V100 2311.16518 suppl. complexity table. Card: SeeSR.

Larger two-encoder base with size conditioning

SDXL uses a UNet about three times the size of SD 1.x and 2.x, with 2.6B parameters, and reads the prompt with two text encoders whose outputs are concatenated 2307.01952 §2.1. Training uses the original image size and the crop offset as extra conditions, which keeps small or oddly shaped images in the data 2307.01952 §2.2 alg. 1. At inference the authors set the crop offset to zero 2307.01952 §2.2. Multi-aspect buckets, with the target size as a further condition, cover landscape and portrait shapes 2307.01952 §2.3. A separate refiner, run on the first 200 of 1000 noise levels, is optional 2307.01952 §2.5. The authors report 48.44% of human votes for refined SDXL against 7.91% for SD 1.5 and 6.71% for SD 2.1, in a four-way study 2307.01952 §2.5. Size conditioning also improves class-conditional ImageNet FID-5k at 512 by 512 from 39.76 with no conditioning to 36.53 2307.01952 §2.2. The two text encoders total 817M parameters, and the refiner is a second full model, so memory is a constraint 2307.01952 §2.1 §3. The weights carry a licence separate from the MIT code, and that licence was not checked. Card: SDXL.

Safety term added to guidance during sampling

SLD changes only the sampler of a Stable Diffusion model. It adds a third noise estimate, conditioned on a text description of unwanted content, and pushes the prompt estimate away from it 2211.05105 §3 eq. 3 eq. 4. The push is element-wise. It is zero where the prompt estimate already exceeds the concept estimate by a threshold, and otherwise scales with a concept factor 2211.05105 eq. 5 eq. 6. A warm-up delays the term, and a momentum term carries it across steps 2211.05105 §3 alg. 1. The default concept is a list of about twenty categories, such as violence, nudity and weapons 2211.05105 §4. No training or external classifier is used, and the method applies to any latent diffusion model with classifier-free guidance 2211.05105 app. D. The authors report that the probability of an inappropriate image on the I2P benchmark falls from 0.39 for the baseline to 0.09 at the strongest preset, and from 0.35 to 0.06 on the sexual subset 2211.05105 tab. 2. Negative prompting reaches 0.18 in the same table, so it is close to the middle presets and behind the strongest one. The cost is measured on image quality: COCO FID-30k rises from 14.43 to 18.28 at the strongest preset and 18.76 at the maximum 2211.05105 tab. 3. The classifier that labels the outputs is described as conservative, so part of the measured rate reflects false positives 2211.05105 tab. 2. The paper's maximum preset uses a threshold outside its recommended range 2211.05105 tab. 1 §4. The authors ask that any deployment state which content is suppressed, because the same mechanism can serve censorship 2211.05105 §7. Card: Safe Latent Diffusion.

Watermark in the decoder, read by a separate extractor

The Stable Signature fine-tunes only a copy of the autoencoder's decoder, so that every output carries one fixed 48-bit key. The diffusion process is not touched 2303.15435 §4.2 fig. 1. The key is read by an extractor that was pre-trained with simulated edits and then whitened, so its bits on ordinary images are close to fair coin flips 2303.15435 §4.1 app. A.1. The decoder fine-tune runs about 100 steps on fewer than 500 images 2303.15435 §4.2. Detection flags an image when enough bits match the key, and identification picks the best of N user keys 2303.15435 §3. The authors report 0.99 bit accuracy on unmodified output and 0.92 after a combined crop, brightness and JPEG edit, with FID moving by at most 0.6 2303.15435 tab. 1. At N = 10^5 users, identification is 98% accurate on unmodified images and 40% after the combined edit 2303.15435 §5.2. The headline text-to-image numbers come from an internal model, not a public one 2303.15435 §5. Card: The Stable Signature.

Denoiser features for matching points, with no training

DIFT takes a real image, noises it to a chosen level t, runs the denoiser once, and keeps the activations of one upsampling block, averaged over several noise draws 2306.03881 §4.2 app. C. Points are matched by nearest neighbour under cosine distance 2306.03881 §3 eq. 1. No parameter is trained and no correspondence label is used 2306.03881 §1. The time step and block index are grid-searched per task. The two backbones are Stable Diffusion 2-1 and an ADM model 2306.03881 §5.1 app. C. On SPair-71k the authors report 52.9 PCK per image for the Stable Diffusion features, against 33.9 for DINO and 38.4 for OpenCLIP 2306.03881 tab. 1. The supervised SCorrSAN reaches 55.3 on the same table. The paper describes noising the image, but the card's Paper vs code notes record that the released Stable Diffusion code noises the VAE latent instead 2306.03881 §4.2. Card: Emergent Correspondence from Image Diffusion.

Depth as the denoised target, with an ensemble

Marigold fine-tunes only the denoising U-Net of Stable Diffusion v2 to output depth, with text conditioning disabled 2312.02145 §3.2 §4.1. The frozen VAE encodes both the image and the depth map, and their latents are concatenated at the U-Net input 2312.02145 §3.2. Training uses only synthetic depth from Hypersim and Virtual KITTI, normalised per image to a fixed percentile range, so the target is affine-invariant 2312.02145 §3.3 eq. 3. At inference, ten runs start from different noise. Each run is aligned by a scale and a shift, and the pixel-wise median is taken [arxiv:2312.02145 §3.4] eq. 4. The authors report AbsRel 5.5 on NYUv2 for the ten-run ensemble, against 6.0 for a single run 2312.02145 tab. 1. Going from one run to ten cuts NYUv2 AbsRel by about 8% 2312.02145 §4.3 fig. 6. On DIODE the method is weaker than DPT, with AbsRel 30.8 against 18.2 [arxiv:2312.02145 tab. 1]. The output is not in metres, so scale must come from elsewhere 2312.02145 §1. Card: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation.

Motion-image latents as an auxiliary target for a robot policy

A pi-series vision-language-action model gets a second head: a diffusion transformer that denoises latents of a predicted motion image, trained jointly with the action head 2512.18007 §3.2. The target is an optical-flow picture between the frame at time t and a frame k steps later, encoded by a frozen SDXL VAE to a 4 × 28 × 28 latent 2512.18007 §3.3 app. A.1 app. A.2. The head has six blocks, each with self-attention over the latent tokens and cross-attention to the policy's prefix app. A.2. Training has two stages. Warm-up trains only the motion head, on flow derived from DROID, and the joint stage then unfreezes everything except the VAE 2512.18007 §3.3. At inference the motion head is not run, so the action path is unchanged 2512.18007 §1 §4.3. The VAE latent is the same kind of object as in the image cards, but the denoised quantity is a motion code for a policy, not a picture for a user. Card: Robotic VLA Benefits from Joint Learning with Motion Image Diffusion.

Diffusion over grasp codes from a conditional VAE

GraspLDM trains a conditional VAE on 6-DoF grasps first. A point-cloud encoder and a grasp encoder and decoder give each grasp a latent, and the VAE is then frozen 2312.11243 §3.1 §3.3. A 1000-step DDPM is trained on those latents, with the shape latent as a condition, so the denoiser works on a short code and not on the grasp pose itself 2312.11243 §3.2 §3.3. The authors motivate this by the gap between the VAE prior and its posterior, and they report that decoding random prior samples alone gives poor grasps 2312.11243 §1 §3. Sampling 100 grasps takes 0.75 s with 100 DDIM steps and 7.39 s with 1000 DDPM steps, on one RTX 3080 Ti. The median success in simulation is 0.756 and 0.792, against 0.660 for the VAE alone 2312.11243 tab. 1. A new task needs only a task-conditioned denoiser trained after the VAE 2312.11243 §4.2. Card: GraspLDM.

Hand-pose latent with an object-conditioned denoiser and an adaptation MLP

FastGrasp encodes a hand as a 768-dimensional latent with an autoencoder, and trains a latent diffusion model over that latent. The denoiser is conditioned on 3000 points sampled from the object surface 2411.14786 §3.2 eq. 8. The latent is read as 256 tokens of three values each 2411.14786 §3.2. After the diffusion model is frozen, a small MLP is trained to add an offset to the sampled latent. The decoder then maps the sum to 61 MANO parameters 2411.14786 §3.3 eq. 9 eq. 10. The authors report that training the diffusion model directly on the physical constraints oscillated and did not converge, which is why the work is split in two 2411.14786 §3.3. On out-of-domain HO-3D, with training on GRAB, they report a penetration volume of 5.23 against 7.37 for GraspTTA, and 0.14 s per pose on an NVIDIA A40 2411.14786 tab. 2 §4.2. The timing does not state the sampler, and the paper does not state a step count 2411.14786 §3.4. Card: FastGrasp.

Next-subgoal image generation from a progress memory

TaKSIE uses a diffusion image generator to propose the next subgoal for a language-conditioned robot. The generator is built on a text-to-image latent diffusion model 2410.11013 §4.2.2. Its inputs are the current view, an embedding of the instruction and a progress embedding from a GRU over the keyframes so far 2410.11013 §3.4.1 §3.3.2. The paper accepts ControlNet or InstructPix2Pix as the conditioned form and reports both 2410.11013 §3.4.1 app. D. Training subgoals are picked from demonstrations by a progress signal, not at fixed intervals [arxiv:2410.11013 §3.3.1]. At run time a progress check moves on to the next subgoal at cosine similarity 0.96, or after 20 steps 2410.11013 §3.3.3 §4.2.3. The authors report a CALVIN average length of 3.18, against 2.80 for SuSIE, which uses fixed intervals 2410.11013 tab. 2. Card: TaKSIE. The released code differs from the paper in several settings, so read the card's Paper vs code section before building.

Surveys, not methods

Two cards are surveys. The state-of-the-art report on diffusion for visual computing explains how Stable Diffusion runs the noise removal on a compressed image, and it summarises the 2021 to 2023 literature 2310.07204 §1 §3.2. It gives almost no hyperparameters, so take training recipes from the original papers it cites. The EEG survey names latent diffusion as one of the decoder families used for EEG-to-image work, next to GANs and VAEs 2502.12048 §4.5. Its numbers come from different setups and should not be compared. Cards: State of the Art on Diffusion Models for Visual Computing, A Survey on Bridging EEG Signals and Generative AI.

Guidance scale across cards

The values differ by card and by task, and the cards do not compare them directly.

When to use which

The speed comparison with MDM is not at matched quality. MDM's FID is 7.334 at 50 DDIM steps and 0.544 at 1000 DDPM steps 2212.04048 tab. 10. Compare at a fixed quality before choosing on speed alone. The same caution applies to the video cards: their speed and quality numbers come from different models and setups 2412.07772 tab. 3 2312.06662 tab. 1. The talking-face and audio-video cards also use different metrics and test sets, so their tables cannot be ranked against each other.

Implementation sketch

Common core of the two stages and of sampling, with the motion paper's settings as comments. Variants differ in the encoder, the denoiser architecture and the condition embedding.

# Stage 1: train the autoencoder once [arxiv:2212.04048 §3.1]
# z holds n tokens of width d; the loss is reconstruction plus KL [app. H.1]
z = encode(x)
x_hat = decode(z)

# Stage 2: freeze the encoder and train the denoiser [arxiv:2212.04048 §3.2]
# Frozen encoder, so no gradient flows into it
z0 = encode(x)
# Condition embedding from a frozen text encoder or a class embedding [§3.3]
c = embed(condition)
# Empty condition for 10% of samples, so one model learns both branches [§3.3]
c = drop(c, p=0.1)
# Training uses 1000 noise levels [arxiv:2212.04048 §4.2]
t = random_step(0, 1000)
eps = random_normal_like(z0)
# Forward noising of the latent [arxiv:2212.04048 eq. 1]
z_t = add_noise(z0, eps, t)
# Noise-prediction MSE with the condition [arxiv:2212.04048 eq. 3]
loss = mse(eps, denoiser(z_t, t, c))

# Sampling with classifier-free guidance [arxiv:2212.04048 eq. 4] [app. E]
# Start from noise in latent space
z = random_normal(n, d)
# 50 DDIM steps with eta = 0 [arxiv:2212.04048 §4.2] [app. C]
for t in ddim_steps(50):
    # Conditional and empty-condition predictions
    eps_c = denoiser(z, t, c)
    eps_u = denoiser(z, t, empty)
    # Guided prediction; s = 7.5 is the paper's default [app. E]
    eps = s * eps_c + (1 - s) * eps_u
    z = ddim_step(z, eps, t)
# Decode the final latent once [arxiv:2212.04048 §3.2]
return decode(z)

Contradictions

• Claimed versus tabulated accuracy. The text says the paper's model is best on FID, R-Precision and MM Distance on both datasets [arxiv:2212.04048 §4.3]. The tables show MotionDiffuse with the best R-Precision and MM Distance on both datasets, and the paper's model second on those metrics 2212.04048 tab. 1 2212.04048 tab. 2. Only FID is clearly best for the paper's model in the tables.
• Speed. The abstract and 2212.04048 §5 describe two orders of magnitude over diffusion on raw motion. The timing table gives about 22 times at 50 DDIM steps and about 30 times at 1000 DDPM steps [arxiv:2212.04048 tab. 10].
• Layer count. The text calls the layer counts similar [arxiv:2212.04048 §5]. The ablation gives FID 0.314 for five layers and 0.473 for nine 2212.04048 tab. 5.
• Latent size in two video and motion ablations. The motion paper finds one token gives the best FID of the sizes tested 2212.04048 tab. 5. W.A.L.T calls 8 channels a sweet spot, but its table shows 16 channels with lower FVD (67.0 against 75.4) at slightly lower IS 2312.06662 tab. 3. The two ablations measure different tasks and metrics, so neither result settles a rule for latent size.
• W.A.L.T ablation baseline. The default configuration scores 60.7 FVD, but the self-conditioning row at p_sc = 0.9 scores 61.4 2312.06662 tab. 3. The paper does not say which row is the baseline.
• FID against human preference for SDXL. On 10k COCO pairs, SDXL has the worst FID of the three models compared, but slightly higher CLIP. Its human study favours it 2307.01952 §2.5. The paper's own discussion says FID and CLIP do not track these results, so the two measures disagree on the winner.
• Consistent hard-mode gains. The robot-policy paper says its method "consistently improves" success in both RoboTwin modes [arxiv:2512.18007 §4.3]. Its table shows lower hard-mode scores than pi0 on Move Can Pot, Move Playingcard and Stack Blocks Two, and a hard-mode average gain of 0.4 points 2512.18007 tab. 2.
• Best FID versus best LPIPS on Places2. PowerPaint's summary says it has the best FID on Places2 context filling 2312.03594 tab. 3. The same table shows LaMa with the lowest LPIPS, 0.1193 against 0.1263 on all samples 2312.03594 tab. 3. The two metrics disagree on the best method, so a single winner depends on which metric matters.
• TikTok video and frame metrics. MagicAnimate has the best FVD and FID-VID in its TikTok table, but its FID of 32.09 is higher than DisCo's 30.75 and than its own ablation without temporal layers, 27.54 2311.16498 tab. 1a tab. 2a. The video and frame metrics pick different winners.
• StableSR's DIV2K score. SeeSR reports StableSR at FID 36.57 on DIV2K-Val 2311.16518 tab. 2. StableSR reports FID 24.44 on DIV2K-Valid 2305.07015 tab. 1. The test sets differ: SeeSR's are 3K random 512 px crops degraded by its training pipeline, and StableSR's are 3K pairs with 128 px LR inputs. The numbers are not comparable.
• Combined driving reward. The text says the combined Drive-WM reward beats each sub-reward 2311.17918 §5.4. Table 4 shows the object reward alone gives the same average L2 of 0.80 m. The combined reward is better only on collision, 0.26% against 0.27% 2311.17918 tab. 4.
• Single-view or multi-view row. The text calls the 12.99 FID row single-view 2311.17918 §5.2, but Table 1 marks it multi-view tab. 1. The card takes the table reading as the more likely one.

Papers

• Executing your Commands via Motion Diffusion in Latent Space: text- and action-conditioned human motion generation with a transformer autoencoder and a latent denoiser.
• SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation: a diffusion model over a 3D VQ-VAE latent, conditioned on partial shapes, images and text with per-input weights.
• SINE: SINgle Image Editing with Text-to-Image Diffusion Models: fine-tuning a Stable Diffusion model on one image, with a guidance switch and patch-based training for editing.
• MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation: averaging a fixed Stable Diffusion model over crops or masked regions, with no training.
• BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing: subject tokens from a vision-language encoder, spliced into the prompt of a Stable Diffusion model.
• DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion: a frame-by-frame video generator from one photo and a pose sequence, with dual guidance.
• A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting: one inpainting model with learned task prompts for insertion, removal and context filling.
• Photorealistic Video Generation with Diffusion Models: one transformer over a shared image and video latent, with window attention.
• EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions: a portrait video generator from audio, with reference attention on a Stable Diffusion 1.5 UNet.
• VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time: diffusion over learned face motion codes, decoded to frames, for real-time talking faces.
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models: a causal, chunked video latent generator distilled to four steps, with a key-value cache.
• JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization: two diffusion branches for video and audio on frozen VAEs, with shared spatio-temporal priors.
• State of the Art on Diffusion Models for Visual Computing: a 2023 survey of diffusion models, including Stable Diffusion's latent space.
• A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond: a survey of EEG-to-media generation, where latent diffusion is one decoder family.
• Scalable Diffusion Models with Transformers: a transformer denoiser over latent patches with adaLN-Zero conditioning, scaled in depth, width and tokens.
• T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models: small control adapters on a frozen Stable Diffusion UNet, composed by weighted sum.
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis: a larger two-encoder latent UNet with size and crop conditioning and an optional refiner.
• Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models: a safety term added to classifier-free guidance during sampling, with no retraining.
• Exploiting Diffusion Prior for Real-World Image Super-Resolution: a frozen Stable Diffusion 2.1 steered by a trained time-aware encoder, with a fidelity dial and patch-blended sampling.
• PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing: six-view body diffusion on a Stable Diffusion unclip model, then SMPL-X-initialised remeshing.
• The Stable Signature: Rooting Watermarks in Latent Diffusion Models: a decoder fine-tune that embeds a fixed 48-bit key in every output, read by a separate extractor.
• Emergent Correspondence from Image Diffusion: point matching from one denoiser pass's features, with no training.
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation: a denoiser fine-tuned on synthetic depth, with a median ensemble of ten runs.
• Robotic VLA Benefits from Joint Learning with Motion Image Diffusion: a motion-image latent diffusion head on a robot policy, trained jointly and unused at inference.
• GraspLDM: Generative 6-DoF Grasp Synthesis using Latent Diffusion Models: a DDPM over the latents of a conditional VAE for grasp poses, with a small denoiser per task.
• FastGrasp: Efficient Grasp Synthesis with Diffusion: latent diffusion over a hand-pose autoencoder, then an adaptation MLP trained for contact and penetration.
• Incorporating Task Progress Knowledge for Subgoal Generation in Robotic Manipulation through Image Edits: a diffusion image generator for the next subgoal, with a progress memory and a progress check.
• MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model: a Stable Diffusion UNet with temporal layers, a reference-attention copy and DensePose control, run over overlapping windows.
• SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution: Stable Diffusion x4 restoration with tag and tagger-embedding prompts, sampled from a noised low-resolution latent.
• Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving: a latent video diffusion world model for six driving cameras, with layout and action tokens and tree planning.