DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion

In plain words

Fashion photos show a garment on a still person, but they do not show how the fabric moves. DreamPose takes one photo of a person and a sequence of body poses, and produces a short video of that same person wearing the same clothes while moving through the poses.

It starts from Stable Diffusion, a model that generates images by removing noise step by step. The model is first taught on many fashion videos to draw frames that follow a pose and look like the photo it is given. Then it is briefly retrained on the one or more photos the user supplies, so the face, the fabric pattern and the print stay recognisable. Each video frame is generated on its own, and the photo and the pose are both steered in the sampling step, with separate strength settings for each.

The point is that a video of a garment can be made from one still image, without filming the garment. The method is slow to set up for each new person, because of the per-subject retraining.

TL;DR

• Input: one or more photos of a person plus a DensePose pose sequence; output: a video of that person in the poses §1 §4.
• Fine-tunes a pretrained Stable Diffusion UNet (conditioned on CLIP and VAE image tokens and five consecutive pose maps), then does a short per-subject fine-tune §4.2–§4.3.
• Author-reported state of the art on UBC Fashion against MRAA, TPSMM and PIDM tab. 1; the code is MIT-licensed.
• Costly per subject (about 10 min UNet and 20 min VAE-decoder tuning, plus 18 s per rendered frame) §6.

Problem

Fashion videos show how a garment drapes and flows, but they are scarce, while still photos are common §1. Earlier video diffusion models trained from text struggle with realism and temporal consistency, and text cannot pin down the exact identity or garment of a person §1 §2.4 §2.5. Pose-guided GAN methods struggle with large pose changes, occlusion and garment style §2.3. The authors report that PIDM, a diffusion pose-transfer method trained from scratch, does not preserve identity or clothing across frames §5.1.2.

Method

Overview. Given an input image x_0 and poses \{p_1, \ldots, p_N\}, each output frame x'_i is sampled independently from a conditioned latent diffusion model and decoded by the VAE §4.1. Frames are not generated jointly, so temporal smoothness comes from the conditioning, not from a video model.

Architecture §4.2.

1. Image conditioning §4.2.1. The CLIP text encoder of Stable Diffusion is replaced by an adapter that receives CLIP image tokens and VAE latent tokens of the input frame. The adapter output c_I = \mathcal{A}(c_{\text{CLIP}}, c_{\text{VAE}}) eq. 3 enters the usual cross-attention layers. The paper argues that concatenating the image to the noise, as InstructPix2Pix does, fits only spatially aligned conditions, and that CLIP alone loses fine detail §4.2.1.
2. Pose conditioning §4.2.2. Pose is spatially aligned with the output, so it is concatenated to the noisy latent. The pose input is five consecutive DensePose frames, \{p_{i-2}, \ldots, p_{i+2}\}, which the authors say reduce frame-to-frame jitter §4.2.2. The UNet input layer gets 10 extra channels (2 per pose), zero-initialised, while the original noise channels keep their pretrained weights §4.2.2.

Training objective §3. The denoising loss is the mean squared error on the predicted noise, conditioned on c eq. 1. The text-conditioned Stable Diffusion model uses the same objective §3.

Two-stage fine-tuning §4.3.

1. Fine-tune the UNet and the adapter on the whole training set, starting from a pretrained Stable Diffusion checkpoint. The CLIP image encoder comes from a separate pretrained checkpoint §4.3. New layers start so that they do not change the initial output §4.3.
2. Subject-specific fine-tuning on one or more photos of the target person: first the UNet and adapter, then the VAE decoder. The decoder step is reported as necessary for sharper, more photorealistic detail §4.3 fig. 5. Augmentation (random crops) is applied at each step, because training on a single frame and pose pair produces texture-sticking artefacts §4.3.

Sampling with dual classifier-free guidance §4.4. The guidance weight for the image, s_I, and the weight for the pose, s_p, are separated eq. 4. Three predictions are needed per step: no conditions, image only, and image plus pose. The paper reports that a large s_I keeps the appearance and a large s_p keeps the pose, and that separating them avoids overfitting to the one pose seen in subject fine-tuning §4.4 fig. 7.

# Training step: denoise frame_j, conditioned on frame_i and a pose window
# Inputs: frame_i, frame_j as (3, H, W) in [0, 1]; poses as (10, H, W)
# Reconstructed from the paper text and the code; not a numbered algorithm

def training_step(frame_i, frame_j, poses):
    # Drop conditions: each of pose, image, or both, with dropout_rate / 3
    # [code: train.py#L210-L218]
    frame_i, poses = apply_condition_dropout(frame_i, poses)
    # Encode the target frame to a latent, scaled by 0.18215
    # [code: train.py#L341-L342]
    latents = vae_encode(frame_j) * 0.18215
    # Sample Gaussian noise and a random timestep, then noise the latent
    # [eq. 1] [code: train.py#L345-L354]
    noise = random_normal(latents.shape)
    timesteps = random_int(0, num_train_timesteps)
    noisy_latents = add_noise(latents, noise, timesteps)
    # Append the poses, resized to latent size, as extra input channels
    # [§4.2.2] [code: train.py#L358]
    noisy_latents = concat(noisy_latents, resize(poses, latent_size))
    # Build the image context from CLIP tokens and VAE latent tokens
    # [§4.2.1] [code: train.py#L361-L370]
    clip_tokens = clip_vision(frame_i)
    vae_tokens = vae_encode(frame_i) * 0.18215
    context = adapter(clip_tokens, vae_tokens)
    # Predict the noise and take the mean squared error  [eq. 1]
    predicted = unet(noisy_latents, timesteps, context)
    loss = mse(predicted, noise)
    return loss

# Adapter: fuse 50 CLIP tokens with 4 VAE tokens into 50 context tokens
# Inputs: clip_tokens (B, 50, 768); vae_latent (B, 4, 80, 64)
# [eq. 3] [code: models/unet_dual_encoder.py#L44-L71]

def adapter(clip_tokens, vae_latent):
    # Halve the latent grid with 2x2 max pooling
    # [code: models/unet_dual_encoder.py#L57]
    pooled = max_pool_2x2(vae_latent)        # (B, 4, 40, 32)
    # Flatten each channel to one 1280-long vector
    flat = reshape(pooled, (B, 4, 1280))     # (B, 4, 1280)
    # Project to the CLIP width, 1280 -> 768  [code: ...#L44]
    vae_tokens = linear_1280_to_768(flat)    # (B, 4, 768)
    # Stack CLIP and VAE tokens along the token axis
    # [code: ...#L63]
    tokens = concat(clip_tokens, vae_tokens, axis=1)  # (B, 54, 768)
    # Mix 54 tokens into 50 along the token axis  [code: ...#L46-L50]
    return linear_54_to_50(tokens)           # (B, 50, 768)

# Sample one video frame with dual classifier-free guidance  [eq. 4]
# Inputs: key_frame (image), pose_window (10, H, W) for frames i-2..i+2
# Output: one decoded frame

def sample_frame(key_frame, pose_window, s_image, s_pose):
    # Context for the image and its null version (zero image)
    # [code: pipelines/dual_encoder_pipeline.py#L218-L261]
    context, null_context = image_contexts(key_frame)
    # Start from Gaussian noise in latent space  [§4.1]
    latent = random_normal(latent_shape)
    for t in pndm_timesteps(num_steps=100):   # [supp. §1] [code: test.py#L30]
        # Three predictions: no conditions, image only, image and pose
        # [code: pipelines/dual_encoder_pipeline.py#L533-L553]
        # The code batches the three calls into one UNet pass
        eps_none = unet(latent, t, null_context, zeros_like(pose_window))
        eps_image = unet(latent, t, context, zeros_like(pose_window))
        eps_full = unet(latent, t, context, pose_window)
        # Weight the image term and the pose term separately  [eq. 4]
        eps = eps_none + s_image * (eps_image - eps_none)
        eps = eps + s_pose * (eps_full - eps_image)
        latent = scheduler_step(eps, t, latent)
    return vae_decode(latent)

Hyperparameters

Results

Author-reported. Setup: UBC Fashion test set (100 videos, evaluated at 256 px), MRAA and TPSMM with their published test scripts in "AVD" mode, and PIDM, all trained from scratch by the authors on UBC Fashion §5.1 §5.1.1.

Quantitative comparison tab. 1 (lower is better except SSIM):

Ablations tab. 2, 100 frames from each of the 100 test videos:

User studies [supp. §2] tab. 3 fig. 10: 50 Amazon Mechanical Turk workers, 50 test input images. Pairwise preference: DreamPose over MRAA 1637 of 2500 (65%), over TPSMM 1417 of 2500 (57%). On a 0 to 5 realism scale, 85% of workers gave DreamPose a 3 or higher.

Further results: multi-image subject fine-tuning with 1, 3, 5 and 7 photos improves pose, face and colour fidelity [supp. §4] fig. 12; DeepFashion retail-image results after subject fine-tuning [supp. §5] fig. 13; pose transfer compared with DynaST and PIDM [supp. §6] fig. 14.

Paper vs code

Official code: https://github.com/johannakarras/DreamPose at commit 5bf30b7df70cf6f2e0bb25556c6ff2cbf0f2b1bf, MIT License (copyright Johanna Karras, 2023) LICENSE. The project page links it § project page. The owner is the first author.

• Pose window. Paper: five consecutive poses p_{i-2} to p_{i+2} §4.2.2. Training code: the loop builds five poses, but every one comes from the same file, frame_j's DensePose map dreampose_dataset.py. At inference the code uses the real window f-2 to f+2, falling back to the current frame's file when a neighbour is missing test.py. So training and inference see different pose inputs.
• Frame size. Paper supplement: 512 × 512 training [supp. §1]. Code: 640 × 512 (H × W), hard-coded, so the --resolution flag in the README has no effect on the dataset dreampose_dataset.py. Test-time size is also 512 × 640 in PIL order test.py. Paper evaluation: 256 px §5.1.1. The text does not reconcile these.
• Training schedule. Paper: 5 epochs, effective batch 16, on 2 A100s [supp. §1]. README: 300 epochs on 4 processes at batch 2 with 4 accumulation steps, which is an effective batch of 32, while the README also says 2 A100s README.md.
• Epochs versus steps. Paper gives step counts (500 UNet steps, 1500 VAE steps). The README gives epoch counts for the same runs README.md README.md. These match only if the dataset has one photo, which I did not verify.
• Guidance weights. The paper does not state s_I and s_p for the reported results §4.4. Code defaults are 0.5 and 0.5 test.py; the README example uses 8 and 3 README.md.
• Guidance formula. Matches eq. 4 when s_I = s1 and s_p = s2. Chunk order and the zero pose and zero image in the unconditional branches match the paper dual_encoder_pipeline.py.
• Adapter zero-initialisation. Paper: VAE weights start at zero §4.2.1. Code: vae2clip is default-initialised, but linear1 starts as an identity over the first 50 of 54 tokens unet_dual_encoder.py, so the VAE columns start at zero. The paper's intent holds, but through linear1, not vae2clip.
• Adapter size arguments. Embedding_Adapter(input_nc=1280, output_nc=1280) is called from train and fine-tune scripts train.py, but the class ignores these and fixes 1280 → 768 and 54 → 50 unet_dual_encoder.py. Changing the arguments changes nothing.
• VAE fine-tune loss. Paper: not stated. Code: pixel MSE between the decoder output and the target frame, clamped to [-1, 1] finetune-vae.py. The perceptual loss (LPIPS) is commented out finetune-vae.py.
• Augmentation. Paper: random cropping §4.3. Code: crop plus rotation, on 70% of samples dreampose_dataset.py.
• Learning-rate scaling. --scale_lr defaults to off parse_args.py, so the lr values in the README are used as given. If enabled, lr is multiplied by accumulation × batch × processes finetune-unet.py.
• Ablation text versus table. Text: the full model outperforms ablations on all four metrics §5.2. Table 2: the one-pose variant ties the full model on L1 and LPIPS and differs by 0.001 on SSIM and VGG. The table caption says the one-pose variant does not match the full method's motion smoothness, which the metrics do not show tab. 2. The smoothing variant's method is not described, and its L1 of 0.767 is not explained tab. 2.
• Dataset count. Section 2.3 says 323 videos §2.3; section 4.5 says 339 training videos §4.5.
• Frames per test video. Section 5.1.1 says 50 frames per video §5.1.1; section 5.2 says 100 §5.2.

Pitfalls

• Pose input in training is one pose repeated five times. The code does not implement the five-frame window that the method section describes (see Paper vs code). Implementing the paper's window and training from the released code would give different models. The code is the only source for what the reported checkpoints saw, but this is not verified against the authors' checkpoints.
• Pixel range may differ between training and inference. Training frames are resized and converted with ToTensor and not rescaled to [-1, 1] dreampose_dataset.py train_vae_dataset.py and are passed to vae.encode as they are train.py. At inference, _encode_image passes the raw image to the VAE as 0–255 values dual_encoder_pipeline.py, and the normalising preprocess runs later dual_encoder_pipeline.py. I read this from the code without running it. Check it before porting, because a 255× scale change would break the VAE branch.
• Null conditions are not learned tokens. A dropped image is an all-zero tensor, and a dropped pose is all zeros train.py. Both still go through CLIP and the VAE train.py, so the "null" context is whatever those networks produce for black input. The same holds at inference dual_encoder_pipeline.py.
• The guidance scale argument does nothing. guidance_scale=7.5 only decides whether the three-pass branch runs. The update uses only s1 and s2 dual_encoder_pipeline.py dual_encoder_pipeline.py.
• Three predictions per step. Dual guidance needs three noise predictions per denoising step, which the code batches into one UNet pass dual_encoder_pipeline.py. Over 100 steps, a frame costs 300 predictions. The paper gives 18 s per frame without naming hardware §6.
• Pose maps are used raw. DensePose UV values are not normalised dreampose_dataset.py. Training resizes them with F.interpolate at its default mode (nearest) train.py; the dataset resizes them with bilinear to 640 × 512 dreampose_dataset.py. The paper does not state either choice.
• Single-frame subject fine-tuning needs augmentation. The paper reports texture-sticking when training on one frame and pose pair, and says random crops prevent it §4.3. It reports the VAE-decoder step as necessary for sharper detail §4.3 fig. 5.
• Frame independence. Each frame is sampled alone, so flicker on large or complex prints is a known limitation §6. The code's stochastic-sampling option is off by default dual_encoder_pipeline.py.
• Licences of inputs and weights. UBC Fashion and DeepFashion are third-party datasets whose terms are not stated here. The base checkpoint and the subject checkpoints are hosted on Google Drive according to the README, and the README gives no licence for them README.md README.md. A commercial port needs those checked; this card contains no data or weights.

Lineage

• Builds on Stable Diffusion for the base model §3, on DreamBooth for subject-specific fine-tuning §2.5 §4.3, and on InstructPix2Pix for dual guidance §4.4.
• Compared against MRAA §2.2 §5.1, TPSMM §2.2 §5.1, and PIDM, a concurrent pose-transfer diffusion method §2.3 §5.1. Also compared with DynaST for pose transfer [supp. §6].
• Related: Tune-A-Video, a text-and-image video diffusion method that DreamPose says has flicker and structural inconsistency §2.4.
• Superseded by: not known from this wiki.