SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

In plain words

SDXL is a model that draws an image from a text prompt. It is a bigger and more careful version of the Stable Diffusion family. It works by starting from random noise in a compressed "latent" picture space and removing the noise step by step, guided by the prompt. Most of the work happens on this small latent picture, and a separate decoder turns the result into a full image.

Three changes make the difference. The main network is much larger, and it reads the prompt with two text encoders instead of one. Training images are no longer thrown away or cut badly when they are too small or oddly shaped: the model is told the original size and the crop position as extra inputs, so it can learn that these affect the look of the result, and at generation time you set them yourself. The model is also trained on several aspect ratios, so landscape and portrait images are not forced into a square.

An optional second model can polish the result. It takes the first output, adds a little noise and denoises it again with the same prompt, which mostly improves fine detail such as backgrounds and faces. The authors report that people preferred the outputs in side-by-side comparisons, while the usual automatic scores (FID) did not agree, and they say so openly.

TL;DR

• Latent diffusion model for text-to-image synthesis; UNet about three times larger than SD 1.x/2.x, with 2.6B parameters §1 §2.1.
• Conditioning on original size and crop coordinates, and on target size for multi-aspect training, replaces discarding or cropping training data §2.2 §2.3.
• Optional refinement LDM applies SDEdit-style noising and denoising to the base output §2.5. The authors report human preference, not automatic metrics.
• Code (MIT) is official and public; model weights have separate licenses (see Pitfalls).

Problem

Latent diffusion models of the Stable Diffusion family need a minimum training image size because of their two-stage design. Discarding every image below the pretraining resolution would remove 39% of the authors' pretraining data at 256² pixels, and upscaling small images introduces artifacts into the outputs §2.2 fig. size distribution, §2.2.

Random crops during training also leak into samples: earlier models often produce cut-off objects §2.2 fig. comparison with previous versions, §2.2. Fixed square outputs at 512² or 1024² are a poor match for landscape and portrait images §2.3. Classical FID and CLIP scores were found to disagree with human judgment for these foundation models [app. On FID Assessment of Generative Text-Image Foundation Models].

Method

The base model keeps the latent diffusion design and changes four things §2: the UNet architecture and scale (§2.1), micro-conditioning on size and crop (§2.2), multi-aspect training (§2.3), and a retrained autoencoder (§2.4). A refinement stage is added on top (§2.5).

Architecture and scale §2.1. Compared with SD 1.x/2.x, the UNet moves most transformer computation to lower-resolution levels. The highest level has no transformer, the lowest level (8× downsampling) is removed, and the model uses 2 and 10 transformer blocks at the remaining levels §2.1 [tab. model comparison, §2.1]. Text is encoded by OpenCLIP ViT-bigG and CLIP ViT-L; the penultimate outputs of the two are concatenated along the channel axis. Cross-attention uses the concatenated sequence, and the pooled OpenCLIP embedding is also added to the time conditioning §2.1. The text encoders together have 817M parameters §2.1.

Size conditioning §2.2. Before any rescaling, the model receives the original height and width, c_size = (h_original, w_original). Each component is embedded with Fourier features, and the embeddings are concatenated and added to the timestep embedding §2.2.

Crop conditioning §2.2. During data loading, the top-left crop offsets (c_top, c_left) are sampled uniformly and embedded the same way. If size and crop are both used, the embeddings are concatenated along the channel dimension before the addition §2.2 alg. 1. At inference the authors set (c_top, c_left) = (0, 0) to get object-centred samples §2.2.

Multi-aspect training §2.3. Images are grouped into buckets of different aspect ratios, keeping the pixel count as close to 1024² as possible, with height and width in multiples of 64. Every batch comes from one bucket, and the bucket alternates between steps. The target size (h_tgt, w_tgt) is a further Fourier- embedded condition. The full list of buckets is in the appendix §2.3 [app. Multi-Aspect Training Hyperparameters].

Autoencoder §2.4. The autoencoder has the same architecture as the original SD autoencoder. It is retrained at batch size 256 instead of 9 and its weights are tracked with an exponential moving average. The authors use it for all their experiments §2.4 tab. autoencoder comparison, §2.4.

Training schedule §2.5. The final model uses the autoencoder from §2.4 and a discrete-time diffusion schedule with 1000 steps §2.5. Training proceeds in three stages: pretraining at 256² for 600,000 steps with batch size 2048 and size and crop conditioning; 200,000 more steps at 512²; and multi-aspect fine- tuning with offset noise of level 0.05 at roughly 1024² pixel area §2.5.

Refinement §2.5. A second LDM in the same latent space is trained on high-quality, high-resolution data. It is specialised on the first 200 discrete noise levels. At inference the base model's latents are noised and denoised by the refiner with the same prompt, following SDEdit. The step is optional and the authors say it mainly helps backgrounds and human faces §2.5.

Diffusion background app. Diffusion Models. The appendix follows the continuous-time framework of score-based models and the EDM presentation. The denoiser D is trained with a weighted denoising loss over 1000 discrete noise levels, with weight σ⁻² app. Diffusion Models, denoising objective. Sampling can use the probability-flow ODE or the SDE [app. Diffusion Models, probability flow and SDE equations].

Classifier-free guidance app. Diffusion Models, guidance equation. The guided prediction mixes the conditional and unconditional denoisers with guidance strength w ≥ 0. The unconditional branch is trained in the same network by replacing the condition with a null embedding, for example 10% of the time [app. Diffusion Models].

The pipeline below is the training-time conditioning procedure of alg. 1, written from the algorithm listing and §2.2:

# Sample one training example with size and crop conditioning  [alg. 1]
# Input:  image (H, W, 3) before any rescaling; target size (h_tgt, w_tgt)
# Output: image (h_tgt, w_tgt, 3), c_size (2,), c_crop (2,)

def sample_training_input(image, target_size):
    # Keep the original size, before any rescaling  [§2.2]
    c_size = (height(image), width(image))
    # Resize the image so that it fits the target size  [alg. 1]
    image = resize(image, target_size)
    # Crop the longer axis at a uniformly sampled offset  [alg. 1]
    if original_height <= original_width:
        c_left = uniform_int(0, width(image) - target_width)
        c_top = 0
    else:
        c_top = uniform_int(0, height(image) - target_height)
        c_left = 0
    c_crop = (c_top, c_left)
    # Cut the image to the target size at that offset  [alg. 1]
    image = crop(image, target_size, c_crop)
    return image, c_size, c_crop

Conditioning embedding and guided denoising at inference, from §2.2, §2.3 and app. Diffusion Models:

# Embed each scalar condition with a Fourier feature  [§2.2]
# c_size, c_crop, c_target: (B, 2)  ->  each embedding (B, 512)
def embed_conditions(c_size, c_crop, c_target):
    # One embedding per scalar, then concatenate  [§2.2] [§2.3]
    size_emb = fourier_features(c_size)
    crop_emb = fourier_features(c_crop)
    target_emb = fourier_features(c_target)
    return concat([size_emb, crop_emb, target_emb], axis=1)

# Guided denoising call with strength w  [app. Diffusion Models]
def guided_denoise(denoiser, latents, sigma, cond, uncond, w):
    # Predict with and without the condition  [app. Diffusion Models]
    pred_cond = denoiser(latents, sigma, cond)
    pred_uncond = denoiser(latents, sigma, uncond)
    # Combine: (1 + w) * cond - w * uncond  [app. Diffusion Models]
    return (1.0 + w) * pred_cond - w * pred_uncond

Inference with the optional refiner, from §2.5:

# Two-stage generation  [§2.5]
def generate(prompt, target_size):
    # Base model at the target size, crop offsets set to zero  [§2.2]
    latents = base_model.sample(prompt, target_size, crop=(0, 0))
    # Optional refinement over the first 200 of 1000 levels  [§2.5]
    latents = refiner.sample_from(latents, prompt, start_level=200)
    return decode(latents)

Hyperparameters

Values marked "not stated" are absent from the paper. Code values come from the official repository at code_commit (MIT). The released repository has inference configs but no SDXL training configs.

Results

Author-reported. Setup: the authors' internal pretraining data and PartiPrompts and COCO evaluations, on the models and baselines named below §2.4 §2.5 app. Comparison to Midjourney v5.1.

Values for the FID-vs-CLIP curves appear only in a figure; they were not extracted. The Midjourney per-category results are in figures only.

Paper vs code

Repository: https://github.com/Stability-AI/generative-models at e8cd657656fa5d61688191730d0e03242bf4ed44, MIT license for the code (the authors' lab). The paper links it in the abstract and the title page. Model weights are not in this repository's license; see Pitfalls.

• Transformer depth. The paper's model table lists [0, 2, 10] §2.1. The base config lists transformer_depth: [1, 2, 10] [code: configs/inference/sd_xl_base.yaml#L31]. The first entry is unused because attention_resolutions: [4, 2] sd_xl_base.yaml excludes the highest level, and the UNet starts downsample count at 1 openaimodel.py and applies attention only where the downsample count is in that list openaimodel.py. The effective depth is therefore [0, 2, 10], as the paper says.
• Text-encoder layer. The paper says the penultimate output of CLIP ViT-L is used §2.1. The base config takes hidden layer 11 of CLIP ViT-L sd_xl_base.yaml. The equivalence with "penultimate" depends on the encoder's indexing, which I did not check.
• Refiner architecture. The paper describes a refiner trained in the same latent space, specialised on the first 200 noise levels §2.5, with no architecture details. The released refiner config is a different UNet with one text encoder and an aesthetic-score input [code: configs/inference/sd_xl_refiner.yaml#L20-L32] [code: configs/inference/sd_xl_refiner.yaml#L63-L66]. The paper's text does not mention an aesthetic input.
• Refinement strength. The paper uses 200 of 1000 noise levels §2.5. The demo's default strength is 0.15 sampling.py. The demo's refiner call also uses fixed aesthetic values sampling.py.
• Guidance formula. The paper writes D^w = (1 + w)·D(c) − w·D(∅) [app. Diffusion Models, guidance equation]. The code computes x_u + s·(x_c − x_u) guiders.py. This is the same formula with s = 1 + w, so a code scale of 5.0 corresponds to w = 4.0.
• Sampler. The paper's comparisons use 50 DDIM steps with cfg 8.0 §2.2 (figure captions). The demo defaults are an EDM-style Euler sampler with 50 steps and cfg 5.0 streamlit_helpers.py streamlit_helpers.py. The paper does not state the sampler used for its headline user study.
• Output parameterisation. The appendix describes a denoiser D that predicts clean data app. Diffusion Models. The base config uses EpsScaling, which sets c_skip = 1 and c_out = −σ, i.e. epsilon prediction sd_xl_base.yaml denoiser_scaling.py. The paper does not state which parameterisation SDXL uses.
• Offset noise. The paper uses offset noise of 0.05 for multi-aspect fine-tuning §2.5. The loss and the img2img helper both default to 0.0 loss.py helpers.py. The training configs for SDXL are not in the repository, so the 0.05 value cannot be checked in code.
• Crop offsets. The paper samples crops during training and sets them to (0, 0) at inference §2.2. The demo's refiner call also sets both values to 0 sampling.py. The training-time crop sampler is not in the repository; the algorithm listing alg. 1 is the only description.
• Conditioning width. The paper concatenates pooled text and three Fourier embeddings fig. concatenation listing, app. Pseudo-code. The base config's adm_in_channels is 2816 sd_xl_base.yaml. Three size, crop and target-size keys with 256 values per scalar account for 1536 of that width, which leaves 1280 for the pooled text embedding (derived, not stated in the paper).
• Autoencoder. The paper says the new autoencoder is trained from scratch tab. autoencoder comparison, §2.4. The released config reuses the SD autoencoder topology with a scale factor of 0.13025 [code: configs/inference/sd_xl_base.yaml#L4] sd_xl_base.yaml. The weights themselves are not in the code.

No discrepancy was found for the discrete-noise count of 1000 or for the text-encoder model names.

Pitfalls

• Training recipe is missing. Learning rate, optimizer, warm-up, batch size for the final stage, EMA settings for the UNet, data filters, and the composition of the internal dataset are not stated. The released repository has only example training configs (e.g. configs/example_training/), not the SDXL recipe. The dataset is not public, so the training result cannot be reproduced from this card alone.
• Crop sampling is ambiguous. Algorithm 1 samples the crop along the longer axis of the resized image, with the branch chosen by the original height and width alg. 1. The bounds of the discrete uniform (inclusive or exclusive) and the rounding of the resize are not stated. Check against your own data loader.
• Size conditioning must use the original size, not the resized size. The paper stresses that original height and width are fed before any rescaling §2.2. The inference helper builds the same tuple from orig_height and orig_width in the value dictionary helpers.py. If you pass the target size instead, the model sees a different condition than it was trained on.
• Bucketing interacts with crop conditioning. Crop conditioning works only within a bucket's boundaries (usually 64 pixels), and the authors kept the control parameter anyway for simplicity §2.3.
• Two text encoders increase memory. The text encoders total 817M parameters §2.1 and the refiner is a second full model. The authors state that loading two large models hurts accessibility and sampling speed §3.
• The refiner is optional and only partly specified. It uses the same autoencoder and prompt fig. user-study and pipeline figure, §1 and §2.5. The paper gives no separate training recipe. Demo and paper strengths differ (0.15 vs 200/1000) sampling.py.
• FID and CLIP do not track the human results. SDXL has the worst FID of the three compared models but the human study favours it app. On FID Assessment. If you select models on FID or CLIP, this card's own results argue against it.
• Author-run comparisons. The Midjourney comparison used AWS GroundTruth workers and a single Midjourney version (v5.1); the paper notes v5.2 was available app. Comparison to Midjourney v5.1. The win rates are from the authors' own study.
• Known failure modes. The authors report weak hands, concept bleeding between objects, and unreliable long text rendering app. Limitations. They also report possible social and racial bias from the data app. Limitations.
• Weight licenses are separate. The code is MIT. The repository's model_licenses/ directory contains separate license texts for the SDXL weights (LICENSE-SDXL1.0 and related files). This card's license field covers the paper only; check the weight license before any commercial use. I did not read the weight licenses.
• Derived, not stated: the 1280-wide pooled text embedding, and the inference from transformer_depth that the first entry is inert. Both are noted in Paper vs code.

Lineage

• Builds on: latent diffusion models (Rombach et al., 2021), cited as the base design in §2 and §2.5. The arXiv id was not checked in this card.
• Builds on: SDEdit for the refinement step §2.5; offset noise §2.5; the EDM formulation app. Diffusion Models; classifier-free guidance [app. Diffusion Models].
• Compared against: Stable Diffusion 1.5 and 2.1 §2.5 [app. Comparison between SD 1.5 vs. SD 2.1 vs. SDXL]; Midjourney v5.1 and v5.2 [app. Comparison to Midjourney v5.1]; DeepFloyd IF, DALL·E 2 and Bing Image Creator in a qualitative comparison (figure) §2.2 app. Comparison to the State of the Art.
• Superseded by: not stated in content already in this wiki.