T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

In plain words

Text-to-image models can draw a lot, but a text prompt is a loose way to say where things go or what shape they take. This paper adds a small helper network, called an adapter, to a large pretrained image generator. The big generator stays frozen. The adapter reads a simple guide image, such as a sketch, a colour blob layout, a pose skeleton or a depth map, and turns it into hints that steer the generator's internal features.

The adapter is cheap to train because it only has to learn how to line up the guide with what the generator already knows. Several adapters can be trained for different guides and mixed together with a weight for each. The authors report that the generated images follow the guide more closely and look better than the plain generator with text alone.

TL;DR

T2I-Adapter is a set of small convolutional adapters trained on top of a frozen Stable Diffusion model (paper version: SD 1.4). Each adapter reads one control map and adds four multi-scale features to the UNet encoder. Adapters for colour, sketch, segmentation, keypose and depth are described; multiple adapters are composed by a weighted sum with no extra training. Author-reported COCO validation FID is 16.78 (text plus segmentation) and 17.36 (text plus sketch), against 24.68 for Stable Diffusion with text only tab. 1. Check the Pitfalls before implementing: the cubic timestep formula, the sketch threshold, the learning rate and the composition weights differ between paper and code.

Problem

Text prompts alone give unstable structure in complex prompts. The paper's examples are unusual compositions the base model renders inconsistently, and it attributes this to the text not carrying enough spatial guidance rather than to missing generation ability §1 fig. 3. Earlier controllable methods either retrain the whole generator, which is costly, or steer through cross-attention manipulation or gradients, which the authors describe as impractical for complex scenes §2.2. The goal is a low-cost, plug-in network that aligns the control signal with the generator's internal knowledge without changing the frozen model §1.

Method

The base model is Stable Diffusion, a latent diffusion model with an autoencoder and a text-conditioned UNet denoiser §3.1 eq. 1 eq. 2. Adapters are trained with the base model frozen §3.4. At generation time the adapter's features are added to the UNet encoder features at each scale §3.3 eq. 4. The four stages of the method are:

1. Condition encoding. The control map (sketch, segmentation, keypose, depth, or a spatial colour map) is passed through the adapter. The adapter is a convolution followed by residual blocks, organised in four scales, with a downsampling step between scales §3.3 fig. 2. It returns one feature map per scale, with the same channel dimension as the UNet encoder at that scale eq. 3.
2. Feature injection. Each adapter feature is added to the matching encoder feature of the UNet. The paper injects into all four encoder scales; injecting into the decoder as well was tested and gave worse FID §3.3 §4.4.1 tab. 2.
3. Multi-adapter composition. Several adapters can be combined as a weighted sum of their features, with no retraining. The weights are set by hand §3.3 eq. 5 §5.
4. Adapter training with non-uniform timesteps. Only the adapter is optimised with a standard noise-prediction loss eq. 6. Training timesteps are drawn from a cubic distribution that favours early (high-noise) steps, because the authors found that guidance added in late steps has little effect §3.4 fig. 4 fig. 5.

Pseudocode for the adapter forward pass. The paper gives the layout in words and in its architecture figure; the code shows the same layout for its released model adapters.py.

# T2I-Adapter forward pass for one condition map  [§3.3] [fig. 2]
# Input:  condition (1 or 3 channels, 512 x 512 in the paper)
# Output: four feature maps, one per UNet encoder scale  [eq. 3]

def adapter_forward(condition):
    # Pixel-unshuffle down to 64 x 64; 512 -> 64 implies factor 8  [§3.3]
    features = pixel_unshuffle(condition, factor=8)  # (C*64, 64, 64)
    # One convolution to the first scale's channel count  [§3.3]
    features = conv3x3(features)
    outputs = []
    for scale in range(4):  # four scales  [§3.3]
        for block in range(2):  # two residual blocks per scale  [§3.3]
            # First block of scales 2 to 4 also halves the resolution  [§3.3]
            is_first = block == 0
            features = residual_block(
                features, downsample=(scale > 0 and is_first))
        outputs.append(features)
    return outputs

Pseudocode for injection and for the adapter-only training step. Both use the cubic timestep rule from §3.4.

# Cubic timestep sampling for one training sample  [§3.4]
# Input:  num_steps (T, the number of diffusion steps; not stated in paper)
# Output: integer timestep t in [0, T]

def sample_cubic_timestep(num_steps):
    # Draw u uniformly from [0, 1)  [§3.4]
    u = uniform(0.0, 1.0)
    # t = (1 - u^3) * T, so most mass sits near high-noise steps  [§3.4]
    return int((1.0 - u ** 3) * num_steps)

# One adapter training step; the base model stays frozen  [§3.4] [eq. 6]
# Input:  images X0, condition maps C, captions y
# Output: scalar loss; only adapter parameters receive gradients

def adapter_training_step(images, conditions, captions, adapter, sd):
    # Encode the image to latent Z0 with the frozen autoencoder  [§3.4]
    latents = sd.autoencoder.encode(images)  # (B, 4, 64, 64)
    # One timestep per sample from the cubic rule  [§3.4]
    timesteps = [sample_cubic_timestep(T) for _ in images]
    # Gaussian noise epsilon and the noised latent Z_t  [§3.1] [§3.4]
    noise = sample_gaussian_noise(latents.shape)
    noisy_latents = add_noise(latents, noise, timesteps)
    # Text states tau(y) from the frozen text encoder  [§3.1]
    text_states = sd.text_encoder(captions)
    # Adapter runs on every training sample, unlike at inference  [§3.4]
    guidance = adapter_forward(conditions)  # list of four maps  [eq. 3]
    predicted = sd.unet(noisy_latents, timesteps, text_states, guidance)
    # Noise-prediction loss on the adapter only  [eq. 6]
    loss = mean_squared_error(predicted, noise)
    return loss

# Generation with one or more adapters  [§3.2] [§3.3] [eq. 5]
# Input:  prompt, list of condition maps, list of weights omega
# Output: image

def generate(prompt, conditions, weights, steps):
    text_states = sd.text_encoder(prompt)
    # Run each adapter once; its features are reused at every step  [§1]
    guidance = compose_adapters(conditions, weights)  # F_c  [eq. 5]
    latent = sample_gaussian_noise((4, 64, 64))  # Z_T  [§3.1]
    # Sampling schedule: DDIM with 50 steps is the setting in [fig. 4]
    for t in schedule(steps):
        predicted = sd.unet(latent, t, text_states, guidance)
        latent = scheduler_step(predicted, t, latent)
    return sd.decoder(latent)

Key equations, in the paper's notation:

Diffusion loss of Stable Diffusion, with \mathbf{Z}_t the noised latent and \mathbf{C} the conditioning eq. 1:

\mathcal{L} = \mathbb{E}\left[\lVert \epsilon - \epsilon_\theta(\mathbf{Z}_t, \mathbf{C})\rVert_2^2\right] eq. 1

Cross-attention used by the UNet to read the text, with \phi and \tau learnable embeddings eq. 2:

\mathbf{Q} = \mathbf{W}_Q \phi(\mathbf{Z}_t);\quad \mathbf{K} = \mathbf{W}_K \tau(\mathbf{y});\quad \mathbf{V} = \mathbf{W}_V \tau(\mathbf{y})

\mathrm{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \mathrm{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d}}\right)\cdot \mathbf{V} eq. 2

Adapter features and their addition to the UNet encoder, for scale i \in \{1,2,3,4\} eq. 3 eq. 4:

\mathbf{F}_c = \mathcal{F}_{AD}(\mathbf{C}) eq. 3

\hat{\mathbf{F}}_{enc}^{i} = \mathbf{F}_{enc}^{i} + \mathbf{F}_c^{i} eq. 4

Composition of K adapters with weights \omega_k eq. 5:

\mathbf{F}_c = \sum_{k=1}^{K} \omega_k\, \mathcal{F}_{AD}^{k}(\mathbf{C}_k) eq. 5

Adapter training objective, with the base model frozen eq. 6:

\mathcal{L}_{AD} = \mathbb{E}_{\mathbf{Z}_0, t, \mathbf{F}_c, \epsilon \sim \mathcal{N}(0,1)}\left[\lVert \epsilon - \epsilon_\theta(\mathbf{Z}_t, t, \tau(\mathbf{y}), \mathbf{F}_c)\rVert_2^2\right] eq. 6

Hyperparameters

Values stated in the paper. Code values are from the official repository at code_commit, which targets SDXL; the paper's SD 1.4 experiments are in the SD branch (see Paper vs code).

Results

Author-reported. Setup: COCO validation set, 5,000 images, one generation per image per method; CLIP score with ViT-L/14; base model Stable Diffusion for the SD row §4.2 tab. 1. Whether baseline numbers were re-run or copied from prior papers is not stated.

Quantitative comparison tab. 1; lower FID and higher CLIP score are better. Bold marks the best result in the paper.

Guidance-injection ablation tab. 2. FID only; dataset and metric setup are not stated for this table.

Other results are qualitative only: visual comparisons fig. 6, single-adapter examples for colour, sketch, depth, segmentation and keypose fig. 7, image editing with sketch in inpainting mode fig. 8, composition of depth with keypose and sketch with colour fig. 9, transfer to SD 1.5 and a custom model fig. 10, and adapter size variants fig. 11. The paper reports no numbers for composition, generalisation, or the cubic timestep rule beyond the figures §4.3 §4.4.2.

Paper vs code

The official repository is github.com/TencentARC/T2I-Adapter. Its README links the paper and says the main branch is the SDXL release, while the SD branch is the SD 1.4/1.5 release README.md. The commit recorded here is main at c408b05, which has no LICENSE file. The SD branch has an Apache-2.0 LICENSE. The SD branch was read through a web view, not cloned at a pinned commit, so its statements below are summaries without line numbers and are unverified against a commit.

Discrepancies found in main (SDXL), which are the cited lines:

• Input size and pixel-unshuffle: paper uses 512 x 512 with factor 8 §4.1 §3.3. main uses 1024 x 1024 with PixelUnshuffle(16) (the sketch config sets cin: 256, which is one channel times 16 times 16) adapters.py Adapter-XL-sketch.yaml. Both give a 64 x 64 grid.
• Adapter channels and blocks: paper gives no channel counts §3.3. main uses channels 320, 640, 1280 and 1280 with two residual blocks per scale and average-pooling downsampling (use_conv: false) Adapter-XL-sketch.yaml adapters.py.
• Injection: paper adds features to four encoder scales eq. 4 §4.4.1. main adds one adapter feature after each of the three UNet down blocks and once after the mid block, with the adapter list consumed in order unet.py unet.py. The SDXL layout has three down blocks, so the four-scale mapping is not one-to-one.
• Learning rate: paper states 1e-5 §4.1. main argparse default is 5e-6 train_sketch.py; the README training command passes 1e-5. The main optimiser is AdamW with PyTorch defaults for betas and weight decay, and the scheduler defaults to constant train_sketch.py train_sketch.py.
• Batch size and steps: paper: batch 8, 10 epochs §4.1. main config: batch 2 Adapter-XL-sketch.yaml; README command: batch 1 with 4-step gradient accumulation, 60,000 steps. The README says it uses 4 A100 GPUs in FP16, not the paper's V100s.
• Training data: paper uses COCO for sketch and LAION-Aesthetics for colour, keypose and depth §4.1. main trains on LAION_6plus and Lexica web datasets sampled 70/30 at 1024 px Adapter-XL-sketch.yaml.
• Sketch threshold: paper thresholds the edge map at 0.5 §4.1. main draws a threshold uniformly between 0.3 and 0.8, rounded to one decimal, for each batch train_sketch.py train_sketch.py. The SD branch summary reports a fixed 0.5 threshold on the PiDiNet output, which would match the paper.
• Edge model: paper cites [edge], which is Pixel Difference Networks [bib: edge]. main loads PiDiNet and takes its last output train_sketch.py train_sketch.py.
• Timestep sampling: paper formula §3.4. main draws u uniformly in [0, 1), computes (1 - u^3) times the scheduler's number of training steps, and truncates to an integer train_sketch.py. This is the same distribution as the paper's formula with u mapped to [0, 1).
• Inference sampler: Figure 4 uses DDIM with 50 steps fig. 4. main sampling uses DDPMScheduler with 50 steps by default and classifier-free guidance with scale 7.5 Sampling.py Sampling.py. The paper states neither a guidance scale nor the CFG setup for Table 1.
• Adapter application at inference: the paper says the adapter runs once per sample §1. main computes the adapter outputs once and passes the same copy to every UNet call Sampling.py.

Pitfalls

• Cubic timestep formula. The paper writes t = (1 - (t/T)^3) T with t uniform on [0, T] §3.4. The symbol t is used on both sides. The code reads it as u uniform on [0, 1) and computes (1 - u^3) times the step count, with integer truncation train_sketch.py. Use the code's reading, and state which convention the step count uses: the paper does not give the number of training steps T.
• Which steps are early. The paper says the cubic rule raises the probability of the early sampling stage §3.4. Early means high noise, so the rule must map to the large-t end of the schedule. Check that the scheduler's indexing matches this before training.
• Adapter once at inference, every step in training. During training the adapter runs on every sample train_sketch.py; at inference it runs once and its output is reused §1 Sampling.py. Running it inside the denoising loop at inference is wasteful and not what the paper describes.
• Fusion is plain addition. Equation 4 adds the adapter feature to the encoder feature without a learned gate or scale eq. 4. Any extra scale is the composition weight or the caller's conditioning scale, not stated in the paper.
• Composition weights are manual. The paper gives no default and says the weights need manual tuning §3.3 §5. The released diffusers example uses adapter_conditioning_scale 0.8 for a single adapter test.py. Treat that as a starting point, not a paper value.
• Colour adapter size. The colour palette uses the small (18M) adapter, while the others use the base (77M) one §4.4.2. The parameter count therefore depends on the condition.
• Colour downsampling. The paper states 64x downsampling and nearest upsampling §3.3. The code reads this as resizing to width/64 and height/64 with bicubic interpolation, then nearest-neighbour resizing back to full size api.py. At 512 px this is an 8 x 8 map.
• Guidance stage. Guidance added in middle and late steps has little effect §3.4 fig. 4. This is a qualitative finding from figures; no numbers are given.
• Decoder injection hurts. Mode 2 (decoder only) and mode 3 (encoder and decoder) are worse than encoder-only injection tab. 2. For the both-sides case the paper says control becomes too strong and limits texture richness §4.4.1; it gives no reason for the decoder-only result.
• Ablation scale drop. When fewer than four scales are used, the paper says it discards the feature at the small scale §4.4.1. "Small" is not defined: the lowest-resolution map is the likely meaning, but this is inferred.
• Ablation setup not stated. Table 2 shows FID with no dataset or sampler given tab. 2. Its encoder-only row (17.36) equals Table 1's text-plus-sketch FID, which suggests the same setup, but the paper does not say so.
• Baseline setup not stated. Whether SPADE, OASIS and PITI were re-run or copied, and what sampler SD used in Table 1, is not stated §4.2.
• Generalisation is qualitative. Transfer to SD 1.5 and a custom model is shown only in figures §4.3.3 fig. 10.
• Missing figure label. The introduction refers to a teaser figure whose label does not exist in the source. The figure number for that example cannot be matched from the source §1.
• Learning rate and schedule. The paper gives a learning rate but no schedule, warmup or optimiser betas §4.1. The code's constant schedule is the default train_sketch.py.
• Attribution and licence. The paper is CC BY 4.0 [raw metadata]. The code repository at the recorded commit has no LICENSE file, so no code is quoted; only pseudocode in our own words and citations are used.

Lineage

• Builds on: Stable Diffusion, referred to as [ldm] in the paper; the arXiv id is not in the source. Adapter-style modules from NLP are cited as [adapter] §2.3.
• Compared against: SPADE [spade], OASIS [oasis], PITI [piti] and Stable Diffusion alone tab. 1.
• Concurrent work: ControlNet [controlnet] learns task-specific control networks, and composer [composer] retrains a diffusion model conditioned on control factors §2.2.