ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

In plain words

A text-to-image model can draw a picture for almost any prompt, but a picture can look fine and still be wrong: it may ignore part of the prompt, distort a body, or look ugly. The usual automatic scores do not capture this well. The way to check is to ask people, and that is slow and expensive.

This paper asks people to compare generated images for many real prompts, and records which image is better. It then trains a scoring model, called ImageReward, on those comparisons. Once trained, the scorer can give any image and prompt a number, so a computer can pick the best of many candidates or compare models without asking people each time.

The paper also uses the scorer to improve the generator itself. Instead of scoring only finished images, the method looks at an image part way through its creation, scores that, and adjusts the generator so that its output moves toward the scorer's preferences. The authors report that the scorer agrees with human rankings better than the common automatic scores, and that the tuned generator is preferred by people in their tests. All results are the authors' own.

TL;DR

ImageReward is a reward model built on the BLIP vision-language backbone. Its data come from 8,878 annotated prompts and 136,892 expert image-pair comparisons, of which 466 prompts are held out for testing §2.1 §4.1. ReFL fine-tunes Stable Diffusion v1.4 by back-propagating ImageReward's score through a single randomly chosen late denoising step §3. The main numbers are author-reported: 65.14% preference accuracy on 6,399 held-out comparisons tab. 3, and ReFL preferred over the baseline in 58.79% of human comparisons tab. 4. The released code differs from the paper in the ReFL loss, the regularisation term and the training length (see Paper vs code).

Problem

• Text-to-image outputs fail in four common ways: weak text-image alignment, malformed bodies, poor aesthetics versus the mainstream preference, and harmful or biased content §1.
• Pre-training data is noisy and differs from what real users type, so better architectures alone do not fix these failures §1.
• FID, the usual metric, averages the distance to real images over a whole set and cannot score a single image. The authors also argue it misses human preference §2.3.
• CLIP similarity and LAION's Aesthetic predictor each capture part of the picture, not the combined judgement of alignment, fidelity and harmlessness §5.
• RLHF for language models uses the likelihood of a whole generation to update the model. Diffusion models produce images through many denoising steps and give no such likelihood for the result §3.
• Earlier methods that use reward models either filter a dataset or re-weight a loss. The authors argue these act only indirectly on the generator §3 §4.2.
• Building a preference dataset is costly. The annotation took about two months and needed a vendor, a written guideline and quality inspectors §2.1 app. A.2.

Method

The paper has three parts: an annotation pipeline that produces the data §2.1, a reward model trained on that data §2.2, and ReFL, which uses the reward model to fine-tune a diffusion model §3. The reward model is also proposed as an evaluation metric for generators §2.3.

1. Annotation pipeline §2.1 app. A.

1. Prompt selection. Prompts come from DiffusionDB, which the authors describe as real user prompts §2.1. They are grouped into 100 sets of about 20k prompts each. Within each set, a graph is built where each prompt is linked to its 150 nearest neighbours by sentence-embedding cosine similarity. Prompts are then picked greedily to favour diversity, 100 per set, for 10,000 prompts app. A.1.
2. Image sampling. Each prompt has 4 to 9 sampled images from DiffusionDB §2.1.
3. Prompt annotation. Annotators assign a category from the Parti scheme (12 categories) and tick boxes for ambiguous or toxic prompts app. A.3 app. A.4.
4. Per-image rating. Each image gets three seven-point Likert ratings for alignment, fidelity and overall quality, plus checkboxes for problems such as body defects, blur and toxic content app. A.3. The annotation screens are shown as screenshots in the appendix.
5. Ranking. All images for the prompt are placed into five preference slots, best to worst. Two images may share a slot, and the authors give trade-off rules, for example that a less toxic image outranks a more aligned one app. A.3 app. B.
6. Quality control. Quality inspectors recheck annotations, and invalid ones are relabelled §2.1 app. A.2.

After annotation, valid labels cover 8,878 prompts and 136,892 compared pairs §2.1 app. A.4.

2. Reward model (ImageReward) §2.2 eq. 1. The score of a prompt and image pair is a scalar. Training uses only the ranking of images for each prompt. Each ranked list with k images yields at most k(k−1)/2 pairs. The loss is a pairwise logistic loss over the score difference eq. 1.

The backbone is BLIP (card): a ViT-L image encoder and a 12-layer text encoder. Image features enter the text encoder through cross-attention. A small MLP turns the text encoder's output into one scalar §2.2 §4.1.

To control overfitting, the authors freeze a fixed fraction of the backbone's transformer layers, and then tune the learning rate and batch size by grid search on the validation set §2.2.

# Preference reward training step  [§2.2] [eq. 1]
# Input:  prompts (B,), better_images, worse_images (each (B, 3, 224, 224))
# Output: scalar loss

def reward_training_step(prompts, better_images, worse_images):
    # Score the preferred and the dispreferred image for each prompt  [§2.2]
    score_better = reward(prompts, better_images)   # (B,)
    score_worse = reward(prompts, worse_images)     # (B,)
    # Pairwise logistic loss on the score gap  [eq. 1]
    gap = score_better - score_worse
    loss = -mean(log_sigmoid(gap))
    return loss

# Score one prompt and image pair  [§2.2]
# Code: ImageReward/ImageReward.py#L84-L100
# Input:  prompt_tokens (T,), image (3, 224, 224)
# Output: scalar reward, normalised

def reward(prompt_tokens, image):
    # Encode the image with the ViT-L backbone  [§2.2]
    image_tokens = vision_encoder(image)
    # The text encoder attends to the image tokens (cross-attention)  [§2.2]
    text_states = text_encoder(prompt_tokens, image_tokens)
    # The first token's state summarises the pair
    # Code: ImageReward/ImageReward.py#L96
    summary = text_states[0]
    raw = mlp(summary)
    # Fixed shift and scale constants, not stated in the paper
    # Code: ImageReward/ImageReward.py#L80-L81, L98
    return (raw - 0.16717362830052426) / 1.0333394966054072

3. Reward feedback learning (ReFL) §3 alg. 1 eq. 2 eq. 3. The authors use the reward model to fine-tune a latent diffusion model directly. They first observe that the reward of the predicted clean image, taken from a single denoising step, becomes informative late in the schedule. With 40 steps, scores are unanimously low up to about step 15, only some high-quality generations stand out between steps 15 and 30, and the generations become clearly distinguishable from about step 30 onward §3 fig. 4. The training loop therefore draws one random late step for each update. It denoises up to that step without gradients, takes one more step with gradients, predicts the clean latent, decodes it, and scores it. The reward is turned into a loss and combined with a pre-training regulariser. The authors report that back-propagating only through the last step was unstable §3.

# One ReFL update for one prompt  [alg. 1] [§3]
# Input:  unet, reward, prompt, schedule of 40 steps
# Output: loss for one gradient step

def refl_step(unet, reward, prompt, schedule):
    # Draw a random stopping step from the late part of the schedule  [§3]
    # Paper text says steps 30 to 40; §4.2 says [1, 10]; code draws 30 to 39
    stop = random_integer(30, 39)
    # Start from Gaussian noise in latent space  [alg. 1]
    latent = sample_normal((4, 64, 64))
    # Denoise up to the stopping step without keeping gradients  [alg. 1]
    for step in schedule[:stop]:
        with no_grad():
            latent = denoise(unet, latent, step, prompt)
    # Take the stopping step with gradients  [alg. 1]
    latent = denoise(unet, latent, schedule[stop], prompt)
    # Predict the clean latent from this step  [alg. 1]
    clean = predict_clean_latent(latent, schedule[stop])
    # Decode to pixels, so the reward sees an image  [alg. 1]
    image = vae_decode(clean)
    # Map the reward to a loss, scaled by lambda  [§4.2] [eq. 2]
    # phi is ReLU in the paper; the code uses relu(2 - reward)
    loss = lambda_scale * phi(reward(prompt, image))
    return loss

The full objective adds a regulariser that keeps the generator close to its pre-training loss, which is the standard denoising loss on a pre-training set §3 eq. 3. The paper says the regulariser prevents overfitting, but does not state its weight.

Key equations:

\mathrm{loss}(\theta) = -\mathbb{E}_{(T, x_i, x_j) \sim \mathcal{D}}\left[\log \sigma\left(f_\theta(T, x_i) - f_\theta(T, x_j)\right)\right] eq. 1

\mathcal{L}_{reward} = \lambda\, \mathbb{E}_{y_i \sim \mathcal{Y}}\left[\phi\left(r(y_i, g_\theta(y_i))\right)\right] eq. 2

\mathcal{L}_{pre} = \mathbb{E}_{(y_i, x_i) \sim \mathcal{D}}\left(\mathbb{E}_{\mathcal{E}(x_i), y_i, \epsilon \sim \mathcal{N}(0, 1), t}\left[\left\lVert \epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y_i)) \right\rVert_2^2\right]\right) eq. 3

\mathcal{L}(\theta) = \mathbb{E}_{(x,z) \sim \mathcal{D}^{model}}\left[-r_\phi(x,z)\log p_\theta(x|z)\right] + \beta\, \mathbb{E}_{(x,z) \sim \mathcal{D}^{pre}}\left[-\log p_\theta(x|z)\right] eq. 4, app. E

Symbols: θ are the diffusion model's weights, g_θ(y) is the image generated for prompt y, r is the reward, φ is a reward-to-loss map, λ is a scale, and z_t is the noised latent at step t. Equation 4 is the reward-weighted baseline from the appendix, not ReFL.

4. Evaluation as a metric §2.3 tab. 1 fig. 3. The authors score six public text-to-image models on 100 real-user prompts. For each model, they pick the best of 10 outputs, then ask researchers (the authors) to rank the six models for each prompt. They compare these human rankings with ImageReward, CLIP and zero-shot FID on MS-COCO 2014 §2.3. The ImageReward and CLIP scores are averaged over 1,000 text-image pairs per model.

Hyperparameters

Values are from the paper unless the "Where" column names code. Where the paper is silent, the code's value is given and marked.

Results

Author-reported throughout. Setup: 466 DiffusionDB test prompts with 6,399 comparisons for preference accuracy tab. 3; 371 further prompts with 8 images each for Recall and Filter tab. 3; human evaluation of ReFL against SD v1.4 on 466 real-user prompts and 90 benchmark prompts tab. 4. Human raters are the paper's annotators unless stated.

Table 1: Model ranking by humans and automatic metrics §2.3 tab. 1. Human ranks come from the authors' annotations of 100 prompts, 10 outputs per model. Spearman rho is with the human ranking. Caption: "Text-to-image model ranking by humans and automatic metrics (ImageReward, CLIP, and FID). *Zero-shot FID (30k) for DALL-E 2 is taken from the cited DALL-E 2 paper; others are evaluated at 256 × 256 on the MS-COCO 2014 validation set."

Table 2: Agreement and ablation §4.1 tab. 2. Agreement is computed on 40 prompts (778 pairs) §4.1. "Annotator ensemble" takes, for each pair, the image most people prefer. "±" is the spread as given in the source.

(a) Agreement

(b) Ablation

Table 3: Human preference prediction §4.1 tab. 3. Preference accuracy is on 466 prompts (6,399 comparisons). Recall@k and Filter@k come from another test set of 371 prompts with 8 images each. The text gives no formulas for them; app. C.2 describes the bucket distribution of the best and worst images. Scores are averaged per prompt. Caption from source: "Results of ImageReward and comparison methods on human preference prediction."

Table 4: Human evaluation of diffusion fine-tuning methods §4.2 tab. 4. WinRate is against SD v1.4. Caption from source: ReFL performs best on total win count and WinRate against the baseline.

Other author-reported results, from the text and figures:

• Human win rates of ImageReward-selected images against other scorers: 77.1% against random, 69.3% against CLIP, 69.8% against Aesthetic, 65.8% against BLIP fig. 5.
• Pairwise win rates between all ReFL and baseline methods are shown as a heat map fig. 6. ReFL is always the preferred method in the pairwise comparisons §4.2.
• In the qualitative comparison, ReFL produces the requested detail (for example, "long pointy ears") where the other fine-tuned models do not fig. 7.

Table 5: Other reward models, recall and filter app. D tab. 5. Source caption: results of ImageReward and other reward models on human preference evaluation (same test sets as tab. 3).

Table 6: Other reward models, best-of-64 and ReFL app. D tab. 6. "Bo64" means picking the highest-reward image from 64 candidates. "ReFL" means the generator was tuned with that reward. "#Win" counts all human comparisons; "WinRate" is against the SD v1.4 baseline. The source does not define the "Reward" column.

Figures, described in words (no images are published here):

• fig. 1: top-1 images from 64 generations under different scorers, and one-shot generations after ReFL training.
• fig. 2: overview of annotation, reward training and ReFL.
• fig. 3: normalised distributions of ImageReward and CLIP scores per model; ImageReward has a wider spread and its medians follow the human ranking §2.3.
• fig. 4: ImageReward scores of one prompt at each denoising step; the scores become distinguishable after about step 30 of 40 §3.
• app. C.1 and app. C.2 plot scores of best and worst images and the bucket distribution of human-picked images.

Paper vs code

Code: https://github.com/THUDM/ImageReward at commit 78f8051059b34fef8e0032ccb40561c23a2ddbbb, Apache-2.0 (root LICENSE). The repository is linked from the paper's abstract. It contains the reward model (ImageReward/), a ReFL training script for Stable Diffusion 1.4 (ImageReward/ReFL.py, scripts/train_refl.sh), and the reward training code (train/).

• ReFL loss. Paper: φ = ReLU with λ = 1e-3 §4.2. Code: relu(-rewards + 2), then times grad_scale = 1e-3 ReFL.py. The offset of 2 and the sign convention are not in the paper. The code's rewards are the normalised scores (see Hyperparameters).
• Regularisation term missing. The paper adds a pre-training loss, Eq. 3, to the ReFL loss §3. The training loop in the released ReFL code computes only the reward loss, then back-propagates it ReFL.py. The script has one training-data argument and no pre-training data option ReFL.py.
• Batch size. Paper: 128 in total, 64 for pre-training and 64 for ReFL §4.2. Script: 2 per GPU × 4 accumulation × 8 GPUs = 64, ReFL only train_refl.sh train_refl.sh.
• Training length. Paper: one epoch over 20,000 samples §4.2. Script: 100 steps train_refl.sh. At 64 samples per step, 100 steps is 6,400 samples (our arithmetic).
• Stopping step. Paper, §3: t in [30, 40]. Paper, §4.2: [T1, T2] = [1, 10]. Code: random.randint(30, 39) on a 40-step schedule ReFL.py ReFL.py. The code is consistent with §3 except for the upper bound.
• Noise scheduler. Paper: PNDM at inference §4.2. Training code: DDPM scheduler loaded from the model folder ReFL.py, with set_timesteps(40) ReFL.py.
• Reward head activations. Paper: "an MLP" §2.2. Code: all ReLU activations are commented out, so the head is a stack of linear layers with dropout ImageReward.py.
• MLP initialisation. Paper: N(0, 1/(d_model+1)) §4.1. Code: standard deviation 1/(768+1) ImageReward.py.
• Reward normalisation. Not in the paper. Code subtracts 0.16717 and divides by 1.03334 ImageReward.py ImageReward.py.
• Frozen layers. Paper: 70% §2.2 §4.1. Code: text layers up to index 8 of 12 and vision blocks up to index 16 of 24 ImageReward.py. The freeze loop sets requires_grad to False on each parameter up to and including the first one whose name contains the marker, so the marked parameter is frozen too.
• Batch split. Paper: 16 per GPU on 4 GPUs §4.1. Script: 4 per GPU with 4 accumulation steps on 4 GPUs train_one_node.sh train_one_node.sh. Effective batch 64 in both.
• Ties. Paper: at most k(k−1)/2 pairs "if no ties", and ties are allowed in annotation §2.1 app. A.3. Code: pairs with tied ranks are skipped rank_pair_dataset.py.
• Model selection. Code keeps the checkpoint with the lowest validation loss train.py. The paper says nothing about model selection.
• Pre-training data. Paper: a 625k subset of LAION-5B selected by aesthetic score §4.2. The released ReFL code has no such dataset path.
• Adam settings. Paper: silent. Code: torch.optim.Adam takes opts.adam_beta1, opts.adam_beta2 and opts.adam_eps train.py, but a search of train/src/ finds no definition of these options.

Pitfalls

• Stopping step is inconsistent. §3 says t in [30, 40] and says the late-step insight holds from about step 30. §4.2 lists [T1, T2] = [1, 10] for the same algorithm. The code uses 30 to 39. Use 30 to 39 unless you have checked otherwise.
• The paper's ReLU is underdefined. A ReLU of the reward alone would push rewards down, so a sign and an offset are needed. The released code uses relu(2 − r). Test the loss on your scorer's output range before using it.
• Regulariser weight not stated. The pre-training term L_pre is named but not weighted. The released code has no such term, so a faithful ReFL needs a pre-training batch and an unstated weight.
• Gradients run through the decoder and the reward model. The loss gradient passes through one UNet step, the VAE decoder and the frozen reward network at 512 × 512. Memory and time are not quantified in the paper beyond the GPU list. The script enables gradient checkpointing train_refl.sh.
• Late-step instability. Back-propagating through only the last step was unstable in the authors' runs §3. Random late steps are the workaround. The paper gives no seed or variance information for ReFL results.
• Reward normalisation changes the loss scale. The ReFL hinge depends on the score range. Use the released normalisation, not raw scores.
• Fraction of frozen layers. 70% in the text becomes 8 of 12 text layers and 16 of 24 vision blocks in code, about two-thirds. The break in the freeze loop leaves the marked layer's first parameter frozen ImageReward.py.
• Optimiser options are referenced but not defined. Adam betas and epsilon are used in the training code but are not defined in train/src/ (only the use at train.py#L106 matches). Set them explicitly if you run the training script.
• Intro percentages are not derivable from the tables. The introduction says ImageReward beats CLIP by 38.6%, Aesthetic by 39.6% and BLIP by 31.6%. Table 3 gives preference accuracies of 54.82, 57.35 and 57.76 against 65.14. Relative gains on that metric are 18.8%, 13.6% and 12.8% (our arithmetic). The metric behind the intro figures is not stated.
• Ablation text overclaims. The text says CLIP used a larger training set, but Table 2 lists CLIP at 4k and 8k, the same maximum as BLIP.
• Human ranking for the metric is small and internal. Table 1 uses 100 prompts, 10 outputs per model, six models and author rankings. Spearman rho over six items is not a strong test.
• Recall and Filter have no formulas. Table 3 reports Recall@k and Filter@k, but the text gives no formula, so these numbers cannot be reproduced from the paper alone.
• Per-prompt averaging. Table 3 averages scores per prompt. Compare only against numbers computed the same way.
• Human studies are small and differ. The Table 1 ranking uses the authors as raters. The selection study in §4.1 used three annotators per prompt §4.1. The number of raters and prompts behind Tables 4 and 6 is not stated. Do not compare these to other papers' human studies without matching the protocol.
• Dataset and weight licences are not checked here. DiffusionDB, the LAION-AES subset, the benchmark prompts, the BLIP and Stable Diffusion weights, and the ImageReward checkpoint each carry their own terms. Check them before commercial reuse.
• Figure and table numbers are derived from LaTeX source order, not from the compiled PDF. If in doubt, cite the section.

Lineage

• Builds on: BLIP, the vision-language backbone for the reward model §2.2, card /papers/2201.12086.
• Builds on: Stable Diffusion v1.4, the generator tuned by ReFL §4.2, latent diffusion, no card here.
• Builds on: DiffusionDB, the source of real user prompts §2.1; LAION's aesthetic predictor, used as a baseline §4.1; neither has a card.
• Compared against: CLIP score, LAION Aesthetic and BLIP score §4.1 tab. 3; HPS (arXiv:2303.14420) and PickScore (Pick-a-Pic) app. D; RAFT (arXiv:2304.06767), Reward Weighted (arXiv:2302.12192) and dataset filtering (the HPS paper) §4.2 app. E. None has a card here.
• Superseded by: not stated in this wiki's content.