Human Preference Score: Better Aligning Text-to-Image Models with Human Preference

In plain words

Text-to-image models often draw things people dislike: a hand with too many fingers, a face that looks wrong, a scene that ignores part of the request. Users cope by generating several images and picking the best one. The authors asked whether the usual scoring tools for these models agree with that human choice, and found that mostly they do not.

So they collected a large set of real choices. People on a Discord server were shown a set of images made from one prompt and picked the one they liked. They then trained a judge on those picks, which scores how well an image matches a prompt in the way people preferred. They call this score the Human Preference Score.

They then used the judge to steer the image model itself. Images the judge liked became training examples, and images it disliked were tagged with a special phrase in their prompt. After training, putting that phrase in the negative prompt steers the model away from the disliked style. People in the authors' small study tended to prefer the adjusted model's images.

TL;DR

• A CLIP ViT-L/14 model is fine-tuned on human choices between images generated from the same prompt, and its image-text similarity is the score (HPS) §5 §7.1.
• Author-reported: HPS predicts human choices better than CLIP or an aesthetic classifier, and it generalises to images from other generators tab. 2 tab. 3.
• Stable Diffusion v1.4 is adapted with a rank-32 LoRA on HPS-selected images, with the non-preferred prompts marked by "Weird image." §6.
• The paper's classifier training code is not in the repository, and the released code differs from the paper in LoRA rank and score scale. See Paper vs code.
• The collected dataset is distributed under a non-commercial licence, so this card does not use it.

Problem

Generated images show artifacts such as malformed limbs and faces, and users often cherry-pick outputs to avoid them §1. To improve a model, one must first measure whether it makes human-preferred images §1. The authors argue that common metrics do not do this: Inception Score and FID rely on an ImageNet classifier that is biased towards texture and sees only the image, not the prompt §1 §4. CLIP score uses the prompt but is not known to track human choice on generated images §1 §4. The paper treats the gap as a missing "awareness" of preference rather than a lack of model capacity §6.

Method

The paper has three parts.

1. Human preference dataset §3. From a public Discord channel of Stable Diffusion images, the authors extracted, by pattern matching on a fixed bot interaction grammar, the prompt, the set of images generated from it and the one image the user sent back as preferred §3, fig. 4. The dataset has 98,807 images from 25,205 prompts, with 25,205 human choices, from 2,659 users §3. Images generated with an uploaded condition image were excluded, and NSFW images were filtered with the bot's detector §3.

2. Existing metrics on this dataset §4. IS and FID show no meaningful difference between preferred and non-preferred images tab. 1. For FID the reference set is a pseudo ground truth: for 10,000 sampled prompts, the nearest LAION image is retrieved §4. CLIP score and an aesthetic classifier predict human choice only slightly better than chance tab. 2 §4.

3. HPS, a human preference classifier §5. The ViT-L/14 CLIP model is fine-tuned so that a prompt's text embedding is close to the embedding of the preferred image and far from those of the other images in its group §5. The paper gives this in words only; the exact loss is not stated. The score is the cosine similarity between the two embeddings, multiplied by 100 §5:

\mathrm{HPS}(\mathrm{img}, \mathrm{txt}) = 100 \cos(\mathit{enc}_v(\mathrm{img}), \mathit{enc}_t(\mathrm{txt})) §5, unnumbered in source

4. Adapting Stable Diffusion with HPS §6.

1. Score every image in DiffusionDB's large_first_1m split with HPS §6 process_diffusiondb.py.
2. Within each prompt's group, compute a softmax over the HPS values. Keep the top image as a preferred sample if its softmax weight is above α/n, where n is the group size and α = 2.0 §6. Apply the same rule to negated scores to get non-preferred samples §6.
3. Add the non-preferred samples to a training set that also contains regularisation images from LAION-5B, filtered to aesthetic score above 6.5 §6.
4. Train a LoRA adapter on the Stable Diffusion UNet only. The VAE and text encoder stay frozen. The LoRA adds low-rank residuals to the key, query, value and output projections of attention §6. Non-preferred captions are prefixed with "Weird image." §6.
5. At inference, use "Weird image." as the negative prompt for classifier-free guidance, with the usual prompt as the positive prompt §6.

The pseudocode below follows these steps. Equation numbers do not appear in the source, so the citations point to sections.

# Human Preference Score of one image-prompt pair  [§5]
# Input:  image (H, W, 3); prompt string
# Output: scalar score, 100 x cosine similarity

def hps(image, prompt, image_encoder, text_encoder):
    # Resize the longest edge to 224 and zero-pad to a square  [§7.1]
    image_vec = image_encoder(pad_to_square(resize(image, 224)))
    # Embed the prompt with the fine-tuned text encoder  [§5]
    text_vec = text_encoder(tokenize(prompt))
    # Cosine similarity, scaled by 100  [§5]
    # The released annotation script also multiplies by 100
    # [code: process_diffusiondb.py#L74]
    return 100 * cosine(image_vec, text_vec)

# Pick preferred and non-preferred images per prompt  [§6]
# Input:  groups: prompt -> list of (image, hps); alpha = 2.0  [§6]
# Output: positives and negatives as (image, prompt) pairs

def softmax(values):
    # Softmax over one prompt's group  [§6]
    exps = [exp(v) for v in values]
    return [e / sum(exps) for e in exps]


def select_training_images(groups, alpha):
    positives = []
    negatives = []
    for prompt, pairs in groups.items():
        # Scores of the images generated for this prompt  [§6]
        scores = [score for image, score in pairs]
        n = len(scores)
        # Keep the highest-scoring image if it clears alpha / n  [§6]
        probs = softmax(scores)
        best = argmax(probs)
        if probs[best] > alpha / n:
            positives.append((pairs[best][0], prompt))
        # Same rule on negated scores picks the lowest-scoring image  [§6]
        worst_probs = softmax([-s for s in scores])
        worst = argmax(worst_probs)
        if worst_probs[worst] > alpha / n:
            negatives.append((pairs[worst][0], prompt))
    return positives, negatives

# One LoRA training step on Stable Diffusion v1.4  [§6]
# Input:  images (B, 3, H, W); captions (B strings); LoRA-equipped unet
# Output: scalar loss; gradients reach only the LoRA weights  [§6]

def training_step(images, captions, unet, vae, text_encoder):
    # The VAE is frozen and encodes images to latents  [§6]
    latents = vae.encode(images)  # (B, 4, H/8, W/8)
    # Add noise at a random timestep (not stated in the paper)
    # [code: train_text_to_image_lora.py#L650-L659]
    noise = sample_gaussian_like(latents)
    timesteps = sample_random_timesteps(latents)
    noisy_latents = add_noise(latents, noise, timesteps)
    # The text encoder is frozen  [§6]
    context = text_encoder(captions)
    predicted = unet(noisy_latents, timesteps, context)
    # Mean squared error to the noise; the paper does not name the loss
    # [code: train_text_to_image_lora.py#L665-L674]
    return mean_squared_error(predicted, noise)

# Generate with the adapter  [§6]
# Input:  prompt; LoRA-equipped pipeline
# Output: image

def generate(prompt, pipeline):
    # The identifier is the negative prompt  [§6]
    negative_prompt = "Weird image."
    # 50 denoising steps and guidance scale 7.5  [§6]
    return pipeline(prompt,
                    negative_prompt=negative_prompt,
                    num_inference_steps=50,
                    guidance_scale=7.5)

Hyperparameters

Derived from §6: 10,000 steps at batch 40 is 400,000 samples, about 1.5 passes over the 258,911 training images (37,572 + 21,108 + 200,231).

Results

Author-reported. Setups are given in each row.

Setup for tab. 4: FID uses 10,000 images from LAION-5B §7.2. CLIP score and HPS use DiffusionDB prompts §7.2. The paper states no sample size or variance for the aesthetic, CLIP or HPS columns, and no significance test is reported. The user study uses the same seed for both models on each prompt §7.2.

Qualitative results: the adapted model is said to follow the prompt more closely and to show fewer limb artefacts §7.2 fig. 7. The regularisation-only model is a head-to-head control, and the paper describes its result only in words §7.2.

Paper vs code

Code: tgxs002/align_sd at 78a49a3b5e30cc738b844bb3cefd6bd5ccbb3e8b, Apache-2.0. The paper's project page links it, and the README calls it the official repository README.md. The owner matches the first author's account; the page does not name the owner explicitly.

• LoRA rank. Paper uses rank 32 §6. The training script's default is 4 train_text_to_image_lora.py, and scripts/train_lora.sh does not pass --lora_rank train_lora.sh. Running the released script as written gives rank 4.
• Score scale. Paper defines HPS as 100 × cosine §5. The annotation script multiplies by 100 process_diffusiondb.py, but the evaluation script does not evaluate_hps.py. Table 4's HPS values, about 0.19, are on the raw cosine scale tab. 4. Selection in §6 is a softmax over the scores, so the scale changes which images pass the α/n rule. Use the 100 × scale for selection, as the annotation script does.
• Image preprocessing. Paper resizes the longest edge to 224 and zero-pads §7.1. The evaluation script uses the OpenAI clip package's preprocessing evaluate_hps.py. That package's default resizes and centre-crops (library behaviour, not in this repository). The paper's padding is not what the released scorer does.
• Selection filters not in the paper. The annotation skips blurry images process_diffusiondb.py and, in the selection script, drops prompts with fewer than four images select_training_images.py. The paper says prompts have two, three or four images §3 and does not describe either filter.
• Selection rule. The script uses a mean-centred softmax, which gives the same probabilities as the paper's softmax select_training_images.py. It applies the threshold 2 / len(probs) select_training_images.py, equal to α/n with α = 2.0 §6.
• Classifier training code. Not in the repository. The README says the training script "will be released soon" README.md. The classifier checkpoint is linked from an external file host README.md. The paper's training details can be compared with no code for them.
• Adapter objective. The paper does not state the loss. The code uses the standard noise-prediction mean squared error train_text_to_image_lora.py.
• Learning-rate schedule. Matches the paper: constant, learning rate 1e-5, batch 40, 10,000 steps train_lora.sh §6. The script sets warmup to 0 train_lora.sh. The script also sets num_train_epochs to 99, with the 10,000-step cap deciding the length train_lora.sh train_lora.sh.
• Inference. Paper: 50 steps, PNDM, guidance 7.5 §6. The generation script sets 50 steps and fp16 generate_images.py generate_images.py and leaves the scheduler and guidance to the pipeline defaults. Whether those defaults match the paper is not verified here.
• Evaluation code. The repository has no script for the human-choice accuracy in tab. 2, the agreement in tab. 3, the FID setup in tab. 1 or tab. 4, or the user study.
• Dataset. The README links a compressed dataset and a classifier checkpoint README.md README.md. Neither is in the repository. This card does not use them.

Pitfalls

• Dataset licence. The paper's datasheet says the dataset is distributed under a non-commercial licence, with terms of use app. datasheet. The paper text is CC BY 4.0, but the dataset is not. Anyone rebuilding the training data must obtain the dataset under its own terms.
• Scale mismatch. The paper's 100 × definition, the released annotation script, and the raw cosine in the scorer and Table 4 disagree. Pick one and check it against the released checkpoint before comparing numbers.
• Preprocessing. Zero-padding to a square (paper) and the CLIP default resize and centre-crop (released scorer) give different inputs. The paper says the padding helps, but gives no ablation §7.1.
• Classifier not reproducible from the paper alone. The loss is described only in words §5. The optimizer's betas and epsilon are not stated. The learning-rate warmup, the number of layers left frozen and the Bayesian search space are not stated §7.1.
• LoRA rank. Use rank 32 from the paper, not the code default of 4, unless you have checked which one matches the released checkpoint.
• Regularisation sampling. The paper does not say how 200,231 images were chosen from the 625k pool §6. The README says about 200K is enough README.md.
• Counts. The classifier was trained on 20,205 prompts and 79,167 images §7.1. The total is 25,205 prompts and 98,807 images §3. How the rest are used beyond the 5,000-sample evaluation split §4 is not stated.
• Negative prompt. Inference depends on the exact identifier, which the paper gives as "Weird image." §6. The released generator uses a trailing space generate_images.py. Keep the identifier in both training and inference.
• Variance and significance. Table 4 and the user study give no variance or significance test tab. 4 §7.2. Table 3's human-vs-human spread is ±4.3 tab. 3, so the gaps between methods are small compared with human noise.
• Bias and scope. The paper marks rather than removes public figures and notes that the dataset reflects a small, active community of users §8. Many prompts are written by experienced users and may not look like everyday language §8.
• Metrics comparison. The paper's IS and FID claims rest on a single dataset, and FID uses a pseudo ground truth retrieved from LAION for each prompt §4.

Lineage

• Builds on: Stable Diffusion v1.4 §6; CLIP ViT-L/14 as the classifier backbone §5; LoRA §6; classifier-free guidance §6; DiffusionDB §3 §6.
• Compared against: Inception Score and FID §4; CLIP score and an aesthetic classifier §4.
• Related concurrent work named by the authors: Lee et al. on text-image alignment with human feedback §2.
• Later work: the README points to HPS v2 at github.com/tgxs002/HPSv2 as a stronger successor README.md. This card does not cover HPS v2.