Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation

In plain words

Text-to-image models are hard to judge. Public benchmarks use generic photo captions, and the large sets of human judgements that could show which picture people actually like are kept inside companies.

This paper builds a small web app where people type a prompt, get two AI-made pictures, and click the one they prefer, or call it a tie. The collected choices are released as an open dataset of real users' preferences. The authors then train a scoring model, a CLIP-style network fine-tuned so that the picture a person chose gets the higher score for that prompt. It predicts held-out choices better than the scorers it was compared with and better than the human annotators used in the study. The authors suggest using it to rank pictures from new models, and to pick the best of several samples for a prompt.

TL;DR

• Open dataset of real users' prompts, image pairs and picks over generated images, with ties. The paper reports 583,747 training examples §2 Statistics.
• PickScore: a CLIP ViT-H/14 model fine-tuned with a two-way softmax preference loss §3.
• Test accuracy on predicting user picks: 70.5 for PickScore, 68.0 for expert annotators, 60.8 for CLIP-H tab. 1.
• Real-user Elo correlation: PickScore 0.790, above HPS (0.670), ImageReward (0.492) and CLIP-H (0.313) §5.
• Released model and dataset are linked from the paper's footnotes §1, but the paper does not state their licences.

Problem

Image models are usually scored with FID over MS-COCO captions, which the authors argue is prompt-agnostic and unlike what users request §5 Model Evaluation. Preference data that could replace this is rare in the open. Crowd workers who rate images lack a reason to care about the prompt's intent, and most large preference sets are not public §1 Introduction §7 Related Work.

Method

Data collection §2

1. A user writes a prompt. The app shows two images generated for it, from backbone models named as Stable Diffusion 2.1, Dreamlike Photoreal 2.0 and Stable Diffusion XL variants, with varied classifier-free guidance scales §2 Pick-a-Pic Dataset.
2. The user picks one image or calls a tie. The rejected image is replaced with a new one, and the loop repeats until the prompt changes §2 The Pick-a-Pic Web App fig. 2.
3. The two-image-with-ties design was chosen over four images without ties and two without ties, on engagement and inter-rater agreement in a pilot §2 Annotation Methodology.
4. Users sign in with a Gmail or Discord account. Accounts are banned for NSFW generation, running several app instances, or judging too fast. Prompts are checked against a list of NSFW phrases, and each user is limited to 1000 interactions §2 Real Data from Real Users.
5. Preprocessing drops NSFW prompts and banned users. Validation and test sets each take one example per prompt, from 1000 prompts split into two equal sets. All remaining examples form the training set, so no prompt appears in two splits §2 Preprocessing.

Scoring function (PickScore) §3

1. Encode the prompt with a text encoder and the image with an image encoder, both producing d-dimensional vectors. The score is the inner product of the two, multiplied by the learned temperature T §3 Model eq. 1.
2. For a prompt and two images, take a softmax over the two scores to get a probability for each image §3 Objective eq. 2.
3. Minimise the KL divergence from the target distribution p to these probabilities. The target is [1, 0] for image 1 preferred, [0, 1] for image 2, and [0.5, 0.5] for a tie §3 Objective eq. 3.
4. Weight each example in a batch in inverse proportion to how often its prompt occurs in the dataset, then average §3 Objective.

# PickScore training step  [§3 Objective]  [eq. 1-3]
# Input:  batch with prompt, image_0, image_1, label_0, label_1, count
#         label_k in {1, 0, 0.5}; count = examples sharing this prompt
# Output: scalar loss

def training_step(model, batch):
    # Encode the prompt and both images, then L2-normalise each  [§3 Model]
    # [code: trainer/criterions/clip_criterion.py#L28-L34]
    text_emb = normalise(model.encode_text(batch.prompt))       # (B, d)
    image_0_emb = normalise(model.encode_image(batch.image_0))  # (B, d)
    image_1_emb = normalise(model.encode_image(batch.image_1))  # (B, d)
    # Scaled inner product with the learned temperature T  [eq. 1]
    # [code: trainer/criterions/clip_criterion.py#L65-L66]
    scale = exp(model.logit_scale)                              # T
    score_0 = scale * dot(text_emb, image_0_emb)                # (B,)
    score_1 = scale * dot(text_emb, image_1_emb)                # (B,)
    # Softmax over the two images  [eq. 2]
    probs_0, probs_1 = softmax_pair(score_0, score_1)           # (B,)
    # Cross-entropy to the target; this is KL plus the target's entropy
    # [eq. 3]  [code: trainer/criterions/clip_criterion.py#L101]
    loss = -(label_0 * log(probs_0) + label_1 * log(probs_1))   # (B,)
    # For ties, subtract the entropy log 2 so the loss is a pure KL
    # [code: trainer/criterions/clip_criterion.py#L102-L106]
    is_tie = (label_0 == label_1)
    loss = loss + is_tie * log(0.5)
    # Weight by inverse prompt frequency, normalised over the batch
    # [§3 Objective]  [code: trainer/criterions/clip_criterion.py#L114-L118]
    weight = (1 / count) / sum(1 / count)                       # (B,)
    return sum(weight * loss)

Inference and tie handling §4 eval_preference_predictor.py

1. Score both images for the prompt with the same model and softmax as in training.
2. Call the result a tie when the two probabilities differ by no more than a threshold t. Otherwise pick the higher one.
3. Choose t on the validation split, then apply it to the test split §4 Tie Threshold Selection.

# Preference prediction with a tie threshold  [§4 Tie Threshold Selection]
# Input:  prompt, image_0, image_1, threshold t (fitted on validation)
# Output: "0", "1" or "tie"

def predict(model, prompt, image_0, image_1, threshold):
    # Same scoring as training, applied to a single example
    # [code: trainer/scripts/eval_preference_predictor.py#L19-L49]
    probs_0, probs_1 = score_pair(model, prompt, image_0, image_1)
    # Two probabilities within the threshold count as a tie
    # [code: trainer/scripts/eval_preference_predictor.py#L76-L78]
    if abs(probs_1 - probs_0) <= threshold:
        return "tie"
    if probs_0 > probs_1:
        return "0"
    return "1"

Best-of-N ranking §6

1. For each of 100 test prompts, generate 100 candidate images: 5 random initial noises crossed with 20 prompt templates, using Dreamlike Photoreal 2.0 at classifier-free guidance 7.5 §6 Text-to-Image Ranking.
2. Keep the candidate with the highest score under the chosen scorer §6.

# Best-of-N selection with a scorer  [§6]
# Input:  prompt, scorer, generator; N = 100 candidates per prompt
# Output: one image

def best_of_n(prompt, scorer, generator):
    # 5 noises x 20 templates = 100 candidates  [§6]
    candidates = generator.sample(prompt, noises=5, templates=20)
    # Keep the candidate the scorer rates highest  [§6]
    return max(candidates, key=lambda image: scorer(prompt, image))

Hyperparameters

Results

Author-reported throughout. The test set is Pick-a-Pic v1 §4 Results. Baselines are CLIP-H, the LAION aesthetics predictor, ImageReward and HPS, plus expert human annotators who are friends and colleagues of the authors §4 Baselines §4 Results.

The Elo, FID and human-ranking results all use the authors' own model-set design (Model Evaluation section §5). The human judgements in §4 come from expert annotators who are friends and colleagues of the authors, and §6 uses expert human annotators; neither section describes crowd workers §4 Baselines §6.

Paper vs code

The official repository is linked from the abstract and the introduction §1 (https://github.com/yuvalkirstain/PickScore). Its owner is the paper's first author, Yuval Kirstain, which makes it an author repository; no third-party code was used. The paper's footnote links directly to it, so no further search was needed. Repository commit f79607ff68609c1e6dfa78cba38afeea4c248df1, MIT licence (LICENSE, copyright 2021).

• Learning-rate schedule: the paper states linear decay after 500 warmup steps §3 Training. The default accelerator is DeepSpeed, which uses WarmupDecayLR, a linearly decaying schedule deepspeed_accelerator.py configs.py. A non-DeepSpeed path uses constant-after-warmup constant_with_warmup.py. Which path produced the paper's run is not stated.
• Batch size: the paper says 128 §3 Training. The code's DeepSpeed block declares gradient accumulation 16 deepspeed_accelerator.py, but the plugin is passed the base value of 1 deepspeed_accelerator.py base_accelerator.py. The 128 total holds only if the base value of 1 is the one used; the repository does not settle this.
• Optimiser: the paper names no optimiser. The code uses AdamW with torch_adam and adam_w_mode set, through DeepSpeed deepspeed_accelerator.py. Weight decay 0.3 dummy_optimizer.py, which the paper does not state. Betas and epsilon are not set in the repository.
• Gradient clipping: not in the paper. The code clips the norm at 1.0 base_accelerator.py deepspeed_accelerator.py.
• Precision: not in the paper. The experiment config uses BF16 clip.yaml.
• Scoring: paper's eq. 1 has no normalisation §3 Model. The code L2-normalises both text and image embeddings before the inner product clip_criterion.py. Temperature is read as exp of CLIP's logit_scale clip_criterion.py clip_model.py.
• Weights: the paper weights by inverse prompt frequency in the dataset §3 Objective. The code divides by the per-prompt count and normalises over the (gathered) batch clip_criterion.py. The weights therefore sum to one per batch, not per dataset.
• Tie comparison: the paper uses a strict inequality, |p̂₁ − p̂₂| < t §4 Tie Threshold Selection. The code uses ≤ eval_preference_predictor.py. This matters only when the difference equals the threshold exactly.
• Threshold grid: the paper says t is chosen on validation §4 Tie Threshold Selection but gives no grid. The code sweeps 0 to 1 in steps of 0.1 eval_preference_predictor.py.
• Training-time validation metric: the paper says accuracy on validation without a tie option §3 Training. The code's evaluate gives a tie label 0.5 whatever the prediction, since each comparison is multiplied by the label, which is 0.5 for a tie clip_task.py. This differs from the tie-aware metric in the eval script eval_preference_predictor.py, which gives full credit when the prediction is also a tie. Reported test numbers come from the eval script.
• Dataset: default is yuvalkirstain/pickapic_v1 with validation_unique and test_unique splits clip_hf_dataset.py.
• Tie handling in the loss: the code adds log 0.5 for tie examples clip_criterion.py. This matches the paper's KL form (eq. 3) once the target entropy is removed.

Pitfalls

• Tie labels earn partial credit. In the eval-script metric, a tie label earns 0.5 against any non-tie prediction and 1 against a tie prediction §4 Metric eval_preference_predictor.py. Training-time validation gives a tie label 0.5 whatever the prediction (see Paper vs code), so compare numbers only within one metric.
• The threshold is a per-model hyperparameter. The paper says it is chosen on validation §4 Tie Threshold Selection, so the threshold for a new scorer must be refitted before comparing accuracies.
• Accumulation is ambiguous in the code. Both 16 and 1 appear as gradient accumulation values (see Paper vs code). An implementer must choose the setting that yields the paper's 128.
• The temperature is inherited. The learned temperature starts from CLIP's logit_scale, and its initial value is not in the paper. Restarting from another checkpoint changes the effective softmax sharpness.
• NSFW content remains. The authors say NSFW images and prompts are still in the dataset and that they will try to reduce them §2 Preprocessing. The filter is a phrase list §2 Real Data from Real Users.
• Data is self-selected. Users come from social media and sign in with Gmail or Discord §2 Real Data from Real Users. The paper argues the data reflects real users' intents better than captions §5 Model Evaluation Prompts; it does not measure how representative the sample is.
• The human baseline is under-described. Expert annotators are friends and colleagues of the authors §4 Baselines. The number of annotators, their instructions and the number of judgements are not stated.
• Backbone count is inconsistent. The 45-model Elo study uses "four different backbone models" §5 Other Evaluation Metrics. §2 lists Stable Diffusion 2.1, Dreamlike Photoreal 2.0 and Stable Diffusion XL variants §2 Pick-a-Pic Dataset. The four backbones are not named.
• Baselines may not be retrained. The paper says it tested ImageReward and HPS on Pick-a-Pic §7 Related Work. Whether they were fine-tuned on it or used as released is not stated.
• Spread is reported for PickScore only. The ± on PickScore is over three seeds §4 Results. The printed Table 1 shows 70.5 without it tab. 1. No other row has a spread.
• Licences of the dataset and model are not stated in the paper or the repository. The repository's MIT licence covers the code only LICENSE.

Lineage

• Builds on: CLIP, whose architecture PickScore follows §3 Model (cited as Radford et al., 2021; no arXiv id in the paper's bibliography); the InstructGPT reward-model objective, which the loss is described as analogous to §3 Objective (cited as instruct; no arXiv id in the bibliography); OpenCLIP's CLIP-H checkpoint §3 Training.
• Compared against: CLIP-H and an aesthetics predictor on CLIP-L §4 Baselines; ImageReward and HPS, both concurrent work §4 Baselines §7 Related Work; expert human annotators §4 Results. The paper gives no arXiv ids for these, so no links are added.
• Superseded by: not known from content already in this wiki.