TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering

In plain words

Checking whether a picture matches the sentence it was made from is hard to automate. Common automatic scores are weak on details such as counting and how objects relate to each other. TIFA takes a more step-by-step route. It first reads the sentence and writes simple questions that have known answers, such as "is there a dog?" or "how many dogs are there?". Then it asks a model that answers questions about images to answer those same questions about the generated picture. The score is the share of questions the picture gets right.

Because each question checks one detail, the score can be broken down by question type, which the paper uses to show where models fail. The question writer and the answering model are both replaceable, so the method can improve as those models improve. The authors also published a ready-made set of questions for 4,081 sentences, so you can score a text-to-image model without running the question writer yourself.

TL;DR

• Reference-free text-to-image faithfulness metric: an LM writes question-answer tuples from the prompt, UnifiedQA filters them, and a VQA model answers them on the generated image. The score is the VQA accuracy §3 eq. 1.
• Released benchmark TIFA v1.0: 4,081 text inputs and 25,829 questions across 12 element categories tab. 1 §4.
• Author-reported Spearman correlation with human faithfulness ratings: 59.7 for TIFA with mPLUG, against 33.2 for CLIPScore and 18.3 to 34.0 for caption-based metrics tab. 2.
• The authors recommend mPLUG-large as the VQA model; it had the highest VQA accuracy against human answers (84.5) and the highest correlation (76.8) among the five models tested §3.3 tab. 3.
• Question generation is a single in-context LM call per prompt with 15 hand-written examples in the paper; the released code uses a different LM and fewer examples (see Paper vs code).

Problem

Text-to-image models often fail to match their prompts, for example by missing objects, mixing up attributes, or getting counts and spatial relations wrong §1.

Existing automatic checks are weak for this purpose. CLIP similarity is unreliable for counting and compositional reasoning §1. Captioning-based checks depend on what the captioner chooses to describe, so details it ignores are not scored §1. DALL-Eval uses object detection, which limits it to synthetic prompts and a few aspects such as objects, counts, colours and spatial relations §1.

The method assumes the VQA model is good enough to answer the questions correctly when the image is faithful. The authors note that this may not hold for domains such as anime and abstract art §6.

Method

Overview §3. For a text input T, the method produces N multiple-choice question-answer tuples (Q_i, C_i, A_i), where A_i is in the choice set C_i. A VQA model answers each question on the image I. The faithfulness score is the fraction of questions answered correctly §3 eq. 1 fig. 2.

\mathit{faithfulness}(T, I) = \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}[A_i^{\text{VQA}} = A_i] eq. 1

The score lies in [0, 1]. Eq. 1 writes the VQA answer as the argmax over choices; for models that are not multiple-choice, the paper decodes a free-form answer and maps it to a choice (see §3.3 below) §3 §3.3. The paper calls the metric fine-grained and interpretable because each question targets one element with one category §1 §3.1.

Question-answer generation §3.1 fig. 3. A single in-context LM call per text input does all of the work. For each element, it writes two questions: one answered "yes" for a faithful image, and one whose answer is the element itself with several choices §3.1.

• Categories: twelve groups covering objects, actions, animals, food, numbers, colours, materials, relations, places, shapes, attributes and a catch-all "other" §3.1.
• Element extraction: the LM pulls out elements that can be noun phrases (named entities included), verbs, adjectives, adverbs, or short constituent spans of up to three words, and labels each one with a category. The in-context examples are annotated by hand §3.1.
• Example from the paper: for the text "A photo of three dogs.", the elements are "photo", "three" and "dogs". "three" is labelled counting, and "dogs" is labelled animal. Two questions are written for "three": "are there three dogs?" with choices yes and no, and "how many dogs are there?" with choices 1, 2, 3 and 4 §3.1.
• The LM is prompted once with 15 hand-annotated examples that cover all question types. The same examples are used for every text input. The paper uses code-davinci-002 at temperature 0 §3.1 app. F.

Question filtering §3.2. UnifiedQA (T5-large, v2 checkpoint) re-checks each generated tuple §3.2. A tuple is kept only if both of these hold:

• UnifiedQA's multiple-choice answer, given the text, the question and the choices, equals the generated answer A_i §3.2.
• UnifiedQA's free-form answer, given the text and the question, has a word-level F1 with A_i greater than 0.7 §3.2.

The authors checked 1,000 kept pairs by hand and found 7 unreasonable ones §3.2 app. C.

# Keep a generated QA tuple only if UnifiedQA agrees  [§3.2]
# Input:  prompt, question, choices, answer, qa_model
# Output: True to keep the tuple

def keep_tuple(prompt, question, choices, answer, qa_model):
    # The multiple-choice answer must match the generated answer  [§3.2]
    mc_answer = qa_model.multiple_choice(prompt, question, choices)
    if mc_answer != answer:
        return False
    # Free-form answer must share more than 0.7 word F1 with the gold  [§3.2]
    free_form = qa_model.free_form(prompt, question)
    return word_f1(free_form, answer) > 0.7

Answering on the image §3.3. Each question is answered by a VQA model on the generated image. The paper uses four general pre-trained vision-language models (GIT-large, ViLT-B/32, OFA-large, mPLUG-large) and BLIP-2 FlanT5-XL §3.3. The first four produce a free-form answer, which is then matched to the closest choice by SBERT similarity. BLIP-2 gets the choices inside its prompt and answers multiple-choice questions directly §3.3. The paper recommends mPLUG-large §3.3 §5.4.

# Score one generated image against its prompt's QA tuples  [§3] [eq. 1]
# Input:  qa_tuples (question, choices, answer) for one prompt; image
# Output: faithfulness in [0, 1]

def tifa_score(qa_tuples, image, vqa_model):
    # Ask every question about the image  [§3.3]
    hits = []
    for question, choices, answer in qa_tuples:
        # Decode a free-form answer, then map it to one choice  [§3.3]
        free_form = vqa_model.answer(image, question, choices)
        # Closest choice by SBERT; the code tries exact match first  [§3.3]
        predicted = closest_choice(free_form, choices)
        # Count 1 when the chosen option equals the gold answer  [eq. 1]
        hits.append(predicted == answer)
    # Average over the questions of this prompt  [eq. 1]
    return sum(hits) / len(hits)

Benchmark TIFA v1.0 §4. The text inputs are 2,000 COCO validation captions with gold images, plus 2,081 prompts with no real image: 161 from DrawBench, 1,420 from PartiPrompt and 500 from PaintSkill. Prompts with fewer than three words are excluded, and some categories are removed (DrawBench misspellings and rare words; PartiPrompt "abstract") §4.1.

The question-answer tuples for the benchmark were all generated by GPT-3 and pre-filtered by UnifiedQA. Users then only need to run a VQA model §4 app. D. For reproducibility, the authors also fine-tuned LLaMA 2 (7B) to generate questions from captions, using the TIFA v1.0 tuples as training data §4.3.

Human evaluation protocol §5.1 app. C. Annotators score each image from 1 to 5 for "does the image match the text?", judged by how many prompt elements are missed or misrepresented. Objects weigh more than attributes, relations and activities, and a missing object also marks its attributes as lost app. C fig. 6.

Human VQA protocol §5.4 app. C. Annotators answer the multiple-choice questions on the generated images. They get one extra option, "None of the above", which the models do not get. Each question has two annotators, with a third on disagreement, and the majority answer is used §5.4 app. C.

Hyperparameters

Code-only settings (from the official code, see Paper vs code):

Results

Author-reported. Setup: the TIFA v1.0 benchmark (4,081 text inputs) with seven text-to-image models and COCO ground-truth images §5.2; the human correlation study uses 800 images from five of those models on 160 prompts §5.1.

Correlation with human faithfulness ratings tab. 2. Spearman's rho and Kendall's tau.

VQA model comparison tab. 3. VQA accuracy uses the human VQA answers as reference. The correlation is Spearman's rho between the TIFA scores from each model and those from human answers.

Benchmark statistics tab. 1.

Benchmark scores by element category tab. 4. VQA accuracy per element category and TIFA score per text source. The paper uses mPLUG as the VQA model unless stated otherwise §5; the appendix table does not repeat the model name.

Findings the paper reports §5.2 §5.3 fig. 7 fig. 8 fig. 9.

• The average TIFA score rises from about 60% to about 75% after DALL-E is released §5.2 fig. 7.
• Stable Diffusion v2.1 has the highest overall score and the highest score in most categories app. B tab. 4.
• Models do well on material, animal/human, color and location. They do poorly on shape, counting and spatial relations. "Other" (abstract-art notions) is also low §5.3 fig. 8.
• COCO captions score higher than free-form prompts; the authors attribute this to real images matching COCO captions §5.3.
• Ground-truth COCO images do not score 100%, which the authors attribute to caption noise and VQA errors §5.3.
• Average score drops when a prompt has more than five entities (objects, animals/humans, food), while real COCO images stay flat §5.3 fig. 9.

VQA errors §5.4 fig. 11. Using human VQA answers, the paper attributes most wrong answers to the generated image rather than the VQA model. For mPLUG, under 25% of errors come from the VQA model §5.4.

Sensitivity to the VQA model §5.4 fig. 10. Scores from different VQA models follow the same trend across generators, and pairwise Spearman correlations between the VQA models are above 0.6 §5.4.

Paper vs code

Code: official repository https://github.com/Yushi-Hu/tifa, Apache-2.0, at commit 8e37050f44ab4ee681e4251f3c8a9d4cc86e901e. The paper's LaTeX source has no repository link; the project page linked from the paper does, and the repository's owner name matches the first author. The repository is cloned at raw/code/2303_11897.

Differences between the paper and the code:

• Filter threshold. The paper keeps a tuple when the free-form word F1 is greater than 0.7 §3.2. The code sets FREE_FORM_THRESHOLD = 0.6 and drops tuples with F1 at or below it question_filter.py question_filter.py.
• Yes/no exemption. The paper applies the F1 check to all tuples §3.2. The code skips the F1 check when the gold answer is "yes" or "no" question_filter.py.
• Duplicate questions. The code drops a repeated question text for the same prompt. The paper does not mention this question_filter.py.
• Numeric answers. The code converts a word-form free-form answer to digits (for example "three" to "3") when the gold answer is numeric question_filter.py. The paper does not mention this.
• Free-form normalisation. The code lower-cases the UnifiedQA answer and strips non-alphanumeric characters unifiedqa.py. The paper does not mention this.
• Question LM. The paper uses code-davinci-002 at temperature 0 §3.1. The code calls gpt-3.5-turbo with temperature 0 and a 700-token limit openai_api.py question_gen.py. The README says code-davinci-002 is deprecated and that the new code's questions "will be slightly different" from TIFA v1.0 [README.md#L141].
• In-context examples. The paper uses 15 examples §3.1. The code prompt has 12 worked examples question_gen.py. The README says the count was reduced because of the shorter context window [README.md#L141].
• Questions per element. The paper writes two questions per element §3.1. The code prompt asks for "one or two" questions per description question_gen.py.
• Prompt text. The paper's appendix prompt says "generate multiple-choice questions" app. F. The code prompt is worded differently, and its examples are different question_gen.py. Do not reproduce the paper's prompt and expect the same questions.
• Element categories. The paper lists 12 categories with no separate human category §3.1. The code uses 13 categories, with "human" as a separate label, and merges "human" and "animal" into "animal/human" question_gen.py question_gen.py.
• SBERT step. The paper maps every free-form answer to a choice with SBERT §3.3. The code first checks for an exact match with a choice and uses SBERT only when that fails vqa_models.py. The SBERT checkpoint is all-mpnet-base-v2 mc_sbert.py vqa_models.py.
• Answer format for BLIP-2. The code passes the choices into the BLIP-2 prompt and does not use SBERT for it vqa_models.py vqa_models.py. This matches the paper's description §3.3.
• Scoring. The benchmark score is the mean over prompts of each prompt's mean accuracy, and the code also reports the standard deviation over prompts tifa_score.py. The paper's description agrees §4 eq. 1.
• Checkpoint for mPLUG. The paper says mPLUG-large. The code uses the COCO-VQA fine-tuned checkpoint vqa_models.py. The paper does not say which checkpoint produced the tables.

Pitfalls

• Threshold and filter. The paper and the code disagree on the F1 cut-off (0.7 vs 0.6) and on whether yes/no tuples skip the F1 check. Choose one and record it; the released benchmark was built with the paper's pipeline, not the released code.
• Regenerating questions changes the benchmark. The released TIFA v1.0 tuples were produced by GPT-3 and are fixed. Questions generated today with another LM will differ in wording, categories and count, so scores are not directly comparable. The paper says the same §4.3 app. D.
• Answer matching is brittle. Choice matching uses exact string equality first in the code vqa_models.py, then embedding similarity. The paper gives no threshold for the similarity step. Free-form answers such as "Yes." or "no," with punctuation, fall through to SBERT.
• Eq. 1 does not describe every model. Eq. 1 takes an argmax over the choices §3. Four of the five models in the paper generate free-form text, so the argmax step is in fact an SBERT match §3.3.
• Humans have a fifth option. Human VQA answers include "None of the above" §5.4 app. C. Model answers do not. The human accuracy in tab. 3 is therefore not a like-for-like upper bound for the model accuracies.
• Hidden cost. Each prompt needs one LM call to create questions, and each question needs one VQA inference per image. The paper's mPLUG figure is about 0.25 s per question on an A40, so the 6.3 questions of an average prompt take about 1.6 s per image (6.3 × 0.25 s) §5.4 tab. 1. The discussion calls this too slow for large-scale data filtering §6.
• Numbers are author-reported on one benchmark. Correlations come from 800 images and 160 prompts, with 2 annotators per image. Captioning baselines use BLIP-2 captions; the paper does not test other captioners §5.1.
• Not stated. The paper does not give the hardware for question generation, the VQA decoding settings, the SBERT checkpoint, the mPLUG checkpoint, or how the 15 examples were chosen beyond being hand-written §3.1 §3.3.
• Known limits. The authors state that TIFA assumes a reasonably good VQA model, which may fail on anime and abstract art §6.
• Ground-truth scores below 100%. The paper attributes this to caption noise and VQA errors. Do not treat 100% as a target: even real COCO images score below it §5.3.

Lineage

• Builds on: GPT-3 (Brown et al.) for in-context question generation §3.1; UnifiedQA for filtering and free-form answers §3.2; Sentence-BERT, [/papers/1908.10084], for the choice-matching step §3.3. Element extraction and categories follow Changpinyo et al. 2022 and Krishna et al. 2019 §3.1.
• VQA models used: GIT (arXiv:2205.14100, bibliography entry only, no card), ViLT, [/papers/2202.03052] (OFA), mPLUG and [/papers/2301.12597] (BLIP-2) §3.3.
• Compared against: CLIPScore (Hessel et al. 2021), caption-based metrics BLEU-4, ROUGE-L, METEOR and SPICE with a BLIP-2 captioner, and DALL-Eval §1 §5.1. The paper also cites DALL-Eval (Cho et al.) for object-detection-based evaluation §1.
• Benchmarked generators: AttnGAN, X-LXMERT, VQ-Diffusion, minDALL-E and Stable Diffusion v1.1, v1.5 and v2.1 §5 app. E. The paper notes a score jump after DALL-E §5.2; that paper's card is [/papers/2102.12092].
• Superseded by: not known from content already in this wiki.