Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

In plain words

Some questions a student answers from school, such as a question about a diagram of a food web, need more than the text on the page. The answerer has to read the picture, remember a rule from class and string the steps together. This paper builds a set of about 21,000 such multiple-choice questions from grade-school science, social studies and language lessons. Most questions come with a short lesson and a worked explanation.

The authors then ask whether a language model does better if it is trained or prompted to write out the reasoning along with its answer. In their setup the model writes the answer first, then the lesson and the explanation. Models that do this score a few points higher than the same models without it. The gains are modest, the dataset is licensed for non-commercial use only, and the generated explanations are not always correct.

TL;DR

• Introduces ScienceQA: 21,208 multiple-choice questions with optional image or text context, plus lectures (general knowledge) and explanations (specific reasons) §3 tab. 1.
• Chain-of-thought (CoT) output format: answer, then lecture, then explanation. It improves fine-tuned UnifiedQA_\text{BASE} by 3.99 points and 2-shot GPT-3 by 1.20 points §4.2 tab. 3.
• Author-reported best result: 75.17% for 2-shot GPT-3 CoT on the 4,241-question test split; human accuracy is 88.40% tab. 3.
• Dataset is CC BY-NC-SA 4.0; the code is MIT. The paper's own CC BY 4.0 license does not cover the dataset [checklist] LICENSE-DATA.

Problem

Science question answering needs multimodal inputs, domain knowledge and multi-step reasoning. Before this work, most science datasets lacked annotated explanations, and those that had them were text-only or small §1 §2. The authors also observe that existing vision-language baselines fail on these questions and do not produce correct explanations §1.

Method

Dataset. Each item has a question, 2 to 5 options, optional text and image context, a correct answer, and optionally a lecture and an explanation. Items are split 60:20:20 at random into train, validation and test, giving 12,726, 4,241 and 4,241 items §3 §3.1 app. A.3 tab. 1. The lecture gives background for a class of problems; the explanation gives the reason for this answer §3.

Baselines. Three families are benchmarked §4.1:

1. Heuristics: random choice, averaged over three trials on the whole test set, and human accuracy from Mechanical Turk workers §4.1.
2. VQA models trained on the train split, fed the question, context and options as text and the image as visual input §4.1 tab. 3.
3. Language models in zero-shot and few-shot form. Images are turned into captions, which are then used as text context §4.1.

Chain-of-thought formats. The paper uses one answer-first layout for both model families §4.2 fig. 5:

1. UnifiedQA_\text{BASE} is fine-tuned to generate the answer, followed by the lecture and the explanation. Other formats are compared in the table (QCM→A, QCM→AE, QCM→ALE, with Q = question, C = context, M = options) §4.2 tab. 3.
2. GPT-3 (text-davinci-002) is prompted with n demonstrations in the same layout and then asked to produce "The answer is X. BECAUSE: lecture explanation" for the test item §4.2 fig. 5. The paper's standard setting is 2 demonstrations §5.2 §5.4.

The idea is to replace the usual "explain, then answer" order (the original CoT form [wei2022chain]) with "answer, then explain". The paper gives the reason: a model that must write a long explanation first is more likely to stop early or run out of tokens before it gives the answer §5.4 tab. 7.

# Build a QCM->ALE prompt for one test item  [§4.2] [fig. 5]
# Input:  shot_ids (n training ids), test_id
# Output: one prompt string; the model writes after the final "Answer:"

def build_prompt(problems, shot_ids, test_id):
    examples = []
    # Demonstrations come from the training split  [§5.1]
    # [code: models/base_prompt.py#L101-L117]
    for shot_id in shot_ids:
        # Each demonstration shows the answer, then lecture, then explanation
        examples.append(format_item(problems[shot_id], with_answer=True))
    # The test item stops at "Answer:" so the model writes the rest  [fig. 5]
    examples.append(format_item(problems[test_id], with_answer=False))
    # Items are separated by a blank line  [code: models/base_prompt.py#L138]
    return "\n\n".join(examples)


# Lay out one item as Question, Context, Options, then Answer  [fig. 5]
def format_item(item, with_answer):
    # Context is the text hint; the paper states an empty string when there
    # is no context  [app. B.1]
    lines = [
        "Question: " + item.question,
        "Context: " + item.context,
        "Options: " + options_text(item.choices),  # "(A) x (B) y ..."
    ]
    if not with_answer:
        lines.append("Answer:")
        return "\n".join(lines)
    # Answer first, then the lecture and explanation after "BECAUSE:"
    # [fig. 5] [§4.2]; newlines inside them are escaped as "\n"  [app. B.1]
    lines.append(
        "Answer: The answer is " + item.answer_letter
        + ". BECAUSE: " + item.lecture + " " + item.solution
    )
    return "\n".join(lines)

# Read the answer letter from one greedy GPT-3 completion  [§5.1] [app. B.1]
# Input:  prompt string
# Output: letter "A".."E", or None when the text has no clear answer

def predict_letter(prompt):
    # Temperature 0, top-p 1, no penalties  [app. B.1]
    # The completion stops at the first newline  [code: models/run_gpt3.py#L49]
    text = complete(prompt, temperature=0.0, max_tokens=512, stop="\n")
    # Look for the phrase "The answer is X."  [code: models/run_gpt3.py#L53]
    matches = find_all("The answer is ([A-Z]).", text)
    if len(matches) == 1:
        return matches[0]
    return None

# Turn a letter into an option index, guessing when the letter is unusable
# [code: models/run_gpt3.py#L63-L70]. The paper does not describe this step.
# Input:  letter or None, number of options
# Output: option index in 0 .. num_options - 1

def resolve_index(letter, num_options):
    options = ["A", "B", "C", "D", "E"]
    if letter in options[:num_options]:
        return options.index(letter)
    # Uniform random guess among the real options
    return random.randrange(num_options)

Explanation metrics. Generated text is scored against the gold lecture plus gold explanation, after the "The answer is X. BECAUSE: " prefix is removed evaluations.py evaluations.py. BLEU-1 and BLEU-4, ROUGE-L and sentence-embedding cosine similarity are reported §5.3 tab. 4. Human raters judge each explanation as relevant, correct and complete, and a majority of three workers decides app. B.3.

Hyperparameters

Values in the paper come first; code-only values are marked as such.

Results

Author-reported. Setup: 4,241 test items; accuracy in percent; GPT-3 runs through the API with text-davinci-002 §5.1. The first three accuracy columns split by subject, the next three by context type, and the last three by grade band and overall average tab. 3. Bold and underline marks from the paper are dropped. Deltas in parentheses are absolute points against the matching row.

Format key: Q question, C context, C_T text context, C_I image context, M options, A answer, AE answer and explanation, ALE answer, lecture and explanation tab. 3.

Explanation quality, author-reported, from tab. 4:

Ablations and upper bounds, author-reported:

• Prompt layout with 4 demonstrations, GPT-3 CoT, on accuracy fig. 7a: the paper reports QCM→ALE as the highest mean and lowest variance; QCM→AE has the largest variance.
• Number of demonstrations fig. 7b: QCM→ALE peaks at 2 demonstrations, then declines as more are added.
• Dynamic demonstration selection tab. 5: QCM→ALE with examples from the same topic, category or skill scores 75.15, 74.58 and 75.10, against 75.17 for random sampling.
• Upper bound with gold lecture or explanation in the test prompt tab. 6: QCME*→A 94.03 (+18.86); QCMLE*→A 94.13 (+18.96) over QCM→ALE 75.17. These use gold answers' explanations, so they are oracle numbers, not a method.
• Output order on 1,000 test items tab. 7: QCM→ALE 73.6; QCM→LA 60.6; QCM→EA 56.0; QCM→LEA 55.4; QCM→ELA 51.5.
• Fewer training items fig. 8: UnifiedQA with CoT reaches the accuracy of plain UnifiedQA with only 40% of the training data §1 §5.4.

Paper vs code

Official repository: lupantech/ScienceQA, owned by first author Pan Lu, at commit 2cbf8318e07b9ece895bb2ae605e71e38d623264. It holds the GPT-3 runner, prompt builder, captioner and evaluation scripts. It does not hold the VQA baseline or UnifiedQA training code. Discrepancies:

• Caption length. The paper says the captioner's maximum caption length is 16 app. B.1. The code uses max_length = 64 generate_caption.py, and the released data/captions.json records max_length 64 captions.json.
• Empty context. The paper says an empty string when there is no context app. B.1. The code writes "N/A" base_prompt.py.
• Captions in the GPT-3 run. Captions are used only with --use_caption, which defaults to off run_gpt3.py. The provided run command does not pass it exp1.sh. The released result file's args has no use_caption key final_test_QCM-ALE_2_seed_3.json, so whether captions reached the 75.17% run cannot be read from the repository. The paper does not say either.
• Demonstration choice. The paper lists fixed trial IDs app. B.1. The code draws demonstrations with Python's random.sample after random.seed(seed) run_gpt3.py run_gpt3.py. The released run's IDs (6493, 16241) are the first two of trial 1, which suggests the seed reproduces the paper's draws, but this is inferred from the outputs, not stated.
• Unparseable outputs. The code guesses at random when no "The answer is X." phrase is found run_gpt3.py run_gpt3.py. The paper does not mention this fallback, and the number of such cases is not reported.
• Similarity model. The paper names Sentence-BERT §5.3. The code loads sentence-transformers/all-MiniLM-L6-v2 evaluate_explaination.py.
• Explanation reference. The code uses lecture plus solution as the BLEU, ROUGE and similarity reference evaluations.py. The paper does not say what the reference is.
• Script name. models/paper.sh calls run_gpt3_v2.py, which is not in the repository paper.sh. The working command is in exp1.sh, which calls run_gpt3.py exp1.sh.
• Shot-ID flag. --shot_qids is declared with type=list run_gpt3.py. Given on the command line, argparse splits the string into single characters, so the flag does not work as written. The IDs have to be set in code.
• Test split default. The runner defaults to val run_gpt3.py; exp1.sh passes test. The paper's 75.17% is on test.

No other differences were found in the GPT-3 prompt format, sampling settings or answer-first layout, which match §4.2 and fig. 5.

Pitfalls

• Dataset license. The paper's CC BY 4.0 covers the paper. The dataset is CC BY-NC-SA 4.0 and "for non-commercial research purposes only", with the copyright held by IXL [checklist] app. A.1. The project page and LICENSE-DATA say the same LICENSE-DATA. Do not reuse the dataset commercially. This card publishes only the paper's aggregate numbers.
• Improvements are absolute points. "3.99%" and "1.20%" are differences between accuracies, not relative gains tab. 3. Against zero-shot GPT-3 (74.04), the best 2-shot CoT result (75.17) is 1.13 points higher (computed from tab. 3); the 1.20-point headline is against 2-shot GPT-3 without CoT.
• Oracle numbers. The 94% upper bounds put gold explanations in the prompt §5.4 tab. 6. They show that explanation quality matters, not that a model can reach that score.
• Output order. Writing the lecture or explanation before the answer costs many points tab. 7, and the code stops generation at a newline run_gpt3.py. A parser that expects the answer on the same line as the explanation will fail if the model inserts a newline.
• Explanation metrics. Automatic scores track surface and embedding similarity to the gold text. The paper itself says these are partial §5.3. Human ratings are majority votes of three workers, and the study pays about $0.5 per HIT for 15 questions app. B.2 app. B.3.
• Dataset inconsistency. §3.1 says 91.3% of questions have an explanation; tab. 1 gives 19,202 of 21,208 (90.5%). Use the table.
• Caption quality. The paper says captions "lack fine-grained semantics" and fail on diagrams app. B.4. Image-context results depend on this text bridge.
• Human baseline. The AMT study uses qualification by exam (4 of 5 correct) and three workers per set app. B.2. Human 88.40% is an author-run number, not a reference ceiling.
• Reproducing 75.17. Needs text-davinci-002 through the OpenAI API (the paper's engine; its current availability is not stated here), the same two demonstration IDs, seed 3, the test split of 4,241 items, greedy decoding, and the answer-first QCM→ALE format. The repository asserts 4,241 items in its accuracy script evaluate_acc.py.
• Missing details. UnifiedQA learning rate, optimizer and exact input string are not stated §5.1 §4.1. The VQA baselines' optimizer is not stated.

Lineage

• Builds on: chain-of-thought prompting [wei2022chain] (2201.11903); the paper reverses its explain-then-answer order §4.2. UnifiedQA [khashabi2020unifiedqa] and GPT-3 [chen2020big] are the base models; neither has a card here.
• Compared against: VQA models Top-Down, BAN, DFAF, ViLT, Patch-TRM and VisualBERT tab. 3; the image-caption baselines use [dosovitskiy2020image] and [radford2019language] §4.1.
• Used by later work: the README of the official repository lists Multimodal-CoT, arXiv:2302.00923 (2302.00923), among follow-ups that use ScienceQA README.md.