Large Language Models Can Be Easily Distracted by Irrelevant Context

In plain words

Maths word problems are often solved by a language model that writes out its steps. This paper asks what happens when the problem also contains a sentence that has nothing to do with the answer, such as an unrelated detail about a person in the story. The authors built a test set by adding one such sentence to easy school maths problems and found that the models' answers often change, even when they solved the original problem correctly.

Three things helped. Showing the model an example that already ignores such a sentence helped. Adding a plain instruction to ignore irrelevant information also helped. The largest gain came from asking the model many times and taking the most common answer, which is also known as voting. Even with these fixes, most of the original problems were not solved reliably once different distractors were added.

The point for an engineer is that a prompt that scores well on clean benchmarks can still be fragile when the input carries extra, unrelated text.

TL;DR

The paper introduces GSM-IC, which adds one irrelevant sentence to each of 100 easy GSM8K base problems and yields 58,052 variants §3.1 app. A. It measures micro, macro and normalised accuracy §3.2. On code-davinci-002, every tested prompting method loses accuracy, and macro accuracy never exceeds 45% tab. 3. Self-consistency (20 samples, majority vote) gives the largest gain, and an instruction to ignore irrelevant information also helps §5.1 §5.3. Relevant if you deploy prompts on inputs that may contain unrelated context.

Problem

Benchmarks usually contain only information that bears on the answer, but real inputs often carry related and unrelated details §1. The paper assumes that a sentence which does not change the standard solution should not change a correct model's answer §1 §3.1. It measures this distractibility with accuracy on the variants and on the base problems §3.2.

Method

The paper has two parts: a dataset construction procedure §3.1, and an evaluation of prompting techniques and mitigations on that dataset §4 §5.

1. Choose base problems. Sample a 1,000-problem development pool from GSM8K's training split. Keep 100 that at least one prompting technique solves correctly. Each base problem needs two to seven reasoning steps, and 60 need two steps §3.1; the step distribution is in app. A.
2. Create variants. Add one irrelevant sentence before the question. Each sentence is produced from a template and is either in-topic (close to the problem) or off-topic §3.1 tab. 1 app. A. Three factors are varied orthogonally §3.1 fig. 1:
   • topic: in-topic or off-topic;
   • role-name overlap: the filled name either reuses a role from the problem (for example "X's father") or uses a name not in the problem;
   • number range: the inserted number is in range or out of range.
3. Verify. The authors manually checked that each sentence reads as correct English and does not change the standard solution §3.1. Where a sentence made a pronoun ambiguous, the question was reworded app. A.
4. Evaluate. Score each method with micro, macro and normalised accuracy §3.2 (see Pseudocode below). Methods: chain-of-thought (CoT), zero-shot CoT, least-to-most (LtM) and program prompts §4.1. Self-consistency (SC) is applied on top of these §4.1.
5. Mitigate. Put irrelevant information in the exemplar §4.2. Add an instruction before the exemplars §4.3. Use SC §4.1 §5.1.

Dataset construction, as pseudocode:

# Build one GSM-IC variant from a base problem  [§3.1] [app. A]
# Inputs: base problem text, standard solution, factor settings
# Output: problem text with one irrelevant sentence inserted

def make_variant(problem, solution, in_topic, overlap, in_range):
    # Numbers the sentence may be compared against  [§3.1]
    known_numbers = numbers_in(problem) + numbers_in(solution)
    # Pick a template for the chosen topic kind  [app. A] [tab. 1]
    template = pick_template(problem, in_topic)
    # Fill the role blank: reuse a role in the problem, or a fixed name  [app. A]
    role = pick_role(problem, overlap)
    # Fill the number blank in range or out of range  [§3.1] [app. A]
    number = pick_number(known_numbers, in_range)
    sentence = template.format(role=role, number=number)
    # Insert the sentence immediately before the question  [tab. 1]
    return insert_before_question(problem, sentence)

Number rule, as pseudocode:

# A number a is in range if some known number b has 1/10 <= a/b <= 10  [§3.1]
def is_in_range(number, known_numbers):
    return any(
        0.1 <= number / other <= 10
        for other in known_numbers
    )

Hyperparameters

Program prompts use a Python program as the solution, executed to obtain the answer §4.1 app. C.

Results

Author-reported. Setup: GSM-IC-4K; code-davinci-002 unless noted §5.

Base problem accuracy on the 100 base problems, code-davinci-002 tab. 2:

For text-davinci-003 the base accuracy is 80.0 for CoT and 81.0 for LtM tab. 3 caption.

Main results on GSM-IC-4K tab. 3. Columns: micro accuracy and macro accuracy, each split into 2-step and more-than-2-step problems, overall, and normalised ("Norm") by base accuracy.

Breakdown by the factors of the inserted sentence tab. 4. Micro then macro; columns are Topic (In / Off), Role overlap (Yes / No), Number range (In / Out).

Instruction and exemplar effects on GSM8K dev and SVAMP test tab. 5. The table as printed is ambiguous, see Pitfalls.

Complicated prompts, micro accuracy tab. 6: a 4-exemplar CoT prompt scores 66.3 on GSM8K dev (1 exemplar: 60.3) but 69.4 on the more-than-2-step split (1 exemplar: 70.8). With the instruction, the 1-exemplar CoT scores 58.8 on dev, 79.0 on 2-step and 76.0 on more-than-2-step; the 4-exemplar version scores 66.5, 79.2 and 70.6 §5.4.

DROP football split, accuracy tab. 7 (code-davinci-002 / text-davinci-003): CoT 67.4 / 68.2; CoT + Inst. 68.9 / 69.9; LtM 73.4 / 70.2; LtM + Inst. 74.4 / 72.8 §5.5.

Headline numbers, quoted from the text §5.1: the recall of the correct answer within 20 samples is 99.7% for CoT and LtM; the best macro accuracy with any prompting technique is 45%.

Paper vs code

No official method code. The paper's abstract footnote links google-research-datasets/GSM-IC. That repository holds the dataset files and a README, and has no LICENSE file. It contains no prompts, sampling or scoring code. So no code, code_commit or code_license is recorded, and nothing is quoted or cited from it. A GitHub search for "GSM-IC" found no other repository by the authors. The authors' data files were used only to check the dataset size: the 2-step file has 34,220 examples, and the more-than-2-step file has 8,864 (3 steps), 7,392 (4), 5,160 (5), 1,776 (6) and 640 (7), for 58,052 in total, which matches §3.1.

Pitfalls

• Macro accuracy is strict. A base problem counts only if every one of its variants is answered correctly §3.2. The 45% ceiling is for macro accuracy; micro accuracy is much higher tab. 3.
• Normalisation uses base accuracy. Divide by the base accuracy from tab. 2 for code-davinci-002, and by 80.0 (CoT) or 81.0 (LtM) for text-davinci-003 tab. 3 caption.
• Text and table disagree in places.
  • The text says LtM "achieves about double macro accuracy of CoT" on code-davinci-002 §5.1. The table gives 18.0 vs 6.0, which is three times tab. 3.
  • The text says SC improves micro accuracy "by more than 11 percentage points" §5.1. From the table, the gains are 15.7 (CoT, 72.4 to 88.1), 15.9 (LtM, 77.5 to 93.4), 20.2 (Program, 54.4 to 74.6) and 35.3 (0-CoT, 29.0 to 64.3). The paper says 35.5 for 0-CoT. The "more than 11" figure cannot be reproduced from the table, and the paper gives no other basis.
• Dev set naming. §3.1 says the 1,000-problem development set was drawn from the GSM8K training set, while later tables report "GSM8K Dev." tab. 5 tab. 6. Which split the dev numbers come from is not stated.
• Table 5 labels. Two rows are labelled "CoT" and "CoT + Inst." in a way that makes the instruction-without-exemplar and exemplar-without-instruction cases unclear. The values above follow the printed order. Check against a fresh run before relying on them.
• Single exemplar. The main prompts use one exemplar §4.2. The paper compares against a 4-exemplar prompt taken from another work §5.4; the paper's appendix does not list that prompt, so its exemplars are not given here.
• Irrelevant sentence can change the answer by use, not just presence. The paper reports that a model can use the irrelevant number in a calculation, or give a wrong answer even without using it §5.1 fig. 3.
• Cost of SC. The SC results use 20 sampled responses per problem §5. The paper does not report token or time costs. Program prompts also need an external Python execution step §4.1.
• Model availability. The models are OpenAI API models named by the paper. Their current availability was not checked, and the results are not guaranteed to transfer to other models. The paper does not claim transfer §6.
• Ambiguous pronouns are fixed by hand. Some inserted sentences made pronouns ambiguous, so the question was reworded app. A. Reproducing the dataset needs the same kind of manual check.
• Metrics are unnumbered. The micro, macro and normalised formulas are unnumbered in the paper §3.2. Cite them by section.

Pseudocode

Metrics, as pseudocode:

# Micro accuracy: every test problem weighs the same  [§3.2]
def micro_accuracy(method, test_problems):
    # standard(p) is the standard solution of problem p
    correct = [
        method(problem) == standard(problem)
        for problem in test_problems
    ]
    return sum(correct) / len(test_problems)

# Macro accuracy: a base problem counts only if all its variants are right  [§3.2]
def macro_accuracy(method, variants_by_base):
    # variants_by_base maps each base problem to its derived variants
    solved = [
        all(method(v) == standard(v) for v in variants)
        for variants in variants_by_base.values()
    ]
    return sum(solved) / len(variants_by_base)

# Normalised accuracy divides by the base-problem accuracy  [§3.2] [tab. 2]
def normalised(accuracy, base_accuracy):
    return accuracy / base_accuracy

Self-consistency, as pseudocode:

# Self-consistency: sample 20 chains at temperature 0.7, then vote  [§5] [§5.1]
def self_consistent_answer(prompt, problem):
    # Sample 20 independent solutions  [§5]
    answers = [
        parse_answer(sample(prompt, problem, temperature=0.7))
        for _ in range(20)
    ]
    # Majority vote; a tie is broken at random among the top answers  [§5.1]
    return vote_with_random_tiebreak(answers)

Prompt assembly, as pseudocode:

# Few-shot prompt: optional instruction, one exemplar, then the problem  [§4.2] [§4.3]
def build_prompt(exemplar, problem, instructed):
    # The instruction is prepended only in the instructed variant  [§4.3] [app. C]
    header = INSTRUCTION if instructed else ""
    # Exemplar may include the irrelevant sentence or not  [§4.2]
    return header + exemplar + "Q: " + problem + "\nA:"

Lineage

• Builds on: chain-of-thought prompting §4.1; zero-shot CoT, which uses the trigger phrase §4.1; least-to-most prompting §4.1; program-based prompting, which the paper takes from prior work §4.1; and self-consistency §4.1.
• Compared against: the same base methods, with and without the instruction and the irrelevant-context exemplar tab. 3.
• Related prior work on distraction and on DROP is cited in the paper but has no card here §2 §5.5.
• Superseded by: not stated.