Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models

In plain words

Language models now accept very long prompts, but it is not clear that they handle a long prompt as well as a short one. A test that compares a long benchmark with a short one is unfair, because the questions also differ.

The authors keep the question fixed and change only how much filler text surrounds the two facts it needs. They then ask the same models the same question at each length. Accuracy drops as the filler grows, even when the filler is copies of the facts themselves. The models also make specific mistakes more often in long prompts: they refuse to answer, they lean towards "False", and in step-by-step answers they often skip the facts they need. Predicting the next word, a common yardstick, does not track this drop.

The practical lesson is that a model should be judged at the lengths it claims to support, not at one length.

TL;DR

• FLenQA has three True/False reasoning tasks. Each question needs two facts that are hidden inside padding of 250 to 3000 tokens §3 §3.3.
• Five LLMs get less accurate as the input grows. The drop appears for duplicated, similar and book padding, and for every placement of the facts §4.1 §4.2 fig. 6 fig. 7 fig. 8.
• Author-reported: average accuracy falls from 0.92 to 0.68 at about 3000 tokens §1.
• Next-word accuracy is negatively correlated with reasoning accuracy across lengths, so it cannot stand in for long-input evaluation §5 fig. 9.
• The authors conclude that a model should be judged at every length it claims to support, not at one length §9.

Problem

Benchmarks of long inputs usually change the task along with the length. The authors say this makes it hard to tell whether a drop comes from the length or from a harder task §1. Their goal is to hold the task fixed and isolate length §1 §2. Their requirements: each sample needs several relevant spans that are all used jointly, the question must rely on text the model has not seen in training, the padding must not contradict the facts, the facts must be placeable at chosen positions, and the input must be cohesive at least by paragraph §2.

The paper also notes that perplexity or next-word accuracy is often used as evidence that models use their whole input, and that these measures do not always track downstream performance §5.

Method

The method has two parts: a dataset whose length can be set freely (FLenQA), and a behavioural protocol that asks the same question at each length §3 §4. Because it only needs text in and text out, it applies to closed and open models alike §2.

Base instances §3.1. Each instance has an optional prefix, two key paragraphs and an optional suffix. Each key paragraph starts with a key sentence that the answer needs. The key sentences are written as simple facts and then expanded into paragraphs with GPT-4, which was told not to add information; the authors checked the expansions by hand §3.1. Key paragraphs are truncated to about 125 tokens on average app. A. The parts are joined by newlines §3.1.

Tasks §3.2 app. A. Each task has 100 base instances and is balanced between True and False §3.

• Monotone relations (MonoRel). Key sentences compare two people on a monotone scale. The question asks about a pair that is not compared directly, so both sentences are needed. Names come from the Faker library and relations from a hand-written list §3.2.
• People in Rooms (PIR). One key sentence places a person in a named room and another gives the room a property. The question asks whether the person is in a room with that property. Rooms and properties were chosen to be mutually exclusive §3.2.
• Simplified Ruletaker. Each sample has two facts and one rule, and the question asks whether a statement follows. The authors generated new samples of this shape; they did not use the original data as is §3.2 app. A.

Length variants §3.3. Each base instance is padded to targets of 250, 500, 1000, 2000 and 3000 tokens. A sample counts as length N if its GPT-4 tokenizer count is in (N-70, N+70) §3.3 (footnote). The actual mean lengths are in the table under Hyperparameters app. A.

Padding sources §3.3.1.

• Duplicate. Copies of the two key paragraphs, alternating, until the target is reached. Used only in the no-irrelevant-text experiment §3.3.1 §4.1.
• Similar. Paragraphs taken from other base instances of the same task. Any paragraph that shares an entity with the key paragraphs is excluded, so the padding does not contradict them §3.3.1.
• Different. A continuous passage from BookCorpus, with each key paragraph inserted at a sentence boundary §3.3.1.

Placement §3.3.2. Four placements: key paragraphs first; split evenly before and after; last; or randomly dispersed, with padding between the two paragraphs as well §3.3.2 fig. 5.

The construction, written as pseudocode from the paper's description:

# Build one prompt at one length  [§3.1] [§3.3] [§3.3.1] [§3.3.2]
# Input:  instance with prefix, two key paragraphs, suffix
# Input:  target in {250, 500, 1000, 2000, 3000} tokens
# Input:  source in {duplicate, similar, different}
# Input:  placement in {first, middle, last, random}
# Output: text of the padded input (question appended by the prompt)

def build_input(instance, target, source, placement):
    # Start from the two key paragraphs, which must both stay  [§3.1]
    key_paragraphs = [instance.key_1, instance.key_2]
    # Padding fills the rest of the target, counted with GPT-4 tokens  [§3.3]
    budget = target - count_tokens(key_paragraphs)
    # Draw padding from the chosen source  [§3.3.1]
    padding = draw_padding(source, budget, instance)
    # Insert the key paragraphs by placement rule  [§3.3.2]
    return place(key_paragraphs, padding, placement)

Prompts app. B.1 FLenQA analysis.ipynb. Each task has a normal and a chain-of-thought prompt. The prompt wraps the padded input and the question, and asks for a True or False answer. The chain-of-thought variant adds a closing request to work the answer out step by step before giving it, worded as in app. B.1 §6.

Answer scoring app. B.3 FLenQA analysis.ipynb. The reply is lowercased. If "true" and "false" both appear, the last occurrence decides. If neither appears, the reply counts as an other or refused answer. Other or refused answers are scored as wrong §4 app. B.3.

# Score a reply as true, false or refused  [app. B.3]
# Input:  reply text from the model
# Output: "true", "false" or "other/refused"

def score_reply(reply):
    # Lowercase the reply  [app. B.3]
    text = reply.lower()
    # Find the last occurrence of each word  [app. B.3]
    last_true = text.rfind("true")
    last_false = text.rfind("false")
    # The later word wins; neither word means refused  [app. B.3]
    if last_true > last_false:
        return "true"
    if last_false > last_true:
        return "false"
    return "other/refused"

Chain-of-thought coverage app. B.4 §7. Each key sentence is counted as covered if it appears in the reply, ignoring case. Full coverage means both key sentences appear app. B.4.

Experiments. Section 4.1 covers duplicate padding with no irrelevant text, adjacent key paragraphs at each placement, and non-adjacent key paragraphs. Section 4.2 compares similar and different padding on non-adjacent paragraphs. Section 5 measures next-word accuracy on the same samples without questions. Section 6 covers chain-of-thought. Section 7 reports four failure modes §4 §5 §6 §7.

Hyperparameters

The paper has no numbered equations. The only formula in the text is a Pearson correlation reported in a footnote §5 (footnote).

Results

Author-reported. Setup: FLenQA (three tasks of 100 base instances each), five LLMs, True/False accuracy, temperature 0 §4 app. B.2. Figure captions give the sample count per point. The paper does not report confidence intervals or repeated runs in the text, and model sizes are not stated §4.

Headline numbers:

The 0.92 figure is an average over all models and tasks; the paper does not name the exact short-input point it is compared with §1.

Table 1 tab. 1 — minimal length accuracy (250 tokens, no irrelevant text).

Ruletaker* is the simplified version §3.2. The table caption says CoT improves performance across almost all tasks and models tab. 1.

Length and placement §4.1 fig. 1 fig. 6 fig. 7:

• With duplicated key paragraphs, which removes the search problem entirely, accuracy still falls as length grows for every model. GPT-4 and GPT-3.5 are less affected when the added text is relevant fig. 6.
• With adjacent key paragraphs, accuracy drops significantly beyond about 500 tokens, whatever the placement fig. 7 §4.1. For most models, adjacent key paragraphs score higher than separated ones §4.1.
• When the key paragraphs appear last, accuracy is often highest, which the authors read as a recency effect. Some models do worse with the paragraphs in the middle §4.1 fig. 7.
• With non-adjacent key paragraphs, the drop is very large as length grows fig. 1 §4.1. The text gives no number for this case.

Padding type §4.2 fig. 8: non-adjacent key paragraphs with similar padding versus book padding. The drop is larger with book padding for most models, the opposite of what the authors expected §4.2.

Next-word prediction §5 fig. 9: next-word accuracy rises with length, while reasoning accuracy falls. Across the samples the two correlate negatively. Gemini Pro is left out because it returned empty replies fig. 9 §5.

Chain-of-thought §6 app. C: CoT raises accuracy at most lengths for most models, but the gain stays roughly the same across lengths. GPT-4 is the exception, with a gap that grows with length. For Gemini Pro, CoT lowers accuracy at longer lengths §6.

Failure modes §7 fig. 10 fig. 11 fig. 12:

• Refusals ("not enough information") grow with length §7 fig. 10.
• Some models favour "False" as length grows §7 fig. 10.
• In CoT, most models give the answer before the reasoning, more so at longer lengths §7 fig. 11.
• In CoT, the fraction of the key facts restated in the steps falls with length for most models §7 fig. 12.

Paper vs code

The official repository is https://github.com/alonj/Same-Task-More-Tokens, linked in the paper's first footnote §1. Its owner is an author's account, so it is the official code. Commit 5a1691c90dcd2b0901f3cd8643d1df9c97778dcd, licence Apache-2.0. The repository has a README, the Apache-2.0 LICENSE, a notebook and FLenQA.zip. The notebook was read in full. The zip could not be opened in this environment, so the dataset generator and the released data were not checked.

Differences and gaps:

• Ruletaker coverage uses the wrong field. The paper defines coverage by the key sentences app. B.4. The notebook counts matches against the statement field for Simplified Ruletaker, not the two key facts FLenQA analysis.ipynb.
• Answer scoring adds rules. The notebook maps "not true" to "false" before scoring, and it removes the articles "a", "an" and "the". The paper describes only the last-occurrence rule app. B.3 FLenQA analysis.ipynb.
• Early-response window. The paper says "before the reasoning steps" with no window. The notebook checks only the first 10 characters of the normalised reply FLenQA analysis.ipynb §7.
• GPT-3.5 call settings. The notebook sets temperature 0, top-p 0, 400 max output tokens and zero penalties FLenQA analysis.ipynb [code: FLenQA analysis.ipynb#cell-10]. The paper gives only temperature and top-p app. B.2.
• Significance tests. The notebook's Fisher tests use a one-sided "greater" alternative and cover GPT-3.5 only FLenQA analysis.ipynb. The paper does not say which alternative it used or which model it used for the odds ratios §7.
• Prompt quotes. The PIR chain-of-thought prompt in the notebook uses curly quotes around "true" and "false"; the appendix uses straight quotes. Wording is otherwise the same FLenQA analysis.ipynb app. B.1.
• Other models not in repo. Only the GPT-3.5 call is in the notebook. The Gemini, Mistral and Mixtral calls are not in the repository [code: FLenQA analysis.ipynb#cell-8].

Pitfalls

• Punctuation is not stripped. The notebook's normaliser tries to remove punctuation with one replace call on the joined punctuation characters. That string does not occur in the text, so the call does nothing [code: FLenQA analysis.ipynb#cell-9]. Coverage matching therefore keeps punctuation from the key sentences. Whether that matters depends on the reply's wording.
• Substring matching. Scoring looks for the substrings "true" and "false". The notebook's rule would therefore read "untrue" as "true" [code: FLenQA analysis.ipynb#cell-9].
• Ruletaker statement shape. The data description says statement is a single string. If so, the coverage loop in the notebook iterates over characters, not sentences. Check the data before reusing that loop [code: FLenQA analysis.ipynb#cell-9] FLenQA analysis.ipynb.
• Refusals count as wrong. Refusals grow with length, so part of the drop is instruction following, not reasoning. Report refusals separately if you reproduce this §4 §7.
• Duplicate padding is not a fix. Even with no irrelevant text, accuracy falls with length. Removing distractors will not remove the effect §4.1 fig. 6.
• Book padding is harder than similar padding for most models, the reverse of the authors' expectation §4.2 fig. 8.
• Next-word accuracy is a poor proxy. The number of points behind the correlation is not stated in the text §5 (footnote).
• Closed-model versions drift. The GPT-4 and GPT-3.5 versions are named, but the Gemini Pro, Mistral Medium and Mixtral 8x7B versions are not. Results from hosted APIs can change over time §1 fn. 2 §4.
• Gemini safety filters were disabled to get non-blank answers app. B.2. Expect different behaviour if you keep the default filters.
• Limited scope. The paper tests five lengths, three simple tasks, and no distance between the key paragraphs [Limitations]. The authors say stronger models may degrade at shorter lengths on harder tasks [Limitations].
• Possible bias from GPT-4 expansion. The key paragraphs were expanded by GPT-4, which may have added some bias. The padding text was not produced by GPT-4 [Limitations] §3.1.
• Dataset licence not verified. The paper says the dataset is released, and the repository links it through Hugging Face. The dataset's licence was not checked here.
• Chain-of-thought effects differ. Gains are mostly steady, and Gemini Pro loses ground at longer lengths. Do not assume CoT fixes long inputs §6.

Lineage

• Builds on: the input-intervention approach of Holtzman et al. (2023) §2; chain-of-thought prompting by Kojima et al. (2022) and Wei et al. (2022) §1 §6, see the Wei et al. card; the elicitation wording from Zhou et al. (2022) §6; Ruletaker of Clark et al. (2021) for the simplified task §3.2; the bAbI tasks of Weston et al. (2016) for PIR §3.2.
• Compared against: the position and length findings of Liu et al. (2023) on extractive QA §4.1; the effect of irrelevant text on GSM8K from Shi et al. (2023) §4.1; the long-input benchmarks ZeroSCROLLS, L-Eval, LongBench and LooGLE §8.
• Superseded by: not stated.