Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks

In plain words

Language models have been reported to pass "theory of mind" questions: short stories where you must work out what a character believes, which can differ from what is actually true. This paper takes two such stories and changes details in ways that should not matter to someone who reasons about other people. The character can see inside the box, or was told the truth by a friend, or the story is about a second character who moved the object.

The author finds that the model's answers flip to the wrong belief after these small changes. The paper argues that one such failure should count for more than many passes, and that a model passing these tests does not by itself show it has the ability they are meant to measure.

TL;DR

• A critique with a small probe, not a new method. It takes two vignettes from Kosinski §2 and makes targeted changes that keep the theory-of-mind logic intact §2.1 §2.2.
• The probe reads the probability of candidate completions from GPT-3.5 §2. In most variants the belief completion moves to the wrong candidate, e.g. chocolate 95% for a bag that the character can see into §2.1.1.
• The author concludes that current LLMs have not acquired theory of mind, and that outlying failures should outweigh average success §1 §3.
• Results are author-reported: one model, one prompt per variant, no statistics, and no code (see Pitfalls).

Problem

Kosinski reported that current models can succeed at these tasks at a level comparable to 9-year-old children §1. The paper asks whether that success is robust. It argues that the standard measures of theory of mind should not be set aside, and that a failure on a trivial variant should count against the claim of success §1 §2.

Method

The paper does not propose a model or training procedure. It runs the same set-up as Kosinski: a vignette, then a belief prompt, and a comparison of the probabilities of different completions §2. Variants change the vignette in ways that should keep the belief of the protagonist the same, and the paper checks which completion the model prefers §2.1 §2.2.

1. Probe. Pose the vignette and prompt to the model, then compare the probability of each candidate completion (for example "popcorn" and "chocolate") §2 §2.1.
2. Study 1, unexpected contents. A bag of popcorn is labelled "chocolate". Four variants §2.1.1 §2.1.2 §2.1.3 §2.1.4 fig. 1:
   • 1A: the bag is transparent, so the contents are visible §2.1.1.
   • 1B: the label is unreadable to the character §2.1.2.
   • 1C: a trusted friend told the character the bag holds popcorn §2.1.3.
   • 1D: the character filled the bag and wrote the label §2.1.4.
3. Study 2, unexpected transfer. A cat moves from a basket to a box while the protagonist is away. Four variants §2.2.1 §2.2.2 §2.2.3 §2.2.4 fig. 2:
   • 2A: transparent containers, a glass chest and a plastic box §2.2.1.
   • 2B: "on" instead of "in", and the protagonist looks around the room §2.2.2.
   • 2C: a call beforehand tells the protagonist about the move §2.2.3.
   • 2D: the questions are about the second character, who moved the cat §2.2.4.
4. Reading the result. A variant counts as a failure when the completion gives the label's content (chocolate) or the old location (basket or chest) as the protagonist's belief. The paper's expectation is that the model should assign the true content, or the new location, after each change §2.1 §2.2.4.

# Probe one vignette with its belief prompts  [§2] [§2.1]
# Input:  model, vignette (str), belief prompts (list of str),
#         candidates (list of str, e.g. "popcorn", "chocolate")
# Output: per prompt, a probability (%) for each candidate

def probe_vignette(model, vignette, belief_prompts, candidates):
    # Same set-up as Kosinski: vignette, then the prompt  [§2]
    # One or two spaces at the join change the result  [§2.1.1]
    results = {}
    for prompt in belief_prompts:
        context = vignette + " " + prompt
        # Probability of each candidate completion  [§2]
        # Normalisation over candidates or whole vocabulary: not stated
        results[prompt] = {
            candidate: completion_probability(model, context, candidate)
            for candidate in candidates
        }
    return results

Hyperparameters

The paper has no training hyperparameters. The settings below are the ones it states for the probe.

Results

Author-reported. Setup: GPT-3.5 (version not stated), one vignette per variant, probabilities in % as printed, no confidence intervals and no repetitions. The baseline rows are Kosinski's numbers, quoted in this paper and not re-measured here §2.1 §2.2.

Study 1, unexpected contents (candidates: popcorn, chocolate):

Study 2, unexpected transfer (candidates: box, basket or chest):

Other reported outcomes without numbers: in 2A, the errors persisted when John was said to look carefully around the room, and an opaque-container variant with the cat's tail sticking out also produced errors §2.2.1. The paper states that the 2D case is the one where the model should have no difficulty, since Mark moved the cat himself §2.2.4.

Paper vs code

No official code. The paper gives no repository or project page, and the arXiv metadata we hold has no comments field. Searched: (1) the paper and metadata for links, none found; (2) GitHub repository search for "tomerullman", which returned one repository, tomeru/tomerullman.github.io, a personal website and not code; (3) GitHub repository search for "trivial alterations theory-of-mind" and "theory of mind alterations LLM", with no results. The materials of Kosinski, which the paper says are public §2, belong to a different paper and are not used here.

Pitfalls

• One model, one version. All results are for GPT-3.5. The paper says other and more powerful models were not tested here, and that less powerful models should fail too §2. Its prediction about future models is an argument, not a measurement §3.
• Probability scoring is not defined. The paper compares probabilities of completions §2 but does not say whether they are normalised over the candidates or over the vocabulary. The printed pairs do not add to 100 (for example 58% + 36% and 14% + 82%), so other tokens take the remaining mass.
• Spacing changes results. A double space before "Sam finds the bag" changed the 1A belief-2 result §2.1.1. An earlier version of the paper reported a flip for that prompt, and the authors attribute it to the double space and withdraw it §2.1.1. Implement the exact strings and the join rule.
• Printed value is ambiguous. The double-space result in 1A reads "P_chocolate = 53%, P_chocolate = 39%" §2.1.1. Both values are labelled chocolate, so the probability for popcorn is not stated.
• Variant 2B changes more than "in" to "on". It also adds that John looks around the room, and the belief prompts are reworded §2.2.2. Do not treat it as a single-word perturbation when comparing with 2A.
• Selection of cases. The paper says it does not report every case it tried §2.1.4. The fraction of tried cases that fail is not reported.
• No repetitions or statistics. Each variant is a single prompt and one vignette in the text, with no seeds, no repeated runs and no confidence intervals §2.1.1–§2.2.4.
• Baselines are quoted. The Kosinski numbers are copied from that work and were not re-measured here §2.1 §2.2. The content-prompt values for study 2 are not given.
• Figures are illustrations. Figures 1 and 2 are generated images that the paper says were not evaluated fig. 1 fig. 2. Use the text of the vignettes as the stimuli.
• "Trivial" is the author's judgement. The paper argues each change keeps the ToM principles, but reports no human baseline, so the claim that people would answer differently is not tested here §2 §3.
• Discussion is argument. The claims that ToM has not emerged and that skepticism should be the default are the author's reading of the results, not tested predictions §3.

Lineage

• Builds on: Kosinski, "Theory of Mind May Have Spontaneously Emerged in Large Language Models", arXiv:2302.02083, whose vignettes and prompts this paper reuses §1 §2. There is no card for it in this wiki.
• Related work named in the paper: Sap et al. (2022), a pessimistic evaluation of theory of mind in neural models §1. No card in this wiki.
• Compared against: the paper compares its variants with Kosinski's baseline numbers only §2.1 §2.2. No other model is tested.
• Superseded by: not stated.