Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs

In plain words

This is an analysis paper, not a new model. The authors ask whether large language models have something like a "theory of mind", the everyday ability to work out what other people feel, want and know. They test GPT-3 on two question-answering sets. One asks about the feelings, intentions and next moves of people in short social situations. The other is a set of tiny stories in which a person leaves a room while someone moves an object, and the model must say where each person will look for it.

GPT-3 does poorly on both. On the social questions it is well below human accuracy. On the object stories it is nearly perfect when asked where the object really is, but much worse when asked where a person thinks it is. Newer models trained with instructions or human feedback do better on the social questions, but they gain little on the mental-state questions. The authors argue that learning from static written text alone may not be enough, and they suggest directions such as socially grounded data and interaction.

TL;DR

• Probe: GPT-3 (Ada, Curie, Davinci) scored by likelihood with k in-context examples, on SocialIQa dev (1954 items) and ToMi dev (1861 items) §2.1 §3.1.
• SocialIQa: best about 55% (Davinci) versus human accuracy more than 30 points higher §2.2.
• ToMi: mental-state questions reach about 60% at best, while factual questions reach 90-100% §3.2 §5.
• Newer models (text-davinci-002/003, gpt-3.5-turbo, gpt-4) are tested zero-shot on 400 items per dataset app. D. GPT-4 with multiple-choice probing reaches 79.3% on SocialIQa, but mental-state accuracy on ToMi stays near 60% app. D.
• Relevant if you evaluate social or theory-of-mind reasoning. Use the released ToMi and SocialIQa data and the probing script, but read the Pitfalls first.

Problem

Social reasoning, including theory of mind (inferring other people's mental states and intents), is needed for NLP systems in social settings §1. The paper asks whether large pretrained language models have it without special training, and whether scale alone will provide it §1 §4. Probing set-ups before this work did not isolate the mental-state part from factual recall, so the authors built a split between factual and mental-state questions §3.

Method

The paper evaluates existing models; it does not train one. It has three parts.

1. SocialIQa probe §2. Use the dev set of SocialIQa (1954 three-way QA items). Each item has a context, a question and three answers. The questions cover nine dimensions from the ATOMIC knowledge graph: six about the protagonist (agent) and three about other participants §2 app. A. The per-dimension counts are in app. A.
2. ToMi probe §3. Use the ToMi dev set (1861 two-way QA items, all generated by the ToMi story generator). Questions are either factual (519 items: where the object was originally, or where it finally is) or mental-state (1342 items: first- and second-order, for true- and false-belief participants) §3 app. B.
3. Probing §2.1 §3.1 app. C. The main setup is k-shot likelihood scoring, following the GPT-3 paper. The k shots are training items of the same reasoning type (ATOMIC dimension for SocialIQa, question type for ToMi). Each candidate answer is scored by its conditional likelihood given the context and question. The prediction is the candidate with the highest score.

# k-shot likelihood probing  [§2.1] [§3.1] [app. C]
# Input:  item (context, question, candidate answers), train pool, k
# Output: index of the predicted candidate

def predict_lm(item, train_pool, k):
    # Choose k training items of the same reasoning type  [§2.1] [§3.1]
    shots = sample_same_type(train_pool, item.type, k)

    # Each shot shows context, question and its gold answer  [app. C]
    prefix = "\n\n".join(shot_with_gold_answer(s) for s in shots)

    # Score each candidate by its conditional likelihood  [app. C]
    scores = []
    for answer in item.candidates:
        # Conditional likelihood of this answer given the prefix and
        # the item's context and question  [app. C]
        text = prefix + "\n\n" + item.context + " " + item.question
        scores.append(conditional_logprob(model, text, answer))

    # The prediction is the candidate with the highest score  [app. C]
    return argmax(scores)

Notes on the method:

• The mental-state questions use a 50/50 split of true-belief and false-belief examples among the k shots §3.1.
• Multiple-choice probing is used only for chat models that cannot score sequences. The item is shown with letters A, B, C (or A, B for ToMi), and the model must output a letter. The paper explains that a system instruction is prepended app. D.
• Other probing variants (QA formats, normalising by the marginal likelihood of each answer) were tried and gave very little difference app. C.
• Section numbers here follow the order of the arXiv v2 LaTeX source (the background text has no heading and sits in §1). The appendices are A to D. Check them against the PDF before citing.

Hyperparameters

Results

Author-reported. Setup: GPT-3 probed through the OpenAI API, k-shot likelihood scoring unless marked as zero-shot chat evaluation. Human accuracy is not restated as a number in the paper; the paper refers to a gap of more than 30 points on SocialIQa §2.2 and about 20 points for the newer models app. D.

Caveat from the paper: the GPT-4 system report says some BIG-Bench test data, which includes the SocialIQa dev set, may have been in GPT-4's training data. The authors say conclusions about near-human performance cannot be drawn without more transparency app. D.

Paper vs code

Code: official repository maartensap/neuralToM-public (owner is the first author), commit 88d4fdde86b0d8195cb5e222503275c06d21f94b. It contains a probing script, example commands, and the SocialIQa dev file, the ToMi train and dev files. It has no LICENSE file, so cite it by path and line only. The repository's README says an apiKeys.py file with an OpenAI key is needed; it is not included. The project page links only the third-party ToMi generator and the data, so this repository was found by a GitHub search of the author's account.

• ToMi shot stratification. The paper says mental-state shots are 50/50 true-belief and false-belief §3.1. The released command stratifies shots by answerMemOrReal instead (memory or reality answer) README.md. The script has no true/false belief stratification option; the only stratification is that column passed on the command line gpt3LMMCprobing.py.
• SocialIQa shot selection. The paper selects shots of the same ATOMIC dimension §2.1. The example SocialIQa command in the repository does not set --examples_of_same_type, so shots are drawn at random from all training items exampleSIQAcommand.sh. The README example does set it README.md.
• Scoring. The paper says the answer with the highest conditional likelihood wins app. C. The code sums the token log-probabilities of the answer, after dropping the first token gpt3LMMCprobing.py. The paper does not say whether this sum is length-normalised.
• Marginal normalisation. The paper says normalising by the answer's marginal likelihood changed little app. C. The flag --normalize_uncond_prob is parsed but never used gpt3LMMCprobing.py.
• Capitalisation. The paper normalises capitalisation app. A. The flag --dont_normalize_answer_capitalization is parsed but never used gpt3LMMCprobing.py.
• Training file size. The paper says 5190 training stories remain after filtering app. B. ToMi.trn.csv has 5994 data rows by line count (5995 lines including the header) ToMi.trn.csv. Whether those rows are stories and whether they were filtered could not be confirmed from the file alone.
• Dev file. ToMi-finalNeuralTOM.csv has 1861 data rows, matching the paper's final dev count app. B ToMi-finalNeuralTOM.csv.
• Chat models. The code asserts that chat models only run in multiple-choice mode gpt3LMMCprobing.py. Requests use temperature 0 and a two-token limit. Any answer that is not A, B or C is recorded as an empty string and counted wrong gpt3LMMCprobing.py. The paper does not describe this handling.
• Candidate order. ToMi candidates are shuffled per item in the code gpt3LMMCprobing.py. The paper does not say this.
• Shot count defaults. The code defaults to 3 shots and model ada gpt3LMMCprobing.py. The paper's k values come from the experiments; the code does not record them.
• Code path. formatExamples reads the global args rather than its arguments, so it only works when the script runs as __main__ gpt3LMMCprobing.py.

Pitfalls

• Scoring is not stated precisely. The paper gives "highest conditional likelihood" without a formula for length or normalisation. The code sums token log-probs. Summed scores favour shorter answers. A reimplementation should choose and report one rule.
• Probing type changes the number. Likelihood probing and multiple-choice probing give different results for the same model, and the paper uses likelihood for GPT-3 and multiple choice for chat models app. D. Compare like with like.
• Shot selection is random and seeded. Shots are sampled with a seed (code default 345), and the paper does not report variance across seeds gpt3LMMCprobing.py. Mental-state numbers depend on which shots are drawn; the k-dependence is not stable across choices, as the recency-bias discussion shows §3.2 app. B.
• Recency bias. Davinci tends to pick the last object location when k is small. This inflates true-belief accuracy at k = 4 and lowers it later §3.2 app. B.
• ToMi is generated, not natural. The stories come from a rule-based generator with random distractor actions. Bananas in closets and similar odd placements can appear §5. The authors also removed pairs that were wrong from an omniscient view app. B.
• Not evidence of theory of mind by itself. The authors call the two tasks necessary but not sufficient §5. A model that solves them does not thereby have theory of mind.
• Chat-model results are zero-shot and contaminated. The 400-item subsets are random but small. The GPT-4 report says BIG-Bench data was in training app. D. The model snapshots named in the paper (davinci, text-davinci-002, -003, gpt-3.5-turbo-0301, gpt-4-0314) are needed to reproduce app. D.
• Model size is unknown for GPT-3. The paper does not state parameter counts for Ada, Curie or Davinci §2.1.
• English only. The paper notes that false-belief reasoning depends on language structure, so these results may not transfer to other languages §5.

Lineage

• Builds on: the Sally-Ann false-belief test §1; SocialIQa (Sap et al., 2019) and its ATOMIC dimensions §2; the ToMi generator (Le et al., 2019) §3; GPT-3 k-shot probing (Brown et al., 2020) §2.1. None of these are cards in this wiki; nothing from them is used here.
• Compared against: BIG-G and PaLM via BIG-Bench results on SocialIQa §2.2. The paper also compares instruction-tuned and RLHF models: text-davinci-002, text-davinci-003, gpt-3.5-turbo and gpt-4 app. D.
• Superseded by: not stated.