Theory of mind

Overview

An agent with theory of mind models other agents' hidden mental states, such as beliefs, goals and emotions, and uses that model to predict their actions 2601.02378 §1. The survey frames this as a second model next to the physical world model. The mental world model adds memory, introspective observations and other agents' actions to the state 2601.02378 §2.1. Every variant below rests on this idea, except the two probing set-ups, which only measure it. The variants differ in who does the reasoning (an LLM, or an explicit probabilistic or symbolic model) and in how the mental states are represented.

Three sources are on this page. The survey runs no experiments and reports numbers quoted from other works 2601.02378 §1. The 2022 study by Sap et al. runs GPT-3 and later chat models itself on ToM-style questions 2210.13312 §2, §3. The 2024 negotiation evaluation by Kwon et al. runs six LLMs on tasks that include inferring the other side's priorities and feelings 2402.13550 §2.2, §4.3.

Variants

ToM Prompting

The LLM is the reasoner. The method changes only the prompt or the context, with no fine-tuning 2601.02378 §4.1. The survey lists about a dozen representative methods, for example Generative Agent, CoT-ToM, SimToM, SymbolicToM, MindForge and DEL-ToM, each with a stated base model and level 2601.02378 tab. 4. Card: Modeling the Mental World for Embodied AI.

Listed strengths are progressive refinement and light training. Listed limits are weak high-order reasoning, shallow multimodal fusion, poor generalisation and narrow coverage 2601.02378 §4.1.

Model-Based Inference

The agent keeps an explicit model of the other agent and infers its state by inverse planning, which backtracks from observed behaviour to the intentions that would explain it 2601.02378 §4.2. Examples include Bayesian inverse planning (BIP-ALM), I-POMDP inverse planning (LIMP), sequential Monte Carlo (Thought-tracing) and a metacognitive cycle (MetaMind) [arxiv:2601.02378 tab. 4]. The survey splits each method into three parts [arxiv:2601.02378 §4.2, fig. 6]:

• Assumption: the hypotheses about what a mental state contains, such as environment state, action and observation.
• Strategy: a forward planner that gives the probability of an action under a hypothesised state. An LLM can serve as this planner, as in BIP-ALM 2601.02378 §4.2.
• Feedback: a loss on the gap between inferred and ground-truth mental states, used to tune the strategy 2601.02378 §4.2.

Hybrid: neural-symbolic coupling

The survey names neural-symbolic integration as the main direction. It takes two forms: a neural model generates and a symbolic check verifies, or symbolic guidance steers neural fine-tuning 2601.02378 §6.1. The most interesting recent concrete route it names is to decode symbolic belief units from an LLM's hidden states, so that symbolic beliefs and distributed activations are used together 2601.02378 §3. See neural-symbolic integration.

Representation choice

Before choosing a reasoner, the survey asks how to represent mental states. It lists four paradigms for reasoning tasks: symbolic belief, probabilistic belief, distributed activation vectors and the BDI (belief, desire, intention) architecture 2601.02378 tab. 3. Its central trade-off is interpretability against flexibility 2601.02378 §3.

Probing with ToM questions

This is a measurement set-up, not a reasoner: the model is scored on ToM-style questions and its weights are not changed. Sap et al. score GPT-3 on the SocialIQa dev set and on the ToMi stories by k-shot likelihood. They split ToMi into factual and mental-state questions 2210.13312 §2.1, §3.1. The best GPT-3 setting reaches about 60% on mental-state questions, while factual questions reach 90–100% 2210.13312 §3.2. Newer chat models do better on SocialIQa but barely improve on mental-state questions 2210.13312 app. D. The authors call these tasks necessary but not sufficient for theory of mind 2210.13312 §5. The scoring rule differs between GPT-3 (likelihood) and chat models (multiple choice), so compare rows only within one probing type 2210.13312 app. D. Card: Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs.

Partner modelling in negotiation dialogue

Another measurement set-up. The model reads a negotiation dialogue and is asked to infer the partner's priorities or feelings, such as whether the partner is satisfied with the deal 2402.13550 §2.2.3. The prompt shows only the agent's own point values, so the partner's private preferences are not given to the model 2402.13550 §2.2. Subjective items are scored against post-game survey answers 2402.13550 §2.2.1. The authors use this as a per-skill test of partner modelling, not as a test of theory of mind itself. Card: Are LLMs Effective Negotiators?.

When to use which

The survey gives no measured cost or latency numbers. The rows below rest on its qualitative labels, and the probing and partner-modelling rows rest on the 2022 and 2024 results.

Implementation sketch

The survey gives no algorithm. The loop below is a generic reading of the model-based inference it describes 2601.02378 §4.2. It is not printed in the paper. No official code exists 2601.02378 Paper vs code.

# Generic model-based ToM loop, read from the survey's description [§4.2]
# Input:  observed actions (list), candidate hypotheses about mental states
# Output: posterior over the hypotheses after the observed actions

def infer_mental_state(actions, hypotheses, planner):
    # Assumption: a prior over the hypothesised mental states [§4.2]
    posterior = prior(hypotheses)
    for action in actions:
        # Strategy: forward planner gives p(action | hypothesis) [§4.2]
        likelihood = planner.action_probability(action, hypotheses)
        # Bayesian update of the hypothesis weights [§4.2]
        posterior = normalize(posterior * likelihood)
    return posterior

Benchmarks and reported results

The survey orders the benchmarks as static text, higher-order text, then multimodal and dynamic 2601.02378 §5. It argues that most are text-only, offline and not interactive 2601.02378 §5. Most rows below are quoted from the survey's cited works and were not re-run by it. The four Sap et al. rows are that paper's own runs 2210.13312 §2.2, §3.2, app. D. The last two rows are the negotiation paper's own runs 2402.13550 §4.3, app. A. Setups differ between rows 2601.02378 §4, §5 2210.13312 app. D 2402.13550 §3.

The negotiation benchmark's sample-size and split rules are in 2402.13550 tab. 1. Its PCC values are not produced by the released scorer 2402.13550 Paper vs code, so the subjective row cannot be reproduced from that repository as shipped.

Benchmark families, as the survey tabulates them 2601.02378 tab. 6:

The negotiation benchmark is not in that table. Its tasks are fixed prompts built from recorded human dialogues, so it is offline 2402.13550 §2.1, §2.3.

When to use which, by benchmark

Pick a benchmark by what it tests. A text-only score says little about embodied or multimodal use. The survey says the field's challenges include inauthentic data, offline question answering and no feedback loop 2601.02378 §6.1. The negotiation paper's partner-modelling tasks are text-only and offline too 2402.13550 §2.1.

Contradictions

• Counts within the survey. The introduction says 25 benchmarks, while the abstract and contributions say 26 2601.02378 §1. Table 6 lists 26 rows. Table 4's Prompting block is labelled 10 but lists 11 methods 2601.02378 tab. 4.
• BigToM conditions. Table 6 gives 6 conditions 2601.02378 tab. 6. §5.1 gives 25 refined control conditions 2601.02378 §5.1.
• CoToMA scenario. §4.1 describes a bomb-disposal scenario, while Table 6 describes rescue missions 2601.02378 §4.1, tab. 6.
• Units. It does not say whether the SimToM and DEL-ToM gains are points or relative changes 2601.02378 §4.1. Do not compare them.
• Equations missing. The MetaMind summary cites two equations that do not appear in the survey, so its scoring cannot be implemented from this source 2601.02378 §4.2.
• Prompting results across sources. The survey gives 80–100% for CoT-ToM on first-order false belief, with GPT-4 perfect 2601.02378 §4.1. Sap et al. report 59% for gpt-4-0314 on ToMi mental-state questions, zero-shot multiple choice 2210.13312 app. D. The survey row does not name its dataset, and the two sources use different items, prompts and scoring. The numbers are not comparable and neither is re-run here.
• Partner modelling versus false-belief scores. The negotiation paper reports 79.2% for GPT-4 on a partner-priority question 2402.13550 app. A, while Sap et al. report 59% for gpt-4-0314 on ToMi mental-state questions 2210.13312 app. D. The tasks, data, prompts and chance levels differ, so neither number should be read as a score on the other's test.

Papers

• Modeling the Mental World for Embodied AI: A Comprehensive Review: a survey that separates physical from mental world models, compares ToM Prompting with Model-Based Inference, and catalogues 19 methods and 26 benchmarks.
• Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs: probes GPT-3 and later chat models on SocialIQa and the ToMi stories, and finds mental-state accuracy far below factual accuracy.
• Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues: evaluates six LLMs on 35 negotiation tasks, including partner-modelling tasks, and finds GPT-4 strongest but weak on subjective judgements.