--- kind: concept slug: theory-of-mind title: Theory of mind summary: >- Theory of mind (ToM) lets an agent infer what other agents believe, intend and feel. A survey sorts the options into two paradigms: prompting an LLM to reason about others, and explicit model-based inverse planning. Prompting is light and black-box; model-based inference is interpretable and costly. Benchmarks are mostly text-only and offline. Results are author-reported. tags: [theory-of-mind, embodied-ai, social-reasoning, belief-modelling, benchmarks] papers: ["2601.02378", "2210.13312", "2402.13550"] related: [world-model, neural-symbolic-integration, partially-observable-markov-decision-process, llm-evaluation, benchmark-validity, in-context-learning] status: draft updated_at: 2026-11-01 --- ## Overview An agent with theory of mind models other agents' hidden mental states, such as beliefs, goals and emotions, and uses that model to predict their actions [arxiv:2601.02378 §1]. The survey frames this as a second model next to the physical world model. The mental world model adds memory, introspective observations and other agents' actions to the state [arxiv:2601.02378 §2.1]. Every variant below rests on this idea, except the two probing set-ups, which only measure it. The variants differ in who does the reasoning (an LLM, or an explicit probabilistic or symbolic model) and in how the mental states are represented. Three sources are on this page. The survey runs no experiments and reports numbers quoted from other works [arxiv:2601.02378 §1]. The 2022 study by Sap et al. runs GPT-3 and later chat models itself on ToM-style questions [arxiv:2210.13312 §2, §3]. The 2024 negotiation evaluation by Kwon et al. runs six LLMs on tasks that include inferring the other side's priorities and feelings [arxiv:2402.13550 §2.2, §4.3]. ## Variants ### ToM Prompting The LLM is the reasoner. The method changes only the prompt or the context, with no fine-tuning [arxiv:2601.02378 §4.1]. The survey lists about a dozen representative methods, for example Generative Agent, CoT-ToM, SimToM, SymbolicToM, MindForge and DEL-ToM, each with a stated base model and level [arxiv:2601.02378 tab. 4]. Card: [Modeling the Mental World for Embodied AI](/papers/2601.02378). Listed strengths are progressive refinement and light training. Listed limits are weak high-order reasoning, shallow multimodal fusion, poor generalisation and narrow coverage [arxiv:2601.02378 §4.1]. ### Model-Based Inference The agent keeps an explicit model of the other agent and infers its state by inverse planning, which backtracks from observed behaviour to the intentions that would explain it [arxiv:2601.02378 §4.2]. Examples include Bayesian inverse planning (BIP-ALM), I-POMDP inverse planning (LIMP), sequential Monte Carlo (Thought-tracing) and a metacognitive cycle (MetaMind) [arxiv:2601.02378 tab. 4]. The survey splits each method into three parts [arxiv:2601.02378 §4.2, fig. 6]: - **Assumption:** the hypotheses about what a mental state contains, such as environment state, action and observation. - **Strategy:** a forward planner that gives the probability of an action under a hypothesised state. An LLM can serve as this planner, as in BIP-ALM [arxiv:2601.02378 §4.2]. - **Feedback:** a loss on the gap between inferred and ground-truth mental states, used to tune the strategy [arxiv:2601.02378 §4.2]. ### Hybrid: neural-symbolic coupling The survey names neural-symbolic integration as the main direction. It takes two forms: a neural model generates and a symbolic check verifies, or symbolic guidance steers neural fine-tuning [arxiv:2601.02378 §6.1]. The most interesting recent concrete route it names is to decode symbolic belief units from an LLM's hidden states, so that symbolic beliefs and distributed activations are used together [arxiv:2601.02378 §3]. See [neural-symbolic integration](/concepts/neural-symbolic-integration). ### Representation choice Before choosing a reasoner, the survey asks how to represent mental states. It lists four paradigms for reasoning tasks: symbolic belief, probabilistic belief, distributed activation vectors and the BDI (belief, desire, intention) architecture [arxiv:2601.02378 tab. 3]. Its central trade-off is interpretability against flexibility [arxiv:2601.02378 §3]. ### Probing with ToM questions This is a measurement set-up, not a reasoner: the model is scored on ToM-style questions and its weights are not changed. Sap et al. score GPT-3 on the SocialIQa dev set and on the ToMi stories by k-shot likelihood. They split ToMi into factual and mental-state questions [arxiv:2210.13312 §2.1, §3.1]. The best GPT-3 setting reaches about 60% on mental-state questions, while factual questions reach 90–100% [arxiv:2210.13312 §3.2]. Newer chat models do better on SocialIQa but barely improve on mental-state questions [arxiv:2210.13312 app. D]. The authors call these tasks necessary but not sufficient for theory of mind [arxiv:2210.13312 §5]. The scoring rule differs between GPT-3 (likelihood) and chat models (multiple choice), so compare rows only within one probing type [arxiv:2210.13312 app. D]. Card: [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](/papers/2210.13312). ### Partner modelling in negotiation dialogue Another measurement set-up. The model reads a negotiation dialogue and is asked to infer the partner's priorities or feelings, such as whether the partner is satisfied with the deal [arxiv:2402.13550 §2.2.3]. The prompt shows only the agent's own point values, so the partner's private preferences are not given to the model [arxiv:2402.13550 §2.2]. Subjective items are scored against post-game survey answers [arxiv:2402.13550 §2.2.1]. The authors use this as a per-skill test of partner modelling, not as a test of theory of mind itself. Card: [Are LLMs Effective Negotiators?](/papers/2402.13550). ## When to use which The survey gives no measured cost or latency numbers. The rows below rest on its qualitative labels, and the probing and partner-modelling rows rest on the 2022 and 2024 results. | Situation | Use | Why | |---|---|---| | Black-box LLM access only, no training budget | ToM Prompting | Only prompt or context design is needed [arxiv:2601.02378 §4.1] | | Need a score on mental-state questions before deploying a model | Probing with ToM questions | Gives a benchmark number for the mental-state split; a pass is not proof of theory of mind [arxiv:2210.13312 §3.2, §5] | | Need to score what a dialogue partner wants or feels | Partner modelling in negotiation dialogue | Per-task scores exist, but the subjective correlations are not reproducible from the released code [arxiv:2402.13550 §4.3, Paper vs code] | | Need an auditable or interpretable belief state | Model-Based Inference | Explicit hypotheses and planner; labelled interpretable and costly [arxiv:2601.02378 §4, tab. 5] | | Deep recursion over others' beliefs (high-order ToM) | Check the method's level before choosing | Prompting is listed as weak here [arxiv:2601.02378 §4.1], yet some prompting methods are assigned levels 3 and 4 [arxiv:2601.02378 tab. 4] | | Video or multimodal input | Look at the base model | Prompting is listed as having shallow multimodal fusion [arxiv:2601.02378 §4.1]. Model-based BIP-ALM uses a video-capable base model [arxiv:2601.02378 tab. 4] | | Need to combine symbolic and learned signals | Hybrid | Named as the main direction [arxiv:2601.02378 §6.1] | ## Implementation sketch The survey gives no algorithm. The loop below is a generic reading of the model-based inference it describes [arxiv:2601.02378 §4.2]. It is not printed in the paper. No official code exists [arxiv:2601.02378 Paper vs code]. ```pseudocode # Generic model-based ToM loop, read from the survey's description [§4.2] # Input: observed actions (list), candidate hypotheses about mental states # Output: posterior over the hypotheses after the observed actions def infer_mental_state(actions, hypotheses, planner): # Assumption: a prior over the hypothesised mental states [§4.2] posterior = prior(hypotheses) for action in actions: # Strategy: forward planner gives p(action | hypothesis) [§4.2] likelihood = planner.action_probability(action, hypotheses) # Bayesian update of the hypothesis weights [§4.2] posterior = normalize(posterior * likelihood) return posterior ``` ## Benchmarks and reported results The survey orders the benchmarks as static text, higher-order text, then multimodal and dynamic [arxiv:2601.02378 §5]. It argues that most are text-only, offline and not interactive [arxiv:2601.02378 §5]. Most rows below are quoted from the survey's cited works and were not re-run by it. The four Sap et al. rows are that paper's own runs [arxiv:2210.13312 §2.2, §3.2, app. D]. The last two rows are the negotiation paper's own runs [arxiv:2402.13550 §4.3, app. A]. Setups differ between rows [arxiv:2601.02378 §4, §5] [arxiv:2210.13312 app. D] [arxiv:2402.13550 §3]. | Setting | Metric | Baseline | Reported value | Source | |---|---|---|---|---| | ToMi mental-state questions, GPT-3 Davinci, k-shot likelihood | accuracy | factual questions on the same model 90–100% | about 60% at best k | [arxiv:2210.13312 §3.2] | | SocialIQa dev, GPT-3 Davinci, 35-shot likelihood | accuracy | human reference not restated as a number; paper cites a gap of more than 30 points | about 55% | [arxiv:2210.13312 §2.2] | | SocialIQa dev, GPT-4, multiple choice, zero-shot (400 items) | accuracy | human reference not restated as a number; paper cites a gap of about 10 points | 79.3% | [arxiv:2210.13312 app. D] | | ToMi mental-state questions, gpt-4-0314, multiple choice, zero-shot (400 items) | accuracy | chance 50% (two options) | 59% | [arxiv:2210.13312 app. D] | | LIMP on MuMA-ToM | accuracy | GPT-4o 50.6%, Gemini 1.5 Pro 56.4% | 76.6% | [arxiv:2601.02378 §4.2] | | MuMA-ToM, human participants | accuracy | state-of-the-art LMMs 50–56% | 93.5% | [arxiv:2601.02378 §5.2] | | SimToM on BigToM | accuracy gain | direct QA; CoT | +22.9% over direct QA; +7.5% over CoT (unit not stated) | [arxiv:2601.02378 §4.1] | | CoT-ToM, first-order false belief | accuracy | zero-shot 40–80%; human 87% | 80–100% (GPT-4 perfect) | [arxiv:2601.02378 §4.1] | | MindForge, Minecraft basic tasks | success rate | not stated | 29% to 79% after 7 dialogue rounds (Mixtral-8x7B) | [arxiv:2601.02378 §4.1] | | MetaMind on 16 LLMs | average ToM accuracy gain | not stated | +6.2%; +35.7% in real social scenarios | [arxiv:2601.02378 §4.2] | | BToM-EL | correlation with human judgements | non-mentalising ρ=0.19; heuristic ρ=0.04 | ρ=0.92 | [arxiv:2601.02378 §4.2] | | ExploreToM | accuracy | not stated | 0–9% for GPT-4o and Llama-3.1 | [arxiv:2601.02378 §5.1] | | SoMi-ToM | gap to humans | not stated | 40.1% (first-person); 26.4% (third-person) | [arxiv:2601.02378 §5.2] | | Partner priority question, CA dataset, during the dialogue, GPT-4 | accuracy | majority answer 29.2%; Flan-T5 fine-tuned 71.7%; GPT-3.5 70.0% | 79.2% | [arxiv:2402.13550 app. A] | | Partner modelling, subjective tasks, GPT-4 | Pearson correlation (PCC) | Flan-T5 accuracy 50.5% on the same tasks (accuracy, not PCC) | 0.39 (average) | [arxiv:2402.13550 §4.3] | The negotiation benchmark's sample-size and split rules are in [arxiv:2402.13550 tab. 1]. Its PCC values are not produced by the released scorer [arxiv:2402.13550 Paper vs code], so the subjective row cannot be reproduced from that repository as shipped. Benchmark families, as the survey tabulates them [arxiv:2601.02378 tab. 6]: | Modality | Examples | Interactive | |---|---|---| | Text | ToMi, BigToM, Hi-ToM, ExploreToM, OpenToM, DynToM | Mostly no; Decrypto, MAgIC, EAI, CoToMA are yes | | Visual 3D | NOPA | no | | Graphs or video | Watch-And-Help | yes | | Video and text | MMToM-QA, MuMA-ToM, ToMLoc, GridToM | no | | Image and text | ChARTOM | no | | Video, image and text | SoMi-ToM | yes | The negotiation benchmark is not in that table. Its tasks are fixed prompts built from recorded human dialogues, so it is offline [arxiv:2402.13550 §2.1, §2.3]. ## When to use which, by benchmark Pick a benchmark by what it tests. A text-only score says little about embodied or multimodal use. The survey says the field's challenges include inauthentic data, offline question answering and no feedback loop [arxiv:2601.02378 §6.1]. The negotiation paper's partner-modelling tasks are text-only and offline too [arxiv:2402.13550 §2.1]. ## Contradictions - **Counts within the survey.** The introduction says 25 benchmarks, while the abstract and contributions say 26 [arxiv:2601.02378 §1]. Table 6 lists 26 rows. Table 4's Prompting block is labelled 10 but lists 11 methods [arxiv:2601.02378 tab. 4]. - **BigToM conditions.** Table 6 gives 6 conditions [arxiv:2601.02378 tab. 6]. §5.1 gives 25 refined control conditions [arxiv:2601.02378 §5.1]. - **CoToMA scenario.** §4.1 describes a bomb-disposal scenario, while Table 6 describes rescue missions [arxiv:2601.02378 §4.1, tab. 6]. - **Units.** It does not say whether the SimToM and DEL-ToM gains are points or relative changes [arxiv:2601.02378 §4.1]. Do not compare them. - **Equations missing.** The MetaMind summary cites two equations that do not appear in the survey, so its scoring cannot be implemented from this source [arxiv:2601.02378 §4.2]. - **Prompting results across sources.** The survey gives 80–100% for CoT-ToM on first-order false belief, with GPT-4 perfect [arxiv:2601.02378 §4.1]. Sap et al. report 59% for gpt-4-0314 on ToMi mental-state questions, zero-shot multiple choice [arxiv:2210.13312 app. D]. The survey row does not name its dataset, and the two sources use different items, prompts and scoring. The numbers are not comparable and neither is re-run here. - **Partner modelling versus false-belief scores.** The negotiation paper reports 79.2% for GPT-4 on a partner-priority question [arxiv:2402.13550 app. A], while Sap et al. report 59% for gpt-4-0314 on ToMi mental-state questions [arxiv:2210.13312 app. D]. The tasks, data, prompts and chance levels differ, so neither number should be read as a score on the other's test. ## Papers - [Modeling the Mental World for Embodied AI: A Comprehensive Review](/papers/2601.02378): a survey that separates physical from mental world models, compares ToM Prompting with Model-Based Inference, and catalogues 19 methods and 26 benchmarks. - [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](/papers/2210.13312): probes GPT-3 and later chat models on SocialIQa and the ToMi stories, and finds mental-state accuracy far below factual accuracy. - [Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues](/papers/2402.13550): evaluates six LLMs on 35 negotiation tasks, including partner-modelling tasks, and finds GPT-4 strongest but weak on subjective judgements.