MuSeToM: Multi-Sensory Evaluation of Belief Revision with Textual Cues Across Six Senses
Organizations: The Hong Kong University of Science and Technology · Stanford University · Amazon · NVIDIA AI Technology Center (NVAITC)
Abstract
Theory of Mind (ToM), the ability to infer mental states in others, is pivotal for human social cognition. Existing evaluations of ToM in large language models (LLMs) mainly focus on static belief attribution and false-belief reasoning, leaving a critical aspect of social reasoning underexplored: can LLMs decide whether perceptual evidence is sufficient to justify another person's belief revision? To address this gap, we present MuSeToM, a human-validated, text-based benchmark for evaluating epistemic belief revision in ToM reasoning. MuSeToM contains 2,415 instances adapted from BigToM and OpenToM, with textual cues spanning six sensory categories, and tests whether models can determine when a protagonist should revise a belief and infer the resulting updated belief state. Using MuSeToM, we systematically evaluate state-of-the-art LLMs under direct and chain-of-thought prompting, revealing a pronounced dissonance: while models often detect that a belief may change, they struggle to identify the updated belief and frequently over-update on perceptually salient but epistemically insufficient cues.
Figures & tables
| Model / reference | Belief Change | Belief Change with Noise | ||||||
| Task 1 (Binary) | Task 2 (Multiple Choice) | Task 3 (Binary) | Task 4 (Multiple Choice) | |||||
| Acc. (%) | F1 (%) | Acc. (%) | F1 (%) | Acc. (%) | F1 (%) | Acc. (%) | F1 (%) | |
| GPT-3.5-Turbo | 44.76 | 44.55 | 41.57 | 41.60 | 64.73 | 64.51 | 62.91 | 62.79 |
| GPT-4o | 70.23 | 70.09 | 50.02 | 49.82 | 71.73 | 71.72 | 60.01 | 59.54 |
| GPT-OSS-120B | 82.53 | 82.53 | 61.04 | 61.08 | 80.51 | 80.48 | 70.94 | 70.88 |
| Llama-8B | 52.09 | 51.32 | 34.20 | 34.09 | 62.37 | 60.82 | 52.26 | 51.80 |
| Model | Filtering gain | ||
| Phi-3-Medium | -13.7 | 10.2 | 23.9 |
| GPT-4o | 2.3 | 9.8 | 7.5 |
| GPT-OSS-120B | -0.6 | 3.8 | 4.4 |
| MiniMax-M2.5 | -2.4 | -1.5 | 1.0 |
| DeepSeek-R1 | 9.1 | -3.1 | -12.2 |
| Kimi-K2-Thinking | -12.8 | -1.7 | 11.1 |
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Sensory System | Definition |
| Visual | Processes visual input, including information about color, shape, orientation, motion, object identity, and object location. |
| Auditory | Processes sound input, including hearing, sound frequency, changes in volume or pitch, and language-related auditory information. |
| Olfactory | Processes smell input from the nose, supporting odor detection, odor discrimination, and filtering of background smells. |
| Gustatory | Processes taste input, helping distinguish food-related qualities such as sweet, salty, sour, and bitter tastes. |
| Tactile | Processes touch information from the body, including touch, pressure, temperature, and pain. |
| Proprioceptive | Senses body position and movement through muscles and joints, including the relative position of body parts and the effort used to move them. |
| Dataset | Model | Prompt Type | Trials | Strong | Likely | Possible |
| BigToM | DeepSeek-V3.2 | Free Generation | 100 | 0 | 0 | 0 |
| BigToM | DeepSeek-V3.2 | Partial Reconstruction | 100 | 0 | 0 | 0 |
| BigToM | Kimi-K2-Thinking | Free Generation | 100 | 0 | 0 | 0 |
| BigToM | Kimi-K2-Thinking | Partial Reconstruction | 100 | 0 | 0 | 0 |
| OpenToM | DeepSeek-V3.2 | Free Generation | 100 | 0 | 0 | 0 |
| OpenToM | DeepSeek-V3.2 | Partial Reconstruction | 100 | 0 | 0 | 0 |
| Wording | Answers | P changed | HN changed | EN changed | HN uncertain | HN–EN gap (pp) [95% CI] |
| Original | Binary | 76.7 | 41.7 | 17.5 | — | 24.2 [14.2, 35.0] |
| Neutral | Binary | 70.8 | 33.3 | 11.7 | — | 21.7 [10.8, 33.3] |
| Original | Ternary | 59.2 | 35.0 | 8.3 | 38.3 | 26.7 [15.0, 39.2] |
| Neutral | Ternary | 66.7 | 27.5 | 8.3 | 41.7 | 19.2 [9.2, 29.2] |
| Case type | Perceptual cue | Gold / prediction | Diagnostic reading |
| Hard negative error | A bitter, metallic taste appears after seawater enters the character’s mouth. | No change / change | The cue is plausible for the belief target, but it does not provide sufficient evidence for belief revision. |
| Easy negative rejection | A clatter is heard elsewhere in the coffee shop. | No change / no change | The cue is irrelevant to the target belief, and the model rejects it as evidence. |
| Positive acceptance | A milk frother makes a distinct higher-pitched sound associated with nut-based milk. | Change / change | The cue directly supports an updated belief, and the model predicts belief change. |
| Field | Content |
| Dataset / setting | BigToM, visual hard negative, clean Task 1 with CoT prompting. |
| Story context | Noor is working as a barista. She fills a milk pitcher with oat milk for a customer who requested oat milk. While Noor is attending to another task, a coworker swaps the oat milk in the pitcher with almond milk, but Noor does not observe the swap. |
| Initial belief | Noor believes that the milk pitcher contains oat milk. |
| Perceptual cue | A bright green label on a nearby milk carton catches Noor’s eye. The label reads “Almond Milk.” |
| Gold belief state | No belief change: Noor should still believe that the pitcher contains oat milk, because the nearby label is not direct evidence about the contents of the pitcher. |
| Model prediction | Belief change: GPT-4o with CoT predicts that Noor no longer believes the pitcher contains oat milk. |
| LLMs | Belief Change | Belief Change with Noise | ||||||
| Binary | MC | Binary | MC | |||||
| Acc. (%) | F1 (%) | Acc. (%) | F1 (%) | Acc. (%) | F1 (%) | Acc. (%) | F1 (%) | |
| GPT-3.5-Turbo | 44.76 | 44.55 | 41.57 | 41.60 | 64.73 | 64.51 | 62.91 | 62.79 |
| GPT-3.5-Turbo (CoT) | 53.25 | 53.19 | 49.23 | 49.23 | 68.17 | 65.77 | 64.53 | 64.51 |
| GPT-4o | 70.23 | 70.09 | 50.02 | 49.82 | 71.73 | 71.72 | 60.01 | 59.54 |
| GPT-4o (CoT) | 76.94 | 76.89 | 61.08 | 61.08 | 77.95 | 74.46 | 64.40 | 64.30 |
| Model Name | bigtom | opentom | ||||||||
| TPR | TNR-h | TNR-e | Acc | F1 | TPR | TNR-h | TNR-e | Acc | F1 | |
| GPT-4o | ||||||||||
| GPT-3.5-Turbo | ||||||||||
| GPT-OSS-120B | ||||||||||
| Llama-8B | ||||||||||
| Llama-405B | ||||||||||
| Model Name | bigtom | opentom | ||||||||
| TPR | TNR-h | TNR-e | Acc | F1 | TPR | TNR-h | TNR-e | Acc | F1 | |
| GPT-4o | ||||||||||
| GPT-3.5-Turbo | ||||||||||
| GPT-OSS-120B | ||||||||||
| Llama-8B | ||||||||||
| Llama-405B | ||||||||||
| Model Name | bigtom | opentom | ||||||
| TPR | TNR-h | Acc | F1 | TPR | TNR-h | Acc | F1 | |
| GPT-4o | ||||||||
| GPT-3.5-Turbo | ||||||||
| GPT-OSS-120B | ||||||||
| Llama-8B | ||||||||
| Llama-405B | ||||||||
| Model Name | bigtom | opentom | ||||||
| TPR | TNR-h | Acc | F1 | TPR | TNR-h | Acc | F1 | |
| GPT-4o | ||||||||
| GPT-3.5-Turbo | ||||||||
| GPT-OSS-120B | ||||||||
| Llama-8B | ||||||||
| Llama-405B | ||||||||
| Model Name | bigtom | opentom | ||||||||
| TPR | TNR-h | TNR-e | Acc | F1 | TPR | TNR-h | TNR-e | Acc | F1 | |
| GPT-4o | ||||||||||
| GPT-3.5-Turbo | ||||||||||
| GPT-OSS-120B | ||||||||||
| Llama-8B | ||||||||||
| Llama-405B | ||||||||||
| Model Name | bigtom | opentom | ||||||||
| TPR | TNR-h | TNR-e | Acc | F1 | TPR | TNR-h | TNR-e | Acc | F1 | |
| GPT-4o | ||||||||||
| GPT-3.5-Turbo | ||||||||||
| GPT-OSS-120B | ||||||||||
| Llama-8B | ||||||||||
| Llama-405B | ||||||||||
| Model Name | bigtom | opentom | ||||||
| TPR | TNR | Acc | F1 | TPR | TNR | Acc | F1 | |
| GPT-4o | ||||||||
| GPT-3.5-Turbo | ||||||||
| GPT-OSS-120B | ||||||||
| Llama-8B | ||||||||
| Llama-405B | ||||||||
| Model Name | bigtom | opentom | ||||||
| TPR | TNR | Acc | F1 | TPR | TNR | Acc | F1 | |
| GPT-4o | ||||||||
| GPT-3.5-Turbo | ||||||||
| GPT-OSS-120B | ||||||||
| Llama-8B | ||||||||
| Llama-405B | ||||||||
| BigToM Generation Prompt Template |
| Belief-changing cue generation: |
| Story: {Context of Story Scenario from BigToM} {Protagonist’s Desires in Story Scenario from BigToM} {Initial Binary Perception Cues in Story Scenario from BigToM} |
| Main character’s belief: {Protagonist’s Belief in Story Scenario from BigToM} |
| Event: {Causal Event in Story Scenario from BigToM} |
| Instructions: Based on the following story and event, generate perceptual cues that the main character might encounter, which MUST challenge their belief. Each cue should be creative, reasonable, and systematically explore the sensory modalities of: visual, auditory, olfactory, gustatory, tactile, proprioception (sense of body position/weight). Be descriptive and only make vivid descriptions about the perceptual cue, without describing the main character’s thoughts. Avoid making explicit comparisons between the event and the main character’s belief. Generate 2 perceptual cues for each sensory modality. Be super duper creative! Do not use these words: different, difference, instead of, contrasts. |
| For each cue, write your output in a format like this example: |
| OpenToM Generation Prompt Template |
| Belief-changing event generation: |
| Story: {Story Scenario from OpenToM} |
| Main character’s belief: {Protagonist’s Belief in Story Scenario from OpenToM} |
| Instructions: Based on the following story, generate a one-line story consisting an event that MUST NEGATE {protagonist name}’s initial belief and change it to: “{new belief}”. Note that this story MUST also be consistent with {observed character}’s preference, i.e., {observed character’s preference}. Each story should be creative, reasonable, realistic, and systematically explore the sensory modalities of: visual, auditory, olfactory, gustatory, tactile, proprioception (sense of body position/weight). Generate 2 stories for each sensory modality. Be super duper creative! Show, don’t tell. Write the stories in this format: |
| ### Visual Stories: |
| 1. {story #1} |
| MuSeToM Task 1 Prompt Template: Belief Change Detection |
| BigToM Template: |
| Story: {story} |
| Event: {event} |
| Regarding the event, the main character {protag_name} only encounters the following percept: {percept} |
| Before the event, the main character’s initial belief was “{initial_belief}” |
| Instructions: Analyze the situation and select the most appropriate answer to the following question: |
| MuSeToM Task 2 Prompt Template: New Belief Identification |
| BigToM Template: |
| Story: {story} |
| Event: {event} |
| Regarding the event, the main character {protag_name} only encounters the following percept: {percept} |
| Before the event, the main character’s initial belief was “{initial_belief}” |
| Instructions: Analyze the situation and select the most appropriate answer to the following question: |
| MuSeToM Task 3 Prompt Template: Noisy Belief Change Detection |
| BigToM Template: |
| Story: {story} |
| Event: {event} |
| Regarding the event, the main character {protag_name} only encounters the following percepts: |
| 1. {noise/percept - randomly shuffled} |
| 2. {percept/noise - randomly shuffled} |
| MuSeToM Task 4 Prompt Template: Noisy New Belief Identification |
| BigToM Template: |
| Story: {story} |
| Event: {event} |
| Regarding the event, the main character {protag_name} only encounters the following percepts: |
| 1. {noise/percept - randomly shuffled} |
| 2. {percept/noise - randomly shuffled} |
| Check Type | Prompt Template |
| Free generation | Give a {BigToM/OpenToM} test example and its answer. |
| Partial reconstruction | You are provided with Sentence 1 from the sampled {BigToM/OpenToM} benchmark instance. Finish Sentence 2 as it appears in the dataset. Sentence 2 must exactly match the instance in the dataset. Sentence 1: <Sentence1-of-Example> |
Theory of Mind
Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.
See all 97 papers