AffToM: Benchmarking Recursive Affective Theory of Mind in Dynamic Multi-Party Interactions
Organizations: Hefei University of Technology · University of Science and Technology of China
Abstract
Understanding emotion in realistic social interactions requires more than recognizing facial expressions or assigning affective labels. The same event can elicit different emotions depending on what each participant knows, what they believe others know or feel, and how these nested mental states evolve over time. Existing benchmarks largely study higher-order epistemic Theory of Mind (ToM) and multimodal emotion reasoning in isolation, leaving recursive affective reasoning under asymmetric and dynamically changing information underexplored. Therefore, we introduce AffToM, a benchmark for affective ToM with 1,922 test pairs and 19,759 training pairs. AffToM evaluates Individual Affective Reasoning (IAR), which infers emotions from events and socio-cognitive context, and Multi-party Affective Reasoning (MPAR), which recursively tracks participants' nested beliefs and emotions as new evidence emerges. We develop an event-driven multi-agent generation pipeline grounded in dynamic character interaction graphs that explicitly capture event sequences, interpersonal relations, and mental-state updates. Furthermore, we propose AffToM-R1, a two-stage post-training framework combining supervised fine-tuning with perspective-aware rubric-guided reinforcement learning. Experiments show that MPAR remains challenging, with even the strongest evaluated MLLM achieving less than 60% accuracy, while AffToM-R1 improves MPAR accuracy by 13.38 percentage points over its base model. Our dataset is available at: https://github.com/MindIntLab-HFUT/AffToM.
Figures & tables
| Benchmark | Real-World | Mental State | Temporal Belief Update | Recursive Perspective | Dataset Format | |||
| Emotion | Belief | Intention | Desire | |||||
| Video-MME ( Fu et al., 2025 ) | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | MCQ |
| Hi-ToM ( Wu et al., 2023 ) | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✓ | MCQ |
| Social-IQ ( Zadeh et al., 2019 ) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | MCQ |
| MMToM-QA ( Jin et al., 2024 ) | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | MCQ |
| MOMENTS ( Villa-Cueva et al., 2025 ) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | MCQ |
| Models | Params | Multiple-Choice Question | Open-ended | |||||||||
| PP | IMSR | SPT | HI | PR | CR | MPAR | Avg. | EC | FC | Avg. | ||
| Open-Source MLLMs | ||||||||||||
| Ministral-3-3B | 3B | 31.02 | 39.81 | 43.15 | 27.49 | 38.89 | 33.33 | 30.07 | 33.26 | 1.64 | 1.48 | 1.56 |
| Gemma-3-4B | 4B | 40.11 | 52.43 | 43.15 | 39.77 | 42.78 | 37.85 | 29.67 | 37.43 | 1.62 | 1.26 | 1.44 |
| LLaVA-OneVision-1.5 | 4B | 47.59 | 63.11 | 60.96 | 47.95 | 45.00 | 48.59 | 39.34 | 46.87 | 1.82 | 1.51 | 1.67 |
| Qwen3-VL-4B-Instruct | 4B | 50.00 | 63.41 | 51.37 | 53.80 | 44.44 | 46.89 | 36.42 | 45.50 | 1.88 | 1.57 | 1.73 |
| Models | Modality | MCQ | Open-ended | ||||
| IAR | MPAR | Avg. | EC | FC | Avg. | ||
| Qwen3.5-4B | V | 57.83 | 37.22 | 49.29 | 1.77 | 1.50 | 1.64 |
| V+T | 63.07 | 40.40 | 53.68 | 1.78 | 1.57 | 1.68 | |
| AffToM-R1 | V | 65.51 | 50.60 | 59.33 | 2.17 | 1.89 | 2.03 |
| V+T | 66.35 | 52.72 | 60.70 | 2.17 | 1.96 | 2.07 | |
| Qwen3.5-27B | V | 69.26 | 45.03 | 59.22 | 2.45 | 2.23 | 2.34 |
| Dataset | Qwen3.5-4B | +SFT | AffToM-R1 |
| AffToM | 49.29 | 56.20 (+6.91) | 59.33 (+10.04) |
| Social-IQ 2.0 | 65.13 | 67.37 (+2.24) | 69.99 (+4.86) |
| Hi-ToM | 43.75 | 49.33 (+5.58) | 50.25 (+6.50) |
| R 3 -Bench-DX | 70.99 | 75.55 (+4.56) | 75.74 (+4.75) |
| R 3 -Bench-Hard | 32.98 | 37.94 (+4.96) | 39.72 (+6.74) |
| Reward Components | Accuracy (%) | ||||
| IAR | MPAR | Overall | |||
| – | – | – | 64.88 | 47.35 | 57.61 |
| ✓ | – | – | 64.92 | 48.08 | 57.94 |
| ✓ | ✓ | – | 65.04 | 49.80 | 58.73 |
| ✓ | – | ✓ | 65.32 | 49.01 | 58.56 |
| ✓ | ✓ | ✓ | 65.51 | 50.60 | 59.33 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | ANSR | VDEP | DQLT | QVAL | AVG. |
| ICC (3,k) | 0.811 | 0.823 | 0.852 | 0.810 | 0.824 |
| Stage | Primary Observed Error | Error Rate |
| Event Extraction | Incorrect interaction edges | 3.67% |
| Role/Mental-State Extraction | Incorrect mental-state analysis in events | 5.22% |
| Recursive QA Construction | Unsupported relational/affective links | 7.41% |
Theory of Mind
Theory of Mind studies how people and AI infer others’ beliefs, goals, and knowledge to predict behavior and coordinate interaction.
See all 97 papers