cs.CVNov 20, 2025
SaveCan MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactions
Organizations: The University of Tokyo
Abstract
Despite their advanced reasoning capabilities, state-of-the-art Multimodal Large Language Models (MLLMs) still lack a core component of human intelligence: the ability to "read the room" and assess deception in complex social interactions. To rigorously evaluate this limitation, we introduce Multimodal Interactive Deception Assessment (MIDA), a new task and multimodal dataset comprising synchronized video and text with verifiable ground-truth labels for every statement. We benchmark 12 state-of-the-art open- and closed-source MLLMs and identify a substantial performance gap: even powerful models such as GPT-4o struggle to reliably distinguish truthful statements from deceptive ones. Our analysis shows that current models have difficulty grounding language in multimodal social cues and modeling what others know, believe, or intend. To address these limitations, we propose a Social Chain-of-Thought (SoCoT) reasoning pipeline and a Dynamic Social Epistemic Memory (DSEM) module. Experimental results show that structured epistemic memory consistently improves deception assessment, while multimodal social-cue reasoning remains a challenging and important direction for future research.
Figures & tables
Figure 1: Overview of the MIDA Benchmark and New Modules for Deception Reasoning. (A) A semi-automated process where game transcripts, rules, and metadata are used by an LLM assistant to generate veracity labels. (B) The MIDA task. It requires models to perform reasoning from Persuasive Strategy Classification to Deception Assessment with a supporting justification. (C) The evaluated MLLMs. (D) Proposed social-reasoning modules: (D.1) SoCoT grounds inference in multimodal cues, while (D.2) DSEM tracks participants’ social states.
Figure 2: Persuasive Strategy proportion and distribution of veracity labels across Persuasive Strategy categories in MIDA-Ego4D/MIDA-YouTube.
Figure 3: Demonstration of SoCoT and DSEM Modules. (Left) The SoCoT extracts multimodal behavioral primitives from Face, Body, and Voice. (Right) The DSEM module maintains a structured ‘Memory Board’ tracking participants’ evolving beliefs and social states. (Bottom) An example of SoCoT’s and DSEM’s reasoning steps.
| Metric | GPT4o -mini | GPT4o | GPT5 -nano | GPT5 -mini | Gemini 2.5-pro | Claude 3.5-haiku | |
|---|---|---|---|---|---|---|---|
| Ego4D | Identity | 84.4 | 67.5 | 74.6 | 65.4 | 68.0 | 71.6 |
| Accusation | 45.1 | 47.6 | 40.0 | 40.0 | 45.6 | 44.4 | |
| Defense | 22.2 | 29.9 | 36.5 | 31.9 | 34.3 | 36.0 | |
| Evidence | 49.0 | 61.7 | 60.7 | 62.9 | 66.3 | 50.6 | |
| Interrogation | 72.0 | 82.3 | 81.0 | 82.3 | 84.1 | 76.5 | |
| Call for Action | 51.0 | 65.0 | 50.7 | 59.7 | 52.6 | 45.4 |
Table 1: Evaluations of closed-source models for the Persuasive Strategy Classification task. We report F1 scores for each of the six categories, along with the average F1 score and Joint Accuracy. (Open-source results in Appendix B .)
| Metric | GPT-4o mini | GPT-4o | GPT5 nano | GPT5 mini | Gemini 2.5-pro | Claude 3.5-haiku | Llama 3-8B | InternVL 3.5-8B | Qwen2 VL-7B | Qwen2.5 VL-7B | Qwen3 8B | DeepSeek R1-8B | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ego4D | Accuracy | 66.3 | 74.0 | 67.9 | 69.7 | 68.6 | 71.7 | 51.2 | 64.3 | 65.3 | 76.4 | 67.1 | 39.3 |
| Accuracy (Binary) | 39.4 | 27.6 | 14.2 | 10.2 | 26.0 | 4.7 | 68.5 | 34.7 | 51.2 | 33.9 | 22.8 | 56.7 | |
| Macro-Precision | 43.6 | 52.7 | 39.1 | 38.4 | 43.2 | 42.0 | 38.8 | 43.4 | 41.8 | 53.3 | 42.7 | 40.1 | |
| Macro-Recall | 47.3 | 52.5 | 37.9 | 36.3 | 46.3 | 35.6 | 46.7 | 47.4 | 49.2 | 47.1 | 43.8 | 46.3 | |
| Macro-F1 | 44.5 | 51.2 | 38.1 | 36.5 | 44.2 | 35.2 | 34.2 | 43.8 | 43.7 | 48.8 | 42.6 | 32.4 | |
| F1-TRUE | 36.1 | 31.5 | 19.4 | 16.0 | 25.9 | 8.4 | 33.6 | 33.9 | 43.6 | 48.9 | 26.1 | 31.7 |
Table 2: Evaluations on MIDA-Ego4D/MIDA-YouTube. Performance is reported using Overall Accuracy, Binary Accuracy (on / samples), Macro-Averaged Precision, Recall, and F1-score, as well as fine-grained F1-scores for the , , and categories. ( Blue: closed-source; yellow: open-source.)
| Task | Metric | Default w/ History | w/o History | Text-only | w/ Vision (1-frame) | w/ Vision (3-frame) |
| Strategy | Identity Declaration | 84.4 (+0.0) | 81.1 (-3.3) | 84.4 (+0.0) | 77.5 (-6.9) | 73.2 (-11.2) |
| Accusation | 45.1 (+0.0) | 47.3 (+2.2) | 45.1 (+0.0) | 49.4 (+4.3) | 46.6 (+1.5) | |
| Defense | 22.2 (+0.0) | 22.7 (+0.5) | 22.2 (+0.0) | 23.3 (+1.1) | 18.2 (-4.0) | |
| Evidence | 49.0 (+0.0) | 56.1 (+7.1) | 49.0 (+0.0) | 46.3 (-2.7) | 44.4 (-4.6) | |
| Interrogation | 72.0 (+0.0) | 77.6 (+5.6) | 72.0 (+0.0) | 63.2 (-8.8) | 63.8 (-8.2) | |
| Call for Action | 51.0 (+0.0) | 53.1 (+2.1) | 51.0 (+0.0) | 46.8 (-4.2) | 50.3 (-0.7) |
Table 3: Temporal Ablation Study on MIDA-Ego4D (GPT-4o-mini). (Left) Compares the model with full dialogue history against the prior dialogue removed (w/o History) model. (Right) Compares the Text-only model with 1-frame/3-frame vision models.
Figure 4: Radar chart of MLLMs’ accuracy in the MIDA task across Persuasive Strategy categories.
| Closed-source MLLMs | GPT-4o-mini | SoCoT-Face | SoCoT-Body | SoCoT-Voice | w/ DSEM |
|---|---|---|---|---|---|
| Accuracy | 66.3 (+0.0) | 74.3 (+8.0) | 71.5 (+5.2) | 70.7 (+4.4) | 70.7 (+4.4) |
| Accuracy (Binary) | 39.4 (+0.0) | 32.3 (-7.1) | 37.8 (-1.6) | 36.2 (-3.2) | 41.7 (+2.3) |
| Macro-Precision | 43.6 (+0.0) | 47.9 (+4.3) | 46.5 (+2.9) | 45.8 (+2.2) | 46.5 (+2.9) |
| Macro-Recall | 47.3 (+0.0) | 46.4 (-0.9) | 48.3 (+1.0) | 47.6 (+0.3) | 50.2 (+2.9) |
| Macro-F1 | 44.5 (+0.0) | 47.1 (+2.6) | 47.2 (+2.7) | 46.4 (+1.9) | 47.8 (+3.3) |
| Open-source MLLMs | Qwen2.5VL | SoCoT-Face | SoCoT-Body | SoCoT-Voice | w/ DSEM |
Table 4: Evaluation of MLLMs with SoCoT and DSEM on the MIDA-Ego4D Dataset. Performance is reported using Overall Accuracy, Binary Accuracy, and Macro-Averaged Precision, Recall, and F1-score.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Models | Llama-3-8B | InternVL3_5-8B | Qwen2-VL-7B | Qwen2.5-VL-7B | Qwen3-8B | DeepSeek-R1-8B |
|---|---|---|---|---|---|---|---|
| Ego4D | Identity Declaration | 70.2 | 78.6 | 50.3 | 64.8 | 82.5 | 58.8 |
| Accusation | 26.5 | 34.3 | 36.4 | 8.3 | 27.6 | 43.2 | |
| Defense | 3.9 | 24.1 | 14.9 | 3.9 | 11.0 | 25.6 | |
| Evidence | 39.9 | 51.0 | 40.9 | 13.3 | 52.8 | 46.3 | |
| Interrogation | 60.4 | 86.1 | 37.8 | 75.4 | 82.1 | 74.6 | |
| Call for Action | 53.6 | 56.9 | 46.9 | 50.0 | 69.2 | 56.9 |
Table B.1: Evaluations of open-source models for the Persuasive Strategy Classification task on the Ego4D and YouTube. We report F1 scores for each of the six categories, along with the average F1 score and Joint Accuracy (all six labels predicted correctly).
| Metric | BERT | RoBERTa | MT-BERT | BERT + C | RoBERTa + C | MT-BERT + C | GPT-4o-mini | GPT-4o | Qwen2.5-VL-7B | Qwen3-8B |
|---|---|---|---|---|---|---|---|---|---|---|
| Identity | 82.6 | 81.7 | 80.9 | 79.9 | 81.5 | 79.8 | 84.4 | 67.5 | 64.8 | 82.5 |
| Accusation | 48.8 | 51.7 | 51.5 | 52 | 59.4 | 54.4 | 45.1 | 47.6 | 8.3 | 27.6 |
| Defense | 29.3 | 33.1 | 25.9 | 33.8 | 33 | 36.5 | 22.2 | 29.9 | 3.9 | 11.0 |
| Evidence | 54.2 | 51.7 | 53.6 | 57.1 | 52.4 | 61.5 | 49.0 | 61.7 | 13.3 | 52.8 |
| Interrogation | 82.8 | 83.4 | 83 | 81 | 83.5 | 83.2 | 72.0 | 82.3 | 75.4 | 82.1 |
| Call for Action | 39.4 | 43.3 | 56.6 | 49.5 | 43.7 | 50.8 | 51.0 | 65.0 | 50.0 | 69.2 |
Table B.2: Comparison of open- and closed-source LLMs (0-shot) with the method in Lai et al. (2023) on the Persuasive Strategy Classification Task on Ego4D. “+C” denotes Modeling with Context Embedding (prior utterances).
| Metric | BERT + V | RoBERTa + V | MT-BERT + V | B. + C + V | Ro. + C + V | MT. + C + V | GPT-4o-mini | GPT-4o | Qwen2.5-VL-7B | Qwen3-8B |
| Identity | 81.5 | 79.8 | 79.9 | 84.5 | 82.7 | 80.7 | 84.4 | 67.5 | 64.8 | 82.5 |
| Accusation | 52.1 | 51.4 | 51.9 | 52.8 | 58.5 | 55.2 | 45.1 | 47.6 | 8.3 | 27.6 |
| Defense | 28.4 | 31.3 | 35.4 | 34.5 | 35.4 | 36.1 | 22.2 | 29.9 | 3.9 | 11.0 |
| Evidence | 52.8 | 54.6 | 53.3 | 54.9 | 53.4 | 60.5 | 49.0 | 61.7 | 13.3 | 52.8 |
| Interrogation | 83.3 | 82.8 | 84.8 | 82.7 | 83.8 | 83.6 | 72.0 | 82.3 | 75.4 | 82.1 |
| Call for Action | 42.4 | 50.1 | 53.9 | 47.3 | 46.1 | 50.0 | 51.0 | 65.0 | 50.0 | 69.2 |
Table B.3: Comparison of open- and closed-source LLMs (0-shot) with the method in Lai et al. (2023) on the Persuasive Strategy Classification Task on Ego4D. “+C” denotes Modeling with Context Embedding (prior utterances), and “+V” denotes Modeling with Video Representation.
Figure B.1: Qualitative analysis of failure modes and the corrective effects of SoCoT and DSEM. (Case A) illustrates a perception failure where the baseline relies on text; SoCoT corrects this by extracting multimodal stress indicators. (Case B) demonstrates a cognitive failure resolved by DSEM, which tracks evolving facts via the Memory Board to detect logical inconsistencies.
| Models | Real-Life Trial | Box of Lies | GPT-4o-mini | GPT-4o | Qwen2.5-VL-7B | Qwen3-8B |
|---|---|---|---|---|---|---|
| Accuracy | 63.8 | 68.6 | 66.3 | 74.0 | 76.4 | 67.1 |
| Accuracy (Binary) | 11.8 | 7.1 | 39.4 | 27.6 | 33.9 | 22.8 |
| Macro-Precision | 32.3 | 40.0 | 43.6 | 52.7 | 53.3 | 42.7 |
| Macro-Recall | 32.8 | 35.5 | 47.3 | 52.5 | 47.1 | 43.8 |
| Macro-F1 | 32.5 | 35.5 | 44.5 | 51.2 | 48.8 | 42.6 |
| F1-TRUE | 13.5 | 12.6 | 36.1 | 31.5 | 48.9 | 26.1 |
Table B.4: Comparison of open- and closed-source LLMs with methods from Real-Life Trial Pérez-Rosas et al. (2015a) and Box of Lies Zadeh et al. (2018b) on the MIDA-Ego4D Dataset. Performance is reported using Overall Accuracy, Binary Accuracy (on / samples), Macro-Averaged Precision, Recall, and F1-score, as well as fine-grained F1-scores for the , , and categories.
| Baseline | Accuracy | Binary | Macro-F1 |
|---|---|---|---|
| Majority-class | 84.5 | 0.0 | 30.5 |
| Uniform-random | 33.5 | 33.1 | 23.7 |
Table B.5: Floor baselines on MIDA-Ego4D.
| Metric | GPT-4o-mini | CoT | Memo-RAG | ToM-Decomp | SoCoT | DSEM |
|---|---|---|---|---|---|---|
| Accuracy | 66.3 | 65.9 | 70.2 | 44.2 | 72.2 | 70.7 |
| Accuracy (Binary) | 39.4 | 40.9 | 29.9 | 12.6 | 35.4 | 41.7 |
| Macro-F1 | 44.5 | 45.9 | 44.3 | 28.5 | 46.9 | 47.8 |
Table B.6: Comparison of SoCoT and DSEM against baseline reasoning paradigms on MIDA-Ego4D: CoT Wei et al. (2022) , Memo-RAG Lewis et al. (2020) , and ToM-Decomp Kosinski (2023) .
| Models | Text-only | Face & Body | Face & Voice | Body & Voice | Vision & DSEM | Face & DSEM | Voice & DSEM | All Modality |
| Accuracy | 66.3 (+0.0) | 59.1 (-7.2) | 59.4 (-6.9) | 56.6 (-9.7) | 69.4 (+3.1) | 70.2 (+3.9) | 69.9 (+3.6) | 54.8 (-11.5) |
| Accuracy (Binary) | 39.4 (+0.0) | 26.0 (-13.4) | 27.6 (-11.8) | 33.9 (-5.5) | 29.9 (-9.5) | 25.2 (-14.2) | 27.6 (-11.8) | 15.0 (-24.4) |
| Macro-Precision | 43.6 (+0.0) | 36.2 (-7.4) | 38.1 (-5.5) | 36.6 (-7.0) | 42.8 (-0.8) | 44.1 (+0.5) | 45.0 (+1.4) | 33.2 (-10.4) |
| Macro-Recall | 47.3 (+0.0) | 37.1 (-10.2) | 39.9 (-7.4) | 38.7 (-8.6) | 43.1 (-4.2) | 43.8 (-3.5) | 44.5 (-2.8) | 30.8 (-16.5) |
| Macro-F1 | 44.5 (+0.0) | 35.7 (-8.8) | 38.3 (-6.2) | 35.7 (-8.8) | 42.9 (-1.6) | 43.8 (-0.7) | 44.4 (-0.1) | 31.4 (-13.1) |
Table C.1: Modality Ablation Study on MIDA-Ego4D (GPT-4o-mini). All models are compared against the Text-only baseline. Values are shown as score ( vs baseline, p.p.) .
| Models | GPT-4o-mini | Human-validated | Qwen2.5VL | Human-validated |
|---|---|---|---|---|
| Accuracy | 57.9 | 57.9 | 76.9 | 76.9 |
| Accuracy (Binary) | 22.2 | 25.6 | 48.9 | 53.5 |
| Macro-Precision | 37.8 | 39.0 | 59.3 | 51.0 |
| Macro-Recall | 34.5 | 35.0 | 54.2 | 51.0 |
| Macro-F1 | 36.1 | 36.9 | 56.1 | 51.0 |
Table D.1: Comparison of Model Performance on the MIDA-Ego4D Subset and the Human-validated Subset. Results for closed-source and open-source models are presented alongside the human-validated subset results for comparative analysis.
| Model | Requested identifier | Served snapshot |
|---|---|---|
| GPT-4o-mini | gpt-4o-mini | gpt-4o-mini-2024-07-18 † |
| GPT-4o | gpt-4o | Provider default (run window) ‡ |
| GPT-5-nano | gpt-5-nano | Provider default (run window) ‡ |
| GPT-5-mini | gpt-5-mini | Provider default (run window) ‡ |
| Gemini-2.5-Pro | gemini-2.5-pro-exp-03-25 | gemini-2.5-pro-exp-03-25 † |
| Claude-3.5-Haiku | claude-3-5-haiku-20241022 | claude-3-5-haiku-20241022 |
Table E.1: Closed-source API identifiers used for the reported results. † Snapshot confirmed from API response metadata. ‡ Alias not pinned; served by the provider default from August 2025 to January 2026.
Figure F.1: The complete and detailed annotation prompt used to generate the MIDA dataset’s veracity labels.(Game Rules - Part 1)
Figure F.2: The complete and detailed annotation prompt used to generate the MIDA dataset’s veracity labels. (Game Rules - Part 2)
Figure F.3: The complete and detailed annotation prompt used to generate the MIDA dataset’s veracity labels. (Game Annotation - Part 1)
Figure F.4: The complete and detailed annotation prompt used to generate the MIDA dataset’s veracity labels.(Game Annotation - Part 2)
Figure F.5: The complete prompt template used to evaluate MLLMs on the MIDA task. The full list of roles and rules was provided to the models during the experiments (Main Task - Part 1).
Figure F.6: The complete prompt template used to evaluate MLLMs on the MIDA task (Main Task - Part 2).
Figure G.1: The complete prompt used in the SoCoT pipeline specifically on facial expressions.
Figure G.2: The complete prompt used in the SoCoT pipeline on holistic body language, including posture, gestures, and expression.
Figure G.3: The prompt used to guide the model in converting voice features (Intensity, Pitch, Jitter) into standardized descriptive labels.
Figure G.4: The prompt used for the model social reasoning, integrating multimodal analyses to infer social motives, knowledge, and alignment within the game context.
Figure G.5: The complete prompt used to instruct a model to act as a Memory-Board Extractor, updating player states based on turn-by-turn evidence in the game “One Night Ultimate Werewolf.”