Despite their advanced reasoning capabilities, state-of-the-art Multimodal Large Language Models (MLLMs) still lack a core component of human intelligence: the ability to "read the room" and assess deception in complex social interactions. To rigorously evaluate this limitation, we introduce Multimodal Interactive Deception Assessment (MIDA), a new task and multimodal dataset comprising synchronized video and text with verifiable ground-truth labels for every statement. We benchmark 12 state-of-the-art open- and closed-source MLLMs and identify a substantial performance gap: even powerful models such as GPT-4o struggle to reliably distinguish truthful statements from deceptive ones. Our analysis shows that current models have difficulty grounding language in multimodal social cues and modeling what others know, believe, or intend. To address these limitations, we propose a Social Chain-of-Thought (SoCoT) reasoning pipeline and a Dynamic Social Epistemic Memory (DSEM) module. Experimental results show that structured epistemic memory consistently improves deception assessment, while multimodal social-cue reasoning remains a challenging and important direction for future research.
Figures & tables
Figure 1: Overview of the MIDA Benchmark and New Modules for Deception Reasoning. (A) A semi-automated process where game transcripts, rules, and metadata are used by an LLM assistant to generate veracity labels. (B) The MIDA task. It requires models to perform reasoning from Persuasive Strategy Classification to Deception Assessment with a supporting justification. (C) The evaluated MLLMs. (D) Proposed social-reasoning modules: (D.1) SoCoT grounds inference in multimodal cues, while (D.2) DSEM tracks participants’ social states.
Figure 2: Persuasive Strategy proportion and distribution of veracity labels across Persuasive Strategy categories in MIDA-Ego4D/MIDA-YouTube.
Figure 3: Demonstration of SoCoT and DSEM Modules. (Left) The SoCoT extracts multimodal behavioral primitives from Face, Body, and Voice. (Right) The DSEM module maintains a structured ‘Memory Board’ tracking participants’ evolving beliefs and social states. (Bottom) An example of SoCoT’s and DSEM’s reasoning steps.
Metric
GPT4o -mini
GPT4o
GPT5 -nano
GPT5 -mini
Gemini 2.5-pro
Claude 3.5-haiku
Ego4D
Identity
84.4
67.5
74.6
65.4
68.0
71.6
Accusation
45.1
47.6
40.0
40.0
45.6
44.4
Defense
22.2
29.9
36.5
31.9
34.3
36.0
Evidence
49.0
61.7
60.7
62.9
66.3
50.6
Interrogation
72.0
82.3
81.0
82.3
84.1
76.5
Call for Action
51.0
65.0
50.7
59.7
52.6
45.4
Table 1: Evaluations of closed-source models for the Persuasive Strategy Classification task. We report F1 scores for each of the six categories, along with the average F1 score and Joint Accuracy. (Open-source results in Appendix B .)
Metric
GPT-4o mini
GPT-4o
GPT5 nano
GPT5 mini
Gemini 2.5-pro
Claude 3.5-haiku
Llama 3-8B
InternVL 3.5-8B
Qwen2 VL-7B
Qwen2.5 VL-7B
Qwen3 8B
DeepSeek R1-8B
Ego4D
Accuracy
66.3
74.0
67.9
69.7
68.6
71.7
51.2
64.3
65.3
76.4
67.1
39.3
Accuracy (Binary)
39.4
27.6
14.2
10.2
26.0
4.7
68.5
34.7
51.2
33.9
22.8
56.7
Macro-Precision
43.6
52.7
39.1
38.4
43.2
42.0
38.8
43.4
41.8
53.3
42.7
40.1
Macro-Recall
47.3
52.5
37.9
36.3
46.3
35.6
46.7
47.4
49.2
47.1
43.8
46.3
Macro-F1
44.5
51.2
38.1
36.5
44.2
35.2
34.2
43.8
43.7
48.8
42.6
32.4
F1-TRUE
36.1
31.5
19.4
16.0
25.9
8.4
33.6
33.9
43.6
48.9
26.1
31.7
Table 2: Evaluations on MIDA-Ego4D/MIDA-YouTube. Performance is reported using Overall Accuracy, Binary Accuracy (on TRUE / FALSE samples), Macro-Averaged Precision, Recall, and F1-score, as well as fine-grained F1-scores for the TRUE , FALSE , and NEUTRAL categories. ( Blue: closed-source; yellow: open-source.)
Task
Metric
Default w/ History
w/o History
Text-only
w/ Vision (1-frame)
w/ Vision (3-frame)
Strategy
Identity Declaration
84.4 (+0.0)
81.1 (-3.3)
84.4 (+0.0)
77.5 (-6.9)
73.2 (-11.2)
Accusation
45.1 (+0.0)
47.3 (+2.2)
45.1 (+0.0)
49.4 (+4.3)
46.6 (+1.5)
Defense
22.2 (+0.0)
22.7 (+0.5)
22.2 (+0.0)
23.3 (+1.1)
18.2 (-4.0)
Evidence
49.0 (+0.0)
56.1 (+7.1)
49.0 (+0.0)
46.3 (-2.7)
44.4 (-4.6)
Interrogation
72.0 (+0.0)
77.6 (+5.6)
72.0 (+0.0)
63.2 (-8.8)
63.8 (-8.2)
Call for Action
51.0 (+0.0)
53.1 (+2.1)
51.0 (+0.0)
46.8 (-4.2)
50.3 (-0.7)
Table 3: Temporal Ablation Study on MIDA-Ego4D (GPT-4o-mini). (Left) Compares the model with full dialogue history against the prior dialogue removed (w/o History) model. (Right) Compares the Text-only model with 1-frame/3-frame vision models.
Figure 4: Radar chart of MLLMs’ accuracy in the MIDA task across Persuasive Strategy categories.
Closed-source MLLMs
GPT-4o-mini
SoCoT-Face
SoCoT-Body
SoCoT-Voice
w/ DSEM
Accuracy
66.3 (+0.0)
74.3 (+8.0)
71.5 (+5.2)
70.7 (+4.4)
70.7 (+4.4)
Accuracy (Binary)
39.4 (+0.0)
32.3 (-7.1)
37.8 (-1.6)
36.2 (-3.2)
41.7 (+2.3)
Macro-Precision
43.6 (+0.0)
47.9 (+4.3)
46.5 (+2.9)
45.8 (+2.2)
46.5 (+2.9)
Macro-Recall
47.3 (+0.0)
46.4 (-0.9)
48.3 (+1.0)
47.6 (+0.3)
50.2 (+2.9)
Macro-F1
44.5 (+0.0)
47.1 (+2.6)
47.2 (+2.7)
46.4 (+1.9)
47.8 (+3.3)
Open-source MLLMs
Qwen2.5VL
SoCoT-Face
SoCoT-Body
SoCoT-Voice
w/ DSEM
Table 4: Evaluation of MLLMs with SoCoT and DSEM on the MIDA-Ego4D Dataset. Performance is reported using Overall Accuracy, Binary Accuracy, and Macro-Averaged Precision, Recall, and F1-score.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Models
Llama-3-8B
InternVL3_5-8B
Qwen2-VL-7B
Qwen2.5-VL-7B
Qwen3-8B
DeepSeek-R1-8B
Ego4D
Identity Declaration
70.2
78.6
50.3
64.8
82.5
58.8
Accusation
26.5
34.3
36.4
8.3
27.6
43.2
Defense
3.9
24.1
14.9
3.9
11.0
25.6
Evidence
39.9
51.0
40.9
13.3
52.8
46.3
Interrogation
60.4
86.1
37.8
75.4
82.1
74.6
Call for Action
53.6
56.9
46.9
50.0
69.2
56.9
Appendix
Table B.1: Evaluations of open-source models for the Persuasive Strategy Classification task on the Ego4D and YouTube. We report F1 scores for each of the six categories, along with the average F1 score and Joint Accuracy (all six labels predicted correctly).
Metric
BERT
RoBERTa
MT-BERT
BERT + C
RoBERTa + C
MT-BERT + C
GPT-4o-mini
GPT-4o
Qwen2.5-VL-7B
Qwen3-8B
Identity
82.6
81.7
80.9
79.9
81.5
79.8
84.4
67.5
64.8
82.5
Accusation
48.8
51.7
51.5
52
59.4
54.4
45.1
47.6
8.3
27.6
Defense
29.3
33.1
25.9
33.8
33
36.5
22.2
29.9
3.9
11.0
Evidence
54.2
51.7
53.6
57.1
52.4
61.5
49.0
61.7
13.3
52.8
Interrogation
82.8
83.4
83
81
83.5
83.2
72.0
82.3
75.4
82.1
Call for Action
39.4
43.3
56.6
49.5
43.7
50.8
51.0
65.0
50.0
69.2
Appendix
Table B.2: Comparison of open- and closed-source LLMs (0-shot) with the method in Lai et al. (2023) on the Persuasive Strategy Classification Task on Ego4D. “+C” denotes Modeling with Context Embedding (prior utterances).
Metric
BERT + V
RoBERTa + V
MT-BERT + V
B. + C + V
Ro. + C + V
MT. + C + V
GPT-4o-mini
GPT-4o
Qwen2.5-VL-7B
Qwen3-8B
Identity
81.5
79.8
79.9
84.5
82.7
80.7
84.4
67.5
64.8
82.5
Accusation
52.1
51.4
51.9
52.8
58.5
55.2
45.1
47.6
8.3
27.6
Defense
28.4
31.3
35.4
34.5
35.4
36.1
22.2
29.9
3.9
11.0
Evidence
52.8
54.6
53.3
54.9
53.4
60.5
49.0
61.7
13.3
52.8
Interrogation
83.3
82.8
84.8
82.7
83.8
83.6
72.0
82.3
75.4
82.1
Call for Action
42.4
50.1
53.9
47.3
46.1
50.0
51.0
65.0
50.0
69.2
Appendix
Table B.3: Comparison of open- and closed-source LLMs (0-shot) with the method in Lai et al. (2023) on the Persuasive Strategy Classification Task on Ego4D. “+C” denotes Modeling with Context Embedding (prior utterances), and “+V” denotes Modeling with Video Representation.
Figure B.1: Qualitative analysis of failure modes and the corrective effects of SoCoT and DSEM. (Case A) illustrates a perception failure where the baseline relies on text; SoCoT corrects this by extracting multimodal stress indicators. (Case B) demonstrates a cognitive failure resolved by DSEM, which tracks evolving facts via the Memory Board to detect logical inconsistencies.
Models
Real-Life Trial
Box of Lies
GPT-4o-mini
GPT-4o
Qwen2.5-VL-7B
Qwen3-8B
Accuracy
63.8
68.6
66.3
74.0
76.4
67.1
Accuracy (Binary)
11.8
7.1
39.4
27.6
33.9
22.8
Macro-Precision
32.3
40.0
43.6
52.7
53.3
42.7
Macro-Recall
32.8
35.5
47.3
52.5
47.1
43.8
Macro-F1
32.5
35.5
44.5
51.2
48.8
42.6
F1-TRUE
13.5
12.6
36.1
31.5
48.9
26.1
Appendix
Table B.4: Comparison of open- and closed-source LLMs with methods from Real-Life Trial Pérez-Rosas et al. (2015a) and Box of Lies Zadeh et al. (2018b) on the MIDA-Ego4D Dataset. Performance is reported using Overall Accuracy, Binary Accuracy (on TRUE / FALSE samples), Macro-Averaged Precision, Recall, and F1-score, as well as fine-grained F1-scores for the TRUE , FALSE , and NEUTRAL categories.
Baseline
Accuracy
Binary
Macro-F1
Majority-class
84.5
0.0
30.5
Uniform-random
33.5
33.1
23.7
Appendix
Table B.5: Floor baselines on MIDA-Ego4D.
Metric
GPT-4o-mini
CoT
Memo-RAG
ToM-Decomp
SoCoT
DSEM
Accuracy
66.3
65.9
70.2
44.2
72.2
70.7
Accuracy (Binary)
39.4
40.9
29.9
12.6
35.4
41.7
Macro-F1
44.5
45.9
44.3
28.5
46.9
47.8
Appendix
Table B.6: Comparison of SoCoT and DSEM against baseline reasoning paradigms on MIDA-Ego4D: CoT Wei et al. (2022) , Memo-RAG Lewis et al. (2020) , and ToM-Decomp Kosinski (2023) .
Models
Text-only
Face & Body
Face & Voice
Body & Voice
Vision & DSEM
Face & DSEM
Voice & DSEM
All Modality
Accuracy
66.3 (+0.0)
59.1 (-7.2)
59.4 (-6.9)
56.6 (-9.7)
69.4 (+3.1)
70.2 (+3.9)
69.9 (+3.6)
54.8 (-11.5)
Accuracy (Binary)
39.4 (+0.0)
26.0 (-13.4)
27.6 (-11.8)
33.9 (-5.5)
29.9 (-9.5)
25.2 (-14.2)
27.6 (-11.8)
15.0 (-24.4)
Macro-Precision
43.6 (+0.0)
36.2 (-7.4)
38.1 (-5.5)
36.6 (-7.0)
42.8 (-0.8)
44.1 (+0.5)
45.0 (+1.4)
33.2 (-10.4)
Macro-Recall
47.3 (+0.0)
37.1 (-10.2)
39.9 (-7.4)
38.7 (-8.6)
43.1 (-4.2)
43.8 (-3.5)
44.5 (-2.8)
30.8 (-16.5)
Macro-F1
44.5 (+0.0)
35.7 (-8.8)
38.3 (-6.2)
35.7 (-8.8)
42.9 (-1.6)
43.8 (-0.7)
44.4 (-0.1)
31.4 (-13.1)
Appendix
Table C.1: Modality Ablation Study on MIDA-Ego4D (GPT-4o-mini). All models are compared against the Text-only baseline. Values are shown as score ( Δ vs baseline, p.p.) .
Models
GPT-4o-mini
Human-validated
Qwen2.5VL
Human-validated
Accuracy
57.9
57.9
76.9
76.9
Accuracy (Binary)
22.2
25.6
48.9
53.5
Macro-Precision
37.8
39.0
59.3
51.0
Macro-Recall
34.5
35.0
54.2
51.0
Macro-F1
36.1
36.9
56.1
51.0
Appendix
Table D.1: Comparison of Model Performance on the MIDA-Ego4D Subset and the Human-validated Subset. Results for closed-source and open-source models are presented alongside the human-validated subset results for comparative analysis.
Model
Requested identifier
Served snapshot
GPT-4o-mini
gpt-4o-mini
gpt-4o-mini-2024-07-18 †
GPT-4o
gpt-4o
Provider default (run window) ‡
GPT-5-nano
gpt-5-nano
Provider default (run window) ‡
GPT-5-mini
gpt-5-mini
Provider default (run window) ‡
Gemini-2.5-Pro
gemini-2.5-pro-exp-03-25
gemini-2.5-pro-exp-03-25 †
Claude-3.5-Haiku
claude-3-5-haiku-20241022
claude-3-5-haiku-20241022
Appendix
Table E.1: Closed-source API identifiers used for the reported results. † Snapshot confirmed from API response metadata. ‡ Alias not pinned; served by the provider default from August 2025 to January 2026.
Figure F.1: The complete and detailed annotation prompt used to generate the MIDA dataset’s veracity labels.(Game Rules - Part 1)
Figure F.2: The complete and detailed annotation prompt used to generate the MIDA dataset’s veracity labels. (Game Rules - Part 2)
Figure F.3: The complete and detailed annotation prompt used to generate the MIDA dataset’s veracity labels. (Game Annotation - Part 1)
Figure F.4: The complete and detailed annotation prompt used to generate the MIDA dataset’s veracity labels.(Game Annotation - Part 2)
Figure F.5: The complete prompt template used to evaluate MLLMs on the MIDA task. The full list of roles and rules was provided to the models during the experiments (Main Task - Part 1).
Figure F.6: The complete prompt template used to evaluate MLLMs on the MIDA task (Main Task - Part 2).
Figure G.1: The complete prompt used in the SoCoT pipeline specifically on facial expressions.
Figure G.2: The complete prompt used in the SoCoT pipeline on holistic body language, including posture, gestures, and expression.
Figure G.3: The prompt used to guide the model in converting voice features (Intensity, Pitch, Jitter) into standardized descriptive labels.
Figure G.4: The prompt used for the model social reasoning, integrating multimodal analyses to infer social motives, knowledge, and alignment within the game context.
Figure G.5: The complete prompt used to instruct a model to act as a Memory-Board Extractor, updating player states based on turn-by-turn evidence in the game “One Night Ultimate Werewolf.”
Deception detection is a critical and highly challenging task within affective computing and behavioral analysis. Existing deep learning methods typically treat this task as a straightforward classification problem; however, this black-box approach lacks interpretability and fails to capture the complex logical deduction processes utilized by human experts when identifying lies. While Multimodal Large Language Models (MLLMs) have shown potential, applying them effectively requires a bridge between low-level audiovisual cues and high-level logical reasoning. In this paper, we propose DeceptionX, a novel MLLM framework that shifts the paradigm of deception detection from black-box classification to an interpretable Observe-Think-Summarize reasoning process. To address the scarcity of high-quality reasoning data, we first constructed DeceptChain, a high-quality dataset developed through a human-in-the-loop process. This dataset synthesizes fine-grained visual and auditory evidence (such as micro-expressions and vocal tremors) into structured chain-of-thought reasoning data. Furthermore, we propose a three-stage training pipeline and a Discrepancy-Aware Redundancy Elimination~(DARE) strategy for DeceptionX to further enhance the model's generalization capabilities. Extensive experiments demonstrate that DeceptionX not only outperforms existing MLLM baselines and state-of-the-art methods on standard real-world benchmarks but also provides transparent, expert-level reasoning paths, bridging the critical gap between accuracy and interpretability in multimodal deception detection.
Jiayu Zhang, Shuo Ye, Jiajian Huang +7
Great Bay University Dongguan, Guangdong, China · Hong Kong Polytechnic University HongKong, China
As large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems, understanding their deceptive capabilities is fundamental to safety. Controlled social deduction games provide a reproducible proxy for isolating and evaluating these complex adversarial behaviors. We present the open-source benchmark framework ParliamentBench based on the game Secret Hitler to evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry. We evaluate 16 LLMs across 1,600 simulated matches playing each other, playing against humans, and compare them against a large set of online games. We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency. Our experiments reveal that frontier models achieve strong performance across cooperative and deceptive roles, with a strong top-four cluster (GPT-5.4, Kimi K2.5, Grok 4.1 Fast, and DeepSeek 3.1 Terminus), whereas the weakest models fall short of random (33%) and simple algorithmic (45%) baselines. Most LLMs struggle to maintain a consistent deceptive persona throughout an entire game, with deception retention dropping below 50%.
Niklas Bauer, Lars Benedikt Kaesberg, Akiko Aizawa +3
University of Göttingen, Germany · National Institute of Informatics, Japan · University of Tokyo, Japan
Multimodal deception detection is critical for identifying fraudulent intentions, yet existing approaches predominantly rely on end to end black--box paradigms. These methods suffer from a severe lack of interpretability failing to provide transparent reasoning trajectories and struggling to explicitly capture the subtle, cross modal inconsistencies inherent in deceptive behaviors. To transcend these limitations, we propose ThinkDeception, a novel and interpretable multimodal deception detection framework. As a pioneering effort, it introduces Multimodal Large Language Models (MLLMs) into this domain, transforming deception detection from a traditional binary classification task into an explicit cognitive reasoning process. Facilitated by the first meticulously annotated step--by--step multimodal Chain of Thought (CoT) dataset, we develop a foundational model, ThinkDeception Base, empirically validating the critical role of modal inconsistency in decoding deception. Building upon this foundation, our core innovation lies in proposing Visual-Audio Consistency Group Relative Policy Optimization(VAC--GRPO) equipped with a progressive training strategy. Distinct from standard GRPO, we stratify the training data into four progressive difficulty tiers, guiding the model through a psychologically grounded easy--to--hard cognitive transition. By innovatively coupling this dynamic curriculum scheduler with a multi dimensional, process aware reward mechanism and a reflective learning paradigm, we significantly elevate the model's overall reasoning quality. Extensive experiments on mainstream benchmarks demonstrate that ThinkDeception establishes a new SOTA, significantly outperforming existing methods in both detection accuracy and rationale quality. Ultimately, this work successfully drives the field of deception detection toward interpretable, multimodal cognitive reasoning.
Jinhao Song, Shan Liang, Yiqun Yue +2
Xi’an Jiaotong-Liverpool University · Suzhou, China