PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos

Heran Hu; Yuhang Dong; Zhucun Xue; Kaiwei Chen; Yong Liu; Jiangning Zhang

cs.CVOct 20, 2026

PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos

Authors: Heran Hu, Yuhang Dong, Zhucun Xue, Kaiwei Chen, Yong Liu, Jiangning Zhang

Organizations: Zhejiang University, APRIL Lab · China University of Geosciences

Abstract

Understanding human needs and selecting responses require distinct judgments that aggregate accuracy can obscure. We introduce PsycheMME, a video benchmark informed by psychology, with eight capabilities spanning grounding, states, needs, interaction decisions, memory, and risk. Drawing on Social Information Processing and the Perception-Action Cycle, this organization separates human understanding from response selection. We construct 1,000 questions from 630 episodes through independent answering with proposed references withheld and expert review. Our Reasoning-Verified protocol weights answer credit by the support and logical consistency of explicit rationales against annotated cues. Across 17 models, Timing has the lowest Verified scores and credit retention, with weaker support judgments for correct Timing answers. Models with similar Need scores still differ in answer completeness on open questions. Input controls and SFT/GRPO examine context dependence and adaptation. Evaluating answers alongside rationales locates where models fail across understanding, response selection, and justification.

Figures & tables

Figure 1: Overview of PsycheMME. (a) Social situations involving people across 16 scene categories. (b) Eight diagnostic capabilities covering social cues, psychological understanding, interaction decisions, memory, and risk. (c) Overall results for 17 models, showing performance differences and gaps after rationale verification.

Benchmark	Videos/Clips	Prompt	Ego	G	W	N	T	S	A	M	R
HumanVideo-MME [ 7 ]	1,200	8,700	✗	✓	✓	✗	✗	✗	✗	✗	✗
Social Genome [ 40 ]	272	1,486	✗	✓	✓	✗	✗	✗	✗	✗	✗
Human Behavior Atlas [ 45 ]	35,046	101,964	✗	$\bullet$	✓	✗	✗	✗	✗	✗	✗
HumanSense [ 50 ]	-	3,882	✗	✓	✓	$\bullet$	✗	✓	✓	$\bullet$	✗
SIV-Bench [ 34 ]	2,792	5,455	✗	✓	✓	✗	✗	✗	✗	✗	✗
R 3 -Bench [ 43 ]	312	4,840+316	✗	✓	✓	$\bullet$	✗	✗	✗	✗	✗

Table 1: Comparison with Existing Multimodal Benchmarks of Human Understanding and Interaction.

Figure 2: Illustrative scenarios and capabilities in PsycheMME. The benchmark spans human understanding, interaction decisions, memory, and social constraints. The illustrated scene is adapted from CASTLE [ 55 ] .

Figure 3: Question construction framework of PsycheMME. Video curation, independent agent answers with proposed answers withheld, verification, iterative refinement, and human quality control.

Figure 4: PsycheMME statistics. (a) Eight capabilities and their subcategories. (b) Question distribution across source groups, capabilities, and formats. (c) Distribution of video durations. (d) Distribution of question lengths in words. (e) Frequent keywords in benchmark questions.

Proprietary Models
Model	Overall		Understanding			Interaction			Contextual
Model	Raw	Ver.	G	W	N	T	S	A	M	R
Gemini 3.6 Flash [ 26 ]	85.68	80.03	91.76	79.68	69.97	47.81	79.32	89.58	90.92	91.21
Claude Opus 5 [ 3 ]	82.34	76.71	89.31	77.80	69.10	40.70	79.18	86.24	86.56	84.82
GPT-5.6 Sol [ 46 ]	81.09	75.98	92.13	78.75	67.28	32.99	74.86	84.25	91.11	86.47
Qwen3.8-Max [ 53 ]	81.48	75.59	90.87	77.61	69.17	39.33	72.84	82.46	84.48	87.94

Table 2: Main results on PsycheMME (%). We report 17 frozen models and two adapted checkpoints. Raw and Ver. are macro scores across eight capabilities; capability columns report Verified scores. Colors mark the top five scores in each column and model group. Bold names mark the adapted Qwen3.5-4B checkpoints.

Figure 5: Performance breakdown. Performance by capability, verification stages, score distributions, and loss attribution.

Figure 6: Illustrative model responses in PsycheMME. The cases illustrate preference mismatch, temporal cue mismatch, disagreement about interaction timing, and goal substitution.

Table 9

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Episodes	Questions	Choice	Open
CASTLE	61	87	84	3
EasyCom	40	70	66	4
Ego4D	142	230	196	34
EgoCom	38	60	48	12
IndEgo	8	8	8	0
MERCI+	67	91	66	25

Appendix

Table 6: Final benchmark composition by source. Episodes are distinct within each source; Choice and Open denote choice and open questions. Counts sum to 630 episodes and 1,000 questions.

Figure 7: Question construction and review. The Answer Agent independently examines the episode without access to the proposed answer or its supporting cues. Psychology Knowledge guides the three agents’ judgments, and critic feedback supports up to two rounds of refinement. Final inclusion requires unanimous human approval.

Agent	Inputs	Assessment	Output
Question Agent	Video, public context, target capability	Observed cues support the question and reference	Candidate question, answer, options, and supporting cues
Answer Agent	Video, question, options, public context	Answerability without the proposed reference or its supporting cues	Independent answer or identified ambiguity
Critic Agent	Proposed and independent answers, source cues	Cue support, logical coherence, construct alignment	Quality assessment and refinement suggestions

Appendix

Table 8: Roles of the three construction agents. Psychology Knowledge guides their interpretation of the target constructs. The Answer Agent receives only the video, question, options, and public context.

Capability	Required cues	Review criterion
G	A uniquely grounded speaker, addressee, person, or interpersonal relation	Corroborate the target person’s visible identity using speech, behavior, and interaction relations.
W	Observable cues supporting a psychological state or its stated cause	Distinguish situated psychological interpretation from clinical diagnosis; check the support for each stated cause.
N	An explicit request, refusal, or contextual basis for the inferred support need	Check whether the available cues indicate willingness to receive the proposed support.
T	Turn structure, ongoing activity, urgency, or a supported opportunity to respond	Assess the intervention opportunity against turn structure, current activity, and urgency.
S	Response content and tone suitable for the supported state, need, and context	Assess the fit of content, strategy, and tone to the person’s state, need, and context.
A	A feasible physical or nonverbal response consistent with the interaction	Distinguish recognition of observed behavior from selection of an appropriate next action.

Appendix

Table 9: Criteria for expert question review by capability. Each target judgment requires support from the available episode and public context.

Figure 8: Evaluation and scoring procedure. Reference answers and cue annotations are provided only to the evaluator. Answer correctness and rationale quality are assessed separately before computing Verified credit.

Field / task	Required representation
answer : choice	Option ID for one required option, e.g., "A" ; array for several, e.g., ["A", "C"]
answer : open	Concise answer in natural language
G/W/N/T/S/A/M/R	Short relevant judgment or JSON null

Appendix

Table 10: Structured response fields and answer representations. The eight rationale fields are short explicit judgments; unused fields are JSON null.

Support / logic label	Interpretation of Evidence Support	Interpretation of Logical Integrity	Weight
Strong / fully coherent	Relevant claims are well supported by frozen cue annotations.	Active judgments jointly support the answer without contradiction.	1.2
Normal / acceptable	Support is adequate, with limited omissions or specificity.	The explanation is consistent and adequate, with minor gaps.	1.0
Weak / weak	A material inference or supporting link lacks adequate support from observed cues.	A material reasoning link is incomplete or inadequately justified.	0.8
Contradicted / contradictory	A material claim conflicts with the supplied cues.	Active judgments contradict one another or the final answer.	0.5
Insufficient / unevaluable	No evaluable support is available for the explicit rationale.	No evaluable logical explanation is present.	0.0

Appendix

Table 11: Interpretation of rationale labels and their scoring multipliers. Evidence Support and Logical Integrity are separate judgments.

Case	$a$	$c$	$e$	$\ell$	Verified (%)
Full support	1	1	1.2	1.2	100.00
Normal / acceptable	1	1	1.0	1.0	69.44
Weak support	1	1	0.8	1.2	66.67
Contradictory logic	1	1	1.2	0.5	41.67
Missing rationale	1	0	0	0	0.00
Incorrect answer	0	1	1.2	1.2	0.00

Appendix

Table 12: Arithmetic examples of Reasoning-Verified credit for individual questions. Each row specifies

(a,c,e,\ell)

; the final column is

100ace\ell/\lambda^{2}

, with

\lambda=1.2

Figure 9: Qwen3.5-4B adaptation on the same 1,000 PsycheMME questions with greedy decoding. Left: overall Raw and Verified macro scores for Base, SFT at step 458, and SFT+GRPO at step 32. Right: Raw scores across the eight capabilities, with Response Timing shaded. Capability abbreviations follow Table 13 .

Model	Overall		Understanding			Interaction			Contextual
	Raw	Ver.	G	W	N	T	S	A	M	R
Base (Qwen3.5-4B)	47.68	37.71	63.98	49.38	47.94	40.28	30.93	53.19	40.98	54.79
SFT (step 458)	72.58	52.13	91.94	79.69	61.86	31.94	65.98	85.11	81.97	82.19
GRPO (step 32)	73.19	52.62	92.47	80.00	62.89	34.72	67.53	85.11	81.97	80.82

Appendix

Table 13: Base, SFT, and GRPO results on PsycheMME (%, 1,000 questions per model). Overall Raw and Ver. denote raw accuracy and Verified scores, respectively, averaged equally over the eight capabilities. Capability columns report Raw scores. Blue shading indicates score magnitude; bold denotes the highest observed value in each column, including ties.

Figure 10: Verified scores (%) for 16 foundation models on 1,000 PsycheMME questions, sorted by Verified macro score within each model group. G: Grounding; W: Psychological Understanding; N: Need; T: Response Timing; S: Verbal Response; A: Action Decision; M: Memory; R: Boundary and Risk. All cells share a 0 to 100 color scale; the gold outline marks Response Timing.

Figure 11: Input controls for Qwen3.5-9B (left) and MiniCPM-V4.5 (right) on the same 1,000 PsycheMME questions. Paired bars show Raw and Verified macro scores. Full supplies video and ASR; Question-only supplies neither; ASR-only omits frames; Visual-only omits ASR. Single-image replaces the video frames with one midpoint image while retaining ASR. The Full condition is shaded for reference.

Figure 12: History controls for Qwen3.5-9B on the same 40 Memory questions from 24 episodes. Bars show Raw and Verified scores with 95% intervals from bootstrapping episodes. Mismatched History averages seeds 42, 2026, and 3407; its interval is computed for that mean.

Video-Language Model Evaluation

Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.

See all 103 papers

SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos
Aug 31, 2026Zheyu Huang, Zijing Shi, Haozhe Luo +4Video QA Multimodal Reasoning
AAII, University of Technology Sydney · Northeastern University · Fudan University +1
PsyJev: Psychological Analysis via System-1 Judgment
Oct 27, 2026Fangyuan Liu, Sirui Zhao, Xiaohao Wang +3Computational Psycholinguistics
State Key Laboratory of Cognitive Intelligence, School of Computer Science and Technology, University of Science and Technology of China · Computer Science, Beijing Institute of Technology · School of Information Science and Technology, University of Science and Technology of China
Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?
Aug 13, 2026Koen P. de Vries, Xavier Alameda-Pineda, Estefanía Talavera +1Supervised Fine-Tuning Multimodal Large Language Models
Inria at Univ. Grenoble Alpes, CNRS, LJK, France · University of Twente, The Netherlands

Aug 31, 2026J/K move · Enter open · S save

SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos

Zheyu Huang, Zijing Shi, Haozhe Luo, Huadong Tang+3

AAII, University of Technology Sydney · Northeastern University · Fudan University+1

Video QA Multimodal Reasoning Video Reasoning Social Reasoning Causal Reasoning in Videos Counterfactual Reasoning

Open paper AskPDF arXiv

Recent advances in Large Multimodal Models (LMMs) have greatly improved video understanding, yet their ability to reason about human-centered social situations remains limited. Existing benchmarks typically rely on videos with a single observed trajectory, making it difficult to determine whether models truly understand social dynamics or merely exploit recurring narrative patterns. We introduce SocialReasonBench, a video multiple-choice QA benchmark for evaluating socially grounded reasoning in scenarios derived from interactive narratives. Built from gameplay videos of Detroit: Become Human, the benchmark leverages branching storylines where player decisions lead to alternative social outcomes that can be checked against the game's own script, flowchart, and recorded branches. We develop a multi-agent curation pipeline that localizes socially meaningful clips, grounds answer labels in game-state signals, and generates theory-guided questions with diagnostic distractors. SocialReasonBench covers seven reasoning dimensions, including intent recognition, emotional empathy, moral dilemma, counterfactual reasoning, and causal antecedent. Experiments on contemporary LMMs show that models perform reasonably well on basic social understanding but struggle with counterfactual and causal reasoning. Further ablation and diagnostic error analyses reveal that models often depend on incomplete modality cues and fall into reasoning traps such as visual shortcuts, highlighting a gap between observable event recognition and deeper reasoning over latent social states.

Abstract

Video-Language Model Evaluation

Explore similar work

SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos

PsyJev: Psychological Analysis via System-1 Judgment

Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?