GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?

Zonghuan Li; Hayley Hung; Chirag Raman

cs.AIOct 10, 2026

GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?

Authors: Zonghuan Li, Hayley Hung, Chirag Raman

Organizations: Department of EEMCS, Delft University of Technology, Delft, Netherlands

Abstract

Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to specific facial, gestural, or prosodic signals that shape social meaning. Current multimodal LLM benchmarks largely evaluate final-answer understanding over complete episodes, rewarding post-hoc recognition of social meaning rather than the unfolding perception required for real-time interaction. Drawing on interactionally embedded Gestalt principles of multimodal human communication, we introduce GesBench: a human-grounded benchmark for cue-level social perception in multimodal LLMs. GesBench uses cumulative sub-utterance clips to measure when models form stable interpretations of speaker intention and social affordance, and human gaze-guided corruption to evaluate whether regions attended by humans are functionally important for model judgments. Built from more than 6000 utterance-level clips from scripted and unscripted conversational datasets, GesBench includes human annotations of unfolding judgments and gaze on a representative subset. Across open and closed multimodal models, we find a clear human-model gap: humans form predictive understandings early and revise more often, while models often hedge or repeat across different clips. GesBench shifts social evaluation from whether models eventually recognize meaning to when and where that meaning is formed. Our codes are publicly available at https://anonymous.4open.science/status/GesBench-share-212C.

Figures & tables

Figure 1: Instead of only asking model perception after the complete utterance, GesBench measures when and how interpretations update. It also collects human gaze during utterance interpretation to compare model judgments of sensitivity to human-attended visual cues. Together, these temporal and spatial probes evaluate when, where, and how social meaning is formed.

Figure 2: Illustration of the evaluation pipeline. The temporal pipeline takes video inputs, cut into small clips, and build CVC which is then given to both human and models. The spatial pipeline measures human eye gaze data, builds a gaze profile, and does corruption to make gaze based corruption videos with varying radius and offset to prompt the models. Finally, we compute specificity, anticipation, revision type, and gaze manipulation similarity for metrics.

Figure 3: Process of building human attention profile. (a) We record participants’ eye gaze data with Pupil Core glasses while watching videos; (b) eye gaze profile by aggregation. Each annotator’s gaze for a frame is shown. (c) Build per-frame corrupted video

\widetilde{D}_{n}(x,r)

with

x=0

, by blocking the region centered at the gaze point. (d) Build

\widetilde{D}_{n}(x,r)

with

x\neq 0

, shifting the manipulation area. A complete set of video manipulation sample is provided in the supplementary material.

Panel of Figure 4 — Figure 4: Specificity vs. anticipation over the first third of sub-utterance answers for each video, for $Q_{1}$ (left) and $Q_{2}$ (right). Anticipation is $0.5$ at chance. Bars are 95% bootstrap intervals over videos.

Panel of Figure 5 — Figure 5: RT classification with NLI-roberta-base model. Humans show notably higher rate of shifting and contradicting than all models, both signs for significant revisions. Many models are occupied with simple copying, and do restating or elaboration more when they do revise.

Panel of Figure 6 — Figure 6: GMS plot for $Q_{1}$ and $Q_{2}$ . Qwen2.5 models has lower GMS with larger $r$ and smaller $x$ , showing they are sensitive to corruption. Gemma shows same pattern for $Q_{1}$ but not obvious for $Q_{2}$ . Other models are not sensitive to gaze based manipulation.

Figure 7: A qualitative illustration of GMS. For Qwen2.5-Omni-7B model on $Q_{1}$ , $(x,r)=(0,0.3)$ gives the lowest GMS, while $(0.5,0.1)$ gives $1$ on GMS, showing its sensitivity over $(x,r)$ . Video source from Mintrec2. A complete sample of video manipulation and the answers is provided in the supplementary material.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Illustration of the Covfee main page for monitoring the annotation tasks.

Panel of Figure 9 — Figure 9: Sample of paid annotators user interface on Covfee when doing the text annotation. They are first asked to sign the consent form to procees( 9(a) ), then go through the instruction( 9(b) ) and example( 9(c) ). 9(d) shows the user actual panel they see during annotation: specifying whether perception has changed, choosing confidence, and recording audios about their perception and reasoning.

Dataset	# utt.	AL (s)	# utt.	AL	# utt.	AL	# utt.	AL	# utt.	AL
	Full		$\mathcal{T}_{1}$		$\mathcal{T}_{2}$		$\mathcal{T}_{3}$		$\mathcal{S}$
MELD	1,870	3.273	216	3.229	141	3.284	8	2.938	219	3.242
MIntRec2	2,974	2.968	215	2.928	143	2.871	8	2.500	219	2.922
Seamless	1,364	4.117	159	4.157	105	4.171	4	5.000	162	4.154
Overall	6,208	3.312	590	3.369	389	3.371	20	3.175	600	3.372

Appendix

Table 1: GesBench data details. Seamless means Seamless Interaction. AL means average utterance length.

Figure 10: Example of the annotator task page for providing gaze annotations. The left is an instruction panel, and the right shows the video. The annotators are asked to think of the two questions ( $Q_{1}$ , $Q_{2}$ ) during watching, and they click "Submit Annotation" button to proceed to the next video. The scrolling bar is intentionally disabled for this task to ensure that the video displaying region stays fixed within the screen.

Model	Answers	Coverage	Exact	Cov. $<$ 0.9
Qwen3-Omni-30B	72735	0.999	0.85	0.00
Qwen2.5-Omni-7B	44656	1.000	0.99	0.00
Qwen2.5-Omni-3B	40659	0.955	0.89	0.05
Ming-Lite-Omni	80142	0.999	0.99	0.00
Gemma 4-E4B	80230	1.000	1.00	0.00
Gemini2.5-flash	11506	1.000	0.96	0.00

Appendix

Table 2: Extraction fidelity, with

Q_{1}

and

Q_{2}

answers pooled. Coverage: mean share of an extracted answer’s words found, in order, in the model’s reply. Exact: share of answers that are one unbroken stretch of the reply. Cov.

<

0.9: share of answers with coverage below 0.9.

Figure 11: Comparison of early commintment type over categories, with raw texts from models without extraction. Humans are the highest in mentioning predictions, with most models providing hedges or saying there is insufficient information.

Panel of Figure 12 — Figure 12: Comparison of early commintment type over categories, on extracted texts for models with both questions. Humans are the highest in mentioning predictions, with most models providing hedges or saying there is insufficient information.

Panel of Figure 13 — Figure 13: Comparison of answer lengths before and after extraction done on $\mathcal{T}_{1}$ . The left part shows original answers while the right part shows after extraction.

Panel of Figure 14 — Figure 14: Comparison of answer lengths before and after extraction done on full set when applicable. The left part shows original answers while the right part shows after extraction.

Panel of Figure 15 — Figure 15: Rank gain $\Delta$ computed for all models with human interval shown in black dotted line. Qwen2.5-Omni-7B/3B and Ming-Lite-Omni are close to 0, while Gemini models are gaining much more positively. The comparison should be made with $0$ instead of human standards.

Panel of Figure 16 — Figure 16: GMS plot for $Q_{1}$ and $Q_{2}$ . Qwen2.5 models has lower GMS with larger $r$ and smaller $x$ , showing they are sensitive to corruption. Gemma shows same pattern for $Q_{1}$ but not obvious for $Q_{2}$ . Other models are not sensitive to gaze based manipulation.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

Social Caption: Evaluating Social Understanding in Multimodal Models
Jan 21, 2026Leena Mathur, Bhaavanaa Thumu, Youssouf Kebe +1Multimodal Large Language Models Multimodal Reasoning
School of Computer Science, Carnegie Mellon University
MIBench: Evaluating LMMs on Multimodal Interaction
Mar 13, 2026Yu Miao, Zequn Yang, Yake Wei +5Multimodal Large Language Models Multimodal Model Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China +1
Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?
Aug 13, 2026Koen P. de Vries, Xavier Alameda-Pineda, Estefanía Talavera +1Supervised Fine-Tuning Multimodal Large Language Models
Inria at Univ. Grenoble Alpes, CNRS, LJK, France · University of Twente, The Netherlands

Jan 21, 2026J/K move · Enter open · S save

Social Caption: Evaluating Social Understanding in Multimodal Models

Leena Mathur, Bhaavanaa Thumu, Youssouf Kebe, Louis-Philippe Morency

School of Computer Science, Carnegie Mellon University

Multimodal Large Language Models Multimodal Reasoning Social Reasoning Multimodal Model Evaluation Multimodal Understanding MLLM Evaluation

Open paper AskPDF arXiv

Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce SOCIAL CAPTION, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to generate relevant information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges demonstrate a path towards scaling automated evaluation of multimodal social understanding.

Abstract

Multimodal Model Evaluation

Explore similar work

Social Caption: Evaluating Social Understanding in Multimodal Models

MIBench: Evaluating LMMs on Multimodal Interaction

Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?