Human-Inspired Social Engagement Analysis via Interpretable Mutual Visual Attention

Urwa Fatima; Mohammad Zohaib; Francesca Odone; Nicoletta Noceti

cs.CVAug 25, 2026

Human-Inspired Social Engagement Analysis via Interpretable Mutual Visual Attention

Authors: Urwa Fatima, Mohammad Zohaib, Francesca Odone, Nicoletta Noceti

Organizations: DIBRIS-Università degli Studi di Genova, Genova, IT · MaLGa - Machine Learning Genoa center, Genova, IT · Istituto Italiano di Tecnologia, Genova, IT

Abstract

Understanding social interactions from non-verbal visual data is important for behavior analysis and activity monitoring. We propose an interpretable computational model of social engagement inspired by psychological theories of mutual visual attention. Rather than learning interaction patterns end-to-end, our framework explicitly models dyadic visual attention and aggregates these cues into interpretable measures of individual and group engagement. The resulting modular framework combines state-of-the-art head orientation estimation with lightweight geometric reasoning, producing explanations that remain accessible to non-technical users. We evaluate the proposed approach on a variety of data through quantitative experiments and demonstrate its practical usefulness with qualitative visualizations designed to support teachers, caregivers, and social workers in understanding group interaction dynamics.

Figures & tables

Figure 1: Proposed four-stage framework.

Dataset	Frames	Method	F1 ${}_{B}\uparrow$	F1 ${}_{U}\uparrow$	F1 ${}_{NO}\uparrow$	Avg. F1 $\uparrow$
GP-Static++	13789	[ 28 ]	0.56	0.44	0.49	0.49
		Ours	0.70	0.45	0.60	0.58
CMU	2368	Ours	0.68	0.29	0.54	0.50

Table 1: Comparison with the frame-based method [ 28 ] on the GP-static++ dataset. In addition to global F1 score on the test set, we also provide the performance of bidirectional events (

F1_{B}

), unidirectional events (

F1_{U}

), and no interaction (

F1_{NO}

). We also report results on the CMU dataset showing the influence of group cardinality (2 for GP-Static++, 3 for CMU).

Panel of Figure 2 — Figure 2: Visualization tool on a sequence from Pexels (Fig. 2(a) ) and CMU (Fig. 2(b) ). In each of them, on the left we report the sequential frames with estimated head directions and interaction states (top), and the interaction states over time for each person with partner IDs (bottom). On the right, we report the corresponding individual interaction analysis (top: frame level interaction counts; bottom: IEI( $p$ ) for each person).

Panel of Figure 3 — Figure 3: Examples from the Pexels sequences with different interaction scenarios. From left, GES is 31.8, 68.7, and 80.6.

Visual Attention

Visual attention studies how people and machines select relevant image regions over time for perception, action, and efficient computation.

See all 62 papers

Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation
Jun 13, 2026Shijun Wan, Xuehai Wu, Jiwen Zhang +2LLM Agent Evaluation AI Agent Benchmarks
Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute
GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions
May 15, 2026Junho Kim, Xu Cao, Houze Yang +6Multimodal QA Multimodal Model Evaluation
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Amazon AGI +1
Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities
Sep 7, 2026Raphael Lorenzo-Louis, Bertrand Luvison, Serena IvaldiHuman Behavior Prediction Human-Robot Interaction
Inria, CNRS, UL, Loria, HUCEBOT F-54600 Villers-les-Nancy France

Jun 13, 2026J/K move · Enter open · S save

Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation

Shijun Wan, Xuehai Wu, Jiwen Zhang, Siyuan Wang+1

Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute

LLM Agent Evaluation AI Agent Benchmarks Multimodal Agents

Open paper AskPDF arXiv

Social interaction depends on both language and visible social signals, such as facial expressions, posture, gaze, and emotional shifts. Yet existing social-agent benchmarks are largely text-based and rarely test whether multimodal agents can use visual cues to guide interaction. We introduce \benchmarkname{}, a benchmark evaluating visual social intelligence in multimodal social simulation. It contains 240 scenarios, 585 role instances, and 2,340 role-task instances, combining aligned textual-visual evidence, structured role profiles, and four role-level tasks: expression task, characteristic task, interaction regulation task, and interaction outcome task. Evaluating seven recent MLLMs under verbalized-vision and direct-vision reveals a clear gap between local role enactment and interaction management: role-specific expression and conflict handling are near saturation, whereas interaction regulation and visually grounded outcome achievement remain substantially more difficult. The code is released at https://github.com/JunsWan/AgentViSS, and the dataset is available at https://huggingface.co/datasets/JunsWan/AgentViSS.

Abstract

Visual Attention

Explore similar work

Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation

GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities