Human-Inspired Social Engagement Analysis via Interpretable Mutual Visual Attention
Authors: Urwa Fatima, Mohammad Zohaib, Francesca Odone, Nicoletta Noceti
Organizations: DIBRIS-Università degli Studi di Genova, Genova, IT · MaLGa - Machine Learning Genoa center, Genova, IT · Istituto Italiano di Tecnologia, Genova, IT
Understanding social interactions from non-verbal visual data is important for behavior analysis and activity monitoring. We propose an interpretable computational model of social engagement inspired by psychological theories of mutual visual attention. Rather than learning interaction patterns end-to-end, our framework explicitly models dyadic visual attention and aggregates these cues into interpretable measures of individual and group engagement. The resulting modular framework combines state-of-the-art head orientation estimation with lightweight geometric reasoning, producing explanations that remain accessible to non-technical users. We evaluate the proposed approach on a variety of data through quantitative experiments and demonstrate its practical usefulness with qualitative visualizations designed to support teachers, caregivers, and social workers in understanding group interaction dynamics.
Figures & tables
Figure 1: Proposed four-stage framework.
Dataset
Frames
Method
F1 B↑
F1 U↑
F1 NO↑
Avg. F1 ↑
GP-Static++
13789
[ 28 ]
0.56
0.44
0.49
0.49
Ours
0.70
0.45
0.60
0.58
CMU
2368
Ours
0.68
0.29
0.54
0.50
Table 1: Comparison with the frame-based method [ 28 ] on the GP-static++ dataset. In addition to global F1 score on the test set, we also provide the performance of bidirectional events ( F1B ), unidirectional events ( F1U ), and no interaction ( F1NO ). We also report results on the CMU dataset showing the influence of group cardinality (2 for GP-Static++, 3 for CMU).
Figure 2: Visualization tool on a sequence from Pexels (Fig. 2(a) ) and CMU (Fig. 2(b) ). In each of them, on the left we report the sequential frames with estimated head directions and interaction states (top), and the interaction states over time for each person with partner IDs (bottom). On the right, we report the corresponding individual interaction analysis (top: frame level interaction counts; bottom: IEI( p ) for each person).
Figure 3: Examples from the Pexels sequences with different interaction scenarios. From left, GES is 31.8, 68.7, and 80.6.
Social interaction depends on both language and visible social signals, such as facial expressions, posture, gaze, and emotional shifts. Yet existing social-agent benchmarks are largely text-based and rarely test whether multimodal agents can use visual cues to guide interaction. We introduce \benchmarkname{}, a benchmark evaluating visual social intelligence in multimodal social simulation. It contains 240 scenarios, 585 role instances, and 2,340 role-task instances, combining aligned textual-visual evidence, structured role profiles, and four role-level tasks: expression task, characteristic task, interaction regulation task, and interaction outcome task. Evaluating seven recent MLLMs under verbalized-vision and direct-vision reveals a clear gap between local role enactment and interaction management: role-specific expression and conflict handling are near saturation, whereas interaction regulation and visually grounded outcome achievement remain substantially more difficult. The code is released at https://github.com/JunsWan/AgentViSS, and the dataset is available at https://huggingface.co/datasets/JunsWan/AgentViSS.
Shijun Wan, Xuehai Wu, Jiwen Zhang +2
Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute
Understanding social interactions requires reasoning over subtle non-verbal cues, yet current multimodal large language models (MLLMs) often fail to identify who interacts with whom in multi-person videos. We introduce GRASP, a large-scale social reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events. GRASP contains 290K question--answer pairs over 46K videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze--gesture reasoning, together with GRASP-Bench for evaluation. Unlike prior resources that focus on either isolated cues or high-level social QA, GRASP builds questions from identity-consistent gaze trajectories, deictic gestures, and their joint compositions into social events. Moreover, we propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction. Experiments show that SGR improves performance on GRASP-Bench while maintaining zero-shot performance on related social video QA benchmarks.
Junho Kim, Xu Cao, Houze Yang +6
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Amazon AGI +1
Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.