SocialCoAct: Enabling and Benchmarking Egocentric Social Co-Action for VLM-Based Humanoid Robots

Jiancong Ma; Kun Li; Jin Jiang; Shengcai Liao

cs.ROOct 17, 2026

SocialCoAct: Enabling and Benchmarking Egocentric Social Co-Action for VLM-Based Humanoid Robots

Authors: Jiancong Ma, Kun Li, Jin Jiang, Shengcai Liao

Organizations: College of Computing and Artificial Intelligence (CCAI), United Arab Emirates University (UAEU), Al Ain, Abu Dhabi, United Arab Emirates.

Abstract

Enabling humanoid robots to participate effectively in social interactions is important for their integration into human-centered environments. Existing methods, however, typically map pre-generated speech directly to continuous motion trajectories but have limited awareness of complex social environments and lack explicit high-level semantic action planning. We introduce SocialCoAct, a unified agent-centered framework for VLM-based egocentric social co-action. We formulate egocentric social co-action as an end-to-end agent task and develop SocialCoAct-Agent, which transforms egocentric observations and dialogue history into verbal responses and atomic action sequences, aligns each action with its corresponding utterance span, and executes the resulting behaviors in MuJoCo and on a physical humanoid robot. To evaluate the VLM capabilities required by this task, we construct SocialCoAct-Bench from continuous egocentric recordings of natural multi-party interactions. The benchmark comprises 423 human-verified multiple-choice questions across three subtasks: Long Episode Understanding (LEU), Speaker Intention Understanding (SIU), and Coordinated Action Planning (CAP). Across a diverse set of closed-source and open-weight VLMs, including general and egocentric models, the highest overall accuracy is 77.54%, while accuracy on the hard subset reaches only 46.51%. Extended reasoning improves Qwen3-VL-2B and Qwen3-VL-4B by 19.62 and 17.02 percentage points, respectively, but reduces inference throughput by 53.6 $\times$ and 20.2 $\times$ . Visual-input ablations further show that the benefits of visual context vary across models. SocialCoAct provides a foundation for developing and evaluating VLM-based humanoid agents that coordinate context-aware speech and action in complex social interactions. The codebase and benchmark will be made publicly available upon publication.

Figures & tables

Fig. 1: Overview of SocialCoAct-Bench, which comprises three subtasks: Long Episode Understanding (LEU), Speaker Intention Understanding (SIU), and Coordinated Action Planning (CAP).

Fig. 2: Overview of the SocialCoAct-Agent architecture. The leftmost panels show examples of atomic robot actions in MuJoCo.

Fig. 3: Overview of the SocialCoAct-Bench construction pipeline.

Benchmark	Venue	Egocentric	Modalities	SS	ESR	AUA
Social understanding benchmarks
Social-IQ [ 12 ]	CVPR’19	$\times$	V+A+T	$\checkmark$	$\times$	$\times$
Social Genome [ 13 ]	EMNLP’25	$\times$	V+A+T	$\checkmark$	$\times$	$\times$
SIV-Bench [ 14 ]	ACL’26	$\times$	V+A+T	$\checkmark$	$\times$	$\times$
Human–robot interaction benchmarks
SCAND [ 28 ]	RA-L’22	$\checkmark$	V	$\checkmark$	$\times$	$\times$

TABLE I: Comparison with representative benchmarks from social, HRI, and egocentric domains. Abbreviations: I, V, A, T = image, video, audio, text; SS = social scene; ESR = embodied social response; AUA = action–utterance alignment.

Fig. 4: Difficulty and source distributions of SocialCoAct-Bench.

		Overall	Task			Difficulty
Model	Params	All	CAP	SIU	LEU	Easy	Medium	Hard
Random Guess	–	25.00	25.00	25.00	25.00	25.00	25.00	25.00
Closed-source models
Gemini-3.1-Pro [ 36 ]	–	69.74	56.90	81.05	92.13	78.31	77.70	37.21
Claude-Sonnet-5 [ 38 ]	–	77.54	71.55	80.00	91.01	92.59	76.35	46.51
Grok-4.6 [ 39 ]	–	47.52	53.14	47.37	32.58	59.26	42.57	30.23

TABLE II: Performance comparison on SocialCoAct-Bench. All values are accuracy (%). The best and second-best results in each column are highlighted in bold and underlined , respectively.

Setting	Overall	LEU	SIU	CAP	Efficiency (samples/s)
Qwen3-VL-2B
Instruct	39.48	42.70	22.11	45.19	1.60
Thinking	59.10 $\uparrow$ 19.62 pp	58.43 $\uparrow$ 15.73 pp	54.74 $\uparrow$ 32.63 pp	61.09 $\uparrow$ 15.90 pp	0.030 $\downarrow$ 53.6 $\times$
Qwen3-VL-4B
Instruct	56.03	68.54	53.68	52.30	1.51
Thinking	73.05 $\uparrow$ 17.02 pp	80.90 $\uparrow$ 12.36 pp	66.32 $\uparrow$ 12.64 pp	72.80 $\uparrow$ 20.50 pp	0.075 $\downarrow$ 20.2 $\times$

TABLE III: Effect of reasoning mode on Qwen3-VL series.

Model	Text only	Vision + Text
Qwen3-VL-2B	43.93	45.19 $\uparrow$ 1.26 pp
InternVL2.5-2B	30.96	26.78 $\downarrow$ 4.18 pp
EgoMind-7B	39.75	48.54 $\uparrow$ 8.79 pp

TABLE IV: Modality ablation study on CAP with different VLMs (%).

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 452 papers

Aug 21, 2026

Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models

Effective social interaction requires agents to translate mental state inferences into coordinated behavioral signals across verbal and nonverbal channels simultaneously. Yet existing benchmarks evaluate theory of mind (ToM) reasoning and embodied behavior in isolation, leaving unmeasured the gap between social inference and social action. We introduce MOSAIC (Multimodal Orchestration of Social Action, Inference, and Communication), a controlled benchmark in which two embodied agents interact across cooperative and competitive scenarios requiring integration of verbal statements, spatial trajectories, gaze direction, and facial expression under systematically varied ToM constraints. Evaluating 13 models, including 11 VLMs, across 200 trials per model, we find that VLMs fail to produce behaviors consistent with the expected outcomes under ToM-order constraints, and that imposing explicit ToM-order constraints produces no reliable behavioral change aligned with the specified reasoning level. Signal-level analysis reveals two sequential bottlenecks: most models cannot produce directionally coherent nonverbal signals, and even when signals are present, VLM agents fail to interpret others behaviors and react to them. PCM-LLM, included as a structured architectural reference point with an explicit ToM module, succeeds across all conditions, suggesting that explicit belief-action coupling is a sufficient ingredient for this class of tasks.

Tonglin Yan, Gregoire Sergeant-Perthuis, David Rudrauf

CIAMS, Université Paris-Saclay · CQSB, Sorbonne Université

Vision-Language Models Multimodal Agents

Ask PDF

Oct 20, 2026

Evaluating Social Reasoning in Embodied Vision-Language Models

Foundation models now serve as reasoning layers for robot teams, but it remains unclear whether embodiment-specialized vision-language models (VLMs) preserve the social-reasoning capabilities of general-purpose VLMs. We evaluate Gemini Robotics-ER on SHREC, a benchmark of real human-robot social interactions, and compare it with published general-purpose VLM baselines. Robotics-ER outperforms GPT-4o-mini on error/competence detection but performs worse on detailed social-attribute judgments; adding a visual frame helps one task while hurting the other. These results suggest embodiment-specialized models may perform differently on social judgments that can be read more directly from observable interaction cues versus those requiring more inference about a person's internal state. Because the comparison spans model families, these differences cannot be attributed to embodiment specialization alone. This matters for multi-robot systems, where a shared foundation model could cause systematic social-perception errors to influence multiple coordination decisions. This work presents an initial evaluation, with broader cross-context and video-based analyses left for future work.

Daniel Weiner, Raj Korpan

The Graduate Center, City University of New York (CUNY), New York, NY, USA · Hunter College and The Graduate Center, City University of New York (CUNY), New York, NY, USA

VLMs for Robotics Reasoning for VLMs Social Robotics

Ask PDF

Jul 13, 2026

Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies

As mobile robots become more integrated into everyday human environments, social robot navigation is becoming essential for ensuring human comfort, safety, and trust. While reinforcement learning (RL) navigation policies provide the fast inference and reactive behavior necessary for real-time deployment, they still lack flexible semantic reasoning capabilities and often fail to generalize to complex social scenarios. Recent approaches have increasingly turned to vision-language models (VLMs) in place of RL policies to improve semantic and social reasoning in robot navigation. Nevertheless, their high computational cost and slow inference remain major barriers to real-time deployment. To overcome these limitations, we introduce HUMA (Hybrid Understanding for Multi-modal social Navigation), a hybrid architecture that dynamically balances the computational efficiency of RL policies with the deep semantic understanding of VLMs. Our approach uses a reactive RL policy to handle low-density, routine navigation tasks, while conditioning it on a post-trained high-level VLM when a human enters sensitive situations, such as the robot's proximity zone. We evaluate HUMA on the Social-MP3D and Social-HM3D benchmarks, where it achieves task success improvements of 20% and 3%, respectively, while significantly reducing personal space violations and human collisions against state-of-the-art baselines. Extensive ablation studies validate each architectural component, and real-world deployment on the Mirokaï mobile robot further demonstrates the practical viability of our approach.

Ali Ahmadi, Hamed Rahimi, Adrien Jacquet Cretides +3

Institut des Syst`emes Intelligents et de Robotique (ISIR) Sorbonne Universit´e France

RL for Robotics Social Robot Navigation Vision-Language Navigation VLMs for Robotics

Ask PDF

Abstract

Human-Robot Interaction

Explore similar work

Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Models

Evaluating Social Reasoning in Embodied Vision-Language Models

Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies