MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interaction

Joerg Deigmoeller; Nakul Agarwal; Stephan Hasler; Daniel Tanneberg; Anna Belardinelli; Reza Ghoddoosian; Chao Wang; Felix Ocker

cs.ROMar 19, 2026

MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interaction

Authors: Joerg Deigmoeller, Nakul Agarwal, Stephan Hasler, Daniel Tanneberg, Anna Belardinelli, Reza Ghoddoosian, Chao Wang, Felix Ocker,

Organizations: Honda Research Institute Europe, 63073 Offenbach, Germany · Honda Research Institute USA, San Jose, CA 95134, USA.

Abstract

We introduce MERGE, a system for situational grounding of actors, objects, and events in dynamic human-robot group interactions. Effective collaboration in such settings requires consistent situational awareness, built on persistent representations of people and objects and an episodic abstraction of events. MERGE achieves this by uniquely identifying physical instances of actors (humans or robots) and objects and structuring them into actor-action-object relations, ensuring temporal consistency across interactions. Central to MERGE is the integration of Vision-Language Models (VLMs) guided with a perception pipeline: a lightweight streaming module continuously processes visual input to detect changes and selectively invokes the VLM only when necessary. This decoupled design preserves the reasoning power and zero-shot generalization of VLMs while improving efficiency, avoiding both the high monetary cost and the latency of frame-by-frame captioning that leads to fragmented and delayed outputs. To address the absence of suitable benchmarks for multi-actor collaboration, we introduce the GROUND dataset, which offers fine-grained situational annotations of multi-person and human-robot interactions. On this dataset, our approach improves the average grounding score by a factor of 2 compared to the performance of VLM-only baselines - including GPT-4o, GPT-5 and Gemini 2.5 Flash - while also reducing run-time by a factor of 4. The code and data are available at www.github.com/HRI-EU/merge.

Figures & tables

Fig. 1: Overview of MERGE . The Action Reasoner is a VLM that integrates three structured inputs: a region of the current camera frame centered on person_y to capture the relevant scene entities and their spatial arrangement; reference images of previously detected objects_x to ground reasoning in known instances; and the four recent images captured by image_i. These inputs are sourced from the Memory module, which aggregates person locations from the Person Tracker and object references from the Object Detector at system initialization. In parallel, a lightweight Action Detector continuously predicts action labels from the image stream, and any change in a person’s predicted action triggers the Action Reasoner, with the updated label itself serving as an additional VLM input.

Fig. 2: Visualization of the prompt provided to VLM. The prompt begins with a general introduction, followed by cropped object and person images (with person ids), each uniquely identifiable via caption. The robot hand is optionally included to assess interaction. The last four images show the recent captured images before the action trigger. The prompt concludes with a task description guiding the VLM through action inference, object assignment, spatial relation, and robot interaction.

Panel of Figure 3 — Fig. 3: GROUND-Train provides a rich set of annotations captured from four synchronized camera views. Each video includes person-wise action segmentation labels, 2D pose annotations along with human and object bounding boxes across all views, with the pose and human box annotations further linked through cross-view tracking.

VLM	Method	Sorting Fruits				Pouring		Handover		Overall GS	Overall Runtime (s)
VLM	Method	1P	2P	1P+R	2P+R	2P	1P+R	2P	1P+R	$\emptyset$	$\emptyset$
GPT-4o [ 2 ]	VLM-only	0.00	0.29	0.37	0.22	0.07	0.22	0.25	0.15	0.23	0.66
GPT-4o [ 2 ]	Ours	0.46	0.42	0.48	0.47	0.38	0.22	0.44	0.50	0.42	0.36
GPT-5 [ 42 ]	VLM-only	0.11	0.29	0.32	0.15	0.17	0.35	0.38	0.00	0.24	5.52
GPT-5 [ 42 ]	Ours	0.46	0.36	0.48	0.29	0.42	0.40	0.50	0.00	0.38	1.39
Gemini 2.5 Flash [ 4 ]	VLM-only	0.00	0.19	0.13	0.07	0.07	0.07	0.13	0.04	0.10	3.23

TABLE I: Grounding Score (GS) based on comparison with ground truth sequence (

\delta

= 5s).

VLM	Method	Sorting Fruits				Pouring				Handover				All
VLM	Method	$x$	$o$	$r$	$i$	$x$	$o$	$r$	$i$	$x$	$o$	$r$	$i$	$x$	$o$	$r$	$i$
GPT-4o [ 2 ]	VLM-only	0.32	0.49	0.33	0.58	0.19	0.25	0.27	0.30	0.20	0.36	0.48	0.48	0.26	0.41	0.34	0.49
GPT-4o [ 2 ]	Ours	0.64	0.68	0.49	0.71	0.36	0.56	0.46	0.56	0.46	0.85	0.85	0.85	0.56	0.68	0.53	0.70
GPT-5 [ 42 ]	VLM-only	0.35	0.39	0.46	0.58	0.34	0.41	0.34	0.41	0.38	0.69	0.85	0.69	0.35	0.43	0.47	0.56
GPT-5 [ 42 ]	Ours	0.52	0.62	0.65	0.83	0.41	0.59	0.47	0.59	0.44	0.67	0.67	0.56	0.49	0.62	0.62	0.76
Gemini 2.5 Flash [ 4 ]	VLM-only	0.18	0.21	0.26	0.27	0.09	0.11	0.11	0.13	0.09	0.17	0.18	0.18	0.13	0.17	0.19	0.20

TABLE II: Grounding Score (GS) of each role contributing in an event tuple

T

for each experiment (

\delta

= 5s).

VLM	Method	$\delta$
VLM	Method	$1s$	$3s$	$5s$
GPT-4o [ 2 ]	VLM-only	0.19	0.22	0.23
	Ours	0.28	0.39	0.42
	Ours (cropped)	0.36	0.52	0.56
GPT-5 [ 42 ]	VLM-only	0.20	0.23	0.24
	Ours	0.27	0.36	0.38

TABLE III: Ablation study on

\delta

and input image configuration with Overall GS score reported for different VLMs.

Vision-Language Grounding

Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.

See all 418 papers

GIVE: Grounding Human Gestures in Vision-Language-Action Models
Jun 11, 2026Pengfei Liu, Gen Li, Junqiao Fan +4Language-Conditioned Robot Manipulation Goal-Conditioned Robotic Manipulation
1MARS Lab, Nanyang Technological University
Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment
Apr 7, 2026Theodor Wulff, Federico Tavella, Rahul Singh Maharjan +2Vision-Language-Action Models Vision-Language Alignment
The University of Manchester, Manchester, United Kingdom
Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration
May 29, 2026Jun Wang, Xiaohao Xu, Xiaonan HuangRobotics Robot Safety
University of Michigan, Ann Arbor

Jun 11, 2026J/K move · Enter open · S save

GIVE: Grounding Human Gestures in Vision-Language-Action Models

Pengfei Liu, Gen Li, Junqiao Fan, Boyu Ma+3

1MARS Lab, Nanyang Technological University

Language-Conditioned Robot Manipulation Goal-Conditioned Robotic Manipulation Vision-Language Grounding Robotic Manipulation Vision-Language-Action Models Human-Robot Interaction

Open paper AskPDF arXiv

Human communication is inherently multimodal, where language is often accompanied by non-verbal cues such as gestures to convey intentions. However, current Vision-Language-Action (VLA) models treat robotic manipulation as a pure text-driven task, overlooking the important role of gestures in Human-Robot Interaction (HRI). This often leads to inaccurate intent grounding and unreliable manipulation when language instructions are ambiguous or underspecified. To address this challenge, we propose GIVE (Gesture Intent via Visual-Semantic Enhancement), an effective approach that enhances pre-trained VLA models with human gesture understanding without architectural modifications. Specifically, GIVE incorporates gesture information through two complementary pathways: a visual pathway that overlays hand skeletons and fingertip rays onto robot observations for explicit object grounding, and a semantic pathway that generates high-level descriptions of human gestures and task instructions for robust intent grounding. By jointly leveraging visual and semantic guidance, GIVE enables VLA policies to better associate gestures with manipulation behaviors and adapt to dynamic interaction intents. In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%, while demonstrating strong robustness and generalization to unseen spatial layouts and diverse participants.

Abstract

Vision-Language Grounding

Explore similar work

GIVE: Grounding Human Gestures in Vision-Language-Action Models

Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignment

Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration