Event-Driven Proactive Robot Assistance through Vision-Language Reasoning

Fengkai Liu; Hao Su; Haozhuang Chi; Rui Geng; Congzhi Ren; Xuqing Liu; Chenfei Xu; Yuichi Ohsita

cs.ROOct 6, 2026

Event-Driven Proactive Robot Assistance through Vision-Language Reasoning

Authors: Fengkai Liu, Hao Su, Haozhuang Chi, Rui Geng, Congzhi Ren, Xuqing Liu, Chenfei Xu, Yuichi Ohsita, +1 more

Organizations: The University of Osaka · Nanyang Technological University · The University of Tokyo

Abstract

Assistance in collaborative manipulation is often initiated by user instructions, making high-level reasoning request-driven. In fluent human teamwork, however, partners often infer the next helpful step from the observed outcome of an action rather than waiting for instructions. Motivated by this, we investigate an event-driven formulation of proactive assistance, where human--object interaction outcomes initiate assistive reasoning without user-provided task specifications at inference time. To this end, we propose an event-driven framework that monitors workspace state changes with an event monitor and, upon event completion, extracts stabilized pre/post snapshots that characterize the resulting state transition. A frozen pretrained Vision-Language Model (VLM) then uses its semantic priors to infer the task context, decide whether assistance is appropriate, and, when needed, generate a sequence of assistive actions from the observed transition. To make outputs executable and verifiable, we restrict actions to a set of action primitives and reference objects via integer IDs.We evaluate the same framework across three distinct real world tabletop collaboration tasks without task-specific training or fine-tuning. The event-driven framework achieves performance comparable to variants given user instructions.

Figures & tables

Fig. 1 : Motivation: request-driven vs. event-driven assistance. Request-driven systems initiate planning from requests provided by the user [ 5 , 6 ] . Our event-driven setting instead triggers on human–object state transitions to infer the user’s goal and act without an additional user request.

Fig. 2 : System overview. Human-induced workspace transitions trigger event processing that produces pre/post evidence and an ID map book. Without an explicit user request, the VLM reasons over this event payload with early stopping to finalize a symbolic plan, which is geometrically grounded, executed, and verified before updating the workspace baseline.

Fig. 3 : Experimental scenarios and VLM prompts. Top: representative interaction sequences for the three tasks. Middle: the common system prompt. Bottom: the early-stop finalization prompt, which retains the generated reasoning while fixing the selected target ID. The JSON output schema is omitted for brevity.

Method	T1: Equation completion			T2: Tabletop sorting			T3: Tool introduction
Method	Appropriate	Succ. / QA / Fail	Latency	Appropriate	Succ. / QA / Fail	Latency	Appropriate	Succ. / QA / Fail	Latency
Ours	100.0	95.0 / 5.0 / 0.0	8.6	80.0	80.0 / 0.0 / 20.0	8.6	100.0	80.0 / 20.0 / 0.0	11.1
w/o Pre-event Context	93.3	91.7 / 1.7 / 6.7	6.8	60.0	53.3 / 6.7 / 40.0	7.4	90.0	80.0 / 10.0 / 10.0	9.4
Explicit Request [ 14 , 18 ]	98.3	91.7 / 6.7 / 1.7	7.2	86.7	80.0 / 6.7 / 13.3	7.2	100.0	53.3 / 46.7 / 0.0	5.0
w/o Pre-event Context	90.0	81.7 / 8.3 / 10.0	4.5	70.0	60.0 / 10.0 / 30.0	6.8	100.0	76.7 / 23.3 / 0.0	4.8
Implicit Request [ 11 , 24 ]	98.3	95.0 / 3.3 / 1.7	8.6	76.7	76.7 / 0.0 / 23.3	12.8	90.0	63.3 / 26.7 / 10.0	14.6

TABLE I : Benchmark results across the three tasks. Appropriate (%) is the sum of autonomous success and valid QA; Succ. / QA / Fail (%) reports autonomous success, valid QA, and failure as percentages; latency is reported in seconds.

Variant	T1	T2	T3
Ours (Full)	100.0	80.0	100.0
w/o Event Triggering ( $\tau=2$ s)	23.3	3.3	60.0
w/o Event Triggering ( $\tau=1$ s)	20.0	0.0	86.7
w/o Early Stopping	86.7	60.0	70.0

TABLE II : Ablation study on the proposed components. All values are reported as Appropriate (%).

Fig. 4 : Overall outcomes across the evaluated events. Appropriate is the sum of autonomous success and valid QA; a QA is valid only when the correct option is retained and leads to a correct plan.

Fig. 5 : Sensitivity analysis of the early-stop threshold

k

k=3

gives the lowest failure rate (5.0%) with 7.5% valid QA and a median latency of 8.93 s.

Method	Optional Intervention	Required Abstention
Ours	82.2	76.7
Explicit Request	83.3	50.0
Implicit Request	80.0	60.0

TABLE III : Boundary event results. All values are reported as Appropriate (%).

Explore similar work

ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation
Jun 16, 2026Tianyi Lu, Hui Zhang, Zijie Diao +8Diffusion-Based Vision-Language-Actions Robotic Manipulation
Fudan University · Shanghai Innovation Institue · China Unicom +1
A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI models
Jun 4, 2026Arash Ghasemzadeh Kakroudi, Roel PietersHuman-Robot Collaboration Robotic Manipulation
Automation Technology and Mechanical Engineering, Tampere University, 33720, Tampere, Finland
Replanning Human-Robot Collaborative Tasks with Vision-Language Models via Semantic and Physical Dual-Correction
Feb 16, 2026Taichi Kato, Takuya Kiyokawa, Namiko Saito +1Human-Robot Collaboration Robotic Manipulation
aDepartment of Systems Science, School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan · bDepartment of Systems Innovation, Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan · cMicrosoft Research Asia, Shinagawa, Tokyo, Japan +1

Jun 16, 2026, cs.ROJ/K move · Enter open · S save

ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation

Tianyi Lu, Hui Zhang, Zijie Diao, Junke Wang+7

Fudan University · Shanghai Innovation Institue · China Unicom+1

Diffusion-Based Vision-Language-Actions Robotic Manipulation Chain-of-Thought Reasoning Reasoning Skills

Open paper AskPDF arXiv

Most Vision-Language-Action (VLA) models map observations directly to actions without explicit reasoning, limiting their capacity for reasoning-intensive long-horizon tasks. To address this, existing approaches adopt Chain-of-Thought (CoT) reasoning to enable subgoal decomposition and spatial anticipation. However, those methods lack a unified architecture for effective cross-modal reasoning and fail to explicitly include inverse reasoning ability based on the target state. We argue that manipulation planning naturally decomposes into prediction, anticipating the next visual state, and inverse dynamics, inferring the actions to reach it. Bridging both requires a unified autoregressive architecture that interleaves textual and visual reasoning in a single generation process. We propose \textbf{ThinkingVLA}, a generative model that realizes this decomposition within a unified Mixture-of-Transformers architecture. ThinkingVLA consists of a forward CoT that identifies the immediate subgoal and guides the visual forecasting; the predicted image then serves as the target state, grounding an inverse CoT that reasons about spatial relationships and action intent based on the predicted image; and the final action is generated conditioned on this full reasoning context. Extensive experiments on simulation and real-world benchmarks demonstrate that ThinkingVLA consistently outperforms state-of-the-art baselines, with particularly large gains on long-horizon manipulation tasks.

Abstract

Explore similar work

ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation

A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI models

Replanning Human-Robot Collaborative Tasks with Vision-Language Models via Semantic and Physical Dual-Correction