cs.RONov 17, 2026

InteractVLA: Turning VLAs into Interactive Tools for Agentic Robotic Manipulation

Authors: Chen Zhao, Zipeng Ye, Shifeng Bao, Yihan Lin, Haitao Shen, Yining Luo, Yuchen Xie, Jing Zhang

Organizations: Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and KnowledgeEngineering, MOE, China · Meituan, Beijing 100102, China

Abstract

Existing agentic manipulation systems treat pretrained vision-language-action (VLA) policies as opaque, language-only primitives: when a policy call fails under unfamiliar instructions or scene perturbations, the agent cannot steer its behavior and is forced into futile retries or costly manual takeover. We present InteractVLA, a framework that turns pretrained flow-based VLAs into geometrically steerable and inspectable tools through a bidirectional geometric interface with vision-language agents. In the forward direction, test-time geometric guidance translates an agent-specified end-effector target into input gradients through the action expert via a differentiable command-to-pose mapping, while soft-time target alignment treats arrival time as a latent execution index to match the target pose without distorting the policy's approach motion. In the reverse direction, action-informed agentic refinement exposes generated action chunks and calibrated camera reprojections back to the agent, allowing it to diagnose spatial discrepancies, revise targets for subsequent VLA calls, or locally adjust motion segments on top of the guided trajectory. Making the VLA controllable not only boosts online agentic tool use, but also unlocks effective recursive self-improvement (RSI), achieving state-of-the-art performance of 99.4% on LIBERO-Plus, 48.62% on RoboDojo, and 89.4% on RoboCasa365. Project page: https://lingwupro.github.io/InteractVLA/.

Figures & tables

Explore similar work

CardsList
  1. Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents

    Jul 9, 2026Yixian Zhang, Huanming Zhang, Feng Gao +14Long-Horizon Robotic ManipulationLanguage-Conditioned Robot Manipulation

  2. In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

    Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2Reasoning for VLA ModelsICL for Robotics

  3. Zero-Shot VLA Adaptation with Retrieved Guidance

    Nov 7, 2026Subin Kim, Anh-Quan Pham, Dinesh JayaramanVLA Model AdaptationVision-Language-Action Models