cs.RONov 14, 2026

Self-Calibrating Vision-Agent-Action: Deploying Pick-and-Place to New Robot Cells without Demonstrations or Fine-Tuning

Authors: Jae Hun Hwang, Yechan Seo, Hong-In Won

Organizations: Manufacturing AI Research Center, Korea Institute of Industrial Technology (KITECH), Incheon, Republic of Korea. · Department of Artificial Intelligence Convergence, Hanyang University, Seoul, Republic of Korea. · Department of Intelligent Robotics, Sungkyunkwan University, Suwon, Republic of Korea. · HYU-KITECH Joint Department, Graduate School, Hanyang University, Seoul, Republic of Korea.

Abstract

In simulation, Vision-Agent-Action (VAA) performs pick-and-place in robot cells with gripper or base-frame changes without robot demonstrations or fine-tuning. It uses pre-trained vision-language and language models for interpretation, measures the command map and camera geometry from its own motion and fingertips, estimates lens distortion, and plans grasps from measured object depth. A controlled study of eight public LIBERO-fine-tuned vision-language-action models (VLAs) varies robot hardware, camera geometry, appearance, and base frame. Gripper and frame changes dominate their losses in the tested fractional design. Across 18 cells with these changes, frozen VAA achieves 230/360 successes (64%), the refined VAA 268/360 (74%), and the strongest released VLA on this subset 76/360 (21%). Paired ablations support the self-calibration bundle, lens estimation under compounded changes, and the measured-depth pipeline. On a two-step LIBERO-Goal chain with object state carried between instructions, VAA completes 54/60 chains in changed cells, while four evaluated VLAs complete none.

Figures & tables

Explore similar work

CardsList
  1. From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model

    Jul 6, 2026Wenhao Li, Xueying Jiang, Quanhao Qian +4Language-Conditioned Robot ManipulationHand-Eye Calibration

  2. InteractVLA: Turning VLAs into Interactive Tools for Agentic Robotic Manipulation

    Nov 17, 2026Chen Zhao, Zipeng Ye, Shifeng Bao +5Agentic AI for RoboticsTool-Using Vision-Language Agents

  3. Vision-Language-Action Models: A Life-Cycle Survey of Design, Training and Deployment

    Nov 13, 2026Suvarna Kadam, Soran ParsaVision-Language-Action ModelsVLA Model Training