Self-Calibrating Vision-Agent-Action: Deploying Pick-and-Place to New Robot Cells without Demonstrations or Fine-Tuning
Authors: Jae Hun Hwang, Yechan Seo, Hong-In Won
Organizations: Manufacturing AI Research Center, Korea Institute of Industrial Technology (KITECH), Incheon, Republic of Korea. · Department of Artificial Intelligence Convergence, Hanyang University, Seoul, Republic of Korea. · Department of Intelligent Robotics, Sungkyunkwan University, Suwon, Republic of Korea. · HYU-KITECH Joint Department, Graduate School, Hanyang University, Seoul, Republic of Korea.
In simulation, Vision-Agent-Action (VAA) performs pick-and-place in robot cells with gripper or base-frame changes without robot demonstrations or fine-tuning. It uses pre-trained vision-language and language models for interpretation, measures the command map and camera geometry from its own motion and fingertips, estimates lens distortion, and plans grasps from measured object depth. A controlled study of eight public LIBERO-fine-tuned vision-language-action models (VLAs) varies robot hardware, camera geometry, appearance, and base frame. Gripper and frame changes dominate their losses in the tested fractional design. Across 18 cells with these changes, frozen VAA achieves 230/360 successes (64%), the refined VAA 268/360 (74%), and the strongest released VLA on this subset 76/360 (21%). Paired ablations support the self-calibration bundle, lens estimation under compounded changes, and the measured-depth pipeline. On a two-step LIBERO-Goal chain with object state carried between instructions, VAA completes 54/60 chains in changed cells, while four evaluated VLAs complete none.
Figures & tables
Fig. 1: VAA retains pick-and-place success under tested gripper and base-frame changes that sharply reduce released VLA performance. Top: X-VLA succeeds in the training setup and fails in a changed cell; VAA self-calibrates in the new cell and completes the task. Green and red badges report episode-level success and failure. Orange arrows show fixed-length image projections of the base directions assigned to command channels ux , uy , uz by the measured command map (Fig. 2 ). Bottom: success of the strongest VLA in each group (open) and of VAA (filled; Tables I and II ); grey dots show the other VLAs. Each tick marks the training-setup success of that group’s strongest VLA.
Fig. 2: A measured cell model connects self-calibration to task execution. Upper lane: in a development cell with an RB5 arm, Hand-E gripper, cloth table, and moved camera, VAA measures command map J , jaw width, settling time, table height, and camera geometry. The camera fit uses 24 fingertip detections (orange crosses) to estimate intrinsics K , pose [R∣t] , and lens term k1 . The 10-cm grid is projected at the table height obtained in the later touch stage. Command arrows use the fixed-length base-direction projections defined in Fig. 1 . Lower lane: logged episode frames show grounding, servoed approach, grasping from object 3-D points, and placement; later instructions reuse the model after a drift check. Orange tags give ablation success rates out of 60 episodes, versus 50/60 (83%) for the full system (Section VI ).
Fig. 3: Gripper and base-frame changes dominate the estimated VLA losses in the tested fraction. Estimated main effects in the resolution-IV fractional design; VAA’s effects are comparatively small.
Fig. 4: π 0.5 succeeds in the training setup and fails in five gripper/frame design cells. LIBERO-spatial task 0, initial state 0, one rollout per cell. Top: first frame (a base-frame rotation changes the command frame and is not visible in the image). Bottom: a frame at 70% of each episode (50% in the 90 ∘ frame cell, where the arm has left the view by 70%); border colour indicates the outcome.
Fig. 5: The same stages in a second changed cell. VAA in development cell X5 (RB5 arm, OnRobot 2FG7 gripper, base frame rotated 90 ∘ , dim light, red bowl and blue plate), LIBERO-spatial task 0; one frame per stage from the episode log. Steps 1–4 measure the cell; steps 5–7 perform the instruction with the measured model.
Released LIBERO-fine-tuned VLAs
VAA (ours)
Change
Cells
X-VLA
π 0.5
π 0
GR00T N1.7
OpenVLA
SmolVLA
τ 0
VLA-JEPA
VAA (frozen)
VAA
Δ
Training setup (reference)
1
95
90
75
90
70
80
90
95
70
85
Gripper changed
4
2
0
10
35
5
1
10
0
56
71
+36
Base frame rotated
2
45
0
0
0
0
0
0
2
78
90
+45
Combined (incl. gripper or frame)
12
23
0
2
6
0
0
0
0
64
73
+50
All gripper or frame changes
18
21
0
4
12
1
0
2
0
64
74
+53
TABLE I: On the gripper/frame design cells, VAA achieves 74% success versus 21% for the strongest released VLA by aggregate success. Success rates (%) of the eight VLAs, frozen VAA, and VAA; each cell contains 20 episodes. Δ : VAA minus the strongest VLA in the row, in percentage points. Grey: the training setup, before any change.
Released LIBERO-fine-tuned VLAs
VAA (ours)
Cell
X-VLA
π 0.5
π 0
GR00T N1.7
OpenVLA
SmolVLA
τ 0
VLA-JEPA
VAA (frozen)
VAA
Δ
Training setup (reference)
18
18
13
19
19
12
20
19
15
17
X1 RB5, Hand-E, cloth, camera
0
0
5
6
0
0
0
0
13
18
+12
X2 Doosan, PGC, dim, 45 ∘
0
0
0
0
0
0
0
0
10
16
+16
X4 Schunk, red bowl, lens
0
0
4
4
0
1
8
0
7
7
−1
X5 RB5, 2FG7, dim, 90 ∘
0
0
0
0
0
0
0
0
5
14
+14
TABLE II: Across whole-cell development stress tests, VAA achieves 68/100 successes and the strongest VLA achieves 10/100. Success counts out of 20 per cell; all changed development cells include a gripper change. The total excludes the training setup (grey). Δ : VAA minus the strongest VLA in the row, in success counts.
Variant
Gripper
Frame 90 ∘
Comb. F07
Total
Δ
p
Full system
15
19
16
50/60
Proposed component removed or replaced
Training-setup calibration reused
17
0
5
22/60
−28
< 0.001
No lens estimate ( k1=0 )
15
16
1
32/60
−18
< 0.001
RGB only (monocular depth)
8
11
1
20/60
−30
< 0.001
Reference
TABLE III: Paired VAA ablations in the PGC gripper, 90∘ frame, and combined F07 cells. Each cell contains 20 episodes (60 total). Δ is the change in successes from the full system; p values are uncorrected exact McNemar tests. The text reports the Holm-adjusted interpretation.
Fig. 6: VAA grasps the bowl from its carried-over position, while π 0.5 moves toward the cabinet without grasping it. Step 2 of the chain moves the bowl from the stove to the top of the cabinet; five frames are spread over the step. Top: π 0.5, training setup, initial state 0. Middle: VAA, same chain. Bottom: VAA in the combined cell F07 (gripper, camera, lens, light), initial state 1, reusing its calibration. The last VAA frames show the bowl still held at the cabinet top, before release; scores in Table IV use the common release-and-settle test in the text.
Released VLAs
Cell
SmolVLA
X-VLA
π 0.5
VLA-JEPA
VAA
Training setup
19 / 0
20 / 0
20 / 0
20 / 0
20 / 16
Gripper PGC
0 / 0
0 / 0
0 / 0
0 / 0
20 / 19
Frame 90 ∘
0 / 0
1 / 0
0 / 0
0 / 0
20 / 19
Combined F07
0 / 0
0 / 0
0 / 0
0 / 0
17 / 16
Changed cells
0 / 0
1 / 0
0 / 0
0 / 0
57 / 54
TABLE IV: VAA completes 54 of 60 two-step chains in the changed cells; none of the four evaluated VLAs completes a chain there. LIBERO-Goal, 20 chains per cell: step 1 puts the bowl on the stove, and step 2 moves it from there to the top of the cabinet. Entries: step 1 successes / completed chains. A step counts only after release and the settling procedure described in the text.
Fig. 7: Off-target placements and missed picks account for most remaining VAA failures. Failure categories for VAA on the 18 gripper/frame cells of Table I , by group. Each bar shows the share of all episodes in the group that failed at each stage; numbers give failed/total episodes.
Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and KnowledgeEngineering, MOE, China · Meituan, Beijing 100102, China