Organizations: School of Software, Shandong University, Jinan 250101, China · Shenzhen Loop Area Institute, Shenzhen 518038, China · School of Life Science and Technology, Xidian University, Xi’an 710126, China · School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), 518055, China
Fine-tuning a vision-language-action (VLA) model on a few dozen demonstrations is the standard way to put a manipulation policy on a robot, yet the resulting policy silently assumes that the camera and the robot base stay exactly where they were during data collection: a bumped tripod or a base re-docked a few centimetres away makes it reach for the wrong place. We show that both failures are one phenomenon, a change of the camera-to-base extrinsic that the policy has memorised, and that the robot carries everything needed to undo it. We introduce WACO, a world-anchored canonicalization framework for VLA models: the arm, placed by forward kinematics at the measured joint angles, serves as a metric calibration target, and the static scene serves as the world anchor that tells a moved camera from a moved robot. From a single RGB-D frame, WACO estimates both shifts without ground-truth extrinsics or extra data, canonicalises the image to the calibration viewpoint and the state, action and start pose to the calibration base frame, and fine-tunes the unmodified policy under the artefacts of its own estimator. We also introduce LIBERO-PoseShift, the first benchmark for RGB VLAs with graded camera, base and compound shifts. Across LIBERO-PoseShift, the LIBERO-Plus camera track and four real-world tasks on a UR5e robot, WACO consistently outperforms the standard few-shot recipe and prior viewpoint-robust methods, raising shifted success by 13.2 points in simulation and by 36.7 points on the real robot, recovering 90% of the simulation gap to a ground-truth oracle and 87% of the real-robot gap, and establishing a new state of the art among methods that use no perturbed training data, with no test-time calibration. Project page: https://peacelwh.github.io/WACO/.
Figures & tables
Fig. 1: Overview. A few-shot fine-tuned VLA memorises the camera-to-base extrinsic of its demonstrations and misses the target once the camera is bumped or the robot is re-docked (left: our UR5e setup at the calibration pose and after the table and camera were moved 10 cm away from the robot). WACO estimates both shifts from a single RGB-D frame, with the arm placed by forward kinematics as the metric anchor and the static scene as the world anchor, and canonicalises the observation back to the calibration viewpoint and base frame, so that an unmodified π0.5 sees the world it was trained in (right; yellow: outline of the calibration reference in the live view). We evaluate on LIBERO-PoseShift, the LIBERO-Plus camera track and four real-world UR5e tasks.
Shift handled
Requires
Method
Cam.
Base
Both
Extr.
Pert. data
Sim
Arch.
Calib.-free
OpenVLA-OFT+ [ 7 ]
✓
✗
✗
✗
✓
✓
✓
✓
RoVi-Aug [ 8 ]
✓
✗
✗
✗
✓
✗
✓
✓
VISTA [ 9 ]
✓
✗
✗
✗
✓
✗
✓
✓
Cross-View AC [ 11 ]
✓
✗
✗
✗
✗
✓
✓
✓
RayViT [ 13 ]
✓
✗
✗
✓
✓
✗
✗
✗
TABLE I: Related Work Summary. Which pose shifts a method handles, what it requires beyond the demonstrations, and whether the policy architecture is left unchanged. Extr. : ground-truth extrinsics at test time; Pert. data : perturbed or multi-view training trajectories; Sim : a simulator that re-renders states; Arch. : policy architecture unchanged; Calib.-free : no test-time calibration object or procedure.
Fig. 2: Model Pipeline. WACO wraps an unmodified π0.5 . At reset, a training-free front end registers the FK-placed arm model to the observed depth ( arm anchor ) and matches the static scene to an empty-workspace reference ( world anchor ); their fusion yields ΔTc , ΔTb and an out-of-plane residual that gates the estimate. Channel A warps the image with ΔTc only; channel B re-expresses state, action and start pose with ΔTb only. The policy is fine-tuned once on the canonicaliser’s own artefacts (bottom left).
Fig. 3: Two-Anchor Estimation. (a) Arm anchor: the visible part of the FK-placed arm cloud is registered to the observed depth by trimmed ICP and ranked by a render-and-compare score. (b) World anchor: the live frame is registered to the empty-workspace reference by SIFT, 3D RANSAC and point-to-plane ICP. (c) Fusion closes Tbc=ΔTb−1Tbc0ΔTc ; the discarded height, roll and pitch of the base shift form a ground-truth-free consistency check.
Fig. 4: Canonicalization and error-aware training. (a) Channel A forward-splats the shifted third-person image to the nominal camera with the measured depth; disocclusions and the out-of-frustum band are grey and are not hallucinated. (b) Channel B maps the reported end-effector pose into the nominal base frame with C=Twb0ΔTb(Twb0)−1 , rotates the policy’s deltas back before execution, and homes the end effector to the start pose the demonstrations began from; in the nominal frame a base shift has become an object displacement on the table. (c) Training splats a demonstration frame to a random virtual camera and back with a perturbed shift drawn from the estimator’s error model, which reproduces the holes, borders and residual misalignment of the test-time warp without changing the action label; the state receives a planar base-shift error about the base origin. Per-frame metric depth (right) removes the ghosting that a static reference depth leaves on the arm (middle).
Camera ΔTc
Base ΔTb
Level
trans.
rot./axis
trans.
yaw
px (t / yaw)
L1
≤ 2 cm
±3∘
≤ 2 cm
±3∘
6 / 12
L2
≤ 5 cm
±5∘
≤ 5 cm
±8∘
15 / 21
L3
≤ 10 cm
±10∘
≤ 10 cm
±15∘
31 / 42
TABLE II: Perturbation Levels of LIBERO-PoseShift. Camera translations are uniform in a ball and base translations in a disc, both between half and the full radius; rotations are uniform per axis. Last column: displacement of a target object in the 2562 third-person view for the largest translation / yaw.
Fig. 5: LIBERO-PoseShift Observations. Episode 0 of a LIBERO-Spatial task (“pick up the black bowl between the plate and the ramekin and place it on the plate”) and of a LIBERO-10 task (“pick up the book and place it in the back compartment of the caddy”): third-person and wrist views at the calibration pose, and the third-person view with the wrist view inset under camera, base and compound shifts of level 1–3. Yellow: outline of the calibration view.
Fig. 6: Shift Distribution and Further Scenes. Left: 80 draws per level from the benchmark’s sampler, seen from above: camera translations, and base translations with the sampled yaw drawn as a tick (rings at 2, 5 and 10 cm). Right: four further tasks at the calibration pose and under level-3 shifts, drawn as in Fig. 5 .
L1
L2
L3
all four suites (240 per level)
1.00
1.00
0.97
Spatial and LIBERO-10 (120 per level)
1.00
1.00
0.94
worst penetration (mm)
0
0
30
TABLE III: Feasibility of the Base Shifts : share of sampled base poses with no robot–environment penetration after reset and a stable arm after settling (six samples per task and level). Invalid samples are kept in the evaluation, so they bound the success at base and compound L3.
Fig. 7: Real-Robot Tasks. The four tabletop tasks form an increasing difficulty ladder: Lemon to Basket needs only coarse localisation; Color Matching adds an instruction-dependent referent and a cup opening only slightly wider than the block; Toy on Shelf requires carrying the toy over the shelf’s wire frame and releasing it on the elevated upper tier; Toast into Toaster requires an insertion whose clearance is far below the object size, into a toaster that stays at the same spot across the demonstrations.
Component
Time
World anchor (SIFT + RANSAC)
124 ms
World anchor refinement (point-to-plane ICP)
83 ms
Arm anchor (multi-hypothesis ICP)
524 ms
Whole front end, once per episode
771 ms
Channel A warp (per control step, when applied)
11 ms
π0.5 inference (per chunk of 10 steps)
≤ 320 ms
TABLE IV: Runtime of the front end (median over 36 resets, one CPU thread) and of the canonicaliser, and homing statistics (12 episodes per condition). Policy inference was timed on a GPU shared with a training run and is an upper bound.
Kind
None
World-frame protocol
WACO runtime
GT oracle
clean
96.0
–
97.5
–
camera
96.0 / 97.0 / 82.5
–
97.5 / 94.5 / 86.0
96.5 / 97.0 / 92.5
base
97.0 / 92.0 / 74.0
94.0 / 97.0 / 85.5
95.5 / 94.5 / 91.0
95.5 / 94.0 / 91.0
both
95.5 / 89.0 / 60.5
94.5 / 92.0 / 73.0
95.5 / 90.5 / 80.5
96.5 / 93.5 / 89.5
Avg over shifts
87.1
89.3 ∗
91.7 +4.6
94.0
TABLE V: Sensitivity and Oracle Bound. The public π0.5 checkpoint on LIBERO-Spatial and LIBERO-10 (200 episodes per cell; cells are L1 / L2 / L3; base-frame protocol unless noted). The WACO runtime is applied without retraining. ∗ World-frame average over base and compound rows only, since a camera shift is protocol-independent.
Requires
Camera
Base
Both
Method
extra data
run- time
Clean
L1
L2
L3
L1
L2
L3
L1
L2
L3
Avg
B1 (official few-shot recipe)
–
–
84.5
82.0
83.5
67.5
84.5
80.0
63.0
87.5
73.5
44.5
74.0
B1 + 2D aug (crop, rotate)
–
–
86.5
87.0
83.0
75.0
87.5
82.5
66.5
84.0
79.0
53.0
77.5 +3.5
B2 (W1 data, no runtime)
–
–
88.5
87.5
86.5
80.0
88.0
82.0
72.0
88.0
80.5
65.0
81.1 +7.1
Align-and-replay (B1 + front end, ch. B)
–
est.
86.0
82.5
80.0
65.5
86.5
90.0
83.0
86.0
77.0
63.0
79.3 +5.3
Base gauge (B1 + arm anchor only)
–
est.
86.5
85.0
80.5
64.0
87.5
74.0
56.0
82.5
73.0
46.0
72.1 − 1.9
TABLE VI: Pose-Shift Robustness on LIBERO-PoseShift (LIBERO-Spatial and LIBERO-10, 200 episodes per cell, base-frame protocol). All learned rows share checkpoint, demonstrations and budget; est. : shift estimated at test time; † : upper bound with ground-truth poses. Red: gain over B1. Base / camera gauge: the arm’s pose change read as a camera / base shift.
Method
Perturbed data
Released
Corrected
OpenVLA-OFT [ 7 ]
–
56.4
–
π0 -FAST [ 7 ]
–
65.1
–
CycleVLA ( π0.5 ) [ 63 ]
–
74.2 ‡
–
Cross-View AC [ 11 ]
sim re-render
87.2 ‡
–
TBD-VLA [ 64 ]
–
87.8 ‡
–
Flex- π [ 62 ]
–
–
91.6
TABLE VII: LIBERO-Plus Camera Track (1,599 tasks, one episode each). Released : the benchmark’s prompts, which append the variant’s parameters; corrected : the task’s own instruction [ 62 ] . ‡ Prompt handling not stated. Ours: evaluation seed 0 / mean over three evaluation seeds.
Demonstrations per task
10
20
50
B1 (standard recipe)
63.2
66.6
74.0
B1 + WACO runtime
73.3
76.3
81.4
W1 (error-aware training only)
71.7
73.3
81.1
WACO (W1 + runtime)
82.4
82.3
87.2
gain of WACO over B1
+19.2
+15.7
+13.2
TABLE VIII: Few-Shot Scaling : average shifted success (nine conditions) on LIBERO-Spatial and LIBERO-10 with the first 10, 20 or all 50 demonstrations per task, same recipe and budget.
Method
Nominal
Camera
Base
Both
Avg
π0.5 (B1)
92.5
60.0
40.0
30.0
43.3
B1 + 2D aug
93.8
72.5
42.5
32.5
49.2 +5.9
WACO runtime on B1
91.3
75.0
72.5
62.5
70.0 +26.7
WACO (W1 + runtime)
95.0
85.0
80.0
75.0
80.0 +36.7
GT oracle on W1 †
95.0
87.5
86.3
82.5
85.4
TABLE IX: Real-Robot Results (UR5e, success rate in %, 20 trials per task and condition, averaged over the four tasks; per-task numbers in Table S4 ). Avg is the mean over the three shifted conditions; red superscripts are gains over B1; † uses the re-calibrated ground-truth shifts and is an upper bound.
Fig. 8: Real-Robot Per-Task Performance. Success under pose shifts (mean of the camera, base and compound conditions) on the four UR5e tasks. WACO improves every task, most on Toy on Shelf (+46.7 points).
Block
Method
Camera
Base
Both
Avg
Gauges
Base gauge
55
25
20
33.3
Camera gauge
30
65
35
43.3
WACO
77.5
72.5
67.5
72.5
Few-shot (10 demos)
B1
45
25
15
28.3
WACO
72.5
67.5
62.5
67.5 +39.2
Occluded
WACO w/o gate
–
40
–
–
TABLE X: Real-Robot Analysis on Toy on Shelf and Toast into Toaster (20 trials per task and condition, averaged over the two tasks). Gauges : arm anchor only, the change read as a camera (base gauge) or base shift (camera gauge). Few-shot : the first 10 of 50 demonstrations per task. Occluded : a box hides about a third of the static background.
Block
Variant
Clean
Cam L3
Base L3
Avg
–
WACO (full)
87.0
84.5
86.5
87.2
Channels
A only (image)
87.0
82.5
67.5
82.0
B only (state, action)
87.0
77.5
81.5
83.6
B without homing
87.0
77.5
78.5
81.7
Anchors
arm only ( ΔTc=I )
87.0
78.5
86.5
82.6
scene only ( ΔTb=I )
87.0
84.0
66.5
81.7
TABLE XI: Ablations. Each block changes one component of the full method (blue row) on LIBERO-Spatial and LIBERO-10. The full row is the same evaluation as WACO in Table VI . Avg is the mean over the nine shifted conditions.
ΔTc error
ΔTb error
Gate
Condition
mm
deg
mm
deg
caught
false
clean
0.0 / 6.6
0.0 / 0.0
0.7 / 7.6
0.1 / 0.2
0/0
0
camera L1–L3
0.8 / 14.5
0.0 / 0.3
1.7 / 18.5
0.2 / 1.0
3/13
1
base L1–L3
0.0 / 3.9
0.0 / 0.0
2.9 / 10.6
0.1 / 0.3
4/4
0
both L1–L3
0.8 / 17.2
0.0 / 1.0
4.8 / 25.0
0.2 / 2.5
16/24
8
all
0.3 / 10.7
0.0 / 0.2
2.4 / 19.0
0.1 / 1.2
23/41
9
TABLE XII: Estimator Accuracy and Gate Behaviour over 240 resets (four suites, three tasks each, two episodes per condition): median / 90th-percentile error against the true shift. A catastrophic estimate exceeds 2 cm or 1.5∘ ; caught counts catastrophic estimates whose channel the gate disabled, false counts accurate estimates of a real shift that it disabled.
Fig. 9: Sensitivity to Calibration Error (base L3, LIBERO-Spatial, 200 episodes per point, 95% Wilson bands): yaw error injected into the ground-truth base canonicalisation, and the histogram of the yaw errors of the estimator’s applied base estimates.
ΔTc error
ΔTb error
Condition
mm
deg
mm
deg
Accepted
nominal
0.6 / 2.4
0.0 / 0.1
1.1 / 3.8
0.1 / 0.2
100%
camera shift
3.4 / 11.8
0.4 / 1.2
4.2 / 14.5
0.5 / 1.6
86.7%
base shift
1.9 / 5.7
0.2 / 0.4
3.6 / 10.2
0.3 / 0.9
96.7%
both
4.7 / 16.5
0.7 / 1.8
5.5 / 19.8
0.8 / 2.1
83.3%
all
2.8 / 11.2
0.3 / 0.9
3.8 / 13.7
0.4 / 1.2
91.7%
TABLE XIII: Real-Robot Estimator Accuracy against hand–eye re-calibration (median / 90th-percentile error; 10 poses per condition, three arm configurations each, 120 estimates). Accepted : share of estimates the gate applies.
Fig. S1: UR5e Setup. The workspace at the calibration pose (left) and after the base shift (right); the wrist D435i is mounted on the gripper.
ΔTc
ΔTb
Condition
cm
deg
cm
deg
Scale
camera
7.2
4.5
0
0
L3
base
0
0
10.0
0
L3
both
6.5
5.8
10.0
0
L3
TABLE S1: Real-Robot Shift Poses : translation and rotation of the camera relative to the table ( ΔTc ) and of the arm’s base relative to the table ( ΔTb ); the base shift as realised, the camera shifts as measured by hand–eye re-calibration. Last column: the corresponding level of LIBERO-PoseShift.
Front-end state
Tasks
WACO
no runtime
accurate, applied
1,423
94.3
64.4
inaccurate, applied
95
76.1
65.3
no usable estimate
67
64.2
46.3
correct but not trusted
14
97.6
71.4
TABLE S2: LIBERO-Plus Outcomes by Front-End State (corrected protocol, all 1,599 tasks; WACO: mean over three evaluation seeds; no runtime: the public checkpoint without canonicalisation). Accurate : within 2 cm and 2∘ of the true camera shift.
Fig. S2: Few-Shot Scaling. Average shifted success on LIBERO-PoseShift with the first 10, 20 and 50 demonstrations per task, same recipe and budget, per suite. The gap between WACO and the standard recipe widens as data shrinks, because the pose variation that the recipe must absorb from data is removed analytically; both the runtime and the error-aware training contribute, and their combination is better than either.
Method
Spatial
LIBERO-10
Mean
B1
79.6
68.4
74.0
B1 + 2D aug
83.6
71.4
77.5
B2
88.4
73.7
81.1
WACO runtime on B1
88.4
74.4
81.4
WACO
90.6
83.8
87.2
GT oracle on W1
91.0
86.3
88.7
TABLE S3: Per-Suite Results of the main comparison ( Table VI ); average over the nine shifted conditions (900 episodes per cell).
Method
Condition
Lemon to Basket
Color Matching
Toy on Shelf
Toast into Toaster
Mean
π0.5 (B1)
Nominal
95
95
90
90
92.5
Camera
80
65
50
45
60.0
Base
60
45
30
25
40.0
Both
50
35
20
15
30.0
Mean (shifted)
63.3
48.3
33.3
28.3
43.3
B1 + 2D aug
Nominal
95
95
95
90
93.8
TABLE S4: Real-Robot Per-Task Results (success rate in %, 20 trials per task and condition). Mean rows are highlighted; † uses the re-calibrated ground-truth shifts.
Fig. S3: Visual Result. LIBERO-Plus views far outside the training distribution: the live frame, the frame canonicalised by WACO, and the calibration reference, with the true viewpoint change and the stage of the front end that produced the estimate. Pixels the live camera did not see are filled from the reference, so objects appear only where the live view observed them.
Fig. S4: More Visual Results on the LIBERO-Plus camera track: live view, the view canonicalised by WACO, and the calibration reference.
Department of Electrical and Computer Engineering, Seoul National University, Seoul 08826, South Korea · Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA