VG-Steer: Visual-Geometric Policy Steering from In-Context Aligned Abstractions
Authors: Arthur Bucker, Pramod Anantharam, Jonathan Francis, Jean Oh
Organizations: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA · Robot Learning Lab, Bosch Center for Artificial Intelligence, Pittsburgh, PA 15222, USA · Lavoro AI, Inc., Pittsburgh, PA 15232, USA
Adapting robot policies to out-of-distribution (OOD) settings with limited real-world data remains a key bottleneck to generalist robots. The multimodality and scarcity of robotics data lead end-to-end approaches to overfit and break under small distribution shifts. Test-time adaptation can alleviate this by correcting frozen policies at inference, but most methods ground their correction in a signal learned implicitly from robotic datasets, making it susceptible to the same shifts as the policy it corrects. Recent methods that instead exploit explicit spatial information are typically limited to coarse corrections, missing the nuanced, task-relevant motion most manipulation tasks require. We propose VG-Steer, which corrects existing policies by transferring the nuanced motion of successful in-distribution rollouts, captured along meaningful geometric axes, known here as object-relational distance pairs. VG-Steer leverages the semantic understanding of Large Vision Models (LVMs) and Vision-Language Models (VLMs) reasoning, which combines the task-relevant entity pairs and their object-relational distance projections into a single tailored visual-geometric guidance signal. In a simulation benchmark, it efficiently corrects flow-matching policies under pose and task-goal shifts; on a real robot experiment, it widens the region of the workspace the frozen policy covers, more than doubling its success rate on cable placements one grid step outside the training positions, and steers a single-task policy toward objects it was never trained on while nearly eliminating object confusion. Project website: https://arthurfenderbucker.github.io/VG-Steer/
Figures & tables
Fig. 1 : Overview of our visual-geometric policy steering framework. Offline analysis (at task initialization, once per episode): (1) we retrieve the most relevant successful in-distribution rollout τ∗ by weighted image (DINOv2) and text (CLIP) similarity; (2) a VLM selects the task-relevant object-relational distance pairs per motion phase (e.g., gripper–cube distance d1 , cube–table distance d2 , gripper orientation θ ), evaluated per timestep into the reference trends Φ ; (3) a VLM with SAM2 matches the demonstration objects to the live OOD scene (e.g., the cylinder takes the cube’s role), re-targeting each pair. Online steering (every action chunk): (4) perception projects the live scene into the same distance space and aligns each pair to Φ within a window, giving the reference d∗ . At every denoising step, a differentiable dynamics model F forecasts the features d(A) of a candidate chunk A , and a line-searched gradient step on the trajectory loss corrects the frozen policy’s velocity field, yielding the steered chunk A′ .
Fig. 2 : Differentiable guidance injection. At each denoising step, the candidate chunk is rolled out by forward kinematics, and the guidance loss (position, orientation, gripper) is differentiated; a golden-section line search sets the step size, and the correction is applied to the policy’s velocity field, steering the trajectory without modifying weights.
Method ( π0.5 )
Task
Pos
Average
No steering
26.3
26.3
26.3
VLS (reproduced, full) ∗
32.0
27.6
29.8
VLS (grad-only) ∗
17.3
21.5
19.4
VG-Steer
40.8
34.3
37.5
TABLE I : Success rate (%) on the task and positional perturbations of LIBERO-Pro ( π0.5 ), each averaged over the four suites, and their mean ( Average ). All rows use simulator-provided object segmentation and labels. ∗ VLS reproduced in our setup using the released VLS codebase ( grad-only disables its population resampling); see Appendix A-N .
Base policy
Steering
Task
Pos
Average
π0.5 (flow)
Base
26.3
26.3
26.3
+ VG-Steer
40.8
34.3
37.5
GR00T N1.7 (flow)
Base
19.3
0.5
9.9
+ VG-Steer
20.0
3.5
11.8
TABLE II : Transfer across base policies (LIBERO-Pro; task and positional success rate %, averaged over suites, and their mean).
Fig. 3 : Per-action-chunk inference overhead on π0.5 (mean over the four suites, CUDA-synchronized). With VG-Steer a chunk takes ∼ 73 ms in total ( 1.2× the base’s ∼ 61 ms), comparable to gradient-only VLS ( ∼ 70 ms total); full VLS costs ∼ 20 × more, scaling with its particle count ( 20 here).
Variant ( π0.5 )
Task
Pos
Average
No guidance
26.3
26.3
26.3
+ gradient guidance (fixed step)
24.2
23.3
23.8
+ line search
34.2
35.8
35.0
+ gradient normalization (VG-Steer)
40.8
34.3
37.5
TABLE III : Guidance-mechanism ablation ( π0.5 , LIBERO-Pro; success rate %, averaged over the four suites). Components are enabled cumulatively on top of the unsteered base. The two intermediate variants use a 3 -trial sweep ( 30 episodes/cell); the base and full method use the 10 -trial main runs.
Fig. 4 : Positional sweep on cable bagging, in the camera view. (a) Training distribution of the 110 cable grasps in the demonstrations. (b) The single reference rollout, with its end-effector path split into sub-trajectories. (c, d) Per-cell outcomes: dots mark the measured cable position of every trial, filled green for success, red rings for failure.
Fig. 5 : Positional sweep on cable bagging, 48 placements per method, every placement run by both. Zones are Mahalanobis bands cut at 2 and 4.5σ . Cells give the success rate (%), successes/rollouts and the 95% Wilson interval; Δ is the gain in points and p a two-sided Fisher exact test.
Fig. 7 : Disambiguating the target object in a cluttered scene. In red and blue, the distance pairs used per sub-trajectory (ST0 approach, ST1 transport).
Fig. 8 : Task variation. Named is the rate at which the arm went for the object the prompt named; Task SR also requires it to be bagged; Confusion is the rate of grasping a distractor. The policy was only trained on the cable, the banana and the spoon are unseen.
Fig. 10 : Real-world outcome breakdown (share of rollouts, counts on bars). Residual failures concentrate in grasp and transport, which the guidance does not address.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 11 : Visual-geometric guidance via VLM-selected distance pairs. Offline (a–c): the demonstration is segmented (Gemini 3 Flash + SAM2) and split into sub-trajectory phases (STRAP); the VLM selects each phase’s entity pairs, projections, and sub-goal constraints. Online (d–e): the live scene is matched to the demonstration, and the guidance steers the frozen policy so the selected projected distances track the reference.
Suite
Pert.
ER-1.6
Gemini 3 Flash
Goal
Task
9.0
11.0
Goal
Pos
23.0
25.0
Spatial
Task
40.0
49.0
Spatial
Pos
33.0
32.0
libero_10
Task
5.0
6.0
libero_10
Pos
9.0
8.0
Appendix
TABLE V : Open-vocabulary grounding-backbone ablation for π0.5 +VG-Steer under real perception (success rate %; “Pos” denotes the positional/swap perturbation). Gemini 3 Flash outperforms Gemini Robotics-ER 1.6 on the task and positional suites.
Stage
Wall time
Retrieval (DINOv2 + CLIP embed and match)
0.43±0.01 s
Demo seed frame: object names (VLM)
2.1±0.3 s
Demo seed frame: box detection (VLM)
2.1±0.4 s
Demo seed frame: SAM2 masks
0.16±0.14 s
Pair and constraint selection (VLM)
9.4±0.6 s
Demo analysis total
14.9±0.8 s
Appendix
TABLE VI: First-frame setup, mean ± sd over 6 runs.
Measurement
Value
Detection, once per episode (VLM boxes + SAM2)
2.6±0.6 s
Track call per replan (replay, 63 frames)
46±4 ms
Tracker step alone (replay)
37.5±3.5 ms
Tracker step alone (in the loop, 80 rollouts)
48±14 ms
Perception block in the live loop
113±2 ms
Appendix
TABLE VII: Per-replan perception cost in the inference loop.
Logistic fit
Base (no steering)
VG-Steer
50% crossing
2.8σ
3.4σ
Slope
3.9 per σ
1.7 per σ
Appendix
TABLE X : Logistic fit of success against Mahalanobis distance, per method. The 50% crossing is annotated in Figure 6 ; the slope is the logit decrease per standard deviation. Both fits are constrained almost entirely by the near-OOD trials, since the other two zones are saturated for both methods, and neither is supported by data beyond 4.5σ .
Fig. 12 : The three images provided to the reasoning VLM for the worked example of Appendix A-O : (a) the raw RGB observation, (b) the segmentation overlay that labels every object with its instance ID, and (c) the end-effector path split into color-coded numbered sub-trajectories.