VG-Steer: Visual-Geometric Policy Steering from In-Context Aligned Abstractions
Authors: Arthur Bucker, Pramod Anantharam, Jonathan Francis, Jean Oh
Organizations: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA · Robot Learning Lab, Bosch Center for Artificial Intelligence, Pittsburgh, PA 15222, USA · Lavoro AI, Inc., Pittsburgh, PA 15232, USA
Adapting robot policies to out-of-distribution (OOD) settings with limited real-world data remains a key bottleneck to generalist robots. The multimodality and scarcity of robotics data lead end-to-end approaches to overfit and break under small distribution shifts. Test-time adaptation can alleviate this by correcting frozen policies at inference, but most methods ground their correction in a signal learned implicitly from robotic datasets, making it susceptible to the same shifts as the policy it corrects. Recent methods that instead exploit explicit spatial information are typically limited to coarse corrections, missing the nuanced, task-relevant motion most manipulation tasks require. We propose VG-Steer, which corrects existing policies by transferring the nuanced motion of successful in-distribution rollouts, captured along meaningful geometric axes, known here as object-relational distance pairs. VG-Steer leverages the semantic understanding of Large Vision Models (LVMs) and Vision-Language Models (VLMs) reasoning, which combines the task-relevant entity pairs and their object-relational distance projections into a single tailored visual-geometric guidance signal. In a simulation benchmark, it efficiently corrects flow-matching policies under pose and task-goal shifts; on a real robot experiment, it widens the region of the workspace the frozen policy covers, more than doubling its success rate on cable placements one grid step outside the training positions, and steers a single-task policy toward objects it was never trained on while nearly eliminating object confusion. Project website: https://arthurfenderbucker.github.io/VG-Steer/
Figures & tables
Fig. 1 : Overview of our visual-geometric policy steering framework. Offline analysis (at task initialization, once per episode): (1) we retrieve the most relevant successful in-distribution rollout τ∗ by weighted image (DINOv2) and text (CLIP) similarity; (2) a VLM selects the task-relevant object-relational distance pairs per motion phase (e.g., gripper–cube distance d1 , cube–table distance d2 , gripper orientation θ ), evaluated per timestep into the reference trends Φ ; (3) a VLM with SAM2 matches the demonstration objects to the live OOD scene (e.g., the cylinder takes the cube’s role), re-targeting each pair. Online steering (every action chunk): (4) perception projects the live scene into the same distance space and aligns each pair to Φ within a window, giving the reference d∗ . At every denoising step, a differentiable dynamics model F forecasts the features d(A) of a candidate chunk A , and a line-searched gradient step on the trajectory loss corrects the frozen policy’s velocity field, yielding the steered chunk A′ .
Fig. 2 : Differentiable guidance injection. At each denoising step, the candidate chunk is rolled out by forward kinematics, and the guidance loss (position, orientation, gripper) is differentiated; a golden-section line search sets the step size, and the correction is applied to the policy’s velocity field, steering the trajectory without modifying weights.
Method ( π0.5 )
Task
Pos
Average
No steering
26.3
26.3
26.3
VLS (reproduced, full) ∗
32.0
27.6
29.8
VLS (grad-only) ∗
17.3
21.5
19.4
VG-Steer
40.8
34.3
37.5
TABLE I : Success rate (%) on the task and positional perturbations of LIBERO-Pro ( π0.5 ), each averaged over the four suites, and their mean ( Average ). All rows use simulator-provided object segmentation and labels. ∗ VLS reproduced in our setup using the released VLS codebase ( grad-only disables its population resampling); see Appendix A-N .
Base policy
Steering
Task
Pos
Average
π0.5 (flow)
Base
26.3
26.3
26.3
+ VG-Steer
40.8
34.3
37.5
GR00T N1.7 (flow)
Base
19.3
0.5
9.9
+ VG-Steer
20.0
3.5
11.8
TABLE II : Transfer across base policies (LIBERO-Pro; task and positional success rate %, averaged over suites, and their mean).
Fig. 3 : Per-action-chunk inference overhead on π0.5 (mean over the four suites, CUDA-synchronized). With VG-Steer a chunk takes ∼ 73 ms in total ( 1.2× the base’s ∼ 61 ms), comparable to gradient-only VLS ( ∼ 70 ms total); full VLS costs ∼ 20 × more, scaling with its particle count ( 20 here).
Variant ( π0.5 )
Task
Pos
Average
No guidance
26.3
26.3
26.3
+ gradient guidance (fixed step)
24.2
23.3
23.8
+ line search
34.2
35.8
35.0
+ gradient normalization (VG-Steer)
40.8
34.3
37.5
TABLE III : Guidance-mechanism ablation ( π0.5 , LIBERO-Pro; success rate %, averaged over the four suites). Components are enabled cumulatively on top of the unsteered base. The two intermediate variants use a 3 -trial sweep ( 30 episodes/cell); the base and full method use the 10 -trial main runs.
Fig. 4 : Positional sweep on cable bagging, in the camera view. (a) Training distribution of the 110 cable grasps in the demonstrations. (b) The single reference rollout, with its end-effector path split into sub-trajectories. (c, d) Per-cell outcomes: dots mark the measured cable position of every trial, filled green for success, red rings for failure.
Fig. 5 : Positional sweep on cable bagging, 48 placements per method, every placement run by both. Zones are Mahalanobis bands cut at 2 and 4.5σ . Cells give the success rate (%), successes/rollouts and the 95% Wilson interval; Δ is the gain in points and p a two-sided Fisher exact test.
Fig. 7 : Disambiguating the target object in a cluttered scene. In red and blue, the distance pairs used per sub-trajectory (ST0 approach, ST1 transport).
Fig. 8 : Task variation. Named is the rate at which the arm went for the object the prompt named; Task SR also requires it to be bagged; Confusion is the rate of grasping a distractor. The policy was only trained on the cable, the banana and the spoon are unseen.
Fig. 10 : Real-world outcome breakdown (share of rollouts, counts on bars). Residual failures concentrate in grasp and transport, which the guidance does not address.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 11 : Visual-geometric guidance via VLM-selected distance pairs. Offline (a–c): the demonstration is segmented (Gemini 3 Flash + SAM2) and split into sub-trajectory phases (STRAP); the VLM selects each phase’s entity pairs, projections, and sub-goal constraints. Online (d–e): the live scene is matched to the demonstration, and the guidance steers the frozen policy so the selected projected distances track the reference.
Suite
Pert.
ER-1.6
Gemini 3 Flash
Goal
Task
9.0
11.0
Goal
Pos
23.0
25.0
Spatial
Task
40.0
49.0
Spatial
Pos
33.0
32.0
libero_10
Task
5.0
6.0
libero_10
Pos
9.0
8.0
Appendix
TABLE V : Open-vocabulary grounding-backbone ablation for π0.5 +VG-Steer under real perception (success rate %; “Pos” denotes the positional/swap perturbation). Gemini 3 Flash outperforms Gemini Robotics-ER 1.6 on the task and positional suites.
Stage
Wall time
Retrieval (DINOv2 + CLIP embed and match)
0.43±0.01 s
Demo seed frame: object names (VLM)
2.1±0.3 s
Demo seed frame: box detection (VLM)
2.1±0.4 s
Demo seed frame: SAM2 masks
0.16±0.14 s
Pair and constraint selection (VLM)
9.4±0.6 s
Demo analysis total
14.9±0.8 s
Appendix
TABLE VI: First-frame setup, mean ± sd over 6 runs.
Measurement
Value
Detection, once per episode (VLM boxes + SAM2)
2.6±0.6 s
Track call per replan (replay, 63 frames)
46±4 ms
Tracker step alone (replay)
37.5±3.5 ms
Tracker step alone (in the loop, 80 rollouts)
48±14 ms
Perception block in the live loop
113±2 ms
Appendix
TABLE VII: Per-replan perception cost in the inference loop.
Logistic fit
Base (no steering)
VG-Steer
50% crossing
2.8σ
3.4σ
Slope
3.9 per σ
1.7 per σ
Appendix
TABLE X : Logistic fit of success against Mahalanobis distance, per method. The 50% crossing is annotated in Figure 6 ; the slope is the logit decrease per standard deviation. Both fits are constrained almost entirely by the near-OOD trials, since the other two zones are saturated for both methods, and neither is supported by data beyond 4.5σ .
Fig. 12 : The three images provided to the reasoning VLM for the worked example of Appendix A-O : (a) the raw RGB observation, (b) the segmentation overlay that labels every object with its instance ID, and (c) the end-effector path split into color-coded numbered sub-trajectories.
Vision-Language-Action (VLA) policies demonstrate remarkable semantic generalization, yet remain highly vulnerable to out-of-distribution (OOD) visual shifts during execution. While existing test-time methods attempt to suppress spurious visual cues, they fundamentally overlook the rich, uncorrupted kinematic priors embedded in the policy's own training experience. We introduce Retrieve to Act (R2A), a test-time scaling framework that formulates OOD robustness purely as a geometric motion primitive retrieval problem. R2A bypasses fragile visual features by structuring successful training demonstrations into an offline spatiotemporal motion primitive graph. During inference, a query-dependent Graph Neural Network dynamically localizes the robot's ongoing execution within this graph, retrieving structurally compatible future motion primitives. These retrieved geometric priors are then seamlessly harmonized with the frozen VLA's predicted actions via a learned, dimension-wise gating mechanism, adaptively rectifying erroneous spatial trajectories while preserving the policy's local reactivity. Extensive experiments across three state-of-the-art VLA architectures, GR00T N1.7, π0.5, and StableVLA, demonstrate that R2A consistently and significantly enhances robustness against diverse contextual perturbations. R2A achieves absolute success rate improvements of up to +12% in simulation benchmarks (LIBERO-Plus, RoboTwin2.0-Plus) and +40% in real-world deployments, without requiring any finetuning of the base policies. Project page: https://retrieve2act.github.io/r2a/
Pham Tri Quang, Bao-Ngoc Dao, Minh Duc Nguyen +15
Center for AI Research, VinUniversity · VinRobotics · Hanoi University of Science and Technology +6
Generalist robot policies carry broad manipulation priors from large-scale data, but specializing them to a new task remains the deployment bottleneck. This requires eliciting task-specific behavior from limited demonstrations without degrading their broad capabilities. We introduce Proxy Policy Steering (PPS), an inference-time adaptation method that resolves this challenge by training two lightweight proxy policies whose calibrated velocity-space difference steers the frozen base sampler. A reference proxy models the frozen base's behavior on target-task observations, and a task proxy, initialized from the reference, captures how this behavior changes under task supervision. Their difference forms a calibrated velocity-space residual that steers the frozen base sampler at every denoising step. We identify the conditions under which this residual isolates the change induced by task supervision, and validate them empirically. Because the base is never directly modified, its broad capabilities remain available at inference, including behaviors such as recovery from failure that the demonstrations themselves do not exercise. Adaptation requires only forward velocity predictions from the base, making PPS lightweight to train and applicable even without access to the base's parameters. On 8 real-world and 4 simulation manipulation tasks, PPS lifts the state-of-the-art pi 0.5 base policy by 53% absolute success rate on average, with zero-to-one gains on tasks the base never solves, while preserving the base's broad capabilities. PPS outperforms LoRA fine-tuning, from-scratch specialists, residual policies, and prior inference-time steering methods.
Inference-time steering adapts pre-trained generative robot policies during deployment by verifying candidate actions before execution. While prior methods typically perform this verification only with visual observations, vision alone is often insufficient for contact-rich manipulation, where success depends on both global task progress and subtle local interactions such as contact force. We introduce ViTaL, a visuo-tactile inference-time steering framework that formulates multimodal guidance as a bi-level optimization problem. At the high level, visual sampling-and-verification performs long-horizon mode selection, deciding what behavior the robot should execute. At the low level, tactile-guided diffusion editing refines the selected action sequence over a shorter horizon to satisfy local contact requirements. To support outcome-based steering, ViTaL learns a visuo-tactile latent world model and employs semantically aligned visual and tactile verifiers, including a novel text-conditioned tactile reward that scores predicted tactile futures directly in latent space. Across three real-world contact-rich manipulation tasks, ViTaL improves overall success by 51% over the base policy, outperforms unimodal steering by at least 33%, and exceeds naive multimodal fusion by at least 20%. Website: https://yilin-wu98.github.io/vital_website.