Organizations: J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA · Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843, USA
Vision-language-action (VLA) and world-action models (WAMs) often degrade under out-of-distribution task variations despite retaining partial task capability. To recover such capability, we propose RoboIRS, an inference-time internal representation steering method that uses successful and failed rollouts to train linear classifiers, select outcome-relevant intervention locations, and derive task-specific steering directions without updating policy parameters. On 15 simulation tasks with a frozen π0.5 policy, RoboIRS improves the average success rate from 44.4% to 66.2%, outperforming alternative inference-time intervention baselines while adding little inference time. We further validate RoboIRS on real-robot manipulation using the same π0.5 policy and demonstrate its applicability to a world-action model Cosmos Policy, where the average success rate improves from 35.4% to 55.4%. These results show that directly steering internal robot-policy representations can improve the performance of robot policies at inference time. Project website is available at https://rollingoat.github.io/roboirs/.
Figures & tables
Fig. 1 : Real-robot setup under visual distribution shift (top) and baseline comparison (bottom).
Fig. 2 : Overview of the proposed RoboIRS framework. (a) RoboIRS intervenes between transformer blocks by adding a scaled steering vector to the selected residual-stream representation without updating the policy parameters. (b) To construct the steering vector, residual-stream representations are collected from successful and failed rollouts and used to train a linear classifier. The classifier weight provides the steering direction that is injected into the residual stream during inference.
Fig. 3 : Steering strength vs. SR from a few tasks to show the typical patterns: monotonic increase/decrease; increase at moderate strengths before degrading at larger strengths.
ID
Task
Unsteered SR (%)
Steered SR (%)
s
Goal suite
L1
Open top drawer; put bowl inside
12.0±7.2
34.0±2.0
−1
L2
Put bowl on plate
32.7±1.2
96.7±3.1
−1
L3
Put bowl on cabinet
72.0±7.2
89.3±2.3
−1
L4
Put wine bottle on cabinet
17.3±3.1
68.0±2.0
+1
L5 *
Put bowl on plate
70.0±4.0
66.7±9.9
+1
TABLE I : Success rates of unsteered and steered π0.5 .
Steering Strategy
Avg. SR (%)
One Random Direction
44.0
Unsteered
44.4
Worst Token per Layer
52.7
Highest-AUC Layer
54.3
Random Token per Layer CAA
56.5
Random Token per Layer
57.3
TABLE II : Ablation study over steering denoise steps, vectors, and locations, ordered by average success rate (SR).
ID
Un- steered
Grad.
Re-rank ( K=16 )
CAA
RoboIRS
L1
12.0
12.7
44.7
34.0
34.0
L2
32.7
30.7
74.0
68.7
96.7
L3
72.0
58.7
79.3
75.3
89.3
L4
17.3
19.3
51.3
94.0
68.0
L5
70.0
63.3
70.7
85.3
66.7
L6
59.3
56.7
79.3
66.0
71.3
TABLE III : Comparison of task-level success rates (%) and average inference time among baselines and RoboIRS.
Fig. 4 : t-SNE visualization of the last-layer’s residuals. Each panel compares residual representations from unsteered and steered policies separately for successful and failed rollouts.
Robots deployed in the real world should learn from their experience and improve over time. This requires a mechanism of practicing and learning from feedback. In this paper, we propose VERITAS, a generator-verifier framework for generalist robot policies for inference-time policy steering and self-improvement. We use a pre-trained generalist robot policy as a generator'' and pair it with a gradient-free visual verifier'' that evaluates actions at inference time. This framework enables inference-time steering that improves policy performance without additional training. We demonstrate that inference-time verification consistently outperforms vanilla generalists without training on additional demonstration data. Additionally, we demonstrate that the verified rollouts provide effective supervision for offline policy improvement: policies fine-tuned on verified self-generated trajectories achieve consistent performance gains. Notably, we find that post-training with verified rollouts achieves comparable efficiency to expert demonstrations, while requiring no human interventions. Our results highlight inference-time verification as a practical and scalable mechanism for improving robotic policies during deployment.
Inference-time steering adapts pre-trained generative robot policies during deployment by verifying candidate actions before execution. While prior methods typically perform this verification only with visual observations, vision alone is often insufficient for contact-rich manipulation, where success depends on both global task progress and subtle local interactions such as contact force. We introduce ViTaL, a visuo-tactile inference-time steering framework that formulates multimodal guidance as a bi-level optimization problem. At the high level, visual sampling-and-verification performs long-horizon mode selection, deciding what behavior the robot should execute. At the low level, tactile-guided diffusion editing refines the selected action sequence over a shorter horizon to satisfy local contact requirements. To support outcome-based steering, ViTaL learns a visuo-tactile latent world model and employs semantically aligned visual and tactile verifiers, including a novel text-conditioned tactile reward that scores predicted tactile futures directly in latent space. Across three real-world contact-rich manipulation tasks, ViTaL improves overall success by 51% over the base policy, outperforms unimodal steering by at least 33%, and exceeds naive multimodal fusion by at least 20%. Website: https://yilin-wu98.github.io/vital_website.
Research on robotic manipulation has developed a diverse set of policy paradigms, including vision-language-action (VLA) models, vision-action (VA) policies, and code-based compositional approaches. Concrete policies typically attain high success rates on specific task distributions, but limited generalization beyond it. Rather than proposing another monolithic policy, we propose to leverage the complementary strengths of existing approaches through intelligent policy routing. We introduce RoboRouter, a training-free framework that maintains a pool of heterogeneous policies and learns to select the best-performing policy for each task through accumulated execution experience. Given a new task, RoboRouter constructs a semantic task representation, retrieves historical records of similar tasks, predicts the optimal policy choice without requiring trial-and-error, and incorporates structured feedback to refine subsequent routing decisions. Integrating a new policy into the system requires only a lightweight evaluation and does not incur training overhead. Across simulation benchmark and real-world evaluations, RoboRouter consistently outperforms individual policies, improving the average success rate by more than 3% in simulation and 13% in real-world settings, while preserving execution efficiency. Our results demonstrate that intelligent routing across heterogeneous, off-the-shelf policies provides a practical and scalable pathway toward building more capable robotic systems.
Yiteng Chen, Zhe Cao, Hongjia Ren +9
South China University of Technology · Nanjing University · University of New South Wales +5