Policy steering is an emerging way to adapt robot behaviors at deployment-time: a learned verifier analyzes low-level action samples proposed by a pre-trained policy (e.g., diffusion policy) and selects only those aligned with the task. While Vision-Language Models (VLMs) are promising general-purpose verifiers due to their reasoning capabilities, existing frameworks often assume these models are well-calibrated. In practice, the overconfident judgment from VLM can degrade the steering performance under both high-level semantic uncertainty in task specifications and low-level action uncertainty or incapability of the pre-trained policy. We propose uncertainty-aware policy steering (UPS), a framework that jointly reasons about semantic task uncertainty and low-level action feasibility, and selects an uncertainty resolution strategy: execute a high-confidence action, clarify task ambiguity via natural language queries, or ask for action interventions to correct the low-level policy when it is deemed incapable at the task. We leverage conformal prediction to calibrate the composition of the VLM and the pre-trained base policy, providing statistical assurances that the verifier selects the correct strategy. After collecting interventions during deployment, we employ residual learning to improve the capability of the pre-trained policy, enabling the system to learn continually but with minimal expensive human feedback. We demonstrate our framework through experiments in simulation and on hardware, showing that UPS can disentangle confident, ambiguous, and incapable scenarios and minimizes expensive user interventions compared to uncalibrated baselines and prior human- or robot-gated continual learning approaches. Videos can be found at https://jessie-yuan.github.io/ups/
Figures & tables
Fig. 1: Uncertainty-Aware Policy Steering. Our framework calibrates the VLM verifier used for policy steering via conformal prediction. This enables the VLM to select an appropriate way to resolve uncertainty, from querying the end-user in natural language to asking to re-train the low-level control policy.
Fig. 2: Outcome Prediction & Narration. The policy and the world model are interleaved to predict long-horizon outcomes induced by the low-level policy. Decoded observations are fed into a VLM which narrates the outcomes in text.
Fig. 3: Uncertainty Quantification Results: Simulation . We compare the combination of Vanilla, CoT and Bayesian Intent (Ours) models for UQ. Dashed lines indicate the target coverage ( 1−ϵ=0.85 ), clarification rate, and set size.
Fig. 4: Uncertainty Quantification Results: Hardware (PnP Cup) . Combination of Vanilla, CoT and Bayesian Intent (Ours) models for UQ. Dashed lines indicate the target coverage rate ( 1−ϵ=0.85 ), clarification rate, and set size.
Fig. 5: Uncertainty Quantification Results: Hardware (Insert Block) . We compare the combination of Vanilla, CoT and Bayesian Intent (Ours) models for UQ. Dashed lines indicate the target coverage, clarification rate, and set size.
Fig. 6: Success Rates Pre- and Post-Continual Learning: Hardware and Simulation. We deploy the robot with 20 straightforward (left) and 20 ambiguous (right) instructions. We average the success rate over 20 trials for each scenario. UPS solicits data in a way which maximizes the final success rate after residual policy training, compared to baselines.
Fig. 7: Hardware: Robot Asking for Interventions. EnsembleDAgger (top): the human intervenes whenever the ensemble disagreement exceeds a threshold. Human-Gated (HG) DAgger (middle): a human monitors and intervenes when the robot’s behavior deviates from their intention. Our approach (bottom two) asks for “cheap” clarifications and only requests interventions when no action sample achieves the task.
Fig. 8: Intervention & Re-deployment for Insert Block Task . (left) Different strategies to elicit human interventions: HG-DAgger (top), EnsembleDAgger (middle) and UPS (bottom). (right) UPS maintains the multi-modality, while other methods exhibit failures or mode collapse.
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 9: Hardware Setup: PnP Cup. We demonstrate our hardware environment setup with a Franka Emika Panda arm and two Zed cameras (left image). In the middle, we show the left-view image captured by a Zed 2i camera and on the right, we show the wrist-view image captured by a Zed Mini camera.
Fig. 10: Hardware Examples of the PnP Cup Task. We show two different ways of achieving the task of placing the cup in the bin. The top row shows the cup placed in the left bin while the bottom row shows the cup placed in the right bin.
Fig. 11: Hardware Setup: Block. We demonstrate our hardware setup for the block task using a Franka Emika Panda arm. The leftmost two images show the initial environment setup, while the right four images show the target configurations for the two behavior modes, as labeled. In each image pair, the left image provides the third-person camera view and the right image provides the wrist view.
Method
Straightforward
Ambiguous
TP
TN
FP
FN
TP
TN
FP
FN
Base Policy
0.35
0.65
–
–
0.35
0.65
–
–
EnsembleDagger + Residual
0.40
0.60
–
–
0.50
0.50
–
–
HG-Dagger + Residual
0.50
0.50
–
–
0.55
0.45
–
–
Forewarn
0.80
0.10
0.10
0.0
0.50
0.0
0.40
0.10
UPS w Clarification
0.85
0.05
0.05
0.05
0.65
0.25
0.10
0.0
Appendix
TABLE I: Hardware - PnP Cup: Policy Steering Results Breakdown . We include the True Positive Rate (TP), True Negative Rate (TN), False Positive Rate (FP) and False Negative Rate (FN) in the table as an extension to the results we show in Fig. 6 . For straightforward and ambiguous scenarios, the reported success rate is averaged across 20 trials, each with a unique task instruction.
Category
User Intent
Task Instruction
Straightforward
left
“on the left side, there is a bin—place the cup there”
left
“the cup should end up in the left bin”
left
“take the cup off the table and finish by putting it in the left bin”
left
“make the left bin the final location for the cup”
left
“ensure the cup is placed inside the left bin”
left
“please place the cup in the left bin”
Appendix
TABLE II: Straightforward Hardware (PnP Cup) Task Instructions. 40 straightforward task instructions used for the calibration dataset are listed here. During testing, we randomly sample 20 instructions from the set and obtain new options when sampling from the policy.
Category
User Intent
Task Instruction
Ambiguous
left or right
“can you deal with the cup and put it away?”
left or right
“the cup should not be on the table anymore—put it somewhere appropriate”
left or right
“tidy this up: move the cup into a bin”
left or right
“pick up the cup and store it in a container”
left or right
“put the cup away in the bin over there”
left or right
“could you move the cup out of the working area?”
Appendix
TABLE III: Ambiguous Hardware (PnP Cup) Task Instructions. 40 ambiguous task instructions used for the calibration dataset are listed here. During testing, we randomly sample 20 instructions from the set and obtain new options with the new action samples from the base policy.
Fig. 12: Simulation Setup. We demonstrate our simulated task setup using a Franka Emika Panda arm. The leftmost two images show the initial environment setup, while the right four images show the target configurations for the two behavior modes, as labeled. In each image pair, the left image provides the third-person camera view and the right image provides the wrist view.
Method
Straightforward
Ambiguous
TP
TN
FP
FN
TP
TN
FP
FN
Base Policy
0.45
0.55
–
–
0.25
0.75
–
–
EnsembleDagger + Residual
0.45
0.55
–
–
0.30
0.70
–
–
HG-Dagger + Residual
0.60
0.40
–
–
0.45
0.55
–
–
Forewarn
0.65
0.20
0.15
0.0
0.0
0.25
0.15
0.60
UPS w Clarification
0.60
0.20
0.10
0.10
0.55
0.25
0.20
0.0
Appendix
TABLE IV: Simulation Policy Steering Results Breakdown . We include the True Positive Rate (TP), True Negative Rate (TN), False Positive Rate (FP) and False Negative Rate (FN) in the table as an extension to the results we show in Fig. 6 . For straightforward and ambiguous scenarios, the reported success rate is averaged across 20 trials respectively, where each trial has an unique task instruction.
Fig. 13: Hardware PnP Cup Histogram for Empirical Quantile. We list the overall histogram with all three categories combined, as well as per category histograms. The red vertical line indicates the q^ value based on user-desired coverage rate 1−ε .
Fig. 14: Hardware Insert Block Histogram for Empirical Quantile. We list the overall histogram with all three categories combined, as well as per category histograms. The red vertical line indicates the q^ value based on user-desired coverage rate 1−ε .
Fig. 15: Simulation Histogram for Empirical Quantile. We list the overall histogram with all three categories combined, as well as per category histograms. The red vertical line indicates the q^ value based on user-desired coverage rate 1−ε .
Fig. 16: Hardware PnP Cup Uncertainty Quantification with Logits . We conduct an ablation study where we directly use the softmax over the logits of the first generated token as the score rather than the self-generated score. Empirically, we find that, compared to Fig. 4 , the self-generated score is a better score function across all scenarios.
Fig. 17: Simulation Uncertainty Quantification with Logits . Similar to hardware experiments, we compare the results of the softmax of logits as the score with self-generated score results in Fig. 3 . None of the methods with logits can achieve the desired coverage while balancing the clarification rate and reaching the desired prediction set size.
Hyperparameter
Value
Batch Size
64
Learning Rate
1e-4
Iterations
10000
Dropout
0.2
Base Action Encoder Output Dimension
128
Action Predictor Hidden Dimension
256
Appendix
TABLE V: Hyperparameters for Residual Policy Training.
User Intent
Task Instruction
left
“Please place the nut on the peg with the handle facing left.”
left
“I want the nut positioned on the peg with the handle directed left.”
left
“Put the nut on the peg so that its handle is oriented leftward.”
left
“Set the nut onto the peg with the handle pointing left.”
left
“The nut goes on the peg with its handle turned to face left.”
left
“Can the nut be placed on the peg so its handle is pointing to the left?”
Appendix
TABLE VI: Straightforward Simulation Task Instructions. 60 straightforward task instructions used for the calibration and test datasets are listed here. 40 of these are randomly sampled to be part of the calibration set, while the remaining 20 make up the test set. We use “c.c.w.” and “c.w.” to denote “counterclockwise” and “clockwise” respectively.
User Intent
Task Instruction
left or right
“Please place the nut on the peg.”
left or right
“I want the nut positioned on the peg.”
left or right
“Pick up the nut and set it on the peg.”
left or right
“Can you set the nut on the peg?”
left or right
“Put the nut onto the peg.”
left or right
“I need you to place the nut on the peg.”
Appendix
TABLE VII: Ambiguous Simulation Task Instructions. 60 ambiguous task instructions used for the calibration and test datasets are listed here. 40 of these are randomly sampled to be part of the calibration set, while the remaining 20 make up the test set.
Category
User Intent
Task Instruction
Straightforward
left
“Insert the purple block into the left hole.”
left
“Attach the purple block to the left-side hole.”
left
“Fit the purple block into the left corner.”
left
“Push the purple block into the left slot.”
left
“Align the purple block with the left hole and insert it.”
left
“Fasten the purple block into the hole on the left.”
Appendix
TABLE VIII: Block Insertion Task Instructions. Straightforward task instructions specify whether the purple block should be inserted into the left or right hole, while ambiguous instructions underspecify the target hole and require disambiguation.
Fig. 18: Hardware Prompt For Behavior Translation (Grasp)
Fig. 19: Hardware Prompt For Behavior Translation (Place)
Fig. 20: Hardware Prompt for VLM Verification in Forewarn
Fig. 21: Hardware Prompt for Generating Possible Intents
Fig. 22: Hardware Prompt for Inferring Intent Probability
Fig. 23: Hardware Prompt for Updating Instructions
Fig. 24: Hardware Prompt for Clarification Questions
Fig. 25: Hardware Prompt for Generating Behavior Probability Conditioned On the Intent
Fig. 26: Hardware Prompt For Asking VLM to Analyze Ambiguity First in CoT Reasoning
Fig. 27: Hardware Prompt For Asking VLM to Score Options Based On Previous Reasoning in CoT Reasoning
Fig. 28: Hardware Prompt For Directly Generating Scores for Available Options
Fig. 29: Simulation Prompt For Behavior Translation (Grasp)
Fig. 30: Simulation Prompt For Behavior Translation (Place)
Fig. 31: Simulation Prompt for VLM Verification in Forewarn
Fig. 32: Simulation Prompt for Generating Possible Intents
Fig. 33: Simulation Prompt for Inferring Intent Probability
Fig. 34: Simulation Prompt For Asking VLM to Analyze Ambiguity First in CoT Reasoning
Fig. 35: Simulation Prompt For Asking VLM to Score Options Based On Previous Reasoning in CoT Reasoning
Fig. 36: Simulation Prompt for Generating Behavior Probability Conditioned On the Intent
Fig. 37: Simulation Prompt For Directly Generating Scores for Available Options
We present Token Steering (TS), a method for dynamically steering trajectories generated by an autoregressive vision-language-action (VLA) model through direct intervention in the action-token space. TS injects low-dimensional user inputs into the model's native action-token representation, allowing users to influence trajectory generation without modifying the underlying vision-language model (VLM) architecture. Because TS operates entirely at inference time, it requires no additional training or finetuning. User inputs guide rather than override the pretrained policy, allowing users to influence robot actions while preserving the dexterity, smoothness, and task priors learned by the VLA. We evaluate TS on two household manipulation tasks -- drawer closing after object placement and state-aware object swapping -- and improve success rates from 10.0% to 72.5% and from 16.7% to 93.8%, respectively. By enabling lightweight, intuitive steering over robot foundation models, our interface has the potential to improve human-robot interaction in consumer environments and broaden accessibility for individuals with limited physical control. Project website: https://jasontchan.github.io/token-steering/ .
Jason Chan, Jonathan C. Kao
Department of Computer Science University of California, Los Angeles · Departments of Electrical and Computer Engineering, Computer Science, and Neurobiology University of California, Los Angeles
Steering an LLM agent means deciding whether to correct it, at which step, and with which mechanism. Uncertainty is often used to decide when to correct an agent, but whether it can guide these decisions remains unclear. We steer agent trajectories separately at every non-terminal step with each of four mechanisms and run each continuation to completion. The resulting stepwise outcome table (SOT) holds about 82,000 counterfactual continuations of 1,864 trajectories from three benchmarks and two agents. It shows that uncertainty can identify failing trajectories, but that no single signal reliably locates the step at which steering helps. We therefore propose VoS (Value of Steering), a trajectory-level monitor, offline or online, that learns from SOT the value of steering at each step and decides where to steer by it. A harm-budgeted trigger decides whether to steer, limiting the fraction of successful trajectories that VoS disturbs. VoS improves on unmodified execution in all 12 settings of benchmark, agent, and offline or online use, by 7.8 points on average, and outperforms the strongest of five existing uncertainty-triggered methods in 11, by 2.9 points on average. Ablations show that training on measured outcomes and a tight harm budget are both essential.
Hanwen Li, Jinhao Duan, Guanhua Zhu +4
New Jersey Institute of Technology · University of North Carolina at Chapel Hill · University of California, Irvine +1
Pretrained vision-language-action (VLA) policies show promising zero-shot generalization, but often fail under deployment-time distribution shift, leading to decreased robustness and inconsistent instruction following. While prior work commonly tackles this by finetuning on in-distribution data, it assumes demonstrations collected on tasks in the target environment. In this work, we propose DREAMSTEER, a deployment-time steering framework for pretrained VLAs without any finetuning or parameter modifications. The key insight in DREAMSTEER is to leverage a latent world model and a value model to steer pretrained VLA policies. During deployment, DREAMSTEER samples candidate action chunks from a VLA policy and predefined motion primitives, imagines their outcomes using an action-conditioned latent world model, and ranks the imagined trajectories with a language-conditioned value model. Across four real-world manipulation benchmarks with unseen objects, DREAMSTEER improves task success rate from 23.75% to 66.25% and instruction-following accuracy from 38.75% to 56.25% over the base VLA policy.
Hanchen Cui, Sergio Arnaud, Arjun Majumdar +5
1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities