When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering

Jessie Yuan; Yilin Wu; Andrea Bajcsy

cs.ROFeb 25, 2026

When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering

Authors: Jessie Yuan, Yilin Wu, Andrea Bajcsy

Organizations: Carnegie Mellon University

Abstract

Policy steering is an emerging way to adapt robot behaviors at deployment-time: a learned verifier analyzes low-level action samples proposed by a pre-trained policy (e.g., diffusion policy) and selects only those aligned with the task. While Vision-Language Models (VLMs) are promising general-purpose verifiers due to their reasoning capabilities, existing frameworks often assume these models are well-calibrated. In practice, the overconfident judgment from VLM can degrade the steering performance under both high-level semantic uncertainty in task specifications and low-level action uncertainty or incapability of the pre-trained policy. We propose uncertainty-aware policy steering (UPS), a framework that jointly reasons about semantic task uncertainty and low-level action feasibility, and selects an uncertainty resolution strategy: execute a high-confidence action, clarify task ambiguity via natural language queries, or ask for action interventions to correct the low-level policy when it is deemed incapable at the task. We leverage conformal prediction to calibrate the composition of the VLM and the pre-trained base policy, providing statistical assurances that the verifier selects the correct strategy. After collecting interventions during deployment, we employ residual learning to improve the capability of the pre-trained policy, enabling the system to learn continually but with minimal expensive human feedback. We demonstrate our framework through experiments in simulation and on hardware, showing that UPS can disentangle confident, ambiguous, and incapable scenarios and minimizes expensive user interventions compared to uncalibrated baselines and prior human- or robot-gated continual learning approaches. Videos can be found at https://jessie-yuan.github.io/ups/

Figures & tables

Fig. 1: Uncertainty-Aware Policy Steering. Our framework calibrates the VLM verifier used for policy steering via conformal prediction. This enables the VLM to select an appropriate way to resolve uncertainty, from querying the end-user in natural language to asking to re-train the low-level control policy.

Fig. 2: Outcome Prediction & Narration. The policy and the world model are interleaved to predict long-horizon outcomes induced by the low-level policy. Decoded observations are fed into a VLM which narrates the outcomes in text.

Fig. 3: Uncertainty Quantification Results: Simulation . We compare the combination of Vanilla, CoT and Bayesian Intent (Ours) models for UQ. Dashed lines indicate the target coverage ( $1-\epsilon=0.85$ ), clarification rate, and set size.

Fig. 4: Uncertainty Quantification Results: Hardware (PnP Cup) . Combination of Vanilla, CoT and Bayesian Intent (Ours) models for UQ. Dashed lines indicate the target coverage rate (

1-\epsilon=0.85

), clarification rate, and set size.

Fig. 5: Uncertainty Quantification Results: Hardware (Insert Block) . We compare the combination of Vanilla, CoT and Bayesian Intent (Ours) models for UQ. Dashed lines indicate the target coverage, clarification rate, and set size.

Fig. 6: Success Rates Pre- and Post-Continual Learning: Hardware and Simulation. We deploy the robot with 20 straightforward (left) and 20 ambiguous (right) instructions. We average the success rate over 20 trials for each scenario. UPS solicits data in a way which maximizes the final success rate after residual policy training, compared to baselines.

Fig. 7: Hardware: Robot Asking for Interventions. EnsembleDAgger (top): the human intervenes whenever the ensemble disagreement exceeds a threshold. Human-Gated (HG) DAgger (middle): a human monitors and intervenes when the robot’s behavior deviates from their intention. Our approach (bottom two) asks for “cheap” clarifications and only requests interventions when no action sample achieves the task.

Fig. 8: Intervention & Re-deployment for Insert Block Task . (left) Different strategies to elicit human interventions: HG-DAgger (top), EnsembleDAgger (middle) and UPS (bottom). (right) UPS maintains the multi-modality, while other methods exhibit failures or mode collapse.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 9: Hardware Setup: PnP Cup. We demonstrate our hardware environment setup with a Franka Emika Panda arm and two Zed cameras (left image). In the middle, we show the left-view image captured by a Zed 2i camera and on the right, we show the wrist-view image captured by a Zed Mini camera.

Fig. 10: Hardware Examples of the PnP Cup Task. We show two different ways of achieving the task of placing the cup in the bin. The top row shows the cup placed in the left bin while the bottom row shows the cup placed in the right bin.

Fig. 11: Hardware Setup: Block. We demonstrate our hardware setup for the block task using a Franka Emika Panda arm. The leftmost two images show the initial environment setup, while the right four images show the target configurations for the two behavior modes, as labeled. In each image pair, the left image provides the third-person camera view and the right image provides the wrist view.

Method	Straightforward				Ambiguous
Method	TP	TN	FP	FN	TP	TN	FP	FN
Base Policy	0.35	0.65	–	–	0.35	0.65	–	–
EnsembleDagger + Residual	0.40	0.60	–	–	0.50	0.50	–	–
HG-Dagger + Residual	0.50	0.50	–	–	0.55	0.45	–	–
Forewarn	0.80	0.10	0.10	0.0	0.50	0.0	0.40	0.10
UPS w Clarification	0.85	0.05	0.05	0.05	0.65	0.25	0.10	0.0

Appendix

TABLE I: Hardware - PnP Cup: Policy Steering Results Breakdown . We include the True Positive Rate (TP), True Negative Rate (TN), False Positive Rate (FP) and False Negative Rate (FN) in the table as an extension to the results we show in Fig. 6 . For straightforward and ambiguous scenarios, the reported success rate is averaged across 20 trials, each with a unique task instruction.

Category	User Intent	Task Instruction
Straightforward	left	“on the left side, there is a bin—place the cup there”
	left	“the cup should end up in the left bin”
	left	“take the cup off the table and finish by putting it in the left bin”
	left	“make the left bin the final location for the cup”
	left	“ensure the cup is placed inside the left bin”
	left	“please place the cup in the left bin”

Appendix

TABLE II: Straightforward Hardware (PnP Cup) Task Instructions. 40 straightforward task instructions used for the calibration dataset are listed here. During testing, we randomly sample 20 instructions from the set and obtain new options when sampling from the policy.

Category	User Intent	Task Instruction
Ambiguous	left or right	“can you deal with the cup and put it away?”
	left or right	“the cup should not be on the table anymore—put it somewhere appropriate”
	left or right	“tidy this up: move the cup into a bin”
	left or right	“pick up the cup and store it in a container”
	left or right	“put the cup away in the bin over there”
	left or right	“could you move the cup out of the working area?”

Appendix

TABLE III: Ambiguous Hardware (PnP Cup) Task Instructions. 40 ambiguous task instructions used for the calibration dataset are listed here. During testing, we randomly sample 20 instructions from the set and obtain new options with the new action samples from the base policy.

Fig. 12: Simulation Setup. We demonstrate our simulated task setup using a Franka Emika Panda arm. The leftmost two images show the initial environment setup, while the right four images show the target configurations for the two behavior modes, as labeled. In each image pair, the left image provides the third-person camera view and the right image provides the wrist view.

Method	Straightforward				Ambiguous
Method	TP	TN	FP	FN	TP	TN	FP	FN
Base Policy	0.45	0.55	–	–	0.25	0.75	–	–
EnsembleDagger + Residual	0.45	0.55	–	–	0.30	0.70	–	–
HG-Dagger + Residual	0.60	0.40	–	–	0.45	0.55	–	–
Forewarn	0.65	0.20	0.15	0.0	0.0	0.25	0.15	0.60
UPS w Clarification	0.60	0.20	0.10	0.10	0.55	0.25	0.20	0.0

Appendix

TABLE IV: Simulation Policy Steering Results Breakdown . We include the True Positive Rate (TP), True Negative Rate (TN), False Positive Rate (FP) and False Negative Rate (FN) in the table as an extension to the results we show in Fig. 6 . For straightforward and ambiguous scenarios, the reported success rate is averaged across 20 trials respectively, where each trial has an unique task instruction.

Fig. 13: Hardware PnP Cup Histogram for Empirical Quantile. We list the overall histogram with all three categories combined, as well as per category histograms. The red vertical line indicates the $\hat{q}$ value based on user-desired coverage rate $1-\varepsilon$ .

Fig. 14: Hardware Insert Block Histogram for Empirical Quantile. We list the overall histogram with all three categories combined, as well as per category histograms. The red vertical line indicates the $\hat{q}$ value based on user-desired coverage rate $1-\varepsilon$ .

Fig. 15: Simulation Histogram for Empirical Quantile. We list the overall histogram with all three categories combined, as well as per category histograms. The red vertical line indicates the $\hat{q}$ value based on user-desired coverage rate $1-\varepsilon$ .

Fig. 16: Hardware PnP Cup Uncertainty Quantification with Logits . We conduct an ablation study where we directly use the softmax over the logits of the first generated token as the score rather than the self-generated score. Empirically, we find that, compared to Fig. 4 , the self-generated score is a better score function across all scenarios.

Fig. 17: Simulation Uncertainty Quantification with Logits . Similar to hardware experiments, we compare the results of the softmax of logits as the score with self-generated score results in Fig. 3 . None of the methods with logits can achieve the desired coverage while balancing the clarification rate and reaching the desired prediction set size.

Hyperparameter	Value
Batch Size	64
Learning Rate	1e-4
Iterations	10000
Dropout	0.2
Base Action Encoder Output Dimension	128
Action Predictor Hidden Dimension	256

Appendix

TABLE V: Hyperparameters for Residual Policy Training.

User Intent	Task Instruction
left	“Please place the nut on the peg with the handle facing left.”
left	“I want the nut positioned on the peg with the handle directed left.”
left	“Put the nut on the peg so that its handle is oriented leftward.”
left	“Set the nut onto the peg with the handle pointing left.”
left	“The nut goes on the peg with its handle turned to face left.”
left	“Can the nut be placed on the peg so its handle is pointing to the left?”

Appendix

TABLE VI: Straightforward Simulation Task Instructions. 60 straightforward task instructions used for the calibration and test datasets are listed here. 40 of these are randomly sampled to be part of the calibration set, while the remaining 20 make up the test set. We use “c.c.w.” and “c.w.” to denote “counterclockwise” and “clockwise” respectively.

User Intent	Task Instruction
left or right	“Please place the nut on the peg.”
left or right	“I want the nut positioned on the peg.”
left or right	“Pick up the nut and set it on the peg.”
left or right	“Can you set the nut on the peg?”
left or right	“Put the nut onto the peg.”
left or right	“I need you to place the nut on the peg.”

Appendix

TABLE VII: Ambiguous Simulation Task Instructions. 60 ambiguous task instructions used for the calibration and test datasets are listed here. 40 of these are randomly sampled to be part of the calibration set, while the remaining 20 make up the test set.

Category	User Intent	Task Instruction
Straightforward	left	“Insert the purple block into the left hole.”
	left	“Attach the purple block to the left-side hole.”
	left	“Fit the purple block into the left corner.”
	left	“Push the purple block into the left slot.”
	left	“Align the purple block with the left hole and insert it.”
	left	“Fasten the purple block into the hole on the left.”

Appendix

TABLE VIII: Block Insertion Task Instructions. Straightforward task instructions specify whether the purple block should be inserted into the left or right hole, while ambiguous instructions underspecify the target hole and require disambiguation.

Fig. 18: Hardware Prompt For Behavior Translation (Grasp)

Fig. 19: Hardware Prompt For Behavior Translation (Place)

Fig. 20: Hardware Prompt for VLM Verification in Forewarn

Fig. 21: Hardware Prompt for Generating Possible Intents

Fig. 22: Hardware Prompt for Inferring Intent Probability

Fig. 23: Hardware Prompt for Updating Instructions

Fig. 24: Hardware Prompt for Clarification Questions

Fig. 25: Hardware Prompt for Generating Behavior Probability Conditioned On the Intent

Fig. 26: Hardware Prompt For Asking VLM to Analyze Ambiguity First in CoT Reasoning

Fig. 27: Hardware Prompt For Asking VLM to Score Options Based On Previous Reasoning in CoT Reasoning

Fig. 28: Hardware Prompt For Directly Generating Scores for Available Options

Fig. 29: Simulation Prompt For Behavior Translation (Grasp)

Fig. 30: Simulation Prompt For Behavior Translation (Place)

Fig. 31: Simulation Prompt for VLM Verification in Forewarn

Fig. 32: Simulation Prompt for Generating Possible Intents

Fig. 33: Simulation Prompt for Inferring Intent Probability

Fig. 34: Simulation Prompt For Asking VLM to Analyze Ambiguity First in CoT Reasoning

Fig. 35: Simulation Prompt For Asking VLM to Score Options Based On Previous Reasoning in CoT Reasoning

Fig. 36: Simulation Prompt for Generating Behavior Probability Conditioned On the Intent

Fig. 37: Simulation Prompt For Directly Generating Scores for Available Options

Robot Policy Adaptation

Robot policy adaptation changes or steers learned robot behaviors to handle new tasks, environments, hardware, and user preferences.

See all 160 papers

Steering Autoregressive Vision-Language-Action Policies via Action Token Intervention
Jun 12, 2026Jason Chan, Jonathan C. KaoRobotic Manipulation Vision-Language-Action Models
Department of Computer Science University of California, Los Angeles · Departments of Electrical and Computer Engineering, Computer Science, and Neurobiology University of California, Los Angeles
From Uncertainty to Action: Learning to Steer LLM Agents
Oct 6, 2026Hanwen Li, Jinhao Duan, Guanhua Zhu +4Language Model Steering LLM Agents
New Jersey Institute of Technology · University of North Carolina at Chapel Hill · University of California, Irvine +1
DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning
Jul 3, 2026Hanchen Cui, Sergio Arnaud, Arjun Majumdar +5World Model-Based Planning Robotic Manipulation
1Fundamental AI Research (FAIR), Meta · University of Minnesota Twin Cities

Jun 12, 2026J/K move · Enter open · S save

Steering Autoregressive Vision-Language-Action Policies via Action Token Intervention

Jason Chan, Jonathan C. Kao

Department of Computer Science University of California, Los Angeles · Departments of Electrical and Computer Engineering, Computer Science, and Neurobiology University of California, Los Angeles

Robotic Manipulation Vision-Language-Action Models Human-Robot Interaction

Open paper AskPDF arXiv

We present Token Steering (TS), a method for dynamically steering trajectories generated by an autoregressive vision-language-action (VLA) model through direct intervention in the action-token space. TS injects low-dimensional user inputs into the model's native action-token representation, allowing users to influence trajectory generation without modifying the underlying vision-language model (VLM) architecture. Because TS operates entirely at inference time, it requires no additional training or finetuning. User inputs guide rather than override the pretrained policy, allowing users to influence robot actions while preserving the dexterity, smoothness, and task priors learned by the VLA. We evaluate TS on two household manipulation tasks -- drawer closing after object placement and state-aware object swapping -- and improve success rates from 10.0% to 72.5% and from 16.7% to 93.8%, respectively. By enabling lightweight, intuitive steering over robot foundation models, our interface has the potential to improve human-robot interaction in consumer environments and broaden accessibility for individuals with limited physical control. Project website: https://jasontchan.github.io/token-steering/ .

Abstract

Robot Policy Adaptation

Explore similar work

Steering Autoregressive Vision-Language-Action Policies via Action Token Intervention

From Uncertainty to Action: Learning to Steer LLM Agents

DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning