ReSteer: Quantifying and Refining the Steerability of Multitask Robot Policies

Zhenyang Chen; Alan Tian; Liquan Wang; Benjamin Joffe; Yingyan Celine Lin; Yuxiao Chen; Siddharth Karamcheti; Danfei Xu

cs.ROMar 18, 2026

ReSteer: Quantifying and Refining the Steerability of Multitask Robot Policies

Authors: Zhenyang Chen, Alan Tian, Liquan Wang, Benjamin Joffe, Yingyan Celine Lin, Yuxiao Chen, Siddharth Karamcheti, Danfei Xu

Organizations: Georgia Institute of Technology

Abstract

Despite strong multi-task pretraining, existing policies often exhibit poor task steerability. For example, a robot may fail to respond to a new instruction put the bowl in the sink" when moving towards the oven, executing close the oven", even though it can complete both tasks when executed separately. We propose ReSteer, a framework to quantify and improve task steerability in multitask robot policies. We conduct an exhaustive evaluation of state-of-the-art policies, revealing a common lack of steerability. We find that steerability is associated with limited overlap among training task trajectory distributions, and introduce a proxy metric to measure this overlap from policy behavior. Building on this insight, ReSteer improves steerability via three components: (i) a steerability estimator that identifies low-steerability states without full-rollout evaluation, (ii) a steerable data generator that synthesizes motion segments from these states, and (iii) a self-refinement pipeline that improves policy steerability using the generated data. In simulation on LIBERO, ReSteer improves steerability by 11% over 18k rollouts. In real-world experiments, we show that improved steerability is critical for interactive use, enabling users to instruct robots to perform any task at any time. We hope this work motivates further study on quantifying steerability and data collection strategies for large robot policies.

Figures & tables

Fig. 1: Daily manipulation is diverse and time-critical, and often requires interruptible behavior, with users revising their intent after execution begins. This demands interactive, steerable policies that can switch behaviors from any intermediate state in response to a new task prompt. We propose ReSteer , a framework to quantify and improve the steerability of multitask robot policies.

	OpenVLA-OFT	MolmoACT	$\boldsymbol{\pi_{0.5}}$
Steerability Score	0.252	0.295	0.403

TABLE I: VLAs steerability comparison on LIBERO.

Fig. 2: Steerability evaluation. We run an original task $i$ , then at sampled timesteps $t$ we switch the language prompt to a target task $j$ and measure whether the policy completes $j$ from that state. Repeating this over all sampled states and all target tasks yields a steerability matrix (time $\times$ target task), whose average success rate gives the overall steerability score in task $i$ .

Fig. 3: We propose an online learning framework to improve the steerability of multitask policies. The framework comprises three components. First, we propose a CMI-based state sampling strategy that prioritizes data collection at the most unsteerable states, improving the sample efficiency of data generation. Second, we introduce a stage-aware steering data generation pipeline that synthesizes cross-task steering motions ( $s^{t}_{task1}\rightarrow s^{t}_{task2}$ ), thereby expanding the steerable states, $\mathcal{S}^{\mathrm{steer}}_{i\leftrightarrow j}$ . Third, we develop a self-refining behavior cloning (SRBC) scheme that finetunes the policy using successful steered trajectories given different instructions $l$ and consequently increases steerability coverage ratio $\mathrm{SCR}_{i\leftrightarrow j}(\pi)$ .

Fig. 4: Illustration of the steerability improvement afforded by ReSteer. Left: the bidirectionally steerable set $\mathcal{S}^{\mathrm{steer}}_{i\leftrightarrow j}$ (green) occupies only a small subset of the policy-induced feasible states $\mathcal{S}_{i}$ and $\mathcal{S}_{j}$ . Middle: stage-aware steering data generation ( $\mathcal{S}^{\text{gen}}_{i}$ and $\mathcal{S}^{\text{gen}}_{j}$ ) introduces instruction-contrasting transitions, expanding the steerable region. Right: self-refining behavior cloning (SRBC) further enlarges steerability within the same feasible state space.

Fig. 5: Evaluating Steerability on LIBERO-Goal. For each source task (x-axis), we sample intermediate execution states and measure the average success rate of switching to each of the other nine target tasks under the corresponding language prompt (bars). ReSteer achieves the highest steerability across all ten tasks and consistently outperforms strong baselines such as CAST [ 23 ] , demonstrating the effectiveness of our two-stage data generation pipeline.

Fig. 6: To evaluate CMI as a proxy for steerability, we track CMI and steerability scores across training checkpoints. As shown in left figure, CMI is positively correlated with steerability, with particularly consistent trends within a single model family (the $\pi_{0.5}$ variants). Motivated by this relationship, we use CMI to prioritize low-steerability states for data collection and fine-tuning. Empirically, MI-guided sampling yields higher sample efficiency than uniform random sampling.

Fig. 7: We conduct a case study on task steering from Push the Plate to the Front of the Stove $\rightarrow$ Put the Bowl on the Plate . The prompt is switched at the indicated timestep, with representative LIBERO environment states shown at select timesteps. SteerGen and ReSteer consistently outperform all baseline methods, with particularly strong gains at later timestep switches. ReSteer further improves upon SteerGen by sharpening the policy in regions where SteerGen cannot consistently complete the task.

Fig. 8: In an example with four single tasks and three steering scenarios, ReSteer (bottom row) successfully switches to the requested next task upon user instruction, whereas the baseline (top row) exhibits failure modes such as getting stuck or continuing the original task (keep grasping the tea bag, put the bowl to the shelf instead of sink) despite the new prompt.

Fig. 9: We evaluate policies on the DROID platform across four single-task evaluations and three steering scenarios. In-domain teleoperation data improves single-task success over the baseline, but yields limited steerability. In contrast, ReSteer achieves

\mathbf{2.2\times}

higher steerability while preserving single-task performance.

Single-task success (%)
Test Case	$\boldsymbol{\pi_{0.5}}$	$\boldsymbol{\pi_{0.5}}$ -FT	$\boldsymbol{\pi_{0.5}}$ - ReSteer
t1-close the oven	0	100	80
t2-put tea bag in drawer	4	60	70
t3-put white bowl on the shelf	20	100	100
t4-put white bowl in the sink	60	100	100
Steering success (%)

TABLE II: Success rates for single-task execution and pregrasping-stage steering across three methods.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 10: Task-level steerability improvement on LIBERO-Goal. Each subplot corresponds to a source task (executed initially), and reports the improvement in steering success rate ( $\Delta$ over the baseline policy) when switching to all other target tasks at different timesteps. The x-axis shows the execution timestep of task switching, the y-axis lists target tasks, and color encodes the success-rate difference, where blue indicates improvement. The number above each subplot indicates the average improvement across all target tasks for that source task ReSteer consistently improves steerability across tasks, with particularly large gains at later timesteps where task commitment is stronger and baseline policies tend to ignore prompt changes.

Fig. 11: We prompt the model to switch from the ”put the bowl on the stove” to ”put the wine bottle on top of the cabinet” at $t=17$ . While baseline follows ”put the bowl on the stove” throughout the rollout, ReSteer manages to switch to the execution of the ”put the wine bottle on top of the cabinet.”

Fig. 12: Comparison of start-step distributions under MI-guided sampling and uniform random sampling. The x-axis denotes the timestep selected as the starting state for steering data generation, and the y-axis shows the sampling count. MI-guided sampling concentrates selections in specific temporal regions corresponding to low-CMI (instruction-blind) states, where language–action coupling is weak. In contrast, random sampling distributes selections more uniformly across the rollout. This targeted concentration demonstrates the improved sample efficiency of CMI-based prioritization, focusing data generation on states that most require steering supervision.

Fig. 13: Visualization of MI-guided sampled states on the DROID platform. Highlighted frames correspond to rollout states selected using the CMI-based prioritization strategy. The sampled states concentrate around high-conflict or decision-critical phases (e.g., object approach, grasp transition, and pre-placement), where baseline policies exhibit weak language–action coupling. This confirms that CMI-guided sampling identifies instruction-blind regions in real-world trajectories and focuses data generation on states that most require steering supervision, thereby improving sample efficiency compared to uniform sampling.

Fig. 14: Visualization of states that were sampled according to the CMI Metric for rollout of task p̈ut the wine bottle on top of the cabinet”. The blue boxes indicate the states selected with mutual information.

	CMI-Sampling	Random Sampling
Steerability Score	0.8	0.75

Appendix

TABLE III: Steerability Score for Different Sampling Strategy on DROID

Fig. 15: We visualize the LIBERO benchmark trajectories. Most of the task trajectories from the dataset are clustered into a small space and lead to a relatively small $\mathcal{S}^{\text{steer}}$ , which potentially makes the learn policies less steerable.

Fig. 16: Stage-dependent linear separability of task semantics from visual latents. We report the performance of a linear probe trained to predict task identity from visual latents as a function of rollout timestep. Top: mean cosine similarity between predicted and ground-truth task prototypes. Bottom: mean classification accuracy. Early in the rollout (pre-grasp stage), both metrics are low and exhibit high variance, indicating that visual latents are not linearly separable across tasks when the scene is visually similar. As execution progresses and physical interaction occurs (post-grasp and transport stages), both cosine similarity and accuracy rapidly increase and saturate near $1.0$ , with reduced variance. This demonstrates that task semantics become strongly encoded in visual latents once task-specific effects are reflected in the environment state, enabling task inference from vision alone.

Training Configuration
Base model	$\pi_{0.5}$
Training steps	2000
Observation history	1
Action horizon	10
Batch size	128
Finetuning method	Full finetuning

Appendix

TABLE IV: Training and evaluation hyperparameters used in all experiments.

Multi-Task Robotic Manipulation

Multi-task robotic manipulation studies how robots can learn reusable, reliable behaviors across diverse objects, tasks, and environments.

See all 56 papers

RoboPrompt: Intuitive Robot Policy Steering with Sparse Human Input
Oct 7, 2026Yanwen Zou, Chenyang Shi, Guoxuan Xu +5Robot Policy Adaptation Interactive Imitation Learning
Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.
SteerAct: Unlocking Action Models through Few-Shot Temporal Steering
Oct 19, 2026Kaichen Zhou, Zhiyan Li, Chongkai Gao +11Data-Efficient Robot Learning Test-Time Adaptation of VLA Models
Massachusetts Institute of Technology · Harvard University · Stanford University +3
Steertuning Policies from a Few Human Videos
Oct 13, 2026Billy Yan, Isaac Tu, Kelly Lee +8Robot Policy Adaptation Data-Efficient Robot Learning
New York University

Oct 7, 2026J/K move · Enter open · S save

RoboPrompt: Intuitive Robot Policy Steering with Sparse Human Input

Yanwen Zou, Chenyang Shi, Guoxuan Xu, Wenye Yu+4

Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.

Robot Policy Adaptation Interactive Imitation Learning

Open paper AskPDF arXiv

End-to-end robot policies trained through imitation learning remain constrained by limited data diversity, making reliable zero-shot deployment in real-world settings challenging. Shared-autonomy methods enable human correction through teleoperation, but specialized hardware and operator training hinder deployment at scale. Other approaches incorporate human guidance as additional policy inputs, often requiring architectural changes and dedicated training for steerability, which limits their applicability across policies. We present RoboPrompt, a general-purpose, lightweight robot policy steering system that enables users to guide policy behavior through intuitive, sparse inputs, including drawn traces, target points, and coarse directional instructions. RoboPrompt decouples human-intention translation from the underlying policy: a reusable module converts human guidance into action drafts, which are refined through the diffusion or flow-matching dynamics of the base policy. By controlling action generation in noise space, RoboPrompt balances human intent with the policy prior without modifying the base policy architecture or fine-tuning it for steerability. Experiments demonstrate effective steering across Diffusion Policy,

π_{0.5}

, and FastWAM. We further use steered rollouts for online policy improvement through DAgger. After 2-3 rounds of iteration, average success rates increase by 15.5% for

π_{0.5}

across three tasks and by 21.3% across three policies(Diffusion Policy,

π_{0.5}

, FastWAM) on the Insert Bread task, while average human intervention counts decrease by 44.0% (2.86 to 1.60) and 81.9% (2.60 to 0.47), respectively.

Abstract

Multi-Task Robotic Manipulation

Explore similar work

RoboPrompt: Intuitive Robot Policy Steering with Sparse Human Input

SteerAct: Unlocking Action Models through Few-Shot Temporal Steering

Steertuning Policies from a Few Human Videos