Steertuning Policies from a Few Human Videos

Billy Yan; Isaac Tu; Kelly Lee; Blake Chang; Kenny Ma; Kanad Patel; Dhawal Kabra; Neer Patel

cs.ROOct 13, 2026

Steertuning Policies from a Few Human Videos

Authors: Billy Yan, Isaac Tu, Kelly Lee, Blake Chang, Kenny Ma, Kanad Patel, Dhawal Kabra, Neer Patel, +3 more

Organizations: New York University

Abstract

Current recipes for robot learning require massive amounts of robot demonstrations to obtain robust and adaptive policies. In this work, we introduce Steertuning, a system that enables robots to learn new tasks in new environments from 5 human demonstrations in under 5 Steertuning consists of two parts. (1) We pretrain a 400M-parameter diffusion policy on our egocentric human dataset, which consists of 100 hours of densely annotated random interactions. This pretraining makes Steertuning robust to background differences and enables spatial and object generalization when deployed on new tasks. (2) Given 5 demonstrations of a new task, we finetune our model for only 2 minutes (2000 training updates) and steer the policy's actions during reverse sampling using the demonstration whose observation is closest to that of the deployment environment. Together with a carefully designed data processing pipeline for embodiment transfer, this enables Steertuning to achieve a 71% success rate across 12 tasks in 12 new environments, ranging from lab settings to kitchens in new homes. The entire process, from data collection for a new task to robot policy deployment, takes less than 5 minutes. Our full dataset, model weights, codebase, and robot videos will be made available at our website: https://steertuning.github.io.

Figures & tables

Figure 2: Policy. Our architecture and action steering mechanism. The policy takes four modalities as input: a point cloud, the manipulated-object mask, the fingertip locations of both hands (10 3D points) in the anchor frame, and a text description of the task. Each modality is passed through its own encoder and the resulting tokens go through a self-attention transformer, after which we mean-pool them into a context latent that conditions the diffusion transformer via AdaLayerNorm [ 50 ] . The observation tokens are also fed to the diffusion transformer so that the noisy action tokens can cross-attend to them [ 51 ] . The point-cloud encoder (Utonia [ 52 ] ) and text encoder (SigLIP [ 53 ] ) are kept frozen; everything else is trained end-to-end with an MSE loss on the predicted $\epsilon$ . During finetuning, LoRA adapters are trained for 2k steps; at deployment, the predicted actions are steered toward the nearest human trajectory. A short finetuning combined with action steering enables robust yet quick task adaptation.

Figure 3: In-the-wild and ablation tasks. We deploy Steertuning on 9 new tasks in new in-the-wild environments. For each task, we collect 5 new human demonstrations (top left) and deploy the policy on our robot YOR (top right). All ablation experiments are run on three tasks with spatial generalization (bottom). Steertuning achieves an overall success rate of $71\%$ (7.1/10) across these 12 tasks; per-task success rates are given on our website.

Figure 4: Full-stack deployments of different policies from a single language command. The left-most column shows the language annotations used by the system. For example, Put the white cup into the trash is split into navigate white cup , pick up white cup , navigate trash , and drop white cup . Demonstrations for picking up and dropping the cup are collected and deployed in the same way as in Steertuning.

Figure 5: Scaling ablations for (a) pretraining compute budget, (b) LoRA compute budget, (c) pretraining data hours, and (d) number of LoRA demonstrations. Points are the mean success rate over the three tasks; bars show

\pm 1

standard deviation. Steertuning settings are marked with *.

Figure 6: Mean landmark error as a function of the steering weight

w

for different fine-tuning budgets, averaged over our 12 tasks. Error is the MSE between the predicted and ground-truth action landmarks on a held-out demonstration; the dotted line is the error of replaying the demonstration directly. Values above 350 (no fine-tuning at

w\leq 0.2

, 0.5k steps at

w=0

) are cut off for readability.

Figure 7: Object generalization. For each task, the left image shows the original object and the right image the new object used at deployment, with the corresponding success rate.

Pipeline	Trash	Flower	Bread	Avg.
Steertuning	9/10	7/10	9/10	8.3/10
Steering only	2/10	2/10	3/10	2.3/10
Short fine-tune (2k)	0/10	0/10	0/10	0/10
Full fine-tune (150k)	7/10	8/10	8/10	7.7/10

Table I: Success rates of the full Steertuning pipeline compared to fine-tuning only and steering only. Ours in bold.

		Trash	Flower	Bread	Avg.
Encoder	PointNeXt	5/10	6/10	7/10	6/10
	PTv3	10/10	4/10	9/10	7.7/10
	Utonia (Ours)	9/10	7/10	9/10	8.3/10
Decoder	UNet	8/10	8/10	6/10	7.3/10
Decoder	DiT (Ours)	9/10	7/10	9/10	8.3/10

Table II: Success rates for encoder and decoder architecture ablations. Ours in bold.

Trajectory selection	Trash	Flower	Bread	Avg.
VINN [ 71 ]	7/10	2/10	7/10	5.3/10
Wrist pose	6/10	7/10	4/10	5.7/10
Object-based (Ours)	9/10	7/10	9/10	8.3/10

Table III: Success rates for trajectory-selection strategies at inference. Ours in bold.

Explore similar work

RoboPrompt: Intuitive Robot Policy Steering with Sparse Human Input
Oct 7, 2026Yanwen Zou, Chenyang Shi, Guoxuan Xu +5Robot Policy Adaptation Interactive Imitation Learning
Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.
Proxy Policy Steering
Sep 8, 2026Chuanruo Ning, Tianrui Wang, Wei-Chiu Ma +1Robot Policy Adaptation Robot Foundation Models
Cornell University
Adaptation of Generalist Robot Policies with Minimal Data
Aug 11, 2026Shreyas Kowshik, Sreyas Venkataraman, Leo Wang +3RL for Robotics Robot Policy Adaptation
Carnegie Mellon University

Oct 7, 2026J/K move · Enter open · S save

RoboPrompt: Intuitive Robot Policy Steering with Sparse Human Input

Yanwen Zou, Chenyang Shi, Guoxuan Xu, Wenye Yu+4

Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.

Robot Policy Adaptation Interactive Imitation Learning

Open paper AskPDF arXiv

End-to-end robot policies trained through imitation learning remain constrained by limited data diversity, making reliable zero-shot deployment in real-world settings challenging. Shared-autonomy methods enable human correction through teleoperation, but specialized hardware and operator training hinder deployment at scale. Other approaches incorporate human guidance as additional policy inputs, often requiring architectural changes and dedicated training for steerability, which limits their applicability across policies. We present RoboPrompt, a general-purpose, lightweight robot policy steering system that enables users to guide policy behavior through intuitive, sparse inputs, including drawn traces, target points, and coarse directional instructions. RoboPrompt decouples human-intention translation from the underlying policy: a reusable module converts human guidance into action drafts, which are refined through the diffusion or flow-matching dynamics of the base policy. By controlling action generation in noise space, RoboPrompt balances human intent with the policy prior without modifying the base policy architecture or fine-tuning it for steerability. Experiments demonstrate effective steering across Diffusion Policy,

π_{0.5}

, and FastWAM. We further use steered rollouts for online policy improvement through DAgger. After 2-3 rounds of iteration, average success rates increase by 15.5% for

π_{0.5}

across three tasks and by 21.3% across three policies(Diffusion Policy,

π_{0.5}

, FastWAM) on the Insert Bread task, while average human intervention counts decrease by 44.0% (2.86 to 1.60) and 81.9% (2.60 to 0.47), respectively.

Abstract

Explore similar work

RoboPrompt: Intuitive Robot Policy Steering with Sparse Human Input

Proxy Policy Steering

Adaptation of Generalist Robot Policies with Minimal Data