Cross-Controller Transfer via In-Context Behavior Adaptation

Jonathan Yang; Chelsea Finn; Dorsa Sadigh

cs.ROOct 29, 2026

Cross-Controller Transfer via In-Context Behavior Adaptation

Authors: Jonathan Yang, Chelsea Finn, Dorsa Sadigh

Organizations: Department of Computer Science, Stanford University, Stanford, CA, USA

Abstract

Generalist robot policies increasingly learn from data collected across diverse tasks and embodiments, yet typically assume a single controller for each robot. This assumption limits transfer because different controllers can assign different physical meanings to the same action and may require different strategies to complete a task. We study how to transfer robot data across both embodiments and controllers without allowing conflicting action interpretations to interfere with learning. Our setting spans six controller types as well as variations in controller gains and compliance. We propose a framework that encodes behavioral interaction under a controller through examples of commanded actions and the resulting robot motions. A transformer context encoder summarizes these interactions, allowing a single policy to adapt in context. Across real-world experiments, our method improves average success by 33 percentage points over naive data pooling, achieves 50% success on unseen controller modes and 52% on unseen controller gains, and transfers to tasks with limited controller-specific data. On contact-rich tasks such as USB insertion and surface wiping, behavioral interaction context enables the policy to adapt not only its actions but also its strategy. Our results provide a path toward generalist robot policies that can leverage data collected across diverse action interpretations. For more information, visit https://controller-transfer.github.io.

Figures & tables

Fig. 1: Cross-Controller Transfer. We train a generalist policy that adapts its manipulation strategy across a dense spectrum of unseen controller types and continuous gains by conditioning on interaction history.

Fig. 2: The optimal strategy differs per controller. Two learned wiping strategies evaluated across a shared $(K_{p},K_{d})$ plane. Low stiffness favors compliant pressing ( $80\%$ – $90\%$ ), whereas high stiffness favors stiff tracing ( $75\%$ – $85\%$ ).

Fig. 3: Controller-informative events are not temporally aligned. We encode every commanded–achieved transition in each trajectory; blue windows indicate where the strongest controller-relevant evidence occurs. Contact, braking, saturation, and recovery appear at different times across tasks, motivating complete trajectory context rather than a fixed temporal crop.

Fig. 4: Cross-task behavioral context provides information beyond controller dynamics. Evaluation on held-out task–controller cells across five manipulation tasks ( $N=20$ trials per cell). Zero-shot evaluates transfer where the target task has no demonstrations under deployment controller relying solely on target-task data collected under another controller. Few-shot provides a budget of $5$ demonstrations from the target (task, controller) pair.

Fig. 5: Generalization across controller modes and gains. Zero-shot success rates across six held-out controller modes (

N=60

, left) and four unseen

(K_{p},K_{d})

joint impedance gains (

N=50

, right) compared to naive pooling and synthetic relabeling (

\pm\text{SE}

Fig. 6: Conditioning and information ablations. Left: Behavioral context resolves the strategy multimodality introduced by relabeling (

0.15\to 0.47

). Right: Focusing context on contact-rich transitions (

0.50

) captures the densest strategy signal, matching or exceeding uniform full-trajectory context (

0.49

Fig. 7: Cross-task behavioral context transfer. Reusing interaction-context tokens $z_{\mathrm{tool},c}$ extracted from tool-bit insertion to condition USB insertion under the same compliant controller achieves $35\%$ performance.

Robot Policy Adaptation

Robot policy adaptation changes or steers learned robot behaviors to handle new tasks, environments, hardware, and user preferences.

See all 175 papers

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer
Sep 18, 2026Yiqi Wang, Mrinal Verghese, Jeff SchneiderCross-Embodiment Transfer Visuomotor Policy Learning
Robotics Institute, School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, United States
Data Analogies Enable Efficient Cross-Embodiment Transfer
Mar 6, 2026Jonathan Yang, Chelsea Finn, Dorsa SadighCross-Embodiment Transfer Generalization in Robotic Manipulation
Stanford University
SPACE: Enabling Learning from Cross-Robot Data Toward Generalist Policies
Jun 23, 2026Haeone Lee, Byeongguk Jeon, Suchae Jeong +2Robotic Control Robot Policy Adaptation
1KAIST · 2Config · 3Yonsei University

Sep 18, 2026J/K move · Enter open · S save

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer

Yiqi Wang, Mrinal Verghese, Jeff Schneider

Robotics Institute, School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, United States

Cross-Embodiment Transfer Visuomotor Policy Learning Robot Policy Generalization Data-Efficient Robot Learning World Models for Robotics

Open paper AskPDF arXiv

The performance of learned robot visuomotor policies depends heavily on the size and quality of their training data, yet collecting high-quality demonstrations remains costly for robots in the real world. Although large-scale robot and human datasets are increasingly available, embodiment gaps and mismatched action spaces make them difficult to leverage directly. Cross-embodiment transfer, reusing experience from other embodiments to improve learning on a target embodiment, is therefore crucial for scaling robot learning beyond per-robot data collection. In this work, we find that efficient transfer can be achieved by learning from what is shared across embodiments, the visual dynamics of how the world responds to motion, and by effectively exploiting the scarce target-embodiment data at test time. The proposed framework, called Latent Policy Steering (LPS), implements an embodiment-agnostic pretraining phase, which trains an image-based World Model (WM) with optical flow across diverse embodiments. The resulting WM is finetuned on the target embodiment with robot actions. It then steers the base policy toward better actions by searching in the WM's latent space for plans that stay close to the finetuning data. LPS is a policy-agnostic framework: it can flexibly accommodate different policies without having to retrain them. In Robomimic and real-world evaluations, LPS improves the average performance of Diffusion Policy relatively by 16% and 62%, and Pi0.5 by 8% and 14%, with only 50 demonstrations on an unseen target embodiment.

Abstract

Robot Policy Adaptation

Explore similar work

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer

Data Analogies Enable Efficient Cross-Embodiment Transfer

SPACE: Enabling Learning from Cross-Robot Data Toward Generalist Policies