Cross-Controller Transfer via In-Context Behavior Adaptation

Jonathan Yang; Chelsea Finn; Dorsa Sadigh

cs.ROOct 29, 2026

Cross-Controller Transfer via In-Context Behavior Adaptation

Authors: Jonathan Yang, Chelsea Finn, Dorsa Sadigh

Organizations: Department of Computer Science, Stanford University, Stanford, CA, USA

Abstract

Generalist robot policies increasingly learn from data collected across diverse tasks and embodiments, yet typically assume a single controller for each robot. This assumption limits transfer because different controllers can assign different physical meanings to the same action and may require different strategies to complete a task. We study how to transfer robot data across both embodiments and controllers without allowing conflicting action interpretations to interfere with learning. Our setting spans six controller types as well as variations in controller gains and compliance. We propose a framework that encodes behavioral interaction under a controller through examples of commanded actions and the resulting robot motions. A transformer context encoder summarizes these interactions, allowing a single policy to adapt in context. Across real-world experiments, our method improves average success by 33 percentage points over naive data pooling, achieves 50% success on unseen controller modes and 52% on unseen controller gains, and transfers to tasks with limited controller-specific data. On contact-rich tasks such as USB insertion and surface wiping, behavioral interaction context enables the policy to adapt not only its actions but also its strategy. Our results provide a path toward generalist robot policies that can leverage data collected across diverse action interpretations. For more information, visit https://controller-transfer.github.io.

Figures & tables

Fig. 1: Cross-Controller Transfer. We train a generalist policy that adapts its manipulation strategy across a dense spectrum of unseen controller types and continuous gains by conditioning on interaction history.

Fig. 2: The optimal strategy differs per controller. Two learned wiping strategies evaluated across a shared $(K_{p},K_{d})$ plane. Low stiffness favors compliant pressing ( $80\%$ – $90\%$ ), whereas high stiffness favors stiff tracing ( $75\%$ – $85\%$ ).

Fig. 3: Controller-informative events are not temporally aligned. We encode every commanded–achieved transition in each trajectory; blue windows indicate where the strongest controller-relevant evidence occurs. Contact, braking, saturation, and recovery appear at different times across tasks, motivating complete trajectory context rather than a fixed temporal crop.

Fig. 4: Cross-task behavioral context provides information beyond controller dynamics. Evaluation on held-out task–controller cells across five manipulation tasks ( $N=20$ trials per cell). Zero-shot evaluates transfer where the target task has no demonstrations under deployment controller relying solely on target-task data collected under another controller. Few-shot provides a budget of $5$ demonstrations from the target (task, controller) pair.

Fig. 5: Generalization across controller modes and gains. Zero-shot success rates across six held-out controller modes (

N=60

, left) and four unseen

(K_{p},K_{d})

joint impedance gains (

N=50

, right) compared to naive pooling and synthetic relabeling (

\pm\text{SE}

Fig. 6: Conditioning and information ablations. Left: Behavioral context resolves the strategy multimodality introduced by relabeling (

0.15\to 0.47

). Right: Focusing context on contact-rich transitions (

0.50

) captures the densest strategy signal, matching or exceeding uniform full-trajectory context (

0.49

Fig. 7: Cross-task behavioral context transfer. Reusing interaction-context tokens $z_{\mathrm{tool},c}$ extracted from tool-bit insertion to condition USB insertion under the same compliant controller achieves $35\%$ performance.

Robot Policy Adaptation

Robot policy adaptation changes or steers learned robot behaviors to handle new tasks, environments, hardware, and user preferences.

See all 175 papers

Sep 18, 2026

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer

The performance of learned robot visuomotor policies depends heavily on the size and quality of their training data, yet collecting high-quality demonstrations remains costly for robots in the real world. Although large-scale robot and human datasets are increasingly available, embodiment gaps and mismatched action spaces make them difficult to leverage directly. Cross-embodiment transfer, reusing experience from other embodiments to improve learning on a target embodiment, is therefore crucial for scaling robot learning beyond per-robot data collection. In this work, we find that efficient transfer can be achieved by learning from what is shared across embodiments, the visual dynamics of how the world responds to motion, and by effectively exploiting the scarce target-embodiment data at test time. The proposed framework, called Latent Policy Steering (LPS), implements an embodiment-agnostic pretraining phase, which trains an image-based World Model (WM) with optical flow across diverse embodiments. The resulting WM is finetuned on the target embodiment with robot actions. It then steers the base policy toward better actions by searching in the WM's latent space for plans that stay close to the finetuning data. LPS is a policy-agnostic framework: it can flexibly accommodate different policies without having to retrain them. In Robomimic and real-world evaluations, LPS improves the average performance of Diffusion Policy relatively by 16% and 62%, and Pi0.5 by 8% and 14%, with only 50 demonstrations on an unseen target embodiment.

Yiqi Wang, Mrinal Verghese, Jeff Schneider

Robotics Institute, School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, United States

Cross-Embodiment Transfer Visuomotor Policy Learning Robot Policy Generalization Data-Efficient Robot Learning World Models for Robotics

Ask PDF

Mar 6, 2026

Data Analogies Enable Efficient Cross-Embodiment Transfer

Generalist robot policies are trained on demonstrations collected across a wide variety of robots, scenes, and viewpoints. Yet it remains unclear how to best organize and scale such heterogeneous data so that it genuinely improves performance in a given target setting. In this work, we ask: what form of demonstration data is most useful for enabling transfer across robot set-ups? We conduct controlled experiments that vary end-effector morphology, robot platform appearance, and camera perspective, and compare the effects of simply scaling the number of demonstrations against systematically broadening the diversity in different ways. Our simulated experiments show that while perceptual shifts such as viewpoint benefit most from broad diversity, morphology shifts benefit far less from unstructured diversity and instead see the largest gains from data analogies, i.e. paired demonstrations that align scenes, tasks, and/or trajectories across different embodiments. Informed by the simulation results, we improve real-world cross-embodiment transfer success by an average of

22.5\%

over large-scale, unpaired datasets by changing only the composition of the data.

Jonathan Yang, Chelsea Finn, Dorsa Sadigh

Stanford University

Cross-Embodiment Transfer Generalization in Robotic Manipulation Learning from Demonstration Data-Efficient Robot Learning

Ask PDF

Jun 23, 2026

SPACE: Enabling Learning from Cross-Robot Data Toward Generalist Policies

In robot learning, scaling training datasets across diverse embodiments and environments has become a dominant paradigm for learning generalizable robot policies. These policies are commonly trained via behavior cloning to imitate actions from pre-collected demonstrations. However, since robot actions are tied to the dynamics of the data collection robot, different robots may require different actions to achieve the same motion. This discrepancy hinders both policy training and deployment across diverse robots. To address this, we propose using Cartesian state delta as a universal action representation across robots, and introduce State Prediction and Adaptive Command Execution (SPACE) framework. SPACE handles robot dynamics variation at three levels: across different embodiments, across hardware units of the same embodiment, and within a single robot during operation. It consists of two components: (i) a Cartesian state delta policy that predicts geometric end-effector displacement, and (ii) Action Adapter, which converts the predicted Cartesian state delta into robot-specific control commands. Experiments show that SPACE substantially outperforms policies that directly predict control commands when learning from data collected across different embodiments and across hardware units of the same embodiment. SPACE also remains robust under dynamics shifts at deployment, including changes in control frequency, object weight, and controller gains. The project page is available at http://haeone.site/space-website/.

Haeone Lee, Byeongguk Jeon, Suchae Jeong +2

1KAIST · 2Config · 3Yonsei University

Robotic Control Robot Policy Adaptation Robot Skill Learning Generalization in Robotic Manipulation Cross-Embodiment Robot Learning Robot Imitation Learning+1

Ask PDF

Abstract

Robot Policy Adaptation

Explore similar work

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer

Data Analogies Enable Efficient Cross-Embodiment Transfer

SPACE: Enabling Learning from Cross-Robot Data Toward Generalist Policies