Organizations: School of Computation, Information and Technology, Technical University of Munich, Garching, Germany. · State Key Laboratory for Novel Software Technology and the School of Science and Technology, Nanjing University (Suzhou Campus), China.
Learning a collective policy for state-based Cartesian manipulation across diverse robotic embodiments presents a fundamental challenge due to embodiment-dependent transition dynamics. Collective learning aggregates multi-robot experience to discover shared task structure, yet naive aggregation can average incompatible action modes when different embodiments realize the same Cartesian command through different transition behavior. This leads to strategy ambiguity and performance collapse on dynamically outlying embodiments. To address this, we propose PACE, a two-stage collective learning framework that separates shared task-space intent from embodiment-dependent transition dynamics. The first stage employs a representation layer pre-trained on offline trajectories to extract task-level context from collective data. The second stage introduces a decision layer equipped with a predictive dynamics adapter. By forcing the latent state to predict future transitions, this adapter acts as a latent regularizer that grounds the policy in the transition behavior of the executing robot while preserving real-time inference without online planning. Evaluations across simulated embodiments demonstrate that our method prevents performance collapse on distinct robots. It achieves over 93% overall success and 89% on dynamically out-of-distribution zero-shot simulated robots, outperforming the Transformer behavior cloning (82%) and diffusion policy baselines (66%). Physical KUKA deployment further validates the feasibility of the full state-based pipeline under zero-shot sim-to-real transfer.
Figures & tables
Fig. 1 : Overview of the PACE framework. The left panel shows the diverse multi-embodiment dataset. The central panel details the collective network, which consists of in-context trajectory encoding and dynamics-aware policy learning. The right panel demonstrates the cross-embodiment generalization to unseen simulated and real-world robots.
Fig. 2 : Structure of PACE. The two-stage distillation framework features 1. an offline representation layer with a trajectory transformer encoder, and 2. an online decision layer with a predictive dynamics adapter and collective agent policy.
Fig. 3 : Cross-embodiment zero-shot transfer across diverse manipulation tasks. Physical specifications are in Table I . For more details, refer to our supplementary video.
Robot Model
Mass(kg)
Payload(kg)
Reach(mm)
Seen Embodiments (Training & Testing)
Unitree Z1
4.5
3.0
740
Kinova Gen3
8.2
4.0
902
UFactory xArm7
13.7
3.5
700
Rethink Sawyer
19.0
4.0
1260
UR10e
33.5
12.5
1300
TABLE I: Physical specifications of evaluated robotic manipulators.
Fig. 4 : Performance Analysis. (a) Per-robot success rate. Our method prevents performance collapse on outliers. (b) Success rate versus the number of seen robots ( N ).
Method
Seen (%)
Unseen (%)
Transformer BC
82.66 ± 5.16
81.75 ± 5.99
Diffusion Policy (DP)
66.40 ± 3.85
65.25 ± 4.12
PACE (Ours)
93.17 ± 2.41
93.21 ± 2.81
TABLE II: Generalization Performance. Success rates (%) on seen ( N=5 ) and unseen ( N=4 ) robots.
Fig. 5 : Zero-Shot vs. In-Domain generalization on UR10e across varying training set sizes.
Fig. 6 : t-SNE cluster Visualization. t-SNE projection of context embeddings colored by task ID.
TABLE III : Stage 1 Component Analysis. Success rates (%) with a frozen Stage 2 dynamics adapter.
Fig. 8 : Qualitative demonstration of zero-shot sim-to-real transfer on the unseen KUKA iiwa robot. For more details, refer to our supplementary video.
Task Name
Successes / 20 Eps
Success Rate (%)
Reach
18 / 20
90.0
Push
17 / 20
85.0
Pick-Place
17 / 20
85.0
Button Press
16 / 20
80.0
Drawer Open
14 / 20
70.0
Window Open
13 / 20
65.0
TABLE IV : Success rates and episode counts across manipulation tasks.
In robot learning, scaling training datasets across diverse embodiments and environments has become a dominant paradigm for learning generalizable robot policies. These policies are commonly trained via behavior cloning to imitate actions from pre-collected demonstrations. However, since robot actions are tied to the dynamics of the data collection robot, different robots may require different actions to achieve the same motion. This discrepancy hinders both policy training and deployment across diverse robots. To address this, we propose using Cartesian state delta as a universal action representation across robots, and introduce State Prediction and Adaptive Command Execution (SPACE) framework. SPACE handles robot dynamics variation at three levels: across different embodiments, across hardware units of the same embodiment, and within a single robot during operation. It consists of two components: (i) a Cartesian state delta policy that predicts geometric end-effector displacement, and (ii) Action Adapter, which converts the predicted Cartesian state delta into robot-specific control commands. Experiments show that SPACE substantially outperforms policies that directly predict control commands when learning from data collected across different embodiments and across hardware units of the same embodiment. SPACE also remains robust under dynamics shifts at deployment, including changes in control frequency, object weight, and controller gains. The project page is available at http://haeone.site/space-website/.
Effective robotic manipulation requires policies that can anticipate physical outcomes and adapt to real-world environments. Effective robotic manipulation requires policies that can anticipate physical outcomes and adapt to real-world environments. In this work, we introduce a unified framework, World-Model-Driven Diffusion Policy with Online Adaptive Learning (AdaWorldPolicy) to enhance robotic manipulation under dynamic conditions with minimal human involvement. Our core insight is that world models provide strong supervision signals, enabling online adaptive learning in dynamic environments, which can be complemented by force-torque feedback to mitigate dynamic force shifts. Our AdaWorldPolicy integrates a world model, an action expert, and a force predictor-all implemented as interconnected Flow Matching Diffusion Transformers (DiT). They are interconnected via the multi-modal self-attention layers, enabling deep feature exchange for joint learning while preserving their distinct modularity characteristics. We further propose a novel Online Adaptive Learning (AdaOL) strategy that dynamically switches between an Action Generation mode and a Future Imagination mode to drive reactive updates across all three modules. This creates a powerful closed-loop mechanism that adapts to both visual and physical domain shifts with minimal overhead. Across a suite of simulated and real-robot benchmarks, our AdaWorldPolicy achieves state-of-the-art performance, with dynamical adaptive capacity to out-of-distribution scenarios.
Collaborative manipulation requires robots to perform complementary actions as interactions unfold. We study single-policy decentralized collaboration: every robot runs the same policy from its visual observations and proprioception, without task prompts, identity labels, or inter-robot messages. The challenge is to learn complementary team behaviors within shared parameters and select appropriate actions from each robot's local observations. We introduce CoRE, which learns Collaboration-Role Experts from pooled multi-task, multi-robot demonstrations. Fused appearance and geometry provide local interaction evidence. Query-conditioned cross-attention experts provide adaptable prediction paths, which a local router combines at each action-chunk position. During training, an action-expert alignment loss supervises expert selection using relative forced-route prediction errors against demonstrations under fixed inputs, without role labels. Across simulation benchmarks, CoRE achieves the highest average performance among evaluated decentralized methods. Physical experiments demonstrate effective collaboration across diverse manipulation tasks and robustness to partner delays and slowdowns. Project page: https://aus.bot/research/core/.
Yanan Zhou, Zhaoyan Qian, Zihao Li +3
School of Computer Science, The University of Sydney, Australia. · College of Computer Science and Technology, Zhejiang University of Technology, China.