Distributed Dexterous Manipulation (DDM) is a novel paradigm that presents significant control challenges due to high action-space redundancy, inter-robot cooperation, and dynamic object-robot interactions. This paper introduces a framework based on spatially conditioned Multi-Agent Transformers (MATs) to efficiently learn robust control policies for a DDM system grounded in an array of 64 soft delta robots arranged in an 8x8 grid. Our three core contributions are: (i) an MAT with adaptive layer norm for compute efficiency, (ii) spatial contrastive embeddings to ground transformer embeddings in the spatial configuration of the robots, and (iii) an MAT-based behavior cloning method fine-tuned using Soft Actor Critic. We also propose an action selection formulation to analyze the trade-off between task performance and the number of robots utilized. Our experiments show that MATs iteratively refine their actions through the stacked attention blocks. This further informs the benefit of spatial conditioning in transformers to learn DDM policies. We demonstrate long-horizon planar manipulation tasks with objects of various geometries in simulation and real-world. Finally, we show how action selection mitigates robot maintenance by reducing wear and tear due to inter-robot collisions while maintaining the ability to manipulate objects along various trajectories in the real-world, achieving an average error of ~1.5 cm, while using ~65% fewer robots.
Figures & tables
Figure 1: We present Distributed Dexterous Manipulation as a novel paradigm for cooperative manipulation using arrays of soft delta robots. We propose a spatially grounded Multi-Agent Transformer to train sample-efficient policies using off-policy reinforcement learning and behavior cloning.
Figure 2: (a) Decoder-only MAT policy network. (b) Decoder-only MAT Q-value network. (c) Neighbourhood adjacency to select positive embeddings as primary neighbors and negative embeddings as farther away neighbors. (d) Adaptive Layer Norm Block. (e) Spatial Contrastive Embeddings using Triplet Margin Loss to create pairwise conformity and contrast in the embedding space.
Figure 3: RoPE fails to generate embeddings that capture similarities among neighboring robots as SCE embeddings are able to do. (left) Heatmap of the cosine similarity among all the robots’ embeddings. (right) Cosine similarity of an arbitrary robot (20) visualized on the delta array structure.
Figure 4: Overall performance for three rigid objects (hexagon, star, trapezium) on (a - c) Simulation, (d - f) Real World. For each, (Left) shows the percentage of active robots involved in each manipulation step. (Middle) shows the error at every execution step. (Right) shows the percentage success rate of reaching subgoals in 3 tries over the trajectories
Figure 5: Trajectory tracking results on our three hardest trajectories with three objects – hexagon, trapezium, star – in the real world.
Figure 6:
Figure 7: Inference Reward Plots a) Attention Mechanisms - AdaLN performs better than Cross Attention and Self Attention. AdaLN transformer fine-tuned using SAC performs better than training tabula-rasa. b) Position Embeddings - Sinusoidal and Rotary Embeddings fail to learn spatial correlations among neighboring sets of robots while SCEs demonstrably do so.
Figure 8:
Figure 9: Ground truth trajectories defined as the inference task for closed-loop DDM tasks.
Figure 10: Object level comparison between all the methods in the real world. The T-shaped object was unseen during training. (a) Shows percentage of active robots involved in each manipulation step. (b) Shows the error at every execution step. (c) Shows the number of attempts needed by an algorithm to complete a 20-step trajectory.
Figure 11: Object level comparison between all the methods. (a) shows percentage of active robots involved in each manipulation step. (b) shows the error at every execution step. (c) shows the number of attempts needed by an algorithm to complete a 20-step trajectory.
Figure 12: Comparison between different methods while tracking objects being manipulated along all target trajectories. For each subplot, we collect data over all 10 objects and run 5 trials in simulation.
Achieving human-level dexterity in robots via imitation learning from heterogeneous datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.
Xiaohan Lei, Min Wang, Bohong Weng +2
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Multi-arm manipulation demands precise spatiotemporal coordination, yet many centralized approaches scale poorly as team size increases. To address this, we propose CLS-DP, a decentralized multi-agent framework that enables implicit coordination under partial observability without shared global views, explicit state information, or inter-agent communication. Under the centralized training and decentralized execution (CTDE) paradigm, CLS-DP distills privileged multi-agent dynamics into a latent space. At deployment, each agent infers a collaborative latent from its local RGB observation and a shared task instruction; it then conditions the diffusion denoising process on this latent. This design enables implicit coordination with a per-agent cost independent of team size. Across six RoboFactory benchmark tasks spanning two to four agents, CLS-DP achieves a 38% mean success rate, outperforming the best centralized baseline (20%) and a decentralized ablation without the collaborative latent (9%). It also maintains superior parameter efficiency across all agent configurations. Attribution maps show that an agent conditioned on the collaborative latent places high attribution on the joints and grippers of both itself and its teammates throughout execution. This suggests that the learned latent efficiently encodes collaborative dynamics from local observation, which facilitates implicit coordination in realistic settings characterized by partial observability.
Chanyoung Park, Minsung Yoon, Andrew Jeong +1
School of Computing at the Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, Republic of Korea
Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an SE(3) wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.
Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5
Zhejiang University · Shanghai Innovation Institute · Fudan University +2