Distributed Dexterous Manipulation with Spatially Conditioned Multi-Agent Transformers

Sarvesh Patil

cs.ROSep 7, 2026

Distributed Dexterous Manipulation with Spatially Conditioned Multi-Agent Transformers

Authors: Sarvesh Patil

Organizations: Carnegie Mellon University, The Robotics Institute Pittsburgh, PA, United States

Abstract

Distributed Dexterous Manipulation (DDM) is a novel paradigm that presents significant control challenges due to high action-space redundancy, inter-robot cooperation, and dynamic object-robot interactions. This paper introduces a framework based on spatially conditioned Multi-Agent Transformers (MATs) to efficiently learn robust control policies for a DDM system grounded in an array of 64 soft delta robots arranged in an 8x8 grid. Our three core contributions are: (i) an MAT with adaptive layer norm for compute efficiency, (ii) spatial contrastive embeddings to ground transformer embeddings in the spatial configuration of the robots, and (iii) an MAT-based behavior cloning method fine-tuned using Soft Actor Critic. We also propose an action selection formulation to analyze the trade-off between task performance and the number of robots utilized. Our experiments show that MATs iteratively refine their actions through the stacked attention blocks. This further informs the benefit of spatial conditioning in transformers to learn DDM policies. We demonstrate long-horizon planar manipulation tasks with objects of various geometries in simulation and real-world. Finally, we show how action selection mitigates robot maintenance by reducing wear and tear due to inter-robot collisions while maintaining the ability to manipulate objects along various trajectories in the real-world, achieving an average error of ~1.5 cm, while using ~65% fewer robots.

Figures & tables

Figure 1: We present Distributed Dexterous Manipulation as a novel paradigm for cooperative manipulation using arrays of soft delta robots. We propose a spatially grounded Multi-Agent Transformer to train sample-efficient policies using off-policy reinforcement learning and behavior cloning.

Figure 2: (a) Decoder-only MAT policy network. (b) Decoder-only MAT Q-value network. (c) Neighbourhood adjacency to select positive embeddings as primary neighbors and negative embeddings as farther away neighbors. (d) Adaptive Layer Norm Block. (e) Spatial Contrastive Embeddings using Triplet Margin Loss to create pairwise conformity and contrast in the embedding space.

Figure 3: RoPE fails to generate embeddings that capture similarities among neighboring robots as SCE embeddings are able to do. (left) Heatmap of the cosine similarity among all the robots’ embeddings. (right) Cosine similarity of an arbitrary robot (20) visualized on the delta array structure.

Figure 4: Overall performance for three rigid objects (hexagon, star, trapezium) on (a - c) Simulation, (d - f) Real World. For each, (Left) shows the percentage of active robots involved in each manipulation step. (Middle) shows the error at every execution step. (Right) shows the percentage success rate of reaching subgoals in 3 tries over the trajectories

Figure 5: Trajectory tracking results on our three hardest trajectories with three objects – hexagon, trapezium, star – in the real world.

Panel of Figure 7 — Figure 7: Inference Reward Plots a) Attention Mechanisms - AdaLN performs better than Cross Attention and Self Attention. AdaLN transformer fine-tuned using SAC performs better than training tabula-rasa. b) Position Embeddings - Sinusoidal and Rotary Embeddings fail to learn spatial correlations among neighboring sets of robots while SCEs demonstrably do so.

Figure 9: Ground truth trajectories defined as the inference task for closed-loop DDM tasks.

Figure 10: Object level comparison between all the methods in the real world. The T-shaped object was unseen during training. (a) Shows percentage of active robots involved in each manipulation step. (b) Shows the error at every execution step. (c) Shows the number of attempts needed by an algorithm to complete a 20-step trajectory.

Panel of Figure 11 — Figure 11: Object level comparison between all the methods. (a) shows percentage of active robots involved in each manipulation step. (b) shows the error at every execution step. (c) shows the number of attempts needed by an algorithm to complete a 20-step trajectory.

Figure 12: Comparison between different methods while tracking objects being manipulated along all target trajectories. For each subplot, we collect data over all 10 objects and run 5 trials in simulation.

Explore similar work

Structural Action Transformer for 3D Dexterous Manipulation
Mar 4, 2026Xiaohan Lei, Min Wang, Bohong Weng +2Robotic Manipulation Robot Imitation Learning
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Distilling Collaborative Dynamics into Latent Space for Implicit Coordination in Decentralized Multi-Agent Manipulation
Jun 22, 2026Chanyoung Park, Minsung Yoon, Andrew Jeong +1Multi-Agent Coordination Multi-Agent Systems
School of Computing at the Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, Republic of Korea
AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning
Aug 14, 2026Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5Robotic Manipulation Learning from Demonstration
Zhejiang University · Shanghai Innovation Institute · Fudan University +2

Mar 4, 2026J/K move · Enter open · S save

Structural Action Transformer for 3D Dexterous Manipulation

Xiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou+1

MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

Robotic Manipulation Robot Imitation Learning Cross-Embodiment Robot Learning Trajectory Representation Learning

Open paper AskPDF arXiv

Achieving human-level dexterity in robots via imitation learning from heterogeneous datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.

Abstract

Explore similar work

Structural Action Transformer for 3D Dexterous Manipulation

Distilling Collaborative Dynamics into Latent Space for Implicit Coordination in Decentralized Multi-Agent Manipulation

AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning