Direct Action-Manifold Modeling for Efficient and Robust Bimanual Manipulation
Authors: Hongren Wang, Mingyu You, Zihan Li, Zhipeng Wang, Shuo Jiang, Bin He
Organizations: Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai 201210, China · College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China
Diffusion and flow matching visuomotor policies require iterative denoising or numerical integration for multimodal action generation, introducing latency and execution mismatch. We instead consider a different formulation: conditioned on vision and proprioception, demonstrated bimanual action chunks concentrate in a structured observation-dependent region of joint-action space, making the action target more direct to model than the generative process. Based on this view, we propose the Direct Manifold Action Transformer (DMAT), which reformulates action generation as the direct modeling of clean, observation-conditioned joint-space action chunks from stochastic interpolants, thereby enabling single-step inference without relying on iterative generative inference or auxiliary trajectory-straightening procedures. Within DMAT, local 3D geometry anchors predictions to spatial structure, while State-Consistent Spectral-Time Refinement (STR) reconciles predicted chunks with proprioception and suppresses high-frequency residuals. Across 567 standard and out-of-distribution (OOD) trials per method, DMAT outperformed Improved 3D Diffusion Policy (IDP3) and DiTFlow3D in aggregate success rate, achieving relative improvements of 129.1% and 95.7%, respectively, while generating action chunks 2.06x and 1.34x faster and remaining robust under repeated 50 N interventions. These results pave the way for a novel generative paradigm for efficient and robust bimanual manipulation.
Figures & tables
Fig. 1: Conceptual comparison of action generation formulations. Representative multistep diffusion and flow matching policies evolve a Gaussian-initialized action state Zinit through intermediate states Zτi . DMAT directly predicts the clean observation-conditioned chunk At in one evaluation. The lower region illustrates hypothesized observation-conditioned structure in demonstrated actions.
Fig. 2: DMAT system and pipeline. (a) System overview integrating direct action-manifold prediction, STR, and receding-horizon execution. (b) Detailed pipeline. Stage 1 encodes recent pointclouds into spatial tokens; Stage 2 directly predicts the clean observation-conditioned action chunk At from a Gaussian-initialized action state and spatial-proprioceptive conditions; Stage 3 combines At , the measured joint state, and STR memory to preserve low-frequency intent while attenuating high-frequency residuals; and Stage 4 executes AtSTR in a receding horizon and returns updated observations.
Fig. 3: Representative successful rollouts under standard conditions: (a) Pick Place, (b) Vase Wipe, and (c) Pour Beans. Rows show time-ordered frames with elapsed time.
Fig. 4: Standard condition comparison. (a) Task success on Pick Place, Vase Wipe, Pour Beans, and the aggregate, with 36 trials per task; arrow labels indicate the absolute differences in success rate between DMAT (w/ STR) and the comparison methods. (b) Equal-weight average action chunk generation latency; labels indicate latency and inference steps, and arrows indicate DMAT speedups. (c) Vase Wipe joint-speed SPARC (mean ± standard deviation).
Method
Fluctuation (deg) ↓
Redundancy ↓
IDP3
0.1662±0.0119
0.4134±0.0095
DiTFlow3D
0.2662±0.0167
0.5716±0.0366
DMAT (Ours)
0.0383±0.0019
0.2053±0.0147
TABLE I: Action Command Quality Under Standard Conditions
Fig. 5: OOD evaluation under object-level and scene-level shifts. Panels (a), (b), and (c) show seen and unseen objects for Pick Place, Vase Wipe, and Pour Beans, respectively; panels (d), (e), and (f) show illumination variation and two cluttered scenes, with no retraining or fine-tuning.
Fig. 6: Representative successful DMAT rollouts under OOD conditions without retraining or fine-tuning. Rows (a)–(c) show unseen-object generalization for Pick Place, Vase Wipe, and Pour Beans, respectively, while rows (d)–(f) show scene-level shifts for the same tasks. Each row presents six time-ordered frames from a single successful rollout.
Fig. 7: Task success under visual and geometric OOD shifts. Panels (a)–(c) show illumination, clutter, unseen object, and aggregate results for Pick Place, Vase Wipe, and Pour Beans, respectively. No policy is retrained or fine-tuned for these OOD evaluations. Arrow labels indicate the absolute differences in success rate between DMAT (w/ STR) and the comparison methods.
Fig. 8: Representative responses to 50 N lateral-pull interventions. (a) DMAT completes Vase Wipe after 30 interventions. Under the three-intervention condition, (b) IDP3 deviates abruptly and collides, whereas (c) DiTFlow3D oscillates repeatedly and fails. Motion overlays and local views highlight the responses; white arrows and labels denote pulls reaching the 50 N threshold.
Fig. 9: Post-intervention task success and motion smoothness under 50 N lateral-pull interventions. (a) and (c) show Vase Wipe success under 3 and 30 interventions, respectively; arrow labels indicate the absolute differences in success rate between DMAT (w/ STR) and the comparison methods. (b) and (d) show joint-speed SPARC (mean ± standard deviation).
Method
Fluctuation (deg) ↓
Redundancy ↓
Three interventions
IDP3
0.176±0.018
0.436±0.016
DiTFlow3D
0.320±0.042
0.595±0.060
DMAT (w/o STR)
0.103±0.005
0.301±0.015
DMAT (w/ STR)
0.038±0.002
0.217±0.007
30 interventions
TABLE II: Action Command Quality Under 50 N Interventions
Variant
Pick Place
Vase Wipe
Pour Beans
Avg. latency
Mean SPARC on Vase Wipe ↑
Overall
DMAT-Direct
22/36
12/36
14/36
8.69 ms
−21.46±2.83
48/108 (44.4%)
DMAT-Geo
26/36
21/36
22/36
22.95 ms
−20.45±2.63
69/108 (63.9%)
DMAT (Full)
27/36
26/36
23/36
24.20 ms
−15.45±0.70
76/108 (70.4%)
TABLE III: Ablation of DMAT Components
Variant
Fluctuation (deg) ↓
Redundancy ↓
DMAT-Direct
0.109±0.011
0.359±0.059
DMAT-Geo
0.097±0.004
0.291±0.010
DMAT (Full)
0.038±0.002
0.205±0.015
TABLE IV: Ablation Results for Action Command Quality
Fig. 10: Representative baseline failure cases under standard and OOD conditions. (a) Pick Place misses the grasp. (b) The wiping tool drops during transport. (c) Visible ink remains after wiping. (d) Beans spill during pouring.
Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm. Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning. We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks. By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising. We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising. We evaluate JAMB on diverse bimanual manipulation tasks in RoboTwin 2.0 and on a real-world robot, comparing it with action-only policies and alternative future-prediction approaches spanning different state representations and learning objectives. Across 16 simulation tasks, JAMB achieves an average success rate of 83.4%, outperforming the strongest baseline by 23.9 percentage points. On three real-world tasks, it outperforms the action-only and auxiliary geometry prediction methods by 50.0 and 21.2 percentage points, respectively. Beyond these performance gains, JAMB shows stronger generalization to cluttered scenes and out-of-distribution backgrounds than the evaluated baselines. Together, these results demonstrate the effectiveness of our joint action-motion modeling framework for coordinated bimanual manipulation. Our project website is available at https://jam-bimanual.github.io/
Chuyang Xiao, Peilin Meng, David Held
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. · University of Michigan, Ann Arbor, MI 48109, USA.
Robotic manipulation is inherently multi-frame: local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame. However, modern diffusion-based visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame. We propose Mixture of Frames Policy (MoF), a diffusion policy that performs synchronized action denoising across multiple coordinate frames. MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame. To make this possible for intermediate noisy diffusion states, we adopt a column-based 6D rotation representation within an SE(3) action parameterization that supports exact, differentiable frame transformations without requiring noisy rotations to lie on the SO(3) manifold. Across nine simulated bimanual manipulation tasks, we show that the best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines. We further evaluate MoF on two real-world bimanual mobile manipulation tasks, demonstrating that it outperforms all constituent single-frame baselines.
Generative models such as diffusion and flow matching have advanced robotic visuomotor policies by modeling multimodal action distributions, but their multi-step sampling or ODE solving introduces inference latency. Existing one-step acceleration methods often compress the whole generation process into a single large update, leading to spatial deviation, frequency distortion, and mode averaging. This paper proposes a high-fidelity one-step generative visuomotor policy framework that addresses these issues with three complementary mechanisms. Recursive Consistent Action Flow (RCAF) uses recursive correction to compensate for spatial truncation errors and align one-step predictions with refined flow trajectories. Dual-Timestep Frequency Consistency (DTFC) preserves high-frequency manipulation details through adaptive spectral consistency across flow timesteps. Contrastive Flow Matching (CFM) separates entangled action flows with a margin-based repulsive objective, reducing ambiguous actions in multimodal manipulation. Experiments on RoboTwin, RoboTwin 2.0, Adroit, DexArt, and real-world robot platforms show that the proposed method achieves competitive or superior performance compared with strong 10-step generative policy baselines while requiring only one forward pass (1 NFE), enabling low-latency visuomotor control.
Yuran Chen, Xinye Cai, Zhonglin Gong +1
School of Safety Science and Engineering, Anhui University of Science and Technology, Huainan 232001, China · State Key Laboratory of Digital Intelligent Technology for Unmanned Coal Mining, Anhui University of Science and Technology, Huainan 232001, China · School of Artificial Intelligence, Anhui University of Science and Technology, Hefei 231131, China