Direct Action-Manifold Modeling for Efficient and Robust Bimanual Manipulation
Authors: Hongren Wang, Mingyu You, Zihan Li, Zhipeng Wang, Shuo Jiang, Bin He
Organizations: Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai 201210, China · College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China
Diffusion and flow matching visuomotor policies require iterative denoising or numerical integration for multimodal action generation, introducing latency and execution mismatch. We instead consider a different formulation: conditioned on vision and proprioception, demonstrated bimanual action chunks concentrate in a structured observation-dependent region of joint-action space, making the action target more direct to model than the generative process. Based on this view, we propose the Direct Manifold Action Transformer (DMAT), which reformulates action generation as the direct modeling of clean, observation-conditioned joint-space action chunks from stochastic interpolants, thereby enabling single-step inference without relying on iterative generative inference or auxiliary trajectory-straightening procedures. Within DMAT, local 3D geometry anchors predictions to spatial structure, while State-Consistent Spectral-Time Refinement (STR) reconciles predicted chunks with proprioception and suppresses high-frequency residuals. Across 567 standard and out-of-distribution (OOD) trials per method, DMAT outperformed Improved 3D Diffusion Policy (IDP3) and DiTFlow3D in aggregate success rate, achieving relative improvements of 129.1% and 95.7%, respectively, while generating action chunks 2.06x and 1.34x faster and remaining robust under repeated 50 N interventions. These results pave the way for a novel generative paradigm for efficient and robust bimanual manipulation.
Figures & tables
Fig. 1: Conceptual comparison of action generation formulations. Representative multistep diffusion and flow matching policies evolve a Gaussian-initialized action state Zinit through intermediate states Zτi . DMAT directly predicts the clean observation-conditioned chunk At in one evaluation. The lower region illustrates hypothesized observation-conditioned structure in demonstrated actions.
Fig. 2: DMAT system and pipeline. (a) System overview integrating direct action-manifold prediction, STR, and receding-horizon execution. (b) Detailed pipeline. Stage 1 encodes recent pointclouds into spatial tokens; Stage 2 directly predicts the clean observation-conditioned action chunk At from a Gaussian-initialized action state and spatial-proprioceptive conditions; Stage 3 combines At , the measured joint state, and STR memory to preserve low-frequency intent while attenuating high-frequency residuals; and Stage 4 executes AtSTR in a receding horizon and returns updated observations.
Fig. 3: Representative successful rollouts under standard conditions: (a) Pick Place, (b) Vase Wipe, and (c) Pour Beans. Rows show time-ordered frames with elapsed time.
Fig. 4: Standard condition comparison. (a) Task success on Pick Place, Vase Wipe, Pour Beans, and the aggregate, with 36 trials per task; arrow labels indicate the absolute differences in success rate between DMAT (w/ STR) and the comparison methods. (b) Equal-weight average action chunk generation latency; labels indicate latency and inference steps, and arrows indicate DMAT speedups. (c) Vase Wipe joint-speed SPARC (mean ± standard deviation).
Method
Fluctuation (deg) ↓
Redundancy ↓
IDP3
0.1662±0.0119
0.4134±0.0095
DiTFlow3D
0.2662±0.0167
0.5716±0.0366
DMAT (Ours)
0.0383±0.0019
0.2053±0.0147
TABLE I: Action Command Quality Under Standard Conditions
Fig. 5: OOD evaluation under object-level and scene-level shifts. Panels (a), (b), and (c) show seen and unseen objects for Pick Place, Vase Wipe, and Pour Beans, respectively; panels (d), (e), and (f) show illumination variation and two cluttered scenes, with no retraining or fine-tuning.
Fig. 6: Representative successful DMAT rollouts under OOD conditions without retraining or fine-tuning. Rows (a)–(c) show unseen-object generalization for Pick Place, Vase Wipe, and Pour Beans, respectively, while rows (d)–(f) show scene-level shifts for the same tasks. Each row presents six time-ordered frames from a single successful rollout.
Fig. 7: Task success under visual and geometric OOD shifts. Panels (a)–(c) show illumination, clutter, unseen object, and aggregate results for Pick Place, Vase Wipe, and Pour Beans, respectively. No policy is retrained or fine-tuned for these OOD evaluations. Arrow labels indicate the absolute differences in success rate between DMAT (w/ STR) and the comparison methods.
Fig. 8: Representative responses to 50 N lateral-pull interventions. (a) DMAT completes Vase Wipe after 30 interventions. Under the three-intervention condition, (b) IDP3 deviates abruptly and collides, whereas (c) DiTFlow3D oscillates repeatedly and fails. Motion overlays and local views highlight the responses; white arrows and labels denote pulls reaching the 50 N threshold.
Fig. 9: Post-intervention task success and motion smoothness under 50 N lateral-pull interventions. (a) and (c) show Vase Wipe success under 3 and 30 interventions, respectively; arrow labels indicate the absolute differences in success rate between DMAT (w/ STR) and the comparison methods. (b) and (d) show joint-speed SPARC (mean ± standard deviation).
Method
Fluctuation (deg) ↓
Redundancy ↓
Three interventions
IDP3
0.176±0.018
0.436±0.016
DiTFlow3D
0.320±0.042
0.595±0.060
DMAT (w/o STR)
0.103±0.005
0.301±0.015
DMAT (w/ STR)
0.038±0.002
0.217±0.007
30 interventions
TABLE II: Action Command Quality Under 50 N Interventions
Variant
Pick Place
Vase Wipe
Pour Beans
Avg. latency
Mean SPARC on Vase Wipe ↑
Overall
DMAT-Direct
22/36
12/36
14/36
8.69 ms
−21.46±2.83
48/108 (44.4%)
DMAT-Geo
26/36
21/36
22/36
22.95 ms
−20.45±2.63
69/108 (63.9%)
DMAT (Full)
27/36
26/36
23/36
24.20 ms
−15.45±0.70
76/108 (70.4%)
TABLE III: Ablation of DMAT Components
Variant
Fluctuation (deg) ↓
Redundancy ↓
DMAT-Direct
0.109±0.011
0.359±0.059
DMAT-Geo
0.097±0.004
0.291±0.010
DMAT (Full)
0.038±0.002
0.205±0.015
TABLE IV: Ablation Results for Action Command Quality
Fig. 10: Representative baseline failure cases under standard and OOD conditions. (a) Pick Place misses the grasp. (b) The wiping tool drops during transport. (c) Visible ink remains after wiping. (d) Beans spill during pouring.
School of Safety Science and Engineering, Anhui University of Science and Technology, Huainan 232001, China · State Key Laboratory of Digital Intelligent Technology for Unmanned Coal Mining, Anhui University of Science and Technology, Huainan 232001, China · School of Artificial Intelligence, Anhui University of Science and Technology, Hefei 231131, China