Towards Dexterous Embodied Manipulation via Deep Multi-Sensory Fusion and Sparse Expert Scaling

Yirui Sun; Guangyu Zhuge; Keliang Liu; Jie Gu; Zhihao xia; Qionglin Ren; Chunxu tian; Zhongxue Ga

cs.ROFeb 23, 2026

Towards Dexterous Embodied Manipulation via Deep Multi-Sensory Fusion and Sparse Expert Scaling

Authors: Yirui Sun, Guangyu Zhuge, Keliang Liu, Jie Gu, Zhihao xia, Qionglin Ren, Chunxu tian, Zhongxue Ga

Organizations: College of Intelligent Robotics and Advanced Manufacturing, Fudan University · School of Information Science and Engineering, Harbin Institute of Technology

Abstract

Realizing dexterous embodied manipulation necessitates the deep integration of heterogeneous multimodal sensory inputs. However, current vision-centric paradigms often overlook the critical force and geometric feedback essential for complex tasks. This paper presents DeMUSE, a Deep Multimodal Unified Sparse Experts framework leveraging a Diffusion Transformer to integrate RGB, depth, and 6-axis force into a unified serialized stream. Adaptive Modality-specific Normalization (AdaMN) is employed to recalibrate modality-aware features, mitigating representation imbalance and harmonizing the heterogeneous distributions of multi-sensory signals. To facilitate efficient scaling, the architecture utilizes a Sparse Mixture-of-Experts (MoE) with shared experts, increasing model capacity for physical priors while maintaining the low inference latency required for real-time control. A Joint denoising objective synchronously synthesizes environmental evolution and action sequences to ensure physical consistency. Achieving success rates of 83.2% and 72.5% in simulation and real-world trials, DeMUSE demonstrates state-of-the-art performance, validating the necessity of deep multi-sensory integration for complex physical interactions.

Figures & tables

Figure 1: The top image shows the layout of the experimental platform and the sources of multimodal inputs. The bottom image shows the tasks involved in real robot operation.

Figure 2: The DeMUSE architecture for multi-modal embodied synthesis. Heterogeneous tokens are processed through a sequence of Transformer blocks. Each block employs AdaMN to ensure representational balance across modalities and a Sparse MoE layer to achieve high-capacity inference under real-time constraints. The framework synchronously generates future visual evolutions and continuous actions via a unified joint denoising stream.

Task Name	Core Physical Challenge	Language Instruction ( $y$ )
Tool Drawer Org.	Long-duration manipulation	Pick up the tool from the table, place it in the open drawer, and close all drawers.
Fill Cup (1/3 Cola)	Precise control	Grab the cup from the table, then fill it with one-third of cola from the dispenser.
Dispense Sanitizer	Pressure sensitivity	Dispense a suitable amount of hand sanitizer onto one hand from the press bottle.
Sweeping to Dustpan	Visual occlusion	Use the broom to sweep the scattered items from the table into the dustpan.

Table 1: Real-World Task Suite and Semantic Instructions. We design four representative tasks in unstructured environments to evaluate embodied manipulation. These tasks present significant physical and coordination challenges, involving intricate constraints such as multi-stage contact and high-precision spatial alignment across diverse linguistic commands.

Method	Success Rate by Difficulty ( $\%$ )			Representative Tasks (SR $\%$ )				Avg. SR (%)
Method	Simple	Medium	Hard	Hammer	Basket	Dial	Plate-slide	Avg. SR (%)
DP	45.5	27.4	15.3	8.0	10.0	52.0	60.0	29.2
RT-2	81.3	50.4	25.6	20.0	12.0	52.0	88.0	52.2
PAD	92.3	65.8	61.3	82.0	84.0	56.0	72.0	72.4
ForceVLA	95.6	71.4	62.1	88.0	85.0	82.0	94.0	75.9
RDT-1B	94.2	76.7	63.3	86.0	89.0	78.0	92.0	77.9

Table 2: Main Results on MetaWorld MT50. We report the success rate (

\%

) averaged over 50 evaluation trials per task. The representative tasks highlight DeMUSE’s precision in diverse scenarios.

Figure 3: Qualitative results on MetaWorld. Left: Basketball task; Right: Hammer task. Rows from top to bottom represent input Observations, generated Predictions, and actual Actions. High consistency across rows demonstrates the model’s ability to capture physically-consistent latent dynamics through joint denoising.

Task Name	DP	RDT-1B	Ours
Sweeping	54.0	62.0	76.0
Dispense Sanitizer	42.0	58.0	74.0
Tool Drawer Org.	0.0	64.0	72.0
Fill Cup	10.0	52.0	68.0
Overall Average	26.5	59.0	72.5

Table 3: Real-World Robot Success Rates (%). Comparison of DeMUSE against state-of-the-art baselines across four representative physical tasks. Results are averaged over 50 independent trials per task with randomized initializations.

Figure 4: Z-axis force profile. Comparison of contact dynamics during the pressing task. DeMUSE achieves stable active compliance (

\sim

10N) within 80ms of contact (

t=80

ms)

Figure 5: Terminal liquid level precision. Final liquid level distributions in the ”Fill Cup” task. DeMUSE achieves high-precision ( $0.33\pm 0.02$ ) with minimal variance.

Configuration	Sim. (MetaWorld)			Real-World
Configuration	Simple	Medium	Hard	(Unstructured)
RGB-only	93.6	77.3	60.8	45.5
RGB-D	93.5	77.6	61.2	61.4
RGB-F	94.7	81.8	68.2	58.0
Conditioned	93.8	77.5	61.1	–
Ours (Full)	95.7	83.8	69.8	72.5

Table 4: Ablation of Modality Configurations and Fusion Strategies (Success Rate %). Full fusion (Ours) significantly outperforms both modality-restricted baselines and the conditional modulation strategy (DeMUSE-Conditioned), proving that the integration architecture is as critical as the sensory data itself.

Method	LPIPS ( $\downarrow$ )	Success Rate ( $\uparrow$ )
Standard AdaLN	0.41	78.6%
AdaMN (Ours)	0.35	83.2%

Table 5: Normalization Ablation. AdaMN ensures both higher world-modeling fidelity and execution precision.

Figure 6: Scaling Analysis on MetaWorld MT50. Comparison of model capacity (parameters), inference compute (GFLOPs), and average success rate across model variants.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Routing Strategy	Avg. Success Rate (%)	Inference Cost (GFLOPs)
Top-1 (DeMUSE)	83.2%	135
Top-2	82.5%	174

Appendix

Table 6: Performance Comparison of Top-1 vs. Top-2 Routing Strategies (on MetaWorld MT50)

Architecture	Value	Optimization	Value
Transformer Layers ( $L$ )	28	Optimizer	AdamW
Hidden Dimension ( $D$ )	1152	Learning Rate	$1\times 10^{-4}$
Attention Heads	16	LR Scheduler	Cosine
MLP Expansion Ratio	4.0	Warmup Steps	8,000
Total Experts ( $N$ )	4	Total Fine-tuning Steps	100,000
Shared Experts	1	Global Batch Size	64

Appendix

Table 7: Hyperparameters for DeMUSE Fine-tuning Phase

Figure 7: Qualitative execution sequences across real-world dexterous tasks. The rows from top to bottom correspond to: Tool Drawer Org. , Sweeping to Dustpan , Dispense Sanitizer , and Fill Cup (1/3 Cola) .

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
Feb 5, 2026Xukun Li, Yu Sun, Lei Zhang +10Bimanual Robotic Manipulation Contact-Rich Robotic Manipulation
XYZ Embodied AI, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · School of Computation, Information and Technology, Technical University of Munich, Garching, Germany +3
AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning
Aug 14, 2026Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5Robotic Manipulation Learning from Demonstration
Zhejiang University · Shanghai Innovation Institute · Fudan University +2
VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation
Aug 21, 2026Congsheng Xu, Qiaochu Yang, Fangyuan Shi +9Visuo-Tactile Representation Learning Visuomotor Policy Learning
Shanghai Jiao Tong University · Xense Robotics

Feb 5, 2026J/K move · Enter open · S save

DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter

Xukun Li, Yu Sun, Lei Zhang, Bosheng Huang+9

XYZ Embodied AI, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · School of Computation, Information and Technology, Technical University of Munich, Garching, Germany+3

Bimanual Robotic Manipulation Contact-Rich Robotic Manipulation Tactile Sensing for Robotic Manipulation Multimodal Diffusion Models

Open paper AskPDF arXiv

Bimanual dexterous manipulation relies on integrating multimodal inputs to perform complex real-world tasks. To address the challenges of effectively combining these modalities, we propose DECO, a decoupled multimodal diffusion transformer that disentangles vision, proprioception, and tactile signals through specialized conditioning pathways, enabling structured and controllable integration of multimodal inputs, with a lightweight adapter for parameter-efficient injection of additional signals. Alongside DECO, we release DECO-50 dataset for bimanual dexterous manipulation with tactile sensing, consisting of 50 hours of data and over 5M frames, collected via teleoperation on real dual-arm robots. We train DECO on DECO-50 and conduct extensive real-world evaluation with over 2,000 robot rollouts. Experimental results show that DECO achieves the best performance across all tasks, with a 72.25% average success rate and a 21% improvement over the baseline. Moreover, the tactile adapter brings an additional 10.25% average success rate across all tasks and a 20% gain on complex contact-rich tasks while tuning less than 10% of the model parameters.

Abstract

Robotic Manipulation

Explore similar work

DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter

AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning

VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation