Organizations: Mingcong Li and Jiayu Qu are with the School of Automation, Beijing Institute of Technology, Beijing 100081, China. · LingYu Robotics, Beijing 100086, China. · Yihan Zhou and Xueyang Guo are with the Department of Automation, Tsinghua University, Beijing 100084, China, and also with the Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China. · Yilin Mo is with the Department of Automation, Tsinghua University, Beijing 100084, China, and also with the Beijing Key Laboratory of Embodied Intelligence Systems, Beijing, China, and also with the Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China.
Diffusion-based policies have achieved strong performance on single-task robotic manipulation. However, a single model trained jointly on multiple tasks consistently underperforms the per-task specialists trained on the same demonstrations. We focus on two challenges. First, the convolutional encoders commonly used in diffusion-based policies produce visual conditions that are not discriminative enough. Second, gradients from incompatible action modes update the shared denoiser's dense parameters along the same directions, causing destructive interference during joint training. To address both, we propose SR-DP (Spectrally Routed Diffusion Policy). A frozen VGGT encoder with a spatial-compression adapter produces compact multi-scale semantic-geometric tokens, a task-discriminative visual condition for tasks that share a workspace. We first train the denoiser as a standard diffusion policy, then decompose selected linear layers by SVD into a frozen basis of orthogonal rank-one experts, adapted through spectral routing and Householder input rotations. On six RoboTwin tasks, SR-DP averages a 68.4% success rate, surpassing multi-task VO-DP by 21.8 points and the strongest per-task baseline by 3.8 points. On four real-world bimanual tasks, Stage 2 reaches a 43.0% average success rate and improves over Stage 1 by 19.0 points.
Figures & tables
Fig. 1: Overview of SR-DP. A frozen VGGT encoder with a spatial-compression adapter produces 14 tokens per view, which condition an AdaLN-Zero DiT denoiser. Selected linear layers of the denoiser are decomposed via SVD into rank-one experts and modulated by a compact router (Stage 2).
Fig. 2: Spatial-compression adapter. Three residual blocks form a coarse-to-fine pyramid; each level is pooled at 1×1 , 2×2 or 3×3 , flattened, linearly projected, and concatenated into 14 tokens per view (Eq. 3 ).
Fig. 3: Router that produces token-wise σ^b,q in Eq. 6 . A task-level branch ( Pekb ) is shared across denoiser tokens, while a token-state bottleneck takes xb,q and adds the diffusion-step bias Wnemb(nb) .
Fig. 4: Two-stage training. Stage 1 jointly trains the adapter, task head, and a standard DiT denoiser without task conditioning. Stage 2 freezes the visual encoder and task head, decomposes well-conditioned layers by SVD, and updates only the router and Householder vectors.
Hyperparam.
Value
Hyperparam.
Value
VGGT block ℓ⋆
23
Task emb. Dt
32
Pool resolutions
{1×1,2×2,3×3}
Router dim Dr
32
Tokens / view
14
Householder Lh
4
DiT blocks L
12
Threshold τ
102
Embedding dim
384
DDPM steps
100
Batch / task wt.
200/0.5
Obs./act. horizon
1/8
TABLE I: Key hyperparameters of SR-DP.
Method
Block Hammer Beat
Bottle Adjust
Container Place
Dual Bottles Pick
Blocks Stack
Tool Adjust
Avg. ( ↑ )
Per-task reference (one model per task)
DP [ 1 ]
71.0 ± 1.0
40.3 ± 2.5
15.3 ± 1.2
74.7 ± 0.6
37.3 ± 1.2
34.0 ± 1.0
45.4 ± 1.0
VO-DP [ 18 ]
83.7 ± 1.5
64.0 ± 4.6
43.0 ± 3.0
85.7 ± 1.5
54.3 ± 3.2
57.0 ± 1.7
64.6 ± 0.6
Multi-task (single model on the six tasks)
DP [ 1 ] (multi-task)
40.0 ± 1.7
14.3 ± 2.5
14.3 ± 4.0
62.3 ± 0.6
23.0 ± 1.0
25.7 ± 1.5
29.9 ± 0.5
VO-DP [ 18 ] (multi-task)
42.0 ± 2.6
16.0 ± 4.0
40.7 ± 1.2
91.0 ± 1.7
32.0 ± 2.6
58.0 ± 0.0
46.6 ± 0.3
TABLE II: Success rates (%) on six RoboTwin tasks. Each entry reports mean ± std over three training seeds, with 100 evaluation episodes per task per seed. Per-task rows are specialist references; bold marks the best single multi-task model in each column.
Fig. 5: Encoder feature t-SNE. Visual conditions from the complete FDP, multi-task VO-DP, and SR-DP pipelines over 9,600 held-out frames ( 1,600 /task) across the six RoboTwin tasks. FDP uses ResNet, VO-DP pools VGGT without task loss, and SR-DP pairs VGGT with the spatial adapter and task head.
Fig. 6: Loss-sensitivity conflict in the shared SVD basis. Layer-averaged task-pair sensitivity cosine ( 4 held-out windows/task, n=50 , shared noise; negative: opposing sensitivities). Stage 2 lowers the mean conflict score from 0.267 to 0.104 .
Fig. 7: Router activation along held-out demonstrations. Six RoboTwin tasks for the routed layer with maximum temporal std (self-attention QKV in Block 9), scanned every third frame of one held-out episode/task ( n=50 ). Color shows token-state offset averaged over action tokens.
Fig. 8: Qualitative comparison on the six RoboTwin tasks: each task pairs a selected Stage 1 failure (top) and Stage 2 success (bottom) under the same initial conditions to illustrate failure modes.
Fig. 9: Real-world platform. The DexTeleop TeleAvatar dual-arm robot, teleoperated through a VR headset and controllers for demonstration collection. Three RGB cameras provide observations: a head-mounted view and two wrist-mounted views (one per arm).
Fig. 10: Real-world rollouts of SR-DP on the four TeleAvatar tasks. Rows correspond to T1–T4, and columns show temporally ordered key frames from one successful Stage 2 rollout per task.
Method
T1
T2
T3
T4
Avg. (%)
SR-DP (Stage 1)
7/50
12/50
13/50
16/50
24.0
Task-AdaLN
12/50
8/50
11/50
16/50
23.5
SR-DP (Stage 2)
19/50
23/50
15/50
29/50
43.0
TABLE IV: Real-world success counts over 50 rollouts per task. The final column reports the average success rate (%) across four tasks.
Diffusion-based visuomotor policies operating directly in raw action spaces conflate scene comprehension with trajectory generation within a single denoising process. The resulting velocity field must simultaneously encode scene information and generate precise trajectories, increasing learning complexity and limiting performance on tasks demanding precise temporal coordination across multiple arms. To simplify this joint learning problem, we introduce Latent Diffusion Policy (LDP), a two-stage framework performing flow matching in a deliberately shaped latent space. By absorbing scene understanding into an observation-conditioned CVAE encoder, LDP concentrates the conditional distribution of each observation. Consequently, the flow model avoids implicitly resolving scene-dependent structures; instead, it generates within a pre-concentrated distribution featuring a smoother velocity field, simplifying learning from limited demonstrations. Furthermore, to capture temporal dependencies among latent tokens, LDP trains with per-token diffusion forcing and employs staircase inference sampling to resolve the resulting distributional mismatch. We also propose reconstruction FID (rFID) as a lightweight proxy predicting downstream task success solely from latent space statistics. On coordination-intensive tasks from RoboTwin 2.0, LDP outperforms DP3 by a substantial margin and transfers effectively to real-world bimanual deployments.
Diffusion-based policies have established a new standard for precise robotic manipulation but face a critical scalability bottleneck: high-performance models are computationally expensive, while lightweight alternatives often fail to generalize across diverse multi-task environments. Mixture-of-Experts (MoE) architectures offer a promising path to efficiency by activating only a subset of parameters. However, existing MoE routing mechanisms typically rely on low-level noise or latent statistics, ignoring the compositional nature of manipulation tasks. This can fragment reusable behaviors across experts, limiting interpretability and transferability. We introduce Semantically Structured Mixture-of-Experts Diffusion Policy (SMoDP) for compositional robotic manipulation, a framework that grounds expert specialization in semantic task structure. SMoDP leverages a lightweight, inference-time skill predictor, supervised by offline annotations from Vision-Language Models (VLMs), to route action chunks to experts specialized for specific behavioral phases. To ensure robust assignment, we propose a dual contrastive alignment strategy that grounds multi-modal observations in language-defined skill semantics (Inter-modal) while enforcing routing consistency across visually distinct but functionally related behaviors (Intra-modal). Our approach outperforms representative diffusion and MoE-based baselines on multi-task benchmarks with significantly improved parameter efficiency and demonstrates effective compositional transfer to novel tasks through parameter-efficient fine-tuning. Project website: https://deng-cy20.github.io/SMoDP/
Chengyu Deng, Guanqi Chen, Yizhou Chen +4
The University of Hong Kong · Southern University of Science and Technology
Diffusion-based visuomotor policies perform well in robotic manipulation, yet current methods still inherit image-generation-style decoders and multi-step sampling. We revisit this design from a frequency-domain perspective. Robot action trajectories are highly smooth, with most energy concentrated in a few low-frequency discrete cosine transform modes. Under this structure, we show that the error of the optimal denoiser is bounded by the low-frequency subspace dimension and residual high-frequency energy, implying that denoising error saturates after very few reverse steps. This also suggests that action denoising requires a much simpler denoising model than image generation. Motivated by this insight, we propose Hyper-DP3 (HDP3), a pocket-scale 3D diffusion policy with a lightweight Diffusion Mixer decoder that supports two-step DDIM inference. Our synthetic experiments validate the theory and support the sufficiency of two-step denoising. Futhermore, across RoboTwin2.0, Adroit, MetaWorld, and real-world tasks, HDP3 achieves state-of-the-art performance with fewer than 1% of the parameters of prior 3D diffusion-based policies and substantially lower inference latency.
Jinhao Zhang, Zhexuan Zhou, Huizhe Li +5
Harbin Institute of Technology, Shenzhen · Shanghai Jiao Tong University