Organizations: Mingcong Li and Jiayu Qu are with the School of Automation, Beijing Institute of Technology, Beijing 100081, China. · LingYu Robotics, Beijing 100086, China. · Yihan Zhou and Xueyang Guo are with the Department of Automation, Tsinghua University, Beijing 100084, China, and also with the Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China. · Yilin Mo is with the Department of Automation, Tsinghua University, Beijing 100084, China, and also with the Beijing Key Laboratory of Embodied Intelligence Systems, Beijing, China, and also with the Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China.
Diffusion-based policies have achieved strong performance on single-task robotic manipulation. However, a single model trained jointly on multiple tasks consistently underperforms the per-task specialists trained on the same demonstrations. We focus on two challenges. First, the convolutional encoders commonly used in diffusion-based policies produce visual conditions that are not discriminative enough. Second, gradients from incompatible action modes update the shared denoiser's dense parameters along the same directions, causing destructive interference during joint training. To address both, we propose SR-DP (Spectrally Routed Diffusion Policy). A frozen VGGT encoder with a spatial-compression adapter produces compact multi-scale semantic-geometric tokens, a task-discriminative visual condition for tasks that share a workspace. We first train the denoiser as a standard diffusion policy, then decompose selected linear layers by SVD into a frozen basis of orthogonal rank-one experts, adapted through spectral routing and Householder input rotations. On six RoboTwin tasks, SR-DP averages a 68.4% success rate, surpassing multi-task VO-DP by 21.8 points and the strongest per-task baseline by 3.8 points. On four real-world bimanual tasks, Stage 2 reaches a 43.0% average success rate and improves over Stage 1 by 19.0 points.
Figures & tables
Fig. 1: Overview of SR-DP. A frozen VGGT encoder with a spatial-compression adapter produces 14 tokens per view, which condition an AdaLN-Zero DiT denoiser. Selected linear layers of the denoiser are decomposed via SVD into rank-one experts and modulated by a compact router (Stage 2).
Fig. 2: Spatial-compression adapter. Three residual blocks form a coarse-to-fine pyramid; each level is pooled at 1×1 , 2×2 or 3×3 , flattened, linearly projected, and concatenated into 14 tokens per view (Eq. 3 ).
Fig. 3: Router that produces token-wise σ^b,q in Eq. 6 . A task-level branch ( Pekb ) is shared across denoiser tokens, while a token-state bottleneck takes xb,q and adds the diffusion-step bias Wnemb(nb) .
Fig. 4: Two-stage training. Stage 1 jointly trains the adapter, task head, and a standard DiT denoiser without task conditioning. Stage 2 freezes the visual encoder and task head, decomposes well-conditioned layers by SVD, and updates only the router and Householder vectors.
Hyperparam.
Value
Hyperparam.
Value
VGGT block ℓ⋆
23
Task emb. Dt
32
Pool resolutions
{1×1,2×2,3×3}
Router dim Dr
32
Tokens / view
14
Householder Lh
4
DiT blocks L
12
Threshold τ
102
Embedding dim
384
DDPM steps
100
Batch / task wt.
200/0.5
Obs./act. horizon
1/8
TABLE I: Key hyperparameters of SR-DP.
Method
Block Hammer Beat
Bottle Adjust
Container Place
Dual Bottles Pick
Blocks Stack
Tool Adjust
Avg. ( ↑ )
Per-task reference (one model per task)
DP [ 1 ]
71.0 ± 1.0
40.3 ± 2.5
15.3 ± 1.2
74.7 ± 0.6
37.3 ± 1.2
34.0 ± 1.0
45.4 ± 1.0
VO-DP [ 18 ]
83.7 ± 1.5
64.0 ± 4.6
43.0 ± 3.0
85.7 ± 1.5
54.3 ± 3.2
57.0 ± 1.7
64.6 ± 0.6
Multi-task (single model on the six tasks)
DP [ 1 ] (multi-task)
40.0 ± 1.7
14.3 ± 2.5
14.3 ± 4.0
62.3 ± 0.6
23.0 ± 1.0
25.7 ± 1.5
29.9 ± 0.5
VO-DP [ 18 ] (multi-task)
42.0 ± 2.6
16.0 ± 4.0
40.7 ± 1.2
91.0 ± 1.7
32.0 ± 2.6
58.0 ± 0.0
46.6 ± 0.3
TABLE II: Success rates (%) on six RoboTwin tasks. Each entry reports mean ± std over three training seeds, with 100 evaluation episodes per task per seed. Per-task rows are specialist references; bold marks the best single multi-task model in each column.
Fig. 5: Encoder feature t-SNE. Visual conditions from the complete FDP, multi-task VO-DP, and SR-DP pipelines over 9,600 held-out frames ( 1,600 /task) across the six RoboTwin tasks. FDP uses ResNet, VO-DP pools VGGT without task loss, and SR-DP pairs VGGT with the spatial adapter and task head.
Fig. 6: Loss-sensitivity conflict in the shared SVD basis. Layer-averaged task-pair sensitivity cosine ( 4 held-out windows/task, n=50 , shared noise; negative: opposing sensitivities). Stage 2 lowers the mean conflict score from 0.267 to 0.104 .
Fig. 7: Router activation along held-out demonstrations. Six RoboTwin tasks for the routed layer with maximum temporal std (self-attention QKV in Block 9), scanned every third frame of one held-out episode/task ( n=50 ). Color shows token-state offset averaged over action tokens.
Fig. 8: Qualitative comparison on the six RoboTwin tasks: each task pairs a selected Stage 1 failure (top) and Stage 2 success (bottom) under the same initial conditions to illustrate failure modes.
Fig. 9: Real-world platform. The DexTeleop TeleAvatar dual-arm robot, teleoperated through a VR headset and controllers for demonstration collection. Three RGB cameras provide observations: a head-mounted view and two wrist-mounted views (one per arm).
Fig. 10: Real-world rollouts of SR-DP on the four TeleAvatar tasks. Rows correspond to T1–T4, and columns show temporally ordered key frames from one successful Stage 2 rollout per task.
Method
T1
T2
T3
T4
Avg. (%)
SR-DP (Stage 1)
7/50
12/50
13/50
16/50
24.0
Task-AdaLN
12/50
8/50
11/50
16/50
23.5
SR-DP (Stage 2)
19/50
23/50
15/50
29/50
43.0
TABLE IV: Real-world success counts over 50 rollouts per task. The final column reports the average success rate (%) across four tasks.