cs.RONov 3, 2026

Spectrally Routed Diffusion Policy for Multi-Task Robotic Manipulation

Authors: Mingcong Li, Yihan Zhou, Jiayu Qu, Xueyang Guo, Yilin Mo

Organizations: Mingcong Li and Jiayu Qu are with the School of Automation, Beijing Institute of Technology, Beijing 100081, China. · LingYu Robotics, Beijing 100086, China. · Yihan Zhou and Xueyang Guo are with the Department of Automation, Tsinghua University, Beijing 100084, China, and also with the Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China. · Yilin Mo is with the Department of Automation, Tsinghua University, Beijing 100084, China, and also with the Beijing Key Laboratory of Embodied Intelligence Systems, Beijing, China, and also with the Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China.

Abstract

Diffusion-based policies have achieved strong performance on single-task robotic manipulation. However, a single model trained jointly on multiple tasks consistently underperforms the per-task specialists trained on the same demonstrations. We focus on two challenges. First, the convolutional encoders commonly used in diffusion-based policies produce visual conditions that are not discriminative enough. Second, gradients from incompatible action modes update the shared denoiser's dense parameters along the same directions, causing destructive interference during joint training. To address both, we propose SR-DP (Spectrally Routed Diffusion Policy). A frozen VGGT encoder with a spatial-compression adapter produces compact multi-scale semantic-geometric tokens, a task-discriminative visual condition for tasks that share a workspace. We first train the denoiser as a standard diffusion policy, then decompose selected linear layers by SVD into a frozen basis of orthogonal rank-one experts, adapted through spectral routing and Householder input rotations. On six RoboTwin tasks, SR-DP averages a 68.4% success rate, surpassing multi-task VO-DP by 21.8 points and the strongest per-task baseline by 3.8 points. On four real-world bimanual tasks, Stage 2 reaches a 43.0% average success rate and improves over Stage 1 by 19.0 points.

Figures & tables

Explore similar work

CardsList
  1. Semantically Structured Mixture-of-Experts for Compositional Robotic Manipulation

    May 22, 2026Chengyu Deng, Guanqi Chen, Yizhou Chen +4Robot Policy LearningRobot Skill Learning

  2. Hyper-DP3: Frequency-Aware Right-Sizing of 3D Diffusion Policies for Visuomotor Control

    May 2, 2026Jinhao Zhang, Zhexuan Zhou, Huizhe Li +5Visuomotor Policy LearningDiffusion Policy