Diffusion Policy (DP) excels in embodied control but suffers from high inference latency and computational cost due to multiple iterative denoising steps. Existing diffusion model (DMs) acceleration methods improve inference speed by reducing computational overhead, as DMs for image generation are computation-bound. In this work, we reveal that DP are I/O bound, and thus propose using speculative decoding, a parallel computing framework, to accelerate DP for the first time. However, it is non-trivial to address the following challenges: (1) How to preserve the denoising quality of the base model at reduced computational cost under time-varying task difficulty in embodied settings; and (2) How to dynamically allocate computation according to task difficulty in such environments. Therefore, we propose Temporal-aware Reinforcement-based Speculative Diffusion Policy (TS-DP), the first framework that enables speculative decoding for DP with temporal adaptivity. First, to handle dynamic environments where task difficulty varies over time, we distill a Transformer-based drafter to imitate the base model and replace its costly denoising calls. Second, an RL-based scheduler further adapts to the time-varying task difficulty by adjusting speculative parameters to maintain accuracy while improving efficiency. Extensive experiments show that TS-DP achieves up to a 4.17x NFE reduction and a 3.2x wall-clock speedup with a draft acceptance rate of over 85% in simulation. Real-world experiments further demonstrate a 3.8x NFE reduction while improving success rates on real-time tasks (e.g., a 23% gain on the conveyor-belt task), reaching an inference frequency of 26 Hz and enabling real-time control while maintaining comparable task performance. The code is available at https://github.com/ChildTang/TS-DP.
Figures & tables
Fig. 1: Vanilla Diffusion Policy (DP) vs. Our TS-DP . For embodied tasks, DP completes a task through multiple interactions with the environment, generating several segments of actions. Each segment typically requires hundreds of model calls for denoising, making the process highly time-consuming. TS-DP introduces a customized speculative decoding framework for temporal-complexity-aware accelerating DP. A lightweight drafter generates multiple denoising results, which are verified in parallel by the DP for near-lossless acceleration. A scheduler further adjusts key speculative decoding parameters—such as the number of drafts and the acceptance rate—based on the complexity of different task phases, achieving adaptive and efficient acceleration.
Fig. 2: The framework of TS-DP . TS-DP is the first temporal-aware speculative decoding framework designed to accelerate Diffusion Policy. ① Decision stage: A PPO-based scheduler evaluates time-varying task difficulty using past observations and actions, and produces adaptive speculative parameters. It operates in parallel with the observation encoder, adding no extra inference latency. ② Denoising stage: Guided by the scheduler, a lightweight drafter executes multiple denoising steps sequentially, effectively replacing expensive DP calls and reducing computational load. ③ Verification stage: DP verifies all drafted steps in parallel, accepting those that pass validation and correcting the first rejected draft via reflection-maximal coupling to better align it with the target distribution. ④ Reward generation: Based on task progress, the framework provides process rewards that encourage the scheduler to produce more valid drafts, and a final success-driven reward upon task completion, jointly guiding the scheduler toward efficient and successful task execution.
Method
Success Rate (%, ↑ )
AVG (%, ↑ )
NFE (%, ↓ )
Speed ×
Lift
Can
Square
Transport
Tool
Push-T
Diffusion Policy [ 1 ]
100 / 100
95 / 97
82 / 88
78 / 81
43 / 53
59 / 64
76 / 80
100 / 100
–
Frozen Target Draft [ 19 ]
90 / 92
82 / 84
70 / 70
50 / 64
26 / 34
16 / 17
56 / 60
33 / 32
3.03 / 3.13
SpeCa [ 39 ]
100 / 100
92 / 96
88 / 88
68 / 72
38 / 40
67 / 69
76 / 78
34 / 35
2.94 / 2.86
BAC [ 13 ]
100 / 100
94 / 97
82 / 89
77 / 82
49 / 55
59 / 62
77 / 81
– / –
3.40 / 3.40
TS-DP
100 / 100
95 / 95
86 / 88
87 / 84
61 / 52
79 / 63
85 / 80
24 / 24
4.17 / 4.17
TABLE I: Benchmark on Proficient Human (PH) demonstration data.
Method
Success Rate (%, ↑ )
AVG (%, ↑ )
NFE (%, ↓ )
Speed ×
Lift
Can
Square
Transport
Diffusion Policy [ 1 ]
99 / 100
92 / 97
76 / 79
35 / 46
76 / 81
100 / 100
–
Frozen Target Draft [ 19 ]
66 / 62
60 / 54
50 / 62
20 / 42
49 / 55
34 / 33
2.94 / 3.03
SpeCa [ 39 ]
96 / 100
86 / 90
76 / 68
34 / 40
73 / 75
35 / 37
2.86 / 2.70
BAC [ 13 ]
99 / 98
95 / 97
77 / 79
30 / 46
75 / 80
– / –
3.41 / 3.41
TS-DP
99 / 99
92 / 97
74 / 78
34 / 62
75 / 84
26 / 26
3.84 / 3.84
TABLE II: Benchmark on Mixed Human (MH) demonstration data.
Method
Success Rate (%, ↑ )
AVG (%, ↑ )
NFE (%, ↓ )
Speed ×
BP p1
BP p2
Kit p1
Kit p2
Kit p3
Kit p4
Diffusion Policy [ 1 ]
98 / 98
98 / 96
100 / 100
100 / 100
100 / 100
100 / 100
99 / 99
100 / 100
–
Frozen Target Draft [ 19 ]
85 / 86
1 / 2
100 / 100
100 / 100
100 / 100
100 / 100
81 / 81
33 / 33
3.03 / 3.03
SpeCa [ 39 ]
97 / 97
85 / 85
100 / 100
100 / 100
100 / 100
100 / 100
97 / 97
37 / 37
2.70 / 2.70
BAC [ 13 ]
100 / 99
97 / 95
100 / 99
100 / 99
100 / 99
94 / 97
99 / 98
– / –
3.60 / 3.60
TS-DP
99 / 100
97 / 97
100 / 100
100 / 100
100 / 100
100 / 100
99 / 99
27 / 27
3.70 / 3.70
TABLE III: Benchmark on multi-stage task (Kitchen & BP).
Fig. 3: Conveyor-Belt Pick-and-Place Task. This figure shows a Franka robotic arm picking pineapple buns from a moving conveyor belt and placing them onto a plate. TS-DP finishes in 20s, compared to 63s for DP, yielding a 3.2 × speedup. This improved real-time performance also increases the success rate from 29% to 52%. Here, T denotes time in seconds (s), and S denotes success rate in percentage (%).
Fig. 4: Pen-to-Holder Insertion Task. This task is designed to examine the impact of TS-DP on high-difficulty, high-precision manipulation. As shown in the figure, DP exhibits greater precision at critical moments, such as more accurate alignment with the pen holder, resulting in smaller displacements. Although TS-DP shows minor inaccuracies in these cases, it is able to quickly correct them and successfully complete the task. It ultimately maintains a 3.4 × NFE speedup, with the success rate decreasing by only 2%. Here, N denotes NFE, and S denotes success rate(%).
Fig. 5: Pick-and-Release Task. As shown in the figure, DP exhibits prolonged pauses both at the start and after grasping, whereas TS-DP effectively mitigates this issue, achieving a 3.8 × NFE speedup with only a 2% accuracy drop. The task closely matches the difficulty of the official simulation and real-world benchmarks, providing a clear setting to showcase the acceleration achieved by our method.
Fig. 6: Impact of Drafter Parameters on Draft Acceptance Rate. Evolution of the Metropolis–Hastings acceptance probability throughout the 100-step denoising process.
Fig. 7: Effect of End-Effector Velocity on the Number of Accepted Drafts. Variation of Accepted Drafts and Velocity During the Can-ph Task: (Top) Accepted Drafts per Step; (Bottom) Velocity per Step.
Fig. 8: Real-world experimental platforms. The Pick-and-Release task is conducted on platform (a), while the Conveyor-Belt Pick-and-Place and Pen-to-Holder Insertion tasks are conducted on platform (b).
Method
Success Rate (%, ↑ ) / NFE ( ↓ )
AVG
Pick
Conveyor Belt
Pen Holder
DP [ 1 ]
82 / 100
29 / 100
31 / 100
47 / 100
TS-DP
80 / 26
52 / 27
29 / 29
54 / 27
TABLE IV: Results on real tasks.
Method
Frequency (Hz, ↑ ) / Latency (s, ↓ )
AVG
Lift
Can
Square
DP [ 1 ]
7.75/0.129±
0.004
7.52/0.133±
0.003
7.52/0.133±
0.004
DP: 8.15/0.124 TS-DP : 26.28/0.039
TS-DP
24.39/0.041±
0.002
24.39/0.041±
0.002
23.81/0.042±
0.003
Method
Frequency (Hz, ↑ ) / Latency (s, ↓ )
Transport
Tool
Push-T
DP [ 1 ]
7.69/0.130±
0.005
8.13/0.123±
0.003
10.31/0.097±
0.003
TABLE V: Frequency and Latency Evaluation of TS-DP . Each entry reports frequency (Hz, ↑ ) / latency (s, ↓ ), with latency given as mean ± standard deviation over repeated trials.
Task
NRMSE ↓
Cosine Similarity ↑
Can
0.0324±0.0015
0.9527±0.0082
Lift
0.0384±0.0018
0.9455±0.0091
Square
0.0434±0.0020
0.9387±0.0095
Transport
0.0454±0.0021
0.9304±0.0103
Tool
0.0101±0.0005
0.9958±0.0021
Push-T
0.0213±0.0010
0.9814±0.0020
TABLE VI: Empirical action-level fidelity between DP and TS-DP , reported as mean ± standard deviation.
Method
Success Rate (%, ↑ )
AVG
Lift
Can
Square
Transport
DP [ 1 ]
99 / 100
92 / 97
76 / 79
35 / 46
75.5 / 80.5
Drafter Only
36 / 40
26 / 20
0 / 0
0 / 0
15.5 / 15.0
TS-DP
99 / 99
92 / 97
74 / 78
34 / 62
74.75 / 84
TABLE VII: Drafter-Only Results on the Mixed Human (MH) Demonstration Dataset.
Model
Lift
Can
Square
Transport
Tool
Push-T
Avg
DP
10.03
9.57
10.14
10.46
7.88
8.15
9.37
Drafter
1.22
1.19
1.19
1.25
1.15
1.02
1.17
Ratio
8.22×
8.04×
8.52×
8.37×
6.85×
7.99×
8.01×
TABLE VIII: Block-level latency (ms) of the target backbone and the drafter, averaged over repeated runs.
Method
Success Rate (%, ↑ )
AVG
Speed ×
Lift
Can
Square
Transport
K=10
100
94
76
66
84
2.45
K=25
88
88
62
60
75
3.47
K=40
74
90
64
60
72
3.92
TS-DP
100
96
84
66
87
3.80
TABLE IX: Performance Comparison With and Without the Scheduler on the Mixed Human (MH) Demonstration Dataset.
Fig. 9: Temporal scheduling dynamics on the Square task. The scheduler adjusts the effective variance (Sigma), acceptance threshold, and the high-, intermediate-, and low-noise draft budgets over execution steps.
Method
Success Rate / Draft Accept Rate (%, ↑ )
AVG
NFE ( ↓ )
Lift
Can
Square
Transport
DP [ 1 ]
99.5 / –
94.5 / –
77.5 / –
40.5 / –
78.0 / –
100
VT [ 21 ]
90 / 80
86 / 80
61 / 79
35 / 81
68 / 80
36
TS-DP
99 / 84.2
94.5 / 87.9
76 / 86.0
48 / 89.6
79 / 86.9
26
TABLE X: Results Using a Speed Threshold on the Mixed Human (MH) Demonstration Dataset.
Fig. 10: Effect of TS-DP on Acceptance Rate and Draft Count. (Top) Acceptance Rate; (Bottom) Draft Count. With the integration of TS-DP , more drafts are accepted while maintaining task accuracy.
Benchmark
NFE
Speed
Draft Count
Accept. Rate
PH
24.0
4.17×
93.9
85.2%
MH
26.3
3.83×
92.5
87.1%
Kitchen/BP
27.2
3.77×
91.8
89.8%
TABLE XI: Speculative decoding statistics across benchmarks. We report NFE, speedup, draft count, and acceptance rate on the PH, MH, and Kitchen/BP benchmarks.
Diffusion-based large language models (dLLMs) support parallel text generation via iterative denoising, yet inference remains latency-heavy because many steps are spent on redundant refinement and repeated remasking of tokens whose final values are already determined. Prior acceleration methods mainly depend on step-local confidence heuristics or fixed schedules, which are sensitive to prompt and task variation and ignore strong positional effects within a sequence. We cast diffusion decoding as a dynamic control problem and show that token-wise denoising trajectories provide the key signal for reliable control. We propose a trace-aware decoding framework with two components. First, Temporal-Spatial Parallel Decoding (TSPD) uses a lightweight temporalspatial controller that consumes per-token trajectory features, including confidence, entropy, and momentum, together with token position, to decide when a token has converged and can be safely fixed. Second, we introduce Confidence Extrapolation (CE), a training-free state-space module that forecasts future logit trends with uncertainty to support proactive decisions, including safe look-ahead and targeted stabilization when trajectories are oscillatory or underconfident. Together, TSPD and CE reduce unnecessary denoising iterations while preserving output quality, and they compose cleanly with system optimizations such as KV caching.
Diffusion policies achieve strong visuomotor control by iteratively denoising action chunks, but repeated denoising makes real-time deployment computationally demanding. Cache-based methods reduce inference cost by reusing intermediate activations, but existing training-free schedules typically allocate computation uniformly across blocks, ignoring heterogeneous redundancy across blocks and leading to a suboptimal performance-efficiency trade-off. To bridge this gap, we introduce Evolving Cache Schedules (EVO), a training-free acceleration framework that globally schedules cache refreshes via evolutionary search. EVO represents each candidate as a complete schedule over the block-timestep lattice. Thus, redundant transformer computations during iterative denoising can be skipped through cache reuse while preserving closed-loop rollout performance. To make the search practical, EVO introduces redundancy-aware initialization, which seeds the population with promising schedules, and target-conditioned early stopping, which verifies and terminates once a desired performance target is reached. The offline-optimized schedule can be directly plugged into pretrained diffusion policies without retraining. Extensive manipulation benchmarks show that EVO preserves near-full performance while substantially reducing computation, achieving up to 8.05x action-generation speedup and reducing FLOPs from 15.77G to as low as 1.96G. Source code is available at https://github.com/pillom/EVO.
Siying Wang, Kangye Ji, Di Wang +1
School of Telecommunications Engineering, Xidian University, Xi’an, China · Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · School of Computer Science and Technology, Xidian University, Xi’an, China +1
Speculative decoding is a technique that accelerates the inference speed of autoregressive large language models (LLMs) by sampling from a faster, lower quality draft model. Traditionally, a fundamental characteristic of speculative decoding is that it is lossless with respect to the verifier distribution, guaranteeing speed without sacrificing quality. This differs from recently developed diffusion LLMs (dLLMs), which are notably able to achieve substantial speedups by degrading the quality of their outputs with less diffusion effort. Because diffusion models are able to sample faster than autoregressive models by generating tokens in parallel, they are considered strong candidates for draft models in speculative diffusion. While it is widely understood that diffusion models can sacrifice quality for speed, speculative decoding itself can also be generalized to generate lossy samples faster using more lenient acceptance criteria with respect to the verifier distribution. Presented with the two parameters of diffusion effort and speculative decoding leniency, this paper explores the full achievable Pareto frontier via joint variations of these parameters. Our experiments show that jointly varying diffusion effort and acceptance leniency can extend the speed-quality Pareto frontier over single step diffusion drafting.
Aaron Zorel, Guy Van den Broeck, Aditya Grover +1
Independent Researcher · University of California, Los Angeles