cs.AIMar 2, 2026
SaveState-Action Inpainting Diffuser for Continuous Control with Delay
Organizations: Microsoft Research Asia · University of Pennsylvania · Shanghai Jiaotong University
Abstract
Signal delay poses a fundamental challenge in continuous control and reinforcement learning (RL) by introducing a temporal gap between interaction and perception. Current solutions have largely evolved along two distinct paradigms: model-free approaches which utilize state augmentation to preserve Markovian properties, and model-based methods which focus on inferring latent beliefs via dynamics modeling. In this paper, we bridge these perspectives by introducing State-Action Inpainting Diffuser (SAID), a framework that integrates the inductive bias of dynamics learning with the direct decision-making capability of policy optimization. By formulating the problem as a joint sequence inpainting task, SAID implicitly captures environmental dynamics while directly generating consistent plans, effectively operating at the intersection of model-based and model-free paradigms. Crucially, this generative formulation allows SAID to be seamlessly applied to both online and offline RL. Extensive experiments on delayed continuous control benchmarks demonstrate that SAID achieves state-of-the-art and robust performance. Our study suggests a new methodology to advance the field of RL with delay.
Figures & tables
Figure 1: Illustration of our idea under case delay=3 as an example. Upper: inpainting in images. Bottom: state-action sequence inpainting for decision making with signal delay (our approach).
Figure 2: The being modeled state-action sequence (in the dashed rectangle) of our State-Action Inpainting Diffuser at environment timestep with delay . Black color indicates input conditions and red color indicates the inpainting values.
Figure 3: Handling episode start by padding initial actions and states, using =0 and =1 as examples. The symbols are the same as those in Figure 2 .
Figure 4: The “compounding error” phenomenon of autoregressive state-transition models. See Fig. 11 for more examples.
Figure 5: Performance (mean episodic return) for online RL with delay. One may notice that the experimental result of Hopper at delay=16 is higher than that at delay=8. This is also reported in Table 10 of the reference paper ( Wu et al., 2025 ) .
| Task | Method | Delay steps | |||
|---|---|---|---|---|---|
| 0 | 4 | 8 | 16 | ||
| HC-v5 | SAC | ||||
| DR | — | ||||
| SADR | — | ||||
| VDPO | — | ||||
| SAID | |||||
Table 1: Results across methods under different delays (mean standard error across 4 training random seeds). Method abbreviations: SAC (Vanilla SAC), DR (DR-Critic), SADR (SA-MLP+DR-Critic). For delay=0, DR-Critic and SA-MLP+DR-Critic reduce to Vanilla SAC; hence we report a single SAC(0) baseline and mark other methods with —. The results of SAID are obtained from 4 random seeds with 200 evaluation episodes per seed. Bold indicates the best mean within each task and delay.
| Category | Non-diffusion | Diffusion Policies | Diffusion Planners | Baseline w/ delay | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | BC | CQL | ReBRAC | DQL | DQL* | Diffuser | DV | SAID (ours) | DQL+SA | |||||
| Delay | No delay | 0 | 4 | 8 | 16 | 4 | 8 | 16 | ||||||
| halfcheetah-medium-expert-v2 | 35.8 | 62.4 | 108.1 | 96.8 | 95.9 | 88.9 | 92.7 | 106.2 | 32.8 | 4.5 | 1.9 | 2.9 | 2.1 | 1.4 |
| halfcheetah-medium-replay-v2 | 38.4 | 46.2 | 51.2 | 47.8 | 48.9 | 37.7 | 45.8 | 50.2 | 34.3 | 8.3 | 2.1 | 0.2 | 1.4 | 1.3 |
| halfcheetah-medium-v2 | 36.1 | 44.4 | 66.4 | 51.1 | 52.6 | 42.8 | 50.4 | 58.5 | 45.2 | 5.8 | 1.5 | 2.4 | 2 | 1.3 |
| walker2d-medium-expert-v2 | 6.4 | 111 | 104.6 | 110.1 | 112 | 106.9 | 109.2 | 111.1 | 111.1 | 110.8 | 106.7 | 8.1 | -0.3 | -0.1 |
Table 2: Offline RL results on D4RL MuJoCo locomotion tasks, the performance of SAID is averaged over 4 random model seeds with each evaluated over 200 episodes. Here we omit the confidence interval since the performance of SAID for offline RL is less sensitive to random seed, see Sec. 6.3 for more discussion. DQL* is our replicated results using the same code base as SAID.
Figure 6: Offline RL performance, compared with Diffusion Q-Learning (DQL) with state augmentation (SA).
Figure 7: Violin plot of relative uncertainly (STD) of Online RL vs. offline RL among different same training seeds (averaged over 200 evaluation episdoes). The white point indicates the data mean. Mann-Whitney U-test was conducted and it showed a significantly higher seed-uncertainty of online RL than offline RL.
Figure 8: Violin plot of relative uncertainly (STD) of Online RL vs. offline RL among different evaluation episodes (averaged over 4 random training seeds). The white point indicates the data mean. Mann-Whitney U-test was conducted but no significant difference was observed.
Figure 9: Examining SAID’s performance for online RLwD tasks when using all online experiences (5M steps) or using only the last 20% experiences (1M steps). Data are Mean S.E.M.
Figure 10: Examining SAID’s performance for online RLwD (all-tasks average) with different denoising steps during inference. Data are Mean S.E.M.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Area | Hyperparameter | Value | What it controls |
| Environment / reward shaping | terminal_penalty | -100 | Penalty on terminal condition. |
| Environment / reward shaping | full_traj_bonus | 0 | Bonus for completing full trajectory. |
| Planner diffusion | planner_solver | ddim | Sampling/solver for planner diffusion. |
| Planner architecture | planner_emb_dim | 128 | Planner embedding dimension. |
| Planner architecture | planner_d_model | 256 | Transformer model width (d_model). |
| Planner sampling | planner_sampling_steps | 20 | Steps used when sampling planner diffusion. |
Table 3: Task-independent hyperparameters of SAID.
| Task Name | Delay 0 (pd / tmp) | Delay 4 (pd / tmp) | Delay 8 (pd / tmp) | Delay 16 (pd / tmp) |
| Online RL Tasks | ||||
| Ant-v5 | 8 / 0.6 | 8 / 0.6 | 8 / 0.6 | 8 / 0.8 |
| HalfCheetah-v5 | 8 / 0.6 | 8 / 0.3 | 8 / 0.3 | 8 / 0.1 |
| Hopper-v5 | 8 / 0.6 | 4 / 0.3 | 2 / 0.1 | 8 / 0.6 |
| Walker2d-v5 | 8 / 0.8 | 8 / 0.6 | 4 / 0.8 | 8 / 0.6 |
| HalfCheetah (D4RL) |
Table 4: Best hyperparameters of SAID for each task after grid search (pd: depth of the planner DiT, tmp: temperature of the planner during inference).
Figure 11: Full D4RL locomotion task results for Fig. 4 .
| Task | Delay 0 | Delay 4 | Delay 8 | Delay 16 |
| halfcheetah-medium-expert-v2 | 106.2 ± 0.1 | 32.8 ± 0.8 | 4.5 ± 0.1 | 1.9 ± 0.5 |
| halfcheetah-medium-replay-v2 | 50.2 ± 0.2 | 34.3 ± 0.5 | 8.3 ± 0.6 | 2.1 ± 0.8 |
| halfcheetah-medium-v2 | 58.5 ± 0.2 | 45.2 ± 0.5 | 5.8 ± 0.1 | 1.5 ± 0.3 |
| walker2d-medium-expert-v2 | 111.1 ± 0.0 | 111.1 ± 0.0 | 110.8 ± 0.0 | 106.7 ± 0.9 |
| walker2d-medium-replay-v2 | 95.5 ± 0.5 | 91.1 ± 1.1 | 85.6 ± 1.4 | 45.2 ± 1.7 |
| walker2d-medium-v2 | 84.4 ± 0.0 | 84.8 ± 0.1 | 84.1 ± 0.1 | 80.3 ± 0.6 |
Table 5: SAID’s offline RL performance with error bar (S.E.M. 4 random seeds).
| Task | Delay | denoising step=2 | denoising step=5 | denoising step=10 | denoising step=20 |
|---|---|---|---|---|---|
| HalfCheetah | 0 | 14304.3 ± 272.1 | 14735.6 ± 350.1 | 14817.9 ± 362.5 | 14844.8 ± 366.7 |
| 4 | 9065.5 ± 425.4 | 9412.7 ± 446.3 | 9520.5 ± 455.9 | 9647.9 ± 435.5 | |
| 8 | 5478.1 ± 278.6 | 5510.9 ± 269.5 | 5556.2 ± 278.3 | 5565.2 ± 273.7 | |
| 16 | 5192.4 ± 413.2 | 5289.3 ± 425.9 | 5265.9 ± 424.3 | 5288.9 ± 431.8 | |
| Ant | 0 | 5586.9 ± 102.7 | 5900.8 ± 79.7 | 5964.3 ± 117.6 | 5953.4 ± 83.5 |
| 4 | 4540.8 ± 643.6 | 4615.8 ± 676.2 | 4544.5 ± 635.4 | 4585.6 ± 659.5 |
Table 6: SAID’s online RL performance vs. Denoising steps of the diffusion model during inference.
Figure 12: Episodic returns of SAID for each task (delay=0). The histogram shows the performance distribution of 200 episodes of one trained diffusion planning model.
Figure 13: Episodic returns of SAID for each task (delay=4). The histogram shows the performance distribution of 200 episodes of one trained diffusion planning model.
Figure 14: Episodic returns of SAID for each task (delay=8). The histogram shows the performance distribution of 200 episodes of one trained diffusion planning model.
Figure 15: Episodic returns of SAID for each task (delay=16). The histogram shows the performance distribution of 200 episodes of one trained diffusion planning model.