RefreshGrad: An Online Adaptive Data-sampling Method for Diffusion-based Visuomotor Learning
Organizations: South China University of Technology · Tsinghua University · The Chinese University of Hong Kong, Shenzhen
Abstract
Training vision-language-action (VLA) policies from demonstrations typically samples trajectory segments uniformly or relies on static offline data curation before training. However, we find that demonstration data vary sharply in supervision value across temporal windows, and that each window's value to the policy evolves substantially during training rather than being fixed or captured by raw loss alone. Motivated by this finding, we propose RefreshGrad, an online adaptive data-sampling framework that periodically scores temporal windows from the evolving policy using five policy-native signals and dynamically adjusts their sampling weights during training. RefreshGrad focuses optimization on informative, learnable windows without altering the policy architecture, training objective, or inference cost. Across ten LIBERO-Plus manipulation tasks, RefreshGrad boosts average success from 60.7% (Uniform) and 68.2% (FAKTUAL) to 76.1% under visual perturbations, and from 57.4% and 73.3% to 81.3% in clean scenes.
Figures & tables
| Instruction | Lighting (%) | Noise (%) | Texture (%) | Pert. mean (%) | Clean (%) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| U | FK | RG | U | FK | RG | U | FK | RG | U | FK | RG | U | FK | RG | |
| Place two mugs on plates | 4.0 | 34.0 | 56.0 | 17.0 | 38.0 | 37.0 | 9.0 | 33.0 | 42.0 | 10.0 | 35.0 | 45.0 | 6.0 | 37.0 | 55.0 |
| Place cream cheese in bowl | 48.0 | 40.0 | 63.0 | 56.0 | 62.0 | 59.0 | 56.0 | 46.0 | 65.0 | 53.3 | 49.3 | 62.3 | 64.0 | 57.0 | 79.0 |
| Move bowl from table center to plate | 74.0 | 77.0 | 98.0 | 95.0 | 85.0 | 93.0 | 71.0 | 96.0 | 97.0 | 80.0 | 86.0 | 96.0 | 74.0 | 88.0 | 96.0 |
| Place cream cheese and butter in basket | 87.0 | 46.0 | 80.0 | 77.0 | 45.0 | 80.0 | 74.0 | 76.0 | 86.0 | 79.3 | 55.7 | 82.0 | 76.0 | 64.0 | 83.0 |
| Place ketchup in basket | 68.0 | 94.0 | 83.0 | 84.0 | 85.0 | 89.0 | 68.0 | 90.0 | 81.0 | 73.3 | 89.7 | 84.3 | 85.0 | 98.0 | 91.0 |
| Family (task IDs) | Scene | U | FK | RG | vs. U / FK |
|---|---|---|---|---|---|
| Single-object container (T1, T4, T6) | Pert. | 70.3 | 76.1 | 78.9 | +8.6 / +2.8 |
| Clean | 79.0 | 84.3 | 90.0 | +11.0 / +5.7 | |
| Precision bowl transfer (T2, T9) | Pert. | 76.0 | 79.5 | 90.3 | +14.3 / +10.8 |
| Clean | 67.0 | 87.5 | 94.5 | +27.5 / +7.0 | |
| Multi-object sequential (T0, T3) | Pert. | 44.7 | 45.3 | 63.5 | +18.8 / +18.2 |
| Clean | 41.0 | 50.5 | 69.0 | +28.0 / +18.5 |
| Sampler Variant | Lighting | Noise | Texture | Overall |
| Uniform | 1.0% | 41.0% | 45.0% | 29.0% |
| RefreshGrad (complete) | 48.0% | 66.0% | 84.0% | 66.0% |
| No gradient agreement ( ) | 17.0% | 52.0% | 47.0% | 38.7% |
| No photometric sensitivity ( ) | 36.0% | 49.0% | 61.0% | 48.7% |
| No temporal consistency ( ) | 47.0% | 47.0% | 53.0% | 49.0% |
| No residual disagreement ( ) | 50.0% | 47.0% | 52.0% | 49.7% |
| Task | Uniform | Progress-only | RefreshGrad |
|---|---|---|---|
| 0 | 10.0% | 44.7% | 45.0% |
| 1 | 53.3% | 57.7% | 62.3% |
| 2 | 80.0% | 95.3% | 96.0% |
| 3 | 79.3% | 68.3% | 82.0% |
| 4 | 73.3% | 91.0% | 84.3% |
| 5 | 66.0% | 83.7% | 83.3% |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Role in window allocation |
|---|---|
| Learning progress ( ) | Prioritizes windows where denoising error is actively decreasing between refreshes. |
| Gradient agreement ( ) | Verifies that window gradients align with visually similar windows across episodes. |
| Temporal consistency ( ) | Rewards coherent action-chunk predictions across adjacent overlapping anchors. |
| Residual disagreement ( ) | Captures inter-noise-bank prediction variance normalized by current loss scale. |
| Photometric sensitivity ( ) | Measures photometric action-prediction shifts as a log ratio to current window loss. |
| Rank-based sampling ( ) | Converts utility ranks across the task’s windows into adaptive anchor weights without sensitivity to raw score scale. |
| Parameter (Symbol) | Value / Configuration |
|---|---|
| Policy backbone ( ) | FP32 Diffusion Policy |
| Training budget & batch size | 100,000 steps, batch size 8 |
| Temporal window length ( ) | 40 consecutive anchors |
| Refresh interval ( ) | 10,000 optimizer steps |
| Scoring timesteps ( ) | ( ) |
| Fixed noise banks ( ) | 2 banks ( conditions) |
| Quantity | Measurement |
|---|---|
| Refresh interval | 10,000 optimizer steps |
| Scoring passes per 100k-step run | 9–10 |
| Median eligible windows per task | 1,548 (range 960–3,225) |
| Median cumulative scoring time per task | 3,658 s (61.0 min) |
| Scoring share of training wall time | 10–30% (median 19%) |
| Controlled run, Task 7 (single RTX 6000D) | 1,190 s of 2.1 h (15.7%) |
Visuomotor Policy Learning
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
See all 309 papers