Progress Predicts Policy
Organizations: Department of Computer Science, National University of Singapore
Abstract
A task specifies how the world should change, while a policy specifies what the robot should do. We introduce Progress Predicts Policy (PPP), which learns a language-conditioned scalar field through the commands induced by its differential. We obtain action conditioning by composing a latent progress field with a shared world model and read its action gradient at a fixed reference command. Continuous commands scale this gradient by a learned positive gain, while discrete commands use finite differences of the same scalar. This yields a hybrid policy without a separate action head or iterative search. Fitting demonstration progress values alone does not ensure that action gradients align with expert commands. PPP trains the induced commands by imitation and uses double backpropagation for the continuous gradient readout to shape the field's local control geometry. Demonstration timing calibrates field values on observed states without additional annotation. The continuous readout remains learnable without progress labels but these timing-derived targets support progress monitoring during execution. Experiments in simulated and real robot manipulation show that PPP outperforms behavioral cloning under a shared architecture.
Figures & tables
| Method | Spatial | Object | Goal | Long | Avg. |
| BC | 31.4 | 53.6 | 47.0 | 20.2 | 38.1 |
| BC+Progress | 33.8 | 52.2 | 49.0 | 20.0 | 38.8 |
| Direct-AP | 45.0 | 50.8 | 53.0 | 30.4 | 44.8 |
| PPP-GH | 46.8 | 64.6 | 53.0 | 27.4 | 48.0 |
| PPP w/o settling | 40.4 | 76.2 | 52.4 | 37.0 | 51.5 |
| PPP | 46.0 | 75.4 | 53.6 | 42.2 | 54.3 |
| Method | Success | Success rate | Wrong target | Pose inaccuracy | Gripper inaccuracy |
| PPP | 21/40 | 52.5% | 10 | 8 | 1 |
| BC | 5/40 | 12.5% | 13 | 13 | 9 |
| Method | Success | Success rate | Time (s) |
| PPP | 9/12 | 75.0% | 31.8 |
| BC | 7/12 | 58.3% | 46.0 |
| Method | From | cos | Succ. | |||
| BC | – | – | – | 36.2 | – | – |
| BC+Progress | – | 0.9973 | 0.08 | 36.5 | 0.86 | |
| PPP-GH | pose | 0.9974 | 0.995 | 43.5 | 0.0002 | |
| PPP w/o settling | both | 0.9970 | 0.995 | 50.5 | 0.0075 | |
| Direct-AP | pose | 0.9975 | – | 41.6 | 0.351 | |
| PPP-GH without progress labels: | ||||||
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Loss | ||||
| yes | – | yes | – | |
| yes | yes | yes | yes | |
| – | – | yes | – | |
| yes | yes | – | – |
| Training epoch | ||||||||
| Method | 20 | 30 | 40 | 50 | 70 | 80 | 100 | 120 |
| BC | 20.5 | 20.6 | 26.3 | 27.4 | 33.7 | 35.3 | 37.8 | 38.1 |
| BC+Progress | 20.9 | 24.0 | 26.2 | 30.4 | 34.1 | 34.0 | 39.4 | 38.8 |
| Direct-AP | 26.5 | 32.2 | 31.5 | 37.5 | 39.4 | 41.6 | 40.5 | 44.8 |
| No | 26.2 | 26.6 | 31.6 | 35.2 | 37.7 | 36.9 | 37.8 | 41.8 |
| PPP-GH | 29.7 | 32.8 | 34.8 | 36.3 | 41.9 | 38.9 | 45.1 | 48.0 |
| Run | Act. | Spatial | Object | Goal | Long | Avg. | |
| BC | – | 0 | 28.7 | 53.8 | 45.4 | 17.0 | 36.2 |
| No | no | 0 | 43.8 | 40.5 | 47.7 | 22.3 | 38.6 |
| PPP-GH | yes | 10 | 45.1 | 53.4 | 49.9 | 25.5 | 43.5 |
| PPP w/o settling | yes | 10 | 39.1 | 74.4 | 50.0 | 38.6 | 50.5 |
| Training epoch | ||||||||
| Method | 20 | 30 | 40 | 50 | 70 | 80 | 100 | 120 |
| BC-branch | 27.4 | 35.8 | 40.5 | 34.6 | 43.2 | 40.2 | 47.0 | 46.8 |
| No -branch | 27.2 | 30.8 | 39.3 | 42.8 | 49.8 | 46.6 | 50.9 | 46.3 |
| PPP-GH-branch | 32.0 | 34.7 | 38.0 | 41.0 | 38.9 | 44.0 | 45.7 | 44.4 |
| PPP-branch w/o settling | 37.0 | 42.8 | 49.2 | 47.8 | 49.7 | 52.5 | 55.1 | 55.0 |
| PPP-branch | 36.8 | 44.0 | 48.8 | 46.9 | 51.0 | 55.5 | 55.5 | 56.3 |
| Method | Spatial | Object | Goal | Long | Avg. |
| BC-branch | 46.2 | 54.2 | 57.0 | 30.0 | 46.8 |
| No -branch | 54.4 | 43.2 | 51.0 | 36.6 | 46.3 |
| PPP-GH-branch | 46.8 | 55.2 | 47.8 | 27.8 | 44.4 |
| PPP-branch w/o settling | 45.6 | 88.0 | 48.6 | 37.6 | 55.0 |
| PPP-branch | 50.2 | 86.2 | 49.8 | 39.0 | 56.3 |
| Method | C10 | C20 | C30 | C40 | C50 |
| Seer | 70.30 | 89.10 | 93.30 | 96.35 | 97.30 |
| Seer-AC | 71.15 | 86.90 | 92.35 | 94.35 | 94.70 |
| Seer-AC-PPP | 81.75 | 89.10 | 91.70 | 93.00 | 92.35 |
| Seer-PPP | 54.45 | 80.65 | 91.15 | 96.00 | 97.10 |
| Method | Spatial | Object | Goal | Long | Avg. |
| Seer | 98.8 | 99.0 | 97.8 | 93.6 | 97.3 |
| Seer-AC | 96.6 | 99.8 | 94.6 | 87.8 | 94.7 |
| Seer-AC-PPP | 95.0 | 96.4 | 93.4 | 84.6 | 92.4 |
| Seer-PPP | 99.4 | 99.8 | 98.2 | 91.0 | 97.1 |
| Method | Spatial | Object | Goal | Long | Avg. |
| PPP-AH w/o settling | 31.6 | 76.2 | 50.6 | 30.6 | 47.3 |
| PPP-AH | 35.8 | 73.2 | 52.0 | 31.6 | 48.2 |
| Training epoch | ||||||||
| Method | 20 | 30 | 40 | 50 | 70 | 80 | 100 | 120 |
| PPP-AH w/o settling | 29.8 | 30.2 | 31.1 | 38.8 | 44.5 | 39.9 | 43.1 | 47.3 |
| PPP-AH | 31.4 | 32.9 | 32.2 | 39.5 | 46.9 | 42.4 | 45.6 | 48.2 |
| Rule | Succ. | Override | ||
| Own pick | 36.3 | – | – | 0.0 |
| Progress pick | 35.8 | 0.75 | 72.8 | |
| Progress, margin 0.5 | 36.6 | 0.61 | 9.7 | |
| Random pick | 37.1 | 0.69 | 75.0 |
| Method | Gripper | Spatial | Object | Goal | Long | Avg. | Path (m) | Flips | ||
| PPP w/o settling | – | switch | 40.4 | 76.2 | 52.4 | 37.0 | 51.5 | 1.16 | 0.95 | 2 |
| CEM | 1 | switch | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.87 | 3.28 | 3 |
| CEM | 1 | search | 0.0 | 0.0 | 0.2 | 0.0 | 0.05 | 4.87 | 3.34 | 6 |
| CEM | 4 | switch | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.39 | 3.13 | 3 |
| CEM | 4 | search | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.44 | 3.42 | 18 |
| CEM-Goal | 4 | search | 4.6 | 1.4 | 15.4 | 3.2 | 6.2 | 3.89 | 2.35 | 11 |
| Readout | Cap | Cos. | Expert | |
| PPP (gradient) | – | 0.97 | 2.39 | 52% |
| CEM, | 5.36 | 0.17 | 4.57 | 96% |
| CEM, | 2.0 | 0.59 | 0.54 | 98% |
| CEM, | 1.0 | 0.78 | 0.38 | 100% |
| CEM, | 5.36 | 0.15 | 4.88 | 100% |
| CEM, | 2.0 | 0.24 | 1.54 | 100% |
| Component | Setting |
| Latent and context | , , raw ticks |
| Dynamics predictor | 6 layers, 16 heads, MLP width 2048, dropout 0.1 |
| SIGReg weight | |
| Potential | 4 residual FiLM blocks, width 1024, FiLM hidden width 256 |
| Scalar gain | 2 hidden layers of width 256, scalar output, |
| Progress weights | ; |
| Inference per decision | Training per sample | ||||
| Method | Parameters (M) | Encoder | Readout | Total | Forward + backward |
| BC | 61.816 | 5.757 | 0.181 | 5.938 | 69.805 |
| BC+Progress | 61.817 | 5.757 | 0.181 | 5.938 | 87.197 |
| PPP-GH | 61.976 | 5.757 | 0.357 | 6.115 | 88.188 |
| PPP | 61.712 | 5.757 | 0.718 | 6.475 | 89.079 |
| Layout | green | purple | white | yellow |
| 0 | Success | Success | Pose | Target |
| 1 | Success | Success | Target | Success |
| 2 | Pose | Success | Pose | Success |
| 3 | Target | Success | Pose | Success |
| 4 | Success | Success | Pose | Success |
| 5 | Success | Success | Target | Success |
| Layout | green | purple | white | yellow |
| 0 | Target | Pose | Pose | Success |
| 1 | Success | Target | Gripper | Pose |
| 2 | Target | Pose | Success | Target |
| 3 | Gripper | Gripper | Target | Target |
| 4 | Target | Success | Pose | Target |
| 5 | Gripper | Gripper | Target | Gripper |
| PPP | BC | |||
| Trial | Outcome | Time (s) | Outcome | Time (s) |
| 0 | Success | 13 | Success | 8 |
| 1 | Failure | – | Failure | – |
| 2 | Success | 55 | Failure | – |
| 3 | Success | 33 | Success | 52 |
| 4 | Success | 19 | Success | 51 |
| Field | Epoch | MAE | cos | 95% CI | Orth. | |||||
| Progress-only (a) | 10 | 0.9897 | 0.069 | 0.249 | [0.238, 0.260] | 0.75 | 0.80 | 0.51 | 0.45 | 1.11 |
| Progress-only (a) | 20 | 0.9925 | 0.062 | 0.245 | [0.233, 0.257] | 0.75 | 0.80 | 0.50 | 0.45 | 1.08 |
| Progress-only (a) | 40 | 0.9969 | 0.058 | 0.242 | [0.229, 0.255] | 0.75 | 0.80 | 0.51 | 0.47 | 1.11 |
| Progress-only (a) | 70 | 0.9952 | 0.061 | 0.235 | [0.221, 0.248] | 0.74 | 0.81 | 0.52 | 0.48 | 1.12 |
| Progress-only (a) | 120 | 0.9992 | 0.051 | 0.225 | [0.211, 0.238] | 0.73 | 0.81 | 0.54 | 0.50 | 1.13 |
| Progress-only (b) | 10 | 0.9894 | 0.075 | 0.246 | [0.234, 0.258] | 0.75 | 0.80 | 0.51 | 0.44 | 1.12 |
| Field | step size | ||||||
| 0.03 | 0.1 | 0.3 | 1 | 2 | 4 | ||
| Progress-only (a) | ratio | 1.01 | 1.02 | 1.04 | 0.99 | 0.83 | 0.55 |
| cos | 1.00 | 1.00 | 0.98 | 0.84 | 0.67 | 0.44 | |
| 0.51 | 0.56 | 0.68 | 1.07 | 1.43 | 1.71 | ||
| pref. | 0.73 | 0.73 | 0.74 | 0.77 | 0.83 | 0.85 | |
| BC+Progress | ratio | 1.31 | 1.29 | 0.32 | 0.11 | 0.07 | 0.04 |
| Setting | Readout | Hit | Chance | Swap cos |
| Robot | PPP (300), command | 0.52 | 0.25 | 0.40 |
| BC (300), cmd. | 0.51 | 0.25 | 0.15 | |
| LIBERO-Goal | Progress-only (10), grad. | 0.13 | 0.10 | 0.31 |
| Progress-only, grad. | 0.10 | 0.10 | 0.57 | |
| BC+Progress, grad. | 0.10 | 0.10 | 0.33 | |
| BC+Progress, BC cmd. | 0.33 | 0.10 | 0.71 |
| End | AUROC | ||||||
| Scalar | Succ. | S | F | S | F | end | s |
| PPP | 51.5 | 1.00 | 0.67 | 0.84 [.78,.90] | 0.78 [.71,.84] | ||
| BC+Progress | 38.8 | 1.00 | 0.69 | 0.89 [.84,.93] | 0.75 [.69,.81] | ||
| No | 41.8 | 0.68 | 0.30 | 0.73 [.66,.79] | 0.56 [.49,.63] | ||
| Field | States | pref. | pref. | pref. random | ||||
| PPP | demo | 0.72 | 0.74 | 0.49 | 0.26 | 0.26 | 6.25 / 0.00 | 1.29 / 0.99 |
| own rollouts | 0.72 | – | 0.50 | 0.26 | – | 6.54 / 0.08 | 2.18 / 0.86 | |
| PPP-GH | demo | 0.75 | 0.76 | 0.49 | 0.28 | 0.29 | 0.23 / 0.15 | 1.18 / 1.06 |
| own rollouts | 0.68 | – | 0.50 | 0.23 | – | 0.70 / 0.21 | 1.29 / 0.79 | |
| PPP-GH, frozen WM | demo | – | – | – | 0.31 | 0.30 | 0.29 / 0.05 | – |
| PPP’s rollouts | – | – | – | 0.39 | – | 0.45 / 0.45 | – |
Robot Policy Learning
Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.
See all 403 papers