Learning to Act with World Models by Valuing What Comes Next
Organizations: Department of Statistics and Data Science, National University of Singapore · Department of Statistics and Data Science, University of California, Los Angeles · Department of Data Science, Fudan University
Abstract
A sparse terminal reward tells an agent little about its steps. Much of the missing information is implicit in the environment's dynamics, because each state transition can change the conditions for success long before the reward arrives. A world model trained on real transitions can learn to predict what each action changes, but a predicted observation does not by itself show whether success has become more likely. With a binary terminal reward, PPO's critic makes this judgment by estimating the current policy's probability of success, but only for states the agent reaches. Our method, WM-PPO, elegantly brings the two together. For each action the policy samples, executed or not, the world model predicts the next observation and the critic values the predicted successor. In the policy channel, comparisons among these values give the policy auxiliary advantages; in the value channel, their mean gives the critic an auxiliary target. A shared gate weights both channels according to how reliably comparisons with similar margins predicted real outcomes earlier in training. The policy, critic and world model all learn from the agent's own interaction, with no teacher or privileged information, and only the policy is used at inference. With a comparable number of real environment steps, WM-PPO raises success from PPO's 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld. It matches the strongest PPO refinements we tested on ALFWorld and exceeds the best of them by 6.5 points on ScienceWorld. The trained policy more often completes prerequisite steps before placing an object or committing to an answer.
Figures & tables
| Tasks seen | ||||
| Method | Episodes | Feedback beyond PPO’s advantage | ALFWorld | ScienceWorld |
| WM-PPO | 64 (1 + 3) | candidates valued by the critic at predicted successors; comparisons as auxiliary advantages, mean as critic target; both weighted by real branches | 43% | 100% |
| PPO-160, PPO-256 | 160 / 256 | – | 100% | 100% |
| GRPO, GiGPO | 20 8 | group-relative advantages instead of a critic | 14% | 51% |
| GRPO, GiGPO | 40 4 | as above, more tasks, fewer episodes each | 27% | 91% |
| CAPO, SO-PPO, Turn-ratio PPO | 160 | action- or turn-level ratios and clipping | 100% | 100% |
| ALFWorld, iteration 36 | ScienceWorld, iteration 60 | ||||
| Method | Runs | Greedy | pass@1 | Greedy | pass@1 |
| WM-PPO (ours) | 3 / 3 | 87.5 0.9 | 87.8 1.7 | 52.9 2.3 | 51.6 4.8 |
| WM-PPO (policy channel only) | 3 / 3 | 88.4 2.3 | 89.4 1.6 | 47.2 2.8 | 47.2 1.7 |
| Standard training and refinements | |||||
| PPO-160 | 3 / 3 | 63.5 9.8 | 71.0 6.0 | 39.3 1.5 | 42.6 2.5 |
| PPO-256 | 3 / 3 | 83.1 3.1 | 84.8 1.2 | 43.8 8.7 | 43.5 6.6 |
| Method | What is predicted | How it is evaluated | Where the signal enters | Deployed agent |
|---|---|---|---|---|
| ECHO, PaW, DGS-WM | next observation, by the policy | – | auxiliary loss on the policy | policy |
| Early Experience, SPA | future or true states | – | supervised stage before RL | policy |
| RWML | next state | similarity to the true state | RL stage before task RL | policy |
| WMA, WebDreamer | change caused by each candidate | value model or prompted judge | action selection at test time | policy, model, evaluator |
| ITP | multi-step imagined rollout | the policy’s reflection (training-free variant) | context of the policy; SFT, then actor-critic RL | policy and model |
| CoMAP | imagined next state | the policy’s own reflection | supervised refinement | policy and model |
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Learns or computes | From | Used for |
|---|---|---|---|
| Policy | action distribution given the state | ordinary PPO rows of real executed steps (tree GAE) and auxiliary rows of evaluated candidates | acting; alone at test time |
| World model | next observation after an action | cross-entropy on real executed transitions of chains and branches, every iteration | predicting the consequences of the candidates |
| Critic | expected task return from a state | Monte-Carlo returns of the trees, the sibling-centered term, and, at each evaluated state , the mean transition value of its candidates (value channel) | GAE of real steps; transition values of predicted successors |
| Gate | weight per margin bin | real branch pairs of the previous iterations | scaling the auxiliary advantages and the value-channel loss |
| Base model | Qwen3-4B-Instruct-2507, non-thinking mode, frozen |
|---|---|
| Adapters | policy, critic, world model: separate LoRA, rank 64, , all linear layers |
| Critic head | separate LoRA with a linear value head on the last prompt token, zero-initialized; inputs left-truncated to 8,192 tokens |
| Rollouts | temperature 1, at most 256 response tokens, at most 40 actions per episode |
| Tasks per iteration | 64; one chain and three branches each |
| Candidates | per chain step, first one executed |
| Branches | branch step uniform over the chain; up to 8 samples to obtain a different action |
| It. | Policy | pass@1 | Majority of 4 | True next state | Predicted next state |
|---|---|---|---|---|---|
| 24 | PPO-160 | 68.9 | 68.8 (-0.1) | 74.6 (+5.8 † ) | 73.9 (+5.0 † ) |
| PPO-256 | 72.6 | 69.7 (-2.9) | 81.0 (+8.4 † ) | 77.2 (+4.6 † ) | |
| Wide-Tree-PPO | 64.0 | 59.5 (-4.5 † ) | 79.0 (+15.0 † ) | 76.1 (+12.1 † ) | |
| 36 | PPO-160 | 77.4 | 76.6 (-0.7) | 84.9 (+7.5 † ) | 82.3 (+4.9 † ) |
| PPO-256 | 86.1 | 85.4 (-0.7) | 90.1 (+4.0 † ) | 89.2 (+3.1 † ) | |
| Wide-Tree-PPO | 83.8 | 83.0 (-0.8) | 91.1 (+7.2 † ) | 86.9 (+3.0 † ) |
| Baseline | Setting | Greedy | pass@1 |
|---|---|---|---|
| CAPO | action-ratio clip 0.001 (original) | 46.0 | 63.9 |
| clip 0.1 | 82.8 | 84.1 | |
| clip 0.3 (selected by the pre-set rule; main table) | 87.2 | 86.7 | |
| Turn-ratio PPO | clip (released code) | 56.6 | 74.6 |
| clip 0.1 (selected) | 85.8 | 85.0 | |
| GRPO | 40 tasks 4 | 54.4 | 60.0 |
| Baseline | Setting | Outcome | Greedy | pass@1 |
|---|---|---|---|---|
| CAPO | clip 0.3 (ALFWorld setting) | diverged, it. 14 | – | – |
| clip 0.1 | diverged, it. 28; it. 18 in Table 2 | 21.1 | 20.0 | |
| clip 0.03 | diverged, it. 18 | – | – | |
| SO-PPO | clip 0.2 (published) | collapsed after it. 36 | 15.8 | 16.2 |
| clip 0.02 (KL rule; Table 2 ) | it. 60 | 46.5 | 49.4 | |
| clip 0.1 (clipped-token rule) | no divergence; it. 60 (best 22.9%, it. 36) | 4.2 | 10.8 |
| Model | ALFWorld | ScienceWorld |
|---|---|---|
| Student at the common start (iteration 12) | 27.0 | 20.8 |
| WM-PPO after training (iteration 36 / 60, greedy) | 87.5 | 52.9 |
| Qwen3-32B, non-thinking, greedy (the teacher used) | 13.5 | 26.4 |
| sampled at temperature 1, three runs | 35.0 / 34.0 / 33.2 | – |
| tool-call format also accepted as an action | 48.5 | – |
| Qwen3-30B-A3B-Instruct-2507, greedy | 28.8 | 25.4 |
| iteration 24 | iteration 36 | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | Steps | Greedy | pass@1 | pass@3 | Steps | Greedy | pass@1 | pass@3 |
| WM-PPO (ours) (3) | 60k | 55.7 | 60.8 | 73.5 | 102k | 87.5 | 87.8 | 93.4 |
| WM-PPO (policy channel only) (3) | 58k | 68.4 | 70.9 | 82.5 | 97k | 88.4 | 89.4 | 94.6 |
| PPO-160 (3) | 57k | 47.1 | 54.3 | 67.9 | 105k | 63.5 | 71.0 | 81.1 |
| PPO-256 (3) | 88k | 63.6 | 69.3 | 81.9 | 150k | 83.1 | 84.8 | 92.9 |
| GRPO | 58k | 53.3 | 57.2 | 73.0 | 113k | 59.5 | 67.8 | 83.6 |
| iteration 24 | iteration 36 | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | Steps | Greedy | pass@1 | pass@3 | Steps | Greedy | pass@1 | pass@3 |
| WM-PPO (ours) (3) | 34k | 28.5 | 27.7 | 35.9 | 67k | 38.0 | 38.0 | 45.8 |
| WM-PPO (policy channel only) (3) | 32k | 26.6 | 27.2 | 35.2 | 67k | 33.0 | 32.9 | 42.3 |
| WM-PPO, terminal-only value target | 31k | 29.9 | 31.0 | 37.7 | 64k | 34.5 | 38.0 | 45.8 |
| PPO-160 (3) | 25k | 26.5 | 26.2 | 33.2 | 54k | 32.0 | 31.5 | 39.1 |
| PPO-256 (3) | 38k | 26.8 | 26.5 | 34.7 | 75k | 33.7 | 32.3 | 40.1 |
| Environment | It. | Method | Run 1 | Run 2 | Run 3 |
|---|---|---|---|---|---|
| ALFWorld | 36 | WM-PPO | 88.3 / 89.7 | 87.6 / 86.9 | 86.5 / 86.7 |
| WM-PPO (policy channel only) | 89.4 / 90.8 | 85.8 / 87.7 | 90.1 / 89.8 | ||
| PPO-160 | 70.1 / 77.4 | 52.2 / 65.5 | 68.2 / 70.3 | ||
| PPO-256 | 83.2 / 86.1 | 86.1 / 83.8 | 79.9 / 84.4 | ||
| Q-PPO | 85.8 / 86.1 | 92.7 / 92.3 | 74.8 / 87.2 | ||
| Wide-Tree-PPO | 76.6 / 83.8 | 88.7 / 86.6 | 70.1 / 76.5 |
| Environment | Method | Real steps | Policy samples | WM generations | Wall (min) | GPU h |
|---|---|---|---|---|---|---|
| ALFWorld | WM-PPO | 4,232 | 8,914 | 1,930 | 40 | 4.6 |
| PPO-160 | 4,381 | 4,381 | 0 | 32 | 1.7 | |
| PPO-256 | 6,252 | 6,252 | 0 | 28 | 2.3 | |
| Wide-Tree-PPO | 4,399 | 4,399 | 0 | 31 | 1.5 | |
| Q-PPO | 4,152 | 8,795 | – | 46 | 2.7 | |
| ScienceWorld | WM-PPO | 2,674 | 5,205 | 1,483 | 55 | 6.4 |
| iteration 24 | iteration 36 | |||||
|---|---|---|---|---|---|---|
| Method | Steps | Greedy | pass@1 | Steps | Greedy | pass@1 |
| WM-PPO (ours) | 20k | 80.3 | 69.8 | 36k | 87.0 | 84.4 |
| ProAct (MC-PPO) | 18k | 77.3 | 70.9 | 33k | 77.3 | 68.1 |
| PPO-160 | 19k | 62.0 | 53.4 | 36k | 67.7 | 63.2 |
| Agreement by margin bin | Gain per pair | ||||||||
| Environment | Valued by | Iterations | Pairs | Agreement | small | middle | large | small+middle | large |
| ScienceWorld | WM-PPO | 13–24 | 415 | 0.71 | 0.61 | 0.55 | 0.78 | +0.008 | +0.077 |
| 25–36 | 549 | 0.73 | 0.48 | 0.58 | 0.82 | +0.007 | +0.145 | ||
| 37–48 | 626 | 0.77 | 0.49 | 0.58 | 0.88 | +0.008 | +0.199 | ||
| 49–60 | 654 | 0.76 | 0.55 | 0.58 | 0.84 | +0.010 | +0.222 | ||
| Policy channel only | 13–24 | 397 | 0.67 | 0.49 | 0.65 | 0.71 | +0.009 | +0.060 | |
| Decisive pairs | WM-PPO | Policy ch. only | Q-PPO | WM-PPO Q-PPO |
|---|---|---|---|---|
| All | 76.2 (867) | 72.3 (755) | 69.9 (1017) | +6.3 [+1, +11] |
| Largest-margin bin | 84.4 (604) | 84.7 (483) | 76.3 (683) | +8.1 [+3, +14] |
| A focus on command involved | 85.5 (543) | 84.0 (451) | 77.7 (623) | +7.8 [+2, +13] |
| No focus on command | 60.8 (324) | 54.9 (304) | 57.6 (394) | +3.2 [ , +10] |
| One successor predicted to end (1 / 0) | 94.7 (378) | 94.5 (328) | – | – |
| Both continue, valued by the critic | 62.0 (489) | 55.3 (427) | – | – |
| iteration 36 | iteration 48 | iteration 60 | |||||
|---|---|---|---|---|---|---|---|
| Method | Greedy | pass@1 | Greedy | pass@1 | Greedy | pass@1 | |
| WM-PPO (ours) | calibrated | 40.8 | 41.7 | 45.8 | 46.5 | 54.9 | 56.5 |
| No gate | (1, 1, 1) | 35.6 | 35.9 | 39.1 | 41.4 | 44.0 | 43.1 |
| Top bin | (0, 0, 1) | 34.9 | 34.0 | 43.0 | 45.3 | 48.6 | 48.6 |
| PPO-160 | – | 32.4 | 29.8 | 40.5 | 40.4 | 40.5 | 43.4 |
| Environment | Iterations | Decision pts | Non-zero | Full | Mean : small / middle / large | Agreement | Pairs | Gain: small / middle / large |
|---|---|---|---|---|---|---|---|---|
| ALFWorld | 13–24 | 846 | 470 | 45 | 0.17 / 0.36 / 0.42 | 0.61 | 12 | +0.002 / +0.033 / +0.051 |
| 25–36 | 593 | 339 | 55 | 0.11 / 0.33 / 0.70 | 0.57 | 12 | -0.003 / +0.005 / +0.046 | |
| 37–48 | 448 | 201 | 5 | 0.18 / 0.03 / 0.38 | 0.56 | 7 | +0.008 / -0.013 / +0.026 | |
| 49–55 | 470 | 204 | 6 | 0.17 / 0.01 / 0.26 | 0.54 | 7 | +0.001 / -0.004 / +0.029 | |
| ScienceWorld | 13–24 | 540 | 327 | 56 | 0.15 / 0.27 / 0.64 | 0.71 | 12 | +0.009 / +0.010 / +0.078 |
| 25–36 | 542 | 327 | 98 | 0.11 / 0.25 / 0.98 | 0.73 | 15 | -0.002 / +0.016 / +0.149 |
| Environment | World model | Iterations | Steps | Exact | Rejections: recall / precision | Endings foreseen |
|---|---|---|---|---|---|---|
| ScienceWorld | online | 13–24 | 16,602 | 0.84 | 0.99 / 0.99 | 0.74 |
| 25–36 | 18,066 | 0.86 | 0.99 / 0.99 | 0.73 | ||
| 37–48 | 17,036 | 0.86 | 0.99 / 0.99 | 0.74 | ||
| 49–60 | 16,452 | 0.85 | 0.99 / 0.99 | 0.75 | ||
| fixed | 13–24 | 5,114 | 0.81 | 0.97 / 0.98 | 0.51 | |
| 25–36 | 4,543 | 0.80 | 0.96 / 0.98 | 0.52 |
| ScienceWorld | ALFWorld | |||||
| AUC | Reinforced | Same state | AUC | Reinforced | Same state | |
| PPO-160, tree GAE | 0.51 | 39% | – | 0.57 | 34% | – |
| Wide-Tree-PPO, tree GAE | 0.51 | 45% | 58% (476) | 0.62 | 31% | 86% (21) |
| WM-PPO (policy channel only) | ||||||
| tree GAE | 0.48 | 40% | 56% (476) | 0.65 | 29% | 74% (43) |
| comparisons | – | – | 67% (1,560) | – | – | 66% (116) |
| ScienceWorld | ||||||
| Method | Useless | Useful | Mean: useless / useful | Gap | AUC | Useless with |
| PPO-160 | 11,200 | 14,624 | / | 0.10 | 0.51 | 39% |
| Wide-Tree-PPO | 13,167 | 13,739 | / | 0.08 | 0.51 | 45% |
| WM-PPO | 11,387 | 14,665 | / | 0.11 | 0.51 | 43% |
| policy channel only | 11,410 | 17,150 | / | 0.02 | 0.48 | 40% |
| ALFWorld (wasted = Nothing happens + revisits) | ||||||
| Rejected commands | Repeated commands | Ended by a wrong focus | ||||
|---|---|---|---|---|---|---|
| Method | it. 13–15 | it. 58–60 | it. 13–15 | it. 58–60 | it. 13–15 | it. 58–60 |
| WM-PPO (3 runs) | 49.9 | 29.5 | 65.1 | 54.0 | 64.0 | 28.6 |
| WM-PPO (policy channel only, 3 runs) | 48.5 | 30.5 | 64.6 | 55.3 | 65.9 | 32.5 |
| PPO-160 (3 runs) | 48.1 | 37.5 | 65.3 | 57.7 | 62.1 | 36.2 |
| PPO-256 (3 runs) | 47.2 | 49.9 | 65.3 | 65.0 | 62.4 | 33.6 |
| OCSD (self teacher) | 51.5 | 43.8 | 65.7 | 66.7 | 60.1 | 28.3 |
| Method | Success | Steps (successful) | Repeated actions | go to before first take (successful) |
|---|---|---|---|---|
| WM-PPO (ours) | 83.9 | 16.5 (12.0) | 42.4 | 3.4 |
| WM-PPO (policy channel only) | 86.8 | 16.0 (12.4) | 38.7 | 3.6 |
| PPO-160 | 64.7 | 23.3 (14.1) | 53.2 | 4.5 |
| PPO-256 | 83.2 | 18.1 (13.7) | 42.9 | 3.9 |
| Q-PPO | 86.5 | 16.4 (12.7) | 40.0 | 3.8 |
| Wide-Tree-PPO | 77.3 | 19.5 (13.4) | 47.1 | 4.0 |
| Game | WM-PPO | PPO-160 | What differs |
|---|---|---|---|
| ALFWorld: heat an apple, put it on the dining table | 3/3 | 0/3 | WM-PPO takes the apple and heats it in the microwave (7 steps); PPO-160 never takes the apple and circles the microwave, fridge and stove burners |
| ALFWorld: clean a cloth, put it in a cabinet | 3/3 | 0/3 | WM-PPO finds the cloth in the first cabinet it opens and cleans it at the sink (8 steps); PPO-160 alternates between sink and cabinets without taking it |
| ALFWorld: put a pepper shaker in a drawer | 3/3 | 0/3 | WM-PPO checks the countertops, where the pepper shaker is (7 steps); PPO-160 opens drawers and cabinets in turn and never finds it |
| Candidate mix bowl | Candidate focus on orange paint | |
|---|---|---|
| Current observation (both) | You pour the contents of the wood cup into the bowl. [Current room] … a table. On the table is: a bowl (containing red paint, yellow paint). … [score: 40] | |
| World model predicts | Yellow and red paint mix to produce orange paint. [Current room] … a bowl (containing orange paint). … [score: 50] | No known action matches that input. [Current room] … a bowl (containing red paint, yellow paint). … [score: 40] |
| Critic reads | task and rules (copied); step counter and last five observation–action pairs, now ending with the real current observation and the candidate (updated by the program); as the current observation (generated); referable objects from before the step, still without orange paint (copied) | |
| Comparison | , in the large-margin bin | |
| ALFWorld | ScienceWorld | Source | |
|---|---|---|---|
| Policy and critic input | task; step counter; last 5 observation–action pairs; current observation; admissible actions | task and rules; step counter; last 5 observation–action pairs; current observation (action feedback, room, inventory, score); action templates; referable objects | real interaction |
| World-model input | task; initial observation; all previous observations and actions; current observation; admissible actions; candidate | task; last 8 actions, without observations; current observation; referable objects; candidate | real interaction |
| World-model output | next observation as the environment prints it | action feedback, [Current room] , [Inventory] , [score: S] , and [task completed] or [task failed] when the simulator ends the task | generated |
| Successor state | policy format: window of recent pairs ends with (real current observation, candidate); as current observation; admissible actions from before the step | the same; referable objects from before the step | program (window, step counter); copied (task, lists); generated ( ) |
| Predicted ending | none (observations report no endings) | an end marker in replaces the critic’s value by 1 (completed) or 0 (failed) | parsed from |
| Progress score | none | printed in every observation; read by the policy and the critic; part of the world model’s input and of its next-observation target; not used as a reward or as a separate training target | environment text |
Model-Based RL
Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.
See all 273 papers