Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at https://github.com/ZJU-REAL/decision-agents.
Figures & tables
Figure 1: One decision model queried in three ways, on the task of putting a heated mug on a table. Actor : choosing the next action. Judge : scoring an action of a recorded trajectory under two contexts (Prefix, Hindsight) and two questions (Value, Progress). Teacher : scoring the actions of policy rollouts, with the centered scores added to the GRPO outcome advantage. The middle row lists the six models we evaluate.
Table 1: Models evaluated in this work. Adaptation describes how a decision model is obtained from its base LLM, and Access indicates whether the model weights are released.
Table 2: Actor success rates (%) on ALFWorld, using the same evaluation tasks as the trained policies (Section 3.3 ). Each Actor is evaluated once per task. Bold marks the best result in each column. Appendix E.1 reports results on broader validation tasks.
Figure 2: Actor success rates (%) on ALFWorld.
Figure 3: (a) Action probabilities with (left) and without (right) the placement subgoal. (b) Success rates with the original prompt, a dependency hint, and a hint plus heated-mug example.
Figure 4: Prefix (left) and Hindsight (right) scores on the same states in Section 3.1.1 .
Figure 5: JEV ( ) as a Judge with Hindsight context on ALFWorld. (a) Mean action scores. (b) Milestone Recall@ R , grouped by the number of milestone actions R in a trajectory. (c) Within-trajectory score gaps relative to no-op actions.
Figure 6: JEV’s action rankings relative to Prefix. Each thin line is a trajectory; the dashed line is the mean.
Figure 7: JEV’s scores for selected steps from a trajectory used in Section 3.2 .
ALFWorld
WebShop
No dependency
With dependency
Method
Pick
Look
Pick2
Clean
Heat
Cool
All
Score
Succ.
Qwen2.5-1.5B-Instruct
GRPO
85.3 ±1.5
53.7 ±8.0
53.5 ±5.6
84.5 ±6.8
78.2 ±7.9
59.7 ±5.0
72.8 ±3.6
75.8 ±3.5
56.8 ±3.8
Prefix + Value
95.2 ±4.4
66.7 ±11.8
73.6 ±4.8
81.5 ±9.8
77.1 ±7.2
66.7 ±2.3
79.0 ±3.6
77.2 ±0.3
67.7 ±0.5
Prefix + Progress
87.6 ±1.6
56.4 ±4.4
72.2 ±4.8
81.5 ±3.7
83.3 ±7.2
76.0 ±4.0
78.3 ±0.4
78.6 ±0.4
66.6 ±0.2
Table 3: Policy success rates (%) on ALFWorld and WebShop, averaged over three evaluation seeds. GRPO results are from GiGPO ( Feng et al., 2025 ) . Bold marks the best result within each policy size.
ALFWorld
WebShop
Method
≤10
≤15
≤20
≤30
≤5
≤10
≤15
Prefix + Value
42.4±1.1
59.3±3.7
69.5±5.1
79.0±3.6
33.7±1.1
60.9±0.7
67.7±0.5
Prefix + Progress
40.7±0.7
56.0±0.4
65.5±0.4
78.3±0.4
45.4±1.4
61.9±0.8
66.6±0.2
Hindsight + Value
41.0±3.4
58.6±1.9
63.6±0.7
69.3±0.7
27.7±2.1
62.2±2.4
68.3±1.5
Hindsight + Progress
47.4±2.2
65.7±1.2
74.5±2.2
82.9±2.9
50.7±1.1
71.1±0.4
71.4±0.7
Table 4: Task completion by action cutoff (%) for 1.5B policies.
ALFWorld
WebShop
Training setting
1.5B
7B
1.5B
7B
Training steps
150
150
150
150
Tasks per batch
16
16
16
16
Rollouts per task
8
8
8
8
Max actions per rollout
30
30
15
15
Max policy prompt / response tokens
2,048 / 512
2,048 / 512
4,096 / 512
4,096 / 512
Table 5: Training settings and Teacher API costs for the four Hindsight + Progress policies.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 6: Model identifiers, interfaces, and API providers for the six models evaluated in this work.
Hyperparameter
Value
Hyperparameter
Value
Teacher supervision
Batch configuration
Teacher advantage weight λ
0.5
Tasks per iteration
16
Teacher-score normalization
Mean centering only
Rollouts per task G
8
Default question / context
Progress / Hindsight
PPO mini-batch size
256 / 64
Training iterations
150
Optimization
Sequence and environment
Appendix
Table 7: Training hyperparameters for the policies. Paired values denote ALFWorld / WebShop. Other settings follow GiGPO [ Feng et al., 2025 ] .
Table 8: Actor success rates (%) on ALFWorld, using the same evaluation tasks as the L1 split in Zhang et al. [2026] . Each Actor is evaluated once per task. Bold marks the best result in each column.
Figure 8: Action scores across six Judges on ALFWorld. Mean Hindsight scores under Value and Progress on the same 462 trajectories as Figure 5 . Scores are averaged within each trajectory and then across trajectories. All panels share the [0,1] scale.
Figure 9: Action rankings as more context becomes visible across six Judges. Each thin line shows a trajectory’s mean rank shift relative to Prefix; the gray dashed line is the mean across trajectories. FB denotes immediate feedback.
Step
Action
Prefix
+ FB
25%
50%
75%
Full
1
look
0.47
0.57
0.62
0.61
0.56
0.54
2
look
0.25
0.25
0.24
0.23
0.24
0.23
3
go to coffeemachine 1
0.49
0.72
0.46
0.40
0.46
0.44
4
take mug 3 from coffeemachine 1
0.92
0.92
0.90
0.82
0.83
0.85
5
go to cabinet 1
0.80
0.81
0.69
0.67
0.66
0.68
6
move mug 3 to cabinet 1
0.79
0.90
0.52
0.47
0.52
0.60
Appendix
Table 9: JEV’s Progress scores for the complete heated-mug trajectory. Contexts follow Figure 7 .
Figure 10: Miss rate versus cost for six Judges with Hindsight + Progress on 462 ALFWorld trajectories. (a) Milestone actions. (b) No-op actions. Cost is measured in USD per 1,000 action scores on a logarithmic scale. Lower left is better . Appendix C.2 describes how scoring costs are calculated.
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
World models help agents assess the consequences of their actions before acting. Many real systems, however, involve multiple actors who make their own decisions, can evolve even when the agent takes no action, and are only partially observed through event logs. We formalize these systems as event worlds and ask whether large language models can predict future observations from the log and the agent's proposed actions. We build complementary benchmarks from human Diplomacy games and real Polymarket trades. Our hierarchical framework compares prediction with and without other actors' actions, then tests multi-step rollouts, predictions for actions never taken, and usefulness to agents. Fine-tuning and more data improve prediction when all actors' recorded actions are given. With only the agent's action, models struggle when other actors strongly affect the outcome, and larger models and longer histories help little. This gap also limits multi-step prediction and agent decisions, with models overestimating plan success. Yet imagined experience and initialization from world models can still improve policy learning. Anticipating other actors' actions remains a central challenge for event world modeling.
Large language models (LLMs) are being used as policies for autonomous decision-making and planning in many domains. Despite their strong reasoning capabilities, LLMs struggle with long-horizon tasks, especially under partial observability. World models are a promising way to enhance policy performance, both during training and inference. During inference, agents currently use world models to simulate the consequences of candidate actions before choosing an action, which can improve decision-making. However, we argue that simulation alone is an incomplete interface for decision-making under partial observability: simulation does not adequately capture uncertainty about the current state, which agents may need for accurate decision-making. We address this limitation with Belief-Based World Models (BB-WMs), which maintain a belief that LLMs can query to access information on what is known and uncertain about the current state. Before developing methods to learn accurate BB-WMs, this paper focuses on a more fundamental question: does exposing a world model's belief directly to an LLM policy improve decision-making? Our results show that giving LLM agents access to beliefs improves task performance under partial observability, while remaining complementary to existing simulation-based world models. Code: https://github.com/skumar-ml/belief-world-models.
A sparse terminal reward tells an agent little about its steps. Much of the missing information is implicit in the environment's dynamics, because each state transition can change the conditions for success long before the reward arrives. A world model trained on real transitions can learn to predict what each action changes, but a predicted observation does not by itself show whether success has become more likely. With a binary terminal reward, PPO's critic makes this judgment by estimating the current policy's probability of success, but only for states the agent reaches. Our method, WM-PPO, elegantly brings the two together. For each action the policy samples, executed or not, the world model predicts the next observation and the critic values the predicted successor. In the policy channel, comparisons among these values give the policy auxiliary advantages; in the value channel, their mean gives the critic an auxiliary target. A shared gate weights both channels according to how reliably comparisons with similar margins predicted real outcomes earlier in training. The policy, critic and world model all learn from the agent's own interaction, with no teacher or privileged information, and only the policy is used at inference. With a comparable number of real environment steps, WM-PPO raises success from PPO's 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld. It matches the strongest PPO refinements we tested on ALFWorld and exceeds the best of them by 6.5 points on ScienceWorld. The trained policy more often completes prerequisite steps before placing an object or committing to an answer.
Chenghao Gao, Zhi Zhang, Dian Jin +2
Department of Statistics and Data Science, National University of Singapore · Department of Statistics and Data Science, University of California, Los Angeles · Department of Data Science, Fudan University