Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at https://github.com/ZJU-REAL/decision-agents.
Figures & tables
Figure 1: One decision model queried in three ways, on the task of putting a heated mug on a table. Actor : choosing the next action. Judge : scoring an action of a recorded trajectory under two contexts (Prefix, Hindsight) and two questions (Value, Progress). Teacher : scoring the actions of policy rollouts, with the centered scores added to the GRPO outcome advantage. The middle row lists the six models we evaluate.
Table 1: Models evaluated in this work. Adaptation describes how a decision model is obtained from its base LLM, and Access indicates whether the model weights are released.
Table 2: Actor success rates (%) on ALFWorld, using the same evaluation tasks as the trained policies (Section 3.3 ). Each Actor is evaluated once per task. Bold marks the best result in each column. Appendix E.1 reports results on broader validation tasks.
Figure 2: Actor success rates (%) on ALFWorld.
Figure 3: (a) Action probabilities with (left) and without (right) the placement subgoal. (b) Success rates with the original prompt, a dependency hint, and a hint plus heated-mug example.
Figure 4: Prefix (left) and Hindsight (right) scores on the same states in Section 3.1.1 .
Figure 5: JEV ( ) as a Judge with Hindsight context on ALFWorld. (a) Mean action scores. (b) Milestone Recall@ R , grouped by the number of milestone actions R in a trajectory. (c) Within-trajectory score gaps relative to no-op actions.
Figure 6: JEV’s action rankings relative to Prefix. Each thin line is a trajectory; the dashed line is the mean.
Figure 7: JEV’s scores for selected steps from a trajectory used in Section 3.2 .
ALFWorld
WebShop
No dependency
With dependency
Method
Pick
Look
Pick2
Clean
Heat
Cool
All
Score
Succ.
Qwen2.5-1.5B-Instruct
GRPO
85.3 ±1.5
53.7 ±8.0
53.5 ±5.6
84.5 ±6.8
78.2 ±7.9
59.7 ±5.0
72.8 ±3.6
75.8 ±3.5
56.8 ±3.8
Prefix + Value
95.2 ±4.4
66.7 ±11.8
73.6 ±4.8
81.5 ±9.8
77.1 ±7.2
66.7 ±2.3
79.0 ±3.6
77.2 ±0.3
67.7 ±0.5
Prefix + Progress
87.6 ±1.6
56.4 ±4.4
72.2 ±4.8
81.5 ±3.7
83.3 ±7.2
76.0 ±4.0
78.3 ±0.4
78.6 ±0.4
66.6 ±0.2
Table 3: Policy success rates (%) on ALFWorld and WebShop, averaged over three evaluation seeds. GRPO results are from GiGPO ( Feng et al., 2025 ) . Bold marks the best result within each policy size.
ALFWorld
WebShop
Method
≤10
≤15
≤20
≤30
≤5
≤10
≤15
Prefix + Value
42.4±1.1
59.3±3.7
69.5±5.1
79.0±3.6
33.7±1.1
60.9±0.7
67.7±0.5
Prefix + Progress
40.7±0.7
56.0±0.4
65.5±0.4
78.3±0.4
45.4±1.4
61.9±0.8
66.6±0.2
Hindsight + Value
41.0±3.4
58.6±1.9
63.6±0.7
69.3±0.7
27.7±2.1
62.2±2.4
68.3±1.5
Hindsight + Progress
47.4±2.2
65.7±1.2
74.5±2.2
82.9±2.9
50.7±1.1
71.1±0.4
71.4±0.7
Table 4: Task completion by action cutoff (%) for 1.5B policies.
ALFWorld
WebShop
Training setting
1.5B
7B
1.5B
7B
Training steps
150
150
150
150
Tasks per batch
16
16
16
16
Rollouts per task
8
8
8
8
Max actions per rollout
30
30
15
15
Max policy prompt / response tokens
2,048 / 512
2,048 / 512
4,096 / 512
4,096 / 512
Table 5: Training settings and Teacher API costs for the four Hindsight + Progress policies.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Table 6: Model identifiers, interfaces, and API providers for the six models evaluated in this work.
Hyperparameter
Value
Hyperparameter
Value
Teacher supervision
Batch configuration
Teacher advantage weight λ
0.5
Tasks per iteration
16
Teacher-score normalization
Mean centering only
Rollouts per task G
8
Default question / context
Progress / Hindsight
PPO mini-batch size
256 / 64
Training iterations
150
Optimization
Sequence and environment
Appendix
Table 7: Training hyperparameters for the policies. Paired values denote ALFWorld / WebShop. Other settings follow GiGPO [ Feng et al., 2025 ] .
Table 8: Actor success rates (%) on ALFWorld, using the same evaluation tasks as the L1 split in Zhang et al. [2026] . Each Actor is evaluated once per task. Bold marks the best result in each column.
Figure 8: Action scores across six Judges on ALFWorld. Mean Hindsight scores under Value and Progress on the same 462 trajectories as Figure 5 . Scores are averaged within each trajectory and then across trajectories. All panels share the [0,1] scale.
Figure 9: Action rankings as more context becomes visible across six Judges. Each thin line shows a trajectory’s mean rank shift relative to Prefix; the gray dashed line is the mean across trajectories. FB denotes immediate feedback.
Step
Action
Prefix
+ FB
25%
50%
75%
Full
1
look
0.47
0.57
0.62
0.61
0.56
0.54
2
look
0.25
0.25
0.24
0.23
0.24
0.23
3
go to coffeemachine 1
0.49
0.72
0.46
0.40
0.46
0.44
4
take mug 3 from coffeemachine 1
0.92
0.92
0.90
0.82
0.83
0.85
5
go to cabinet 1
0.80
0.81
0.69
0.67
0.66
0.68
6
move mug 3 to cabinet 1
0.79
0.90
0.52
0.47
0.52
0.60
Appendix
Table 9: JEV’s Progress scores for the complete heated-mug trajectory. Contexts follow Figure 7 .
Figure 10: Miss rate versus cost for six Judges with Hindsight + Progress on 462 ALFWorld trajectories. (a) Milestone actions. (b) No-op actions. Cost is measured in USD per 1,000 action scores on a logarithmic scale. Lower left is better . Appendix C.2 describes how scoring costs are calculated.
LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.
Department of Statistics and Data Science, National University of Singapore · Department of Statistics and Data Science, University of California, Los Angeles · Department of Data Science, Fudan University