Organizations: Shanghai Institute of AI for Education, East China Normal University · Shanghai Innovation Institute · Hithink Research · School of Data Science and Engineering, East China Normal University · The Hong Kong University of Science and Technology
Long-horizon LLM agents make multi-step decisions, yet outcome-reward-only reinforcement learning provides little direct supervision for individual steps. We introduce AttnSR, a step-level reward augmentation method that derives reward allocations from policy attention extracted during training replay, without a separately trained reward model or additional rollouts. AttnSR traces backward through the policy model attention graph to estimate the relevance of earlier steps to the final response. The terminal step, measured relative to a running baseline, determines whether these rewards reinforce or penalize the selected steps. In a controlled Qwen2.5-7B-Instruct study, AttnSR consistently outperforms the PPO baseline across all evaluated checkpoints on ALFWorld, raising final held-out success from 91.2% to 97.1%. The ablation experiment shows that the benefit depends on how intermediate rewards are assigned, rather than simple accumulation of step rewards over time. AttnSR also improves ScienceWorld at every checkpoint while cutting environment interactions by 29.9%. These results support policy attention as a practical prior for step-level reward allocation in long-horizon agentic RL. Code is available at https://github.com/fairyshine/AttnSR .
Figures & tables
Figure 1: Overview of AttnSR . Actor replay supplies attention for a step graph. Backtracking from the terminal step, recency correction, and layer fusion yield an allocation, and confidence and the centered outcome scale it into signed intermediate rewards. The outcome reward is retained. Graph edges link later steps to earlier dependencies, and episode indices are omitted.
Figure 2: Integration of AttnSR into actor–critic RL. (a) The PPO integration of AttnSR adds intermediate rewards while retaining the outcome reward. (b) Detached actor attention supplies rewards, and pre-update critic values support GAE and target construction. Normalized advantages guide the actor, while estimated returns supervise the critic as in Equation 14 . Targets remain fixed during optimization. Solid arrows carry signals or targets, and dashed arrows denote updates.
Table 3
Figure 3: Environment outcome reward during eight-epoch training (learning rate 1e-6), excluding shaping rewards. These traces supplement the four-epoch held-out comparison.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Metric
Epoch 1
Epoch 2
Epoch 3
Epoch 4
ALFWorld PPO
score (%)
36.50
69.34
89.42
91.24
ALFWorld AttnSR
score (%)
37.59
70.44
91.24
97.08
WebShop PPO
score (%)
63.72
67.53
66.71
70.74
WebShop AttnSR
score (%)
65.74
68.35
69.58
69.71
ScienceWorld PPO
score (%)
22.89
31.49
34.59
44.53
ScienceWorld AttnSR
score (%)
27.12
35.57
41.15
46.35
Appendix
Table 4: Official scores (%) at four checkpoints from the canonical held-out evaluations. ALFWorld official score equals binary success.
Method
Epoch 1
Epoch 2
Epoch 3
Epoch 4
PPO
36.50
69.34
89.42
91.24
Linear Step Reward
43.43
59.85
79.20
92.34
AttnSR
37.59
70.44
91.24
97.08
Appendix
Table 5: ALFWorld success (%) across checkpoints for the canonical positional control. All checkpoints use the full 274-task held-out set.
Pick
Look
Clean
Heat
Cool
Pick2
Overall
n
59
31
58
39
46
41
274
PPO (%)
100.00
96.77
86.21
89.74
86.96
87.80
91.24
AttnSR (%)
100.00
96.77
93.10
100.00
97.83
95.12
97.08
Δ (pp)
0.00
0.00
+6.89
+10.26
+10.87
+7.32
+5.84
Appendix
Table 6: Fine-grained ALFWorld task-family success at epoch 4 on the full held-out set. Columns use the compact family names Pick (pick and place simple), Look (look at object in light), Clean , Heat , Cool , and Pick2 (pick two objects and place). Counts sum to 274 tasks.
Benchmark
Training tasks
Updates/epoch
Updates at epoch 4
ALFWorld
3,553
444
1,776
WebShop
10,587
1,323
5,292
ScienceWorld
3,592
449
1,796
Appendix
Table 7: Canonical training-set sizes and update budgets, per method. One update uses one collected batch of eight trajectories.
Component
Configuration
Base model
Qwen2.5-7B-Instruct
Training framework
VeRL with Megatron-based training
Rollout engine
vLLM
Hardware
8 × H200 GPUs; 140 GB memory per GPU
Fine-tuning
LoRA rank 64; frozen base model
Trainable modules
Actor and critic LoRA adapters
Appendix
Table 8: Framework and model configuration for the canonical runs.
Component
Configuration
Batch size
8
PPO update epochs
1
Rollouts per prompt
1
Advantage estimator
GAE with trajectory-level agent advantage
Actor / critic learning rate
5×10−6 / 5×10−6
Reference-policy KL coefficient
0.02
Appendix
Table 9: PPO, evaluation, and AttnSR hyperparameters used for the main results.
Task
Pairs
Wins
Losses
McNemar p
Δ Success (pp)
Δ Score (pp)
ALFWorld
274
20
4
0.002
+5.84 [+2.55, +9.49]
+5.84 [+2.55, +9.49]
WebShop
500
48
44
0.755
+0.80 [-3.00, +4.40]
-1.04 [-3.36, +1.20]
ScienceWorld
1488
173
136
0.040
+2.49 [+0.20, +4.70]
+1.85 [-0.37, +4.06]
Appendix
Table 10: Paired evaluation-set uncertainty for the final-checkpoint comparison, computed over exact-input-matched episodes. Wins and losses count tasks solved by exactly one policy, and p is the unadjusted two-sided exact McNemar test on those discordant pairs. Intervals are 10,000-resample percentile bootstrap intervals for the mean paired difference (AttnSR − PPO) in points. ALFWorld official score equals binary success. Intervals quantify sampling uncertainty over matched tasks.
Successful
Unsuccessful
Task
Method
n
Mean steps
n
Mean steps
ALFWorld
PPO
250
11.16
24
103.83
AttnSR
266
12.69
8
109.12
WebShop
PPO
201
5.22
299
6.41
AttnSR
205
6.62
295
9.00
ScienceWorld
PPO
520
11.32
1299
38.43
Appendix
Table 11: Full-set trajectory lengths on each benchmark. Success and failure groups differ across policies and are not matched task subsets. Table 12 provides the restricted paired comparison.
Task
Eligible
Both succeed
PPO
AttnSR
Δ steps
ALFWorld
274
246
11.19
11.99
+0.80
WebShop
500
157
5.11
6.19
+1.08
ScienceWorld
1488
268
11.06
7.40
-3.66
Appendix
Table 12: Environment steps on exact-input-matched episodes where both policies succeed. Eligible pairs require the same evaluation row and identical input text, unique within each benchmark set. All ALFWorld and WebShop episodes qualify, while ScienceWorld retains 1,488 of 1,819 episodes, excluding 331 with different or non-unique input text. Results are descriptive.
Task
Method
Parse/ep.
Parse%
Format%
Natural%
Repeat%
ALFWorld
PPO
0.084
8.39
0.00
0.00
0.04
AttnSR
0.044
3.28
0.00
0.00
0.00
WebShop
PPO
0.000
0.00
0.00
0.00
0.04
AttnSR
0.006
0.60
0.00
0.00
0.03
ScienceWorld
PPO
0.240
23.97
0.60
12.15
15.35
AttnSR
0.135
13.47
0.22
18.42
16.30
Appendix
Table 13: Tool-trace and termination diagnostics at epoch 4. Parse/ep. counts exported trace-parser events, and Parse% is the fraction of episodes with any such event. Format% and Natural% are the evaluator terminal flags. Repeat% is the pooled fraction of adjacent parsed-call pairs with identical function names and arguments, and repetition need not imply an invalid or redundant action.
Figure 4: Qwen2.5-3B-Instruct held-out ALFWorld success after each of four PPO training epochs, shown overall (274 tasks), seen (140), and unseen (134). Each point is a full-set greedy evaluation from one training run per method. Lines connect checkpoints without smoothing, and no estimate of training-seed uncertainty is available.
(a) Overall success and termination
Method
Overall / 274
Seen / 140
Unseen / 134
Format errors
Natural term.
TARL
265 (96.72)
135 (96.43)
130 (97.01)
0
0
AgentPRM
252 (91.97)
130 (92.86)
122 (91.04)
0
8
FlowTracer
262 (95.62)
135 (96.43)
127 (94.78)
0
0
Appendix
Table 14: Task outcomes of the additional ALFWorld baselines at the final checkpoint (update 1,776). (a) Successful tasks, with success rates (%) in parentheses. Format and termination columns count episodes. (b) Successful tasks by category, with task counts in the header. Each row of (b) sums to the overall success count in (a).
Symbol
Meaning
i , t
episode index; collected training-batch index
Ki
number of assistant steps in episode i
k , j
step indices, 1≤k<j≤Ki
Si,k
interaction step: assistant message and observation, when present