Reinforcement learning now trains language-model agents that act over dozens of steps in live environments. The gains are large, and they are read as better decision-making. An agent in a closed loop writes its own inputs. Each observation follows from its own earlier actions, so the states it meets late in an episode are partly of its own making. An SFT checkpoint and an RL checkpoint are then scored from different states, even on identical tasks. Endpoint success mixes two changes: where the agent arrives, and what it does once it is there. Restricting the comparison to states both policies reach does not separate them. That restriction selects on an outcome, and in our data it flips the sign of the effect. We introduce checkpoint handoff, an evaluation protocol that clones a state one released checkpoint reached and hands it to another, with no retraining. Crossing a reacher role and a solver role over SFT and RL splits an endpoint gain into REACH and SOLVE. REACH is how often a policy arrives at a state the environment confirms is a fixed number of actions from success. SOLVE is how often it finishes from an identical cloned state. Across two benchmarks and two independently released pipelines, the reacher by solver interaction is positive in all five conditions. An RL history is worth more to an RL solver than the same history is to an SFT solver. On ALFWorld, RL improves both terms, and the SFT solver never succeeds where the RL solver fails. Independent REACH and SOLVE gaps predict the aggregate interaction. Handoff asks only that one checkpoint's history can be replayed under another, so long-horizon evaluation can report arrival and completion beside endpoint success.
Figures & tables
Figure 1: The endogenous state problem. Two checkpoints receive one task and act. Their early actions leave them in different states, and each is then scored from the state it produced.
Figure 4: Checkpoint handoff and interaction prediction. (A) A benchmark-specific rule selects a handoff from each reacher’s trajectory. Both solvers continue with matched inputs, yielding four endpoint success rates and their interaction. (B) On ALFWorld, target arrival gaps and independent Solve gaps predict the interaction for comparison with the observed value (Equation 9 ).
Endpoint success (%)
Setting
n
W =SFT
W =SFT
W =RL
W =RL
Interaction
R =SFT
R =RL
R =SFT
R =RL
(pp)
TravelPlanner / Agent-STAR
1.5B
180
6.1
14.4 +8.3
7.8
33.9 +26.1
+ 17.8 [9.4, 26.1]
3B
180
12.2
24.4 +12.2
22.2
48.9 +26.7
+ 14.4 [6.1, 22.8]
7B
180
20.0
29.4 +9.4
38.9
58.9 +20.0
+ 10.6 [0.6, 20.6]
Table 1: Endpoint success for the four reacher ( W ) by solver ( R ) assignments. Subscripts on the R =RL columns give the solver effect under that reacher, and the last column is their difference, Iτ of Equation 4 . Brackets are 95% paired-bootstrap intervals.
(a) REACH
(b) SOLVE from identical replayed states
Split
Reacher
Arrival %
Steps
States n
R =SFT %
R =RL %
RL − SFT (pp)
Seen split
SFT
15.8
41.6
19
31.6
57.9
26.3 —
RL
81.7
15.7
98
53.1
79.6
26.5 —
Pooled
117
49.6
76.1
26.5 [11.2, 41.3]
Unseen split
SFT
13.3
44.0
16
43.8
93.8
50.0 [15.4, 83.3]
RL
82.5
17.8
99
58.6
96.0
37.4 [27.7, 47.0]
Table 2: Reach and Solve on ALFWorld. Each row is a reacher. (a) Arrival at F2 over 120 trajectories, with mean actions to arrival or stopping. (b) Completion by each solver from identical replayed states that the reacher produced. Brackets are task-stratified paired-bootstrap 95% intervals; a dash marks a stratum too small for one.
Split
Solver
Bridge %
Finish ∣ bridge %
Complete %
RL only
SFT only
Seen split
SFT
72.6 [64.1, 80.3]
68.2 [57.6, 77.6]
49.6 [40.2, 59.0]
41
10
RL
92.3 [87.2, 96.6]
82.4 [75.0, 88.9]
76.1 [68.4, 83.8]
Unseen split
SFT
72.2 [63.5, 80.0]
78.3 [68.7, 86.7]
56.5 [47.8, 65.2]
45
0
RL
97.4 [93.9, 100.0]
98.2 [95.5, 100.0]
95.7 [91.3, 99.1]
Table 3: Solve split into choosing a bridge action and finishing from it. Shares of reached states with 95% bootstrap intervals; the last two columns count states completed by only one solver ( n=115 on the unseen split, 117 on the seen split).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Checkpoint
Revision
TravelPlanner
xxwu/Agent-STAR-SFT-1.5B
b3d55c10e763231d13f591bfa2ad2cb99d19b981
TravelPlanner
xxwu/Agent-STAR-RL-1.5B
279b7c75586b2a6ada3eac5a5f406a147522c787
TravelPlanner
xxwu/Agent-STAR-SFT-3B
dfb0d943ad8c7a1d6a5e38064a75a6f4e8b4e503
TravelPlanner
xxwu/Agent-STAR-RL-3B
705723c62e82ed15844783f221572ea1c78f4185
TravelPlanner
xxwu/Agent-STAR-SFT-7B
945532c2ca454caaf1f62d1c42916cbec0a2973a
TravelPlanner
xxwu/Agent-STAR-RL-7B
7013d9fd0fff692d8f6f352ae5db4351d174eab9
Appendix
Table 4: Released checkpoints and fixed revisions.
Design question
Protocol control
Reported quantity
Are states equally close to success?
Exhaustive F2 replay check that never consults a checkpoint, and a matched two-action solver budget
Arrival at a common solvable frontier
Do solvers receive the same input?
Exact replay with equal environment observations, history, admissible actions, budget, prompt, and paired seeds
Same-state Solve effect
Are difficult non-arrivals removed?
Every natural non-arrival remains a zero endpoint outcome; reached-state results are reported separately
Deployed-agent endpoint interaction
Does scoring introduce the interaction?
Environment-native won in ALFWorld; archived Formatter replies and exact official-score recomputation in TravelPlanner
Stable benchmark outcomes
Is the interaction prediction fitted to the target split?
Conversion is measured on an independently selected sample before it is combined with target arrival
Out-of-sample interaction prediction
Appendix
Table 5: Design questions and the protocol control that answers each.
Target-split arrival %
External SOLVE %
Interaction (pp)
Task type
SFT
RL
SFT
RL
Pred.
Obs.
Resid.
look at obj in light
20.0
80.0
10.0
35.0
15.0
40.0
25.0
pick & place
35.0
70.0
85.0
100.0
5.2
− 5.0
− 10.2
clean then place
10.0
70.0
20.0
100.0
48.0
60.0
12.0
cool then place
0.0
100.0
80.0
100.0
20.0
25.0
5.0
heat then place
15.0
100.0
60.0
80.0
17.0
5.0
− 12.0
Appendix
Table 6: Per-task-type interaction prediction (Equation 9 ). Arrival is from the target valid_unseen split and the Solve profile from the independent sample. The overall row is the unweighted mean over types, the stratified bootstrap unit; dashes mark cells it leaves undefined.
(a) Arrival by reacher action budget, seen split
Budget (steps)
Reacher
Arrival %
95% CI
8
SFT
10.8
[4.2, 17.5]
8
RL
47.5
[34.2, 60.0]
16
SFT
14.2
[7.5, 20.8]
16
RL
75.8
[62.5, 88.3]
48
SFT
15.8
[8.3, 22.5]
Appendix
Table 7: Frontier arrival by reacher budget on valid_seen . The bolded row contrasts RL at 8 actions with SFT at 48. Brackets are task-stratified paired-bootstrap 95% intervals from 100,000 resamples.
Step
SFT solver
RL solver
1
look “I should head straight to the nearest shelf. I need to locate a shelf first … I’ll scan the room.”
go to shelf 1
2
go to shelf 1
move pencil 1 to shelf 1
Outcome
budget exhausted, not won
won
Appendix
Table 8: Two solvers on one cloned state. Bridge actions in blue. The SFT solver’s first-step reasoning is quoted from its archived reply.
Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.
RLHF evaluation should track how failures emerge, where they localize, and which warning signals appear before external quality degrades. We study this problem with a compact RLHF pipeline built for this paper, including PPO, DPO, uncertainty-penalized PPO (UP-PPO), reward-model uncertainty, approximate policy drift, diversity and repetition diagnostics, and two external LLM judges. Rather than treating reward hacking as a single terminal event, we classify matched checkpoint and prompt-level transitions by the directions of learned reward R_phi, judge scores R_dag and R2_dag, and their average R_dag. The main empirical findings are that aggressive PPO produces the clearest localized reward-hacking signal, UP-PPO reduces but does not eliminate that signal, row-level diagnostics reveal failures hidden by checkpoint averages, and pre-transition features partially anticipate future localized reward hacking. The central conclusion is methodological: RLHF failures are training dynamics that can be classified, localized, and partially anticipated, not only final-model pathologies. The repository is available at github.com/zabahana/rlhf-failure-modes-diagnostics. The pipeline is also deployed as a live interactive web demo for model comparison and diagnostic views at rlhf-failures.zelalem.ai.
Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan +1
Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training. SFT is expected to provide a useful behavioral prior for RL to further enhance model capabilities. However, checkpoints with excessive SFT often show limited improvement during RL. We attribute this failure to the loss of model plasticity: the reduced ability of an SFT-initialized policy to be effectively reshaped by subsequent RL. To better understand this phenomenon, we conduct detailed analysis from multiple perspectives, including parameter changes, output spaces, and RL optimization dynamics. Our results show that models from excessive SFT tend to produce over-confident token distributions and exhibit sharp parameter landscapes, which make them harder to optimize in the RL stage. To enable a more robust SFT-to-RL handoff, we propose \texttt{Rejuvenation}, a simple yet effective method that restores plasticity while preserving useful SFT-acquired priors. Rejuvenation leverages base-anchored model fusion to reduce excessive SFT-induced drift with targeted neuron reset to mitigate model rigidity. Experimental results on both math reasoning tasks and agentic tasks demonstrate that our approach consistently improves RL performance on over-trained SFT models, while also enhancing generalization to out-of-distribution tasks.