Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs

Xuan Liu; Jingbin Qian

cs.AISep 17, 2026

Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs

Authors: Xuan Liu, Jingbin Qian

Organizations: Shanghai Jiao Tong University · Rice University

Abstract

Reinforcement learning now trains language-model agents that act over dozens of steps in live environments. The gains are large, and they are read as better decision-making. An agent in a closed loop writes its own inputs. Each observation follows from its own earlier actions, so the states it meets late in an episode are partly of its own making. An SFT checkpoint and an RL checkpoint are then scored from different states, even on identical tasks. Endpoint success mixes two changes: where the agent arrives, and what it does once it is there. Restricting the comparison to states both policies reach does not separate them. That restriction selects on an outcome, and in our data it flips the sign of the effect. We introduce checkpoint handoff, an evaluation protocol that clones a state one released checkpoint reached and hands it to another, with no retraining. Crossing a reacher role and a solver role over SFT and RL splits an endpoint gain into REACH and SOLVE. REACH is how often a policy arrives at a state the environment confirms is a fixed number of actions from success. SOLVE is how often it finishes from an identical cloned state. Across two benchmarks and two independently released pipelines, the reacher by solver interaction is positive in all five conditions. An RL history is worth more to an RL solver than the same history is to an SFT solver. On ALFWorld, RL improves both terms, and the SFT solver never succeeds where the RL solver fails. Independent REACH and SOLVE gaps predict the aggregate interaction. Handoff asks only that one checkpoint's history can be replayed under another, so long-horizon evaluation can report arrival and completion beside endpoint success.

Figures & tables

Figure 1: The endogenous state problem. Two checkpoints receive one task and act. Their early actions leave them in different states, and each is then scored from the state it produced.

Figure 4: Checkpoint handoff and interaction prediction. (A) A benchmark-specific rule selects a handoff from each reacher’s trajectory. Both solvers continue with matched inputs, yielding four endpoint success rates and their interaction. (B) On ALFWorld, target arrival gaps and independent Solve gaps predict the interaction for comparison with the observed value (Equation 9 ).

		Endpoint success (%)
Setting	$n$	$W$ =SFT	$W$ =SFT	$W$ =RL	$W$ =RL	Interaction
		$R$ =SFT	$R$ =RL	$R$ =SFT	$R$ =RL	(pp)
TravelPlanner / Agent-STAR
1.5B	180	6.1	14.4 +8.3	7.8	33.9 +26.1	$+$ 17.8 [9.4, 26.1]
3B	180	12.2	24.4 +12.2	22.2	48.9 +26.7	$+$ 14.4 [6.1, 22.8]
7B	180	20.0	29.4 +9.4	38.9	58.9 +20.0	$+$ 10.6 [0.6, 20.6]

Table 1: Endpoint success for the four reacher (

W

) by solver (

R

) assignments. Subscripts on the

R

=RL columns give the solver effect under that reacher, and the last column is their difference,

I_{\tau}

of Equation 4 . Brackets are

95\%

paired-bootstrap intervals.

Split	Reacher	Arrival %	Steps	States $n$	$R$ =SFT %	$R$ =RL %	RL $-$ SFT (pp)
		(a) REACH		(b) SOLVE from identical replayed states
Seen split	SFT	15.8	41.6	19	31.6	57.9	26.3 —
	RL	81.7	15.7	98	53.1	79.6	26.5 —
	Pooled			117	49.6	76.1	26.5 [11.2, 41.3]
Unseen split	SFT	13.3	44.0	16	43.8	93.8	50.0 [15.4, 83.3]
	RL	82.5	17.8	99	58.6	96.0	37.4 [27.7, 47.0]

Table 2: Reach and Solve on ALFWorld. Each row is a reacher. (a) Arrival at

\mathcal{F}_{2}

over

120

trajectories, with mean actions to arrival or stopping. (b) Completion by each solver from identical replayed states that the reacher produced. Brackets are task-stratified paired-bootstrap

95\%

intervals; a dash marks a stratum too small for one.

Split	Solver	Bridge %	Finish $\mid$ bridge %	Complete %	RL only	SFT only
Seen split	SFT	72.6 [64.1, 80.3]	68.2 [57.6, 77.6]	49.6 [40.2, 59.0]	41	10
	RL	92.3 [87.2, 96.6]	82.4 [75.0, 88.9]	76.1 [68.4, 83.8]
Unseen split	SFT	72.2 [63.5, 80.0]	78.3 [68.7, 86.7]	56.5 [47.8, 65.2]	45	0
	RL	97.4 [93.9, 100.0]	98.2 [95.5, 100.0]	95.7 [91.3, 99.1]

Table 3: Solve split into choosing a bridge action and finishing from it. Shares of reached states with

95\%

bootstrap intervals; the last two columns count states completed by only one solver (

n=115

on the unseen split,

117

on the seen split).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Checkpoint	Revision
TravelPlanner	xxwu/Agent-STAR-SFT-1.5B	b3d55c10e763231d13f591bfa2ad2cb99d19b981
TravelPlanner	xxwu/Agent-STAR-RL-1.5B	279b7c75586b2a6ada3eac5a5f406a147522c787
TravelPlanner	xxwu/Agent-STAR-SFT-3B	dfb0d943ad8c7a1d6a5e38064a75a6f4e8b4e503
TravelPlanner	xxwu/Agent-STAR-RL-3B	705723c62e82ed15844783f221572ea1c78f4185
TravelPlanner	xxwu/Agent-STAR-SFT-7B	945532c2ca454caaf1f62d1c42916cbec0a2973a
TravelPlanner	xxwu/Agent-STAR-RL-7B	7013d9fd0fff692d8f6f352ae5db4351d174eab9

Appendix

Table 4: Released checkpoints and fixed revisions.

Design question	Protocol control	Reported quantity
Are states equally close to success?	Exhaustive $\mathcal{F}_{2}$ replay check that never consults a checkpoint, and a matched two-action solver budget	Arrival at a common solvable frontier
Do solvers receive the same input?	Exact replay with equal environment observations, history, admissible actions, budget, prompt, and paired seeds	Same-state Solve effect
Are difficult non-arrivals removed?	Every natural non-arrival remains a zero endpoint outcome; reached-state results are reported separately	Deployed-agent endpoint interaction
Does scoring introduce the interaction?	Environment-native won in ALFWorld; archived Formatter replies and exact official-score recomputation in TravelPlanner	Stable benchmark outcomes
Is the interaction prediction fitted to the target split?	Conversion is measured on an independently selected sample before it is combined with target arrival	Out-of-sample interaction prediction

Appendix

Table 5: Design questions and the protocol control that answers each.

	Target-split arrival %		External SOLVE %		Interaction (pp)
Task type	SFT	RL	SFT	RL	Pred.	Obs.	Resid.
look at obj in light	20.0	80.0	10.0	35.0	15.0	40.0	25.0
pick & place	35.0	70.0	85.0	100.0	5.2	$-$ 5.0	$-$ 10.2
clean then place	10.0	70.0	20.0	100.0	48.0	60.0	12.0
cool then place	0.0	100.0	80.0	100.0	20.0	25.0	5.0
heat then place	15.0	100.0	60.0	80.0	17.0	5.0	$-$ 12.0

Appendix

Table 6: Per-task-type interaction prediction (Equation 9 ). Arrival is from the target valid_unseen split and the Solve profile from the independent sample. The overall row is the unweighted mean over types, the stratified bootstrap unit; dashes mark cells it leaves undefined.

(a) Arrival by reacher action budget, seen split
Budget (steps)	Reacher	Arrival %	95% CI
8	SFT	10.8	[4.2, 17.5]
8	RL	47.5	[34.2, 60.0]
16	SFT	14.2	[7.5, 20.8]
16	RL	75.8	[62.5, 88.3]
48	SFT	15.8	[8.3, 22.5]

Appendix

Table 7: Frontier arrival by reacher budget on valid_seen . The bolded row contrasts RL at 8 actions with SFT at 48. Brackets are task-stratified paired-bootstrap 95% intervals from

100{,}000

resamples.

Step	SFT solver	RL solver
1	look “I should head straight to the nearest shelf. I need to locate a shelf first … I’ll scan the room.”	go to shelf 1
2	go to shelf 1	move pencil 1 to shelf 1
Outcome	budget exhausted, not won	won

Appendix

Table 8: Two solvers on one cloned state. Bridge actions in blue. The SFT solver’s first-step reasoning is quoted from its archived reply.

Explore similar work

CAST: Game Solvers as Turn-Level Teachers for LLM Agents
Jul 28, 2026Yu Wang, Yi-Kai Zhang, Wentao Shi +8Reinforcement Learning With Verifiable Reward Large Language Model Agents
When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
Jun 2, 2026Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan +1Reinforcement Learning From Human Feedback Collapse
When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff
Jun 7, 2026Runze Liu, Jiashun Liu, Xu Wan +2Supervised Fine-Tuning Plasticity

Jul 28, 2026 · cs.CLJ/K move · Enter open · S save

CAST: Game Solvers as Turn-Level Teachers for LLM Agents

Yu Wang, Yi-Kai Zhang, Wentao Shi, Ziang Ye+7

University of Science and Technology of China · 4Meituan, China · 2Nanjing University · 3Wuhan University

Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.

Reinforcement Learning With Verifiable Reward Large Language Model Agents Teacher

Open paper Ask about it PDF

Abstract

Explore similar work

CAST: Game Solvers as Turn-Level Teachers for LLM Agents

When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff