Reinforcement learning is now the mainstream paradigm for training LLMs on agentic tasks, yet standard practice keeps only the best-validation checkpoint and discards the rest. We show that this is lossy: in GRPO training of LLM agents, the best-validation checkpoint fails on problems that earlier checkpoints from the same run had already solved, and the union of problems solved across checkpoints far exceeds any single one. The discarded checkpoints thus form a teacher pool obtained at no additional training cost, which on-policy distillation (OPD) can mine to recover the lost coverage. Mining this pool raises two challenges: i) which checkpoint to distill from, as no checkpoint dominates the others and complementarity to the student is problem-specific; ii) where to trust it, as even a well-chosen teacher is only locally superior, so indiscriminate distillation overwrites behavior the student has already mastered. We propose Agentic-OPD, which treats the best-validation checkpoint as the student and all saved checkpoints as frozen teacher candidates, using the student's evolving capability gaps to guide both teacher selection and supervision filtering: the HCN policy (Highest Coverage Next) dynamically selects teachers by weighted coverage of the student's unsolved problems, while the LAD module (Locality-Aware Distillation) confines transfer to failed student rollouts on problems where the teacher performs better. Across five agentic task domains and 24 benchmarks and task subsets, Agentic-OPD improves over its initialization checkpoint by up to 7.4 points, with a 4B student surpassing the latest frontier models GPT-5.6 and DeepSeek-V4.1 on deep search. Agentic-OPD recasts a finished RL run as a reusable pool of policies and establishes post-RL capability recovery as a stage orthogonal to improvements in the RL algorithm itself.
Figures & tables
Figure 1: The temporal capability gap. (a) DeepSearch on HotpotQA: the current checkpoint against U≥k(i) , the tasks solved by at least k of the checkpoints through index i , for k=1,2,3 . (b) Best single checkpoint against the historical union U≥1 in five domains, with the relative gap annotated.
Figure 2: Overview of Agentic-OPD. Top: the consolidation loop. (a) Frozen checkpoints from one RL run form the teacher pool. (b) HCN selects teachers based on coverage of the student’s unsolved tasks. (c) LAD filters supervision at the problem, rollout, and turn levels.
In-domain
Out-of-domain
Model
NQ
HotpotQA
PopQA
TriviaQA
2Wiki
MuSiQue
Bamboogle
Avg.
Proprietary models
GPT-5.6 ( OpenAI, 2026 )
7.7
22.2
32.2
56.5
17.4
11.2
30.4
25.4
Claude Sonnet 5 ( Anthropic, 2026 )
23.0
16.8
22.6
64.8
4.5
5.9
40.8
25.5
Grok-4.6 ( xAI, 2026 )
20.7
33.4
27.3
59.8
25.6
12.3
46.4
32.2
Open-source models, >100 B parameters
Table 1: DeepSearch exact match accuracy (%) on the test sets ( Dtest ) of seven benchmarks. Bold marks the best score within each backbone block. Reference models use the same retriever, four-turn limit, and scoring, without domain-specific training.
Figure 3: (a) Teacher indices and NQ/HotpotQA exact match during consolidation; highlighted points mark peak accuracy. (b) Domain scores (%): mean exact match for DeepSearch, mean pass@1 for Tool-use, success rate for ALFWorld, mean of Score and Success for WebShop, and Overall score for SciWorld. Base denotes the instruction-tuned model.
Figure 4: Capability recovery and teacher complementarity on the Qwen3-4B routing-validation set. (a) Question-level changes after OPD update 20. (b) Teacher accuracy versus unweighted coverage 100∣Qi∖P20∣/2,000 . Color indicates RL step; the dashed line marks student accuracy.
Method
NQ
HotpotQA
GRPO (initialization)
42.1
34.8
Training-free merging
Uniform soup
42.5
35.1
Greedy soup
43.4
35.9
TIES
43.0
35.6
DARE-TIES
43.7
36.2
Table 2: Checkpoint reuse and ablations on DeepSearch with Qwen2.5-3B-Instruct: exact match (%).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Benchmarks / tasks
Metric
Turns
DeepSearch
NQ ( Kwiatkowski et al., 2019 ) , HotpotQA ( Yang et al., 2018 ) (in-domain); PopQA ( Mallen et al., 2023 ) , TriviaQA ( Joshi et al., 2017 ) , 2Wiki ( Ho et al., 2020 ) , MuSiQue ( Trivedi et al., 2022 ) , Bamboogle ( Press et al., 2023 ) (out-of-domain)
Exact match
4
Tool-use
AIME 2024, AIME 2025, MATH-500 ( Lightman et al., 2024 ) , AMC23, OlympiadBench ( He et al., 2024 )
pass@1
16
WebShop ( Yao et al., 2022 )
Goal-directed shopping on simulated e-commerce pages
Score, success rate
20
SciWorld ( Wang et al., 2022 )
Measure, Test-Cond., Find, Chem-Mix, Lifespan
Overall score
15
ALFWorld ( Shridhar et al., 2020 )
Embodied household instruction following
Success rate
50
Appendix
Table 3: Benchmarks, evaluation metrics, and maximum interaction turns per episode.
Figure 5: Validation tasks solved during RL in four domains: current checkpoint (blue) and historical union U≥1(d)(i) (red). Shading marks the gap Gi(d) .
Method
AIME 2024
AIME 2025
MATH-500
AMC23
OlympiadBench
Mean
Base
3.0
3.1
43.0
27.6
17.9
18.9
GRPO
7.9
12.3
49.5
36.3
24.8
26.1
Agentic-OPD
15.1
15.5
55.3
44.8
29.9
32.1
Appendix
Table 4: Tool-use pass@1 accuracy (%). Mean is the average across five benchmarks.
Method
Measure
Test-Cond.
Find
Chem-Mix
Lifespan
Overall
Base
1.1
4.2
8.4
4.7
10.6
4.8
GRPO
7.6
9.3
22.9
15.4
41.1
14.4
Agentic-OPD
14.8
10.0
25.0
26.6
50.0
18.0
Appendix
Table 5: SciWorld scores by task subset and the aggregate Overall score.
Method
Score
Success
Mean
Base
10.4
2.5
6.5
GRPO
80.6
63.0
71.8
Agentic-OPD
84.0
66.0
75.0
Appendix
Table 6: WebShop task score and success rate (%). Mean averages the two metrics.
Method
Pick
Look
Clean
Heat
Cool
Pick2
All
Base
45.7
5.3
10.8
8.0
10.7
17.8
19.5
GRPO
97.8
73.7
94.6
96.0
82.1
80.0
88.5
Agentic-OPD
100.0
78.9
94.6
92.0
85.7
86.7
91.0
Appendix
Table 7: ALFWorld success rate (%) by task type. All denotes the full evaluation set.
Figure 6: SciWorld case study. The Agentic-OPD agent identifies the crocodile as the animal with the longest life span, but fails to issue the final focus action and receives partial reward.
Figure 7: WebShop case study. The agent searches for the requested product, selects the required attributes, and completes the purchase.
Figure 8: ALFWorld case study. The agent locates and cleans a tomato before placing it on the sidetable, completing the task.
Figure 9: MATH-500 case study. The agent uses the code interpreter to verify the maximum GCD and returns 13 .
Figure 10: DeepSearch case study (incorrect). At OPD update 20, the agent confuses the character Slipknot with the band, follows an unsupported link to Officer Downe , and answers Shawn Crahan instead of David Ayer (score 0.0).
Figure 11: DeepSearch case study (correct). At OPD update 20, the agent identifies Colin Cowdrey as Fabian Cowdrey’s grandfather, then retrieves his 22 Test centuries for England and returns the correct answer (score 1.0).