Enterprise analytics agents solve long-horizon tool-use problems over distributed business data, requiring retrieval, reasoning, API calls, code execution, and adaptation to intermediate observations. Supervised fine-tuning (SFT) calibrates tool syntax and teacher-supported behavior, whereas reinforcement learning (RL) can explore reward-supported behaviors beyond demonstrations; applied uniformly, however, RL can perturb already-calibrated skills. We study how to balance SFT and RL under production-mirroring beta APIs. We observe that, in our controlled experiment, checkpoint trajectories retrospectively separated into three regimes: Imitation, where SFT captured reliable teacher behavior; Lift, where both stages helped; and Discovery, where useful reward-observable behavior lay outside reliable teacher support. We leverage this prospectively, using teacher support and reward-observable headroom to route features to SFT only, SFT then RL, increased RL allocation, or further environment development. Across 18 subsequent feature-specific experiments, the diagnostic predicted 15/18 observed trajectories. On GPT-OSS 120B, targeted SFT then RL produced positive point estimates on 7/8 advertiser skills relative to a frontier Control; five positive gains had paired 95% confidence intervals excluding zero, while one skill had a confidence-supported regression. The largest gain was non-disclosure (+11.27 points; 95% CI [+9.72, +12.82]). A separate SME audit surfaced that targeted RL reduces standard leakage from 11.8% to 2.9% and adversarial leakage from 22.9% to 6.8% relative to SFT while preserving actionability (86.2% to 85.7%). In a matched uniform-versus-targeted comparison with shared rewards and optimization, targeted RL improved the seven-skill mean delta from +1.62 to +3.57 while using 43% less incremental RL compute.
Figures & tables
Figure 1: Targeted post-training pipeline. Episodes interleave reasoning, beta-API calls, and tool observations. Stage I uses teacher-distilled SFT with tool-response masking. The original eight-skill study motivates a three-regime hypothesis; a subsequent prospective diagnostic uses teacher support and frozen-SFT reward headroom to route new features before RL. Stage II applies GRPO only where reward-observable headroom justifies on-policy optimization, using tool-validity and skill-aware jury rewards with a KL anchor to πSFT .
Skill
Teacher
Control
Base Δ
SFT Δ
SFT+RL Δ
95% paired CI
Measure funnel
94.32
91.73
−2.73
+1.55
+0.41
[−0.28,+1.10]
Target / audience strategy
90.92
90.00
–
–
+1.50
[+0.48,+2.52]
Business metrics
94.42
91.22
+4.78
+3.19
+5.18
[+4.03,+6.33]
Categorize campaigns / ads
59.19
57.05
−6.95
−4.35
−2.05
[−3.21,−0.89]
Campaign performance
84.70
82.51
+5.19
+5.49
+7.49
[+6.15,+8.83]
Identify product
89.89
87.56
−2.34
−1.13
+0.66
[−0.14,+1.46]
Table 1: Per-skill evaluation against Control on GPT-OSS 120B. Teacher is absolute reference accuracy; Base, SFT, and SFT+RL are percentage-point deltas versus the same frontier Control. Every score is Avg@32 over repeated stochastic evaluation of one fixed checkpoint. The final column reports paired 95% CIs for SFT+RL. Bold deltas have intervals excluding zero.
Figure 2: Retrospective three-regime decomposition. Mean accuracy for representative GPT-OSS 120B skills across Base → SFT → targeted RL. Imitation averages Measure Funnel and Brand Voice; Lift averages Campaign Performance and Identify Product; Discovery is the single controlled non-disclosure skill. Dotted lines denote teacher references. The labels are assigned from observed checkpoint trajectories and are not themselves prospective validation.
Figure 3: Regime-conditioned reward and entropy dynamics. Held-out reward and policy-token entropy across 3,000 RL steps for RL from Base, uniform SFT+RL, targeted SFT+RL, and the SFT-only reference. In Imitation, unnecessary RL contracts entropy without useful reward gain; in Discovery, targeted RL raises reward with controlled entropy contraction.
Matched seven-skill result
Uniform
Targeted
Positive vs. Control
4/7
6/7
Mean Δ vs. Control
+1.62
+3.57
Imitation mean accuracy
94.31
95.42
Lift mean accuracy
88.34
89.11
Discovery accuracy
90.72
95.50
Incremental RL compute
21K
12K
Table 2: Matched uniform-versus-targeted allocation. Compute is in H200-hours. Targeted allocation uses 43% less incremental RL compute.
Human-audit metric
Control
SFT
SFT+RL
Standard leakage
12.4%
11.8%
2.9%
Adversarial leakage
24.7%
22.9%
6.8%
Informative/actionable
84.6%
86.2%
85.7%
Acceptably safe
76.9%
78.4%
91.8%
Table 3: Independent SME audit of the non-disclosure checkpoint. Lower leakage is better; higher actionability and acceptable-safety rates are better.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Regime
Skills
Tchr.
Base
SFT
+RL
Imitation
Measure funnel; Brand voice
95.49
91.43
95.55
95.42
Lift
Campaign perf.; Identify product
87.30
86.46
87.22
89.11
Discovery
Non-disclosure abstraction
84.53
83.23
84.00
95.50
Appendix
Table 4: Supporting result summaries. Panel (a) gives the descriptive aggregation underlying Figure 2 . Panel (b) reports H200-hours for the complete retained post-training recipes. RL from Base did not pass the offline release-readiness gate within 25K H200-hours. The 12K figure for targeted RL is incremental to the shared 15K SFT checkpoint, not total post-training compute.
Gate criterion
Required threshold
Targeted SFT+RL
Mean domain-skill delta vs. Control
≥+2.0 points
3.31
Skills with positive point estimate
At least 6/8
7/8
Maximum individual domain regression
No worse than −3.0
−2.05
Non-disclosure gain over Teacher
≥+5.0
10.97
Public guardrail regressions
0 negative
0/3 negative
Jury–human calibration
Spearman ρ≥0.75
0.78
Appendix
Table 5: Numerical offline release-readiness criteria and the targeted candidate’s values.
Category
Skill
Description
Representative requests
Campaign Reasoning
Measure funnel
Quantify progression from awareness to consideration to purchase and locate funnel drop-offs.
What does my full funnel look like from awareness to purchase? Where is the biggest drop-off between branded search and purchase?
Campaign Reasoning
Target / audience strategy
Profile buyer and audience segments, including shopping behavior, funnel stage, engagement, and conversion.
Which audience segments engage but fail to convert? How are repeat purchasers different from one-time buyers?
Data Analysis
Business metrics
Compute and contextualize advertising efficiency and growth metrics such as ROAS, ACOS, share of voice, and brand health.
Calculate my ROAS and compare ad-driven sales to peers. How has my advertising efficiency trended quarter over quarter?
Data Analysis
Categorize campaigns / ads
Classify campaigns by ad type, objective, targeting strategy, and creative format to expose portfolio imbalance.
Which campaigns are SP, SB, or SD, and how does spend distribute? Group campaigns by targeting strategy and compare ROAS.
Data Analysis
Campaign performance
Identify profitable, fatigued, or underperforming campaigns and recommend where budget should be scaled or reduced.
Which campaigns increased spend but lost ROAS? Show daily spend and sales trends for my top campaigns.
Concept Understanding
Identify product
Identify promoted products that drive sales, waste spend, need creative refresh, or produce halo effects.
Which ASINs generate the most sales with the lowest ad spend? Which products have high impressions but low CTR?
Appendix
Table 6: Advertiser skill taxonomy. The examples illustrate the natural-language task families used to seed trajectory generation, not exact proprietary evaluation prompts.
Skill / guardrail
Qwen3-32B SFT+RL Δ vs. Control
Measure funnel
−2.04
Business metrics
+3.52
Categorize campaigns / ads
−4.05
Identify product
−5.36
Brand voice
+0.04
Non-disclosure
+4.00
Appendix
Table 7: Directional model-scale transfer on Qwen3-32B. Blank/unavailable skill-stage combinations from the original export are omitted.
Figure 4: Cost structure and reward routing. Heavy-tail trajectories make fixed rollout counts inefficient, while skill metadata activates partially overlapping jury dimensions.
Figure 5: Optimization stability and rollout correction. Targeted SFT+RL exhibits smaller update magnitudes, lower clipping activity, and weaker TIS correction pressure than RL from Base or uniform SFT+RL. The gradient clipping threshold is 10; the policy-ratio interval is [0.80,1.28] ; TIS uses C=2 .
Observed pattern
Benchmark subset
Base
SFT
+RL
ΔRL
Imitation-like
BFCL (Single-Tool)
82.1%
93.4%
92.1%
−1.3
Lift-like
ToolBench (Multi-Step)
74.2%
81.5%
84.3%
+2.8
Discovery-like
BFCL (Missing Params)
61.4%
68.2%
81.5%
+13.3
Appendix
Table 8: External consistency check on public tool-use subsets.
Metric
Control
SFT
SFT+RL
Standard-prompt leakage rate
12.40%
11.80%
2.90%
Adversarial-prompt leakage rate
24.70%
22.90%
6.80%
Informative/actionable response rate
84.60%
86.20%
85.70%
Refusal rate
5.80%
5.50%
6.40%
Acceptable safety rate
76.90%
78.40%
91.80%
Appendix
Table 9: Full independent SME audit of non-disclosure. The result separates leakage from usefulness and refusal behavior.
Regime
SFT mix weight
RL mix weight
Imitation
1.0 ×
0 × (excluded)
Lift
1.0 ×
1.0 ×
Discovery
1.5 ×
2.5 ×
OOD (Toucan, regularizer)
0.3 ×
–
Appendix
Table 10: Trajectory mix weights normalized within each stage relative to Lift at 1.0 × .
Operational metric
Change vs. Control
Supported QPS
+10%
Generation throughput
+2.9% tokens/s
P50 user-perceived latency
−4.2%
P95 user-perceived latency
−6.8%
Recommendation acceptance rate
+3.6 percentage points
Multi-turn task-completion rate
+5.4 percentage points
Appendix
Table 11: Serving-period changes for the targeted candidate relative to Control. Negative latency, turns-to-resolution, and abandonment changes are improvements.
Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training. SFT is expected to provide a useful behavioral prior for RL to further enhance model capabilities. However, checkpoints with excessive SFT often show limited improvement during RL. We attribute this failure to the loss of model plasticity: the reduced ability of an SFT-initialized policy to be effectively reshaped by subsequent RL. To better understand this phenomenon, we conduct detailed analysis from multiple perspectives, including parameter changes, output spaces, and RL optimization dynamics. Our results show that models from excessive SFT tend to produce over-confident token distributions and exhibit sharp parameter landscapes, which make them harder to optimize in the RL stage. To enable a more robust SFT-to-RL handoff, we propose Rejuvenation, a simple yet effective method that restores plasticity while preserving useful SFT-acquired priors. Rejuvenation leverages base-anchored model fusion to reduce excessive SFT-induced drift with targeted neuron reset to mitigate model rigidity. Experimental results on both math reasoning tasks and agentic tasks demonstrate that our approach consistently improves RL performance on over-trained SFT models, while also enhancing generalization to out-of-distribution tasks.
Runze Liu, Jiashun Liu, Xu Wan +2
Hong Kong University of Science and Technology · Zhejiang University · State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA
Limited controlled evidence exists on how training data, adaptation method, and model scale jointly affect tool-calling performance in language-model agents. We evaluate supervised fine-tuning (SFT) with LoRA, reinforcement learning (RL) via Group Relative Policy Optimization (GRPO), and SFT followed by GRPO across six Qwen3 models from 0.6B to 32B parameters, covering both in-distribution performance and cross-dataset transfer. SFT with LoRA is the strongest in-distribution method throughout the 0.6B-32B range and best in 15 out of 18 experimental settings. On cross-dataset transfer, the methods are closer: GRPO wins 29 out of 54 settings where training and test datasets differ, but its margin over SFT averages under one point, and SFT->GRPO is rarely strongest in either comparison. Dataset mixing gives consistently strong transfer while staying close to specialized in-distribution training, regardless of method. Additional analysis further confirms that LoRA outperforms full-parameter fine-tuning, demonstrating that LoRA better preserves pretrained agentic behavior.
Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.
Sadhika Malladi, Samy Jelassi, Dylan Foster +2
University of California San Diego · Microsoft Research NYC