Before the Next Rollout: Diagnosing Skill Utility Shifts from Policy Updates

Yifan Wang; Zhenyi Gai; Yiwen Zhao; Mingxuan Jiang; Keyang Chen; Zeping Li; Hongfeng Chai; Yixin Cao

cs.AIOct 29, 2026

Before the Next Rollout: Diagnosing Skill Utility Shifts from Policy Updates

Authors: Yifan Wang, Zhenyi Gai, Yiwen Zhao, Mingxuan Jiang, Keyang Chen, Zeping Li, Hongfeng Chai, Yixin Cao

Organizations: Fudan University · Pyra

Abstract

Skill-evolution methods typically analyze execution trajectories to guide agent skill refinement. However, policy updates can change the behavioral effects and marginal utility of an unchanged skill. Attribution based on earlier trajectories may therefore no longer reflect the skill's contribution under the updated policy. Moreover, trajectory failures can arise from multiple factors, making individual skill contributions difficult to isolate. We evaluate whether policy-update signals contain predictive information about changes in skill marginal utility. At fixed contexts from pre-update training rollouts, we measure how policy updates change the contrast in token preferences between skill and control conditions. Combining these interaction shifts with advantage estimates yields reward-directed readouts for diagnosing utility changes. The readouts reuse existing training rollouts and require only forward passes for single-step action scoring, without additional post-update rollout evaluation. Our experiments show that policy updates can alter skill marginal utility and that policy-update readouts carry diagnostic information for identifying skills with declining utility. Integrating readout-guided prioritization into skill evolution improves downstream performance while reducing policy and editor token consumption.

Figures & tables

Figure 1: Policy updates alter skill marginal utility. Distributions before and after policy updates, with skill content fixed. Darker shades indicate higher concentrations of skills at the corresponding utility values.

Figure 2: Diagnostic value and downstream benefits of policy-update readouts. (a) Association between reward-directed readout percentiles and measured utility declines across three benchmarks. Positive $-\Delta M$ indicates declining skill utility; $\rho$ denotes Spearman correlation. (b) ALFWorld unseen success rate versus policy token consumption under full-failure and readout-guided revision. Bubble size indicates cumulative editor token usage.

Figure 3: Overview of our method. With the skill library fixed within each update window, we reuse pre-update rollouts to compare action preferences under skill and matched-control conditions before and after the policy update. Combining these fixed-state comparisons with batch advantages yields reward-directed readouts that prioritize skills for revision without additional diagnostic rollouts.

Benchmark	Skills	Decline	Readout	Decline diagnosis				Magnitude
Benchmark	Skills	Decline	Readout	AP $\uparrow$	$\mathrm{AUROC}_{-/+}\uparrow$	$\rho_{\mathrm{dir}}$ $\uparrow$	P@5 $\uparrow$	$\rho_{\mathrm{mag}}$
ALFWorld	25/44	32.0%	Magnitude	0.483	0.643	$-0.032$	0.200	0.170
ALFWorld	25/44	32.0%	Reward	0.567	0.833	0.416	0.400	$-0.174$
WebShop	42/54	50.0%	Magnitude	0.739	0.600	0.334	0.400	0.505
WebShop	42/54	50.0%	Reward	0.853	0.800	0.778	0.800	0.073
HotpotQA	22/30	36.4%	Magnitude	0.560	0.542	0.080	0.400	0.390

Table 1: Skill-level diagnostic performance of policy-update readouts. Magnitude-only and reward-directed readouts use the same underlying skill–control interaction shifts. Skills reports evaluated/total skill counts, and Decline denotes the percentage of evaluated skills with declining utility. AP and P@5 assess decline ranking, while

\mathrm{AUROC}_{-/+}

distinguishes declines from increases.

\rho_{\mathrm{dir}}

and

\rho_{\mathrm{mag}}

denote Spearman correlations with

-\Delta M

and

|\Delta M|

, respectively. Bold indicates the higher value in each comparison.

Method	ALFWorld		WebShop	HotpotQA	NQ	Spreadsheet Bench	LogicBench
Method	Seen SR	Unseen SR	Score	EM	EM	Acc.	Acc.
GRPO (no skills)	65.36 3.7	64.55 4.3	55.10 2.6	50.10 1.6	47.60 1.1	18.50 2.6	81.90 1.5
GRPO (fixed skills)	68.21 3.1	67.16 3.0	58.49 2.5	51.30 2.2	49.30 1.8	22.50 2.3	83.37 1.7
SkillRL	75.36 3.3	74.63 3.5	59.32 2.2	52.40 1.2	51.70 0.7	27.00 2.4	86.20 1.3
WikiSkill	69.64 2.8	68.28 3.7	58.84 1.8	51.90 1.7	50.40 0.8	26.60 2.1	85.30 1.2
EvoSkill	67.86 3.6	66.04 4.3	58.72 2.4	51.20 1.1	49.80 1.2	24.70 2.8	84.50 1.8

Table 2: Performance comparison across six benchmarks. Subscripts denote standard deviations; bold indicates the best mean.

\dagger

marks significant improvements over the highest-scoring baseline for each metric.

Benchmark	Method	Inference			Revision
Benchmark	Method	Steps $\downarrow$	Policy in. (k) $\downarrow$	Policy out. (k) $\downarrow$	Editor in. (M) $\downarrow$	Editor out. (M) $\downarrow$	Accepted/ Proposed
ALFWorld	Full-failure	24.76	30.46	3.07	14.940	0.098	22/30
ALFWorld	Ours	18.31	16.48	4.76	3.475	0.092	25/29
WebShop	Full-failure	10.74	14.81	0.19	1.868	0.026	18/30
WebShop	Ours	6.88	10.21	0.17	1.389	0.046	19/26
HotpotQA	Full-failure	3.45	7.25	1.12	6.200	0.084	21/30

Table 3: Inference and revision efficiency. Comparison with full-failure attribution in terms of average interaction steps, policy input/output tokens (k), editor input/output tokens (M), and accepted/proposed skill revisions. Bold indicates the better result for each metric, with revision acceptance compared by rate.

Selector	ALFWorld		WebShop
	Seen SR $\uparrow$	Unseen SR $\uparrow$	Score $\uparrow$
Random top-5	$72.14$	$70.52$	$58.07$
Failure-frequency	$76.43$	$75.75$	$59.88$
Magnitude-only	$74.64$	$73.88$	$58.91$
Reward-directed	$\mathbf{81.43}$	$\mathbf{80.22}$	$\mathbf{63.08}$

Table 4: Effect of candidate selection on downstream performance. All selectors share the same top-5 budget and revision protocol.

Figure 4: Skill library evolution and editing dynamics on WebShop. (a) Library composition by skill category. (b) Applied editing operations per 25 iterations. Labels indicate accepted/proposed candidate libraries.

Figure 5: Case study of readout-guided skill revision on WebShop. Policy-update readouts prioritize a candidate skill for revision, while the editor uses its associated failure trajectories to propose targeted improvements instead of reviewing all failures.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Interaction	Train	Eval
ALFWorld	Multi-Step	3,553	140 / 134
WebShop	Multi-Step	10,587	500
HotpotQA	Multi-Step	90,447	7,405
SpreadsheetBench	Multi-Step	600	212
NQ	Multi-Step	79,168	3,610
LogicBench	Single-Step	11,524	760

Appendix

Table 5: Benchmark task counts and interaction modes. ALFWorld reports Seen/Unseen evaluation tasks.

Benchmark	Source	Skills
ALFWorld	SkillRL	44
WebShop	SkillRL	54
HotpotQA	SkillRL (Search)	30
NQ	SkillRL (Search)	30
SpreadsheetBench	Trajectory distillation	32
LogicBench	SRA-Bench	19

Appendix

Table 6: Initial skill libraries shared across comparison methods. HotpotQA and NQ reuse the same 30 search skills.

Setting	Default value
Model size	Qwen3.5-4B (4B parameters)
Compute hardware	8 $\times$ NVIDIA B200 GPUs
Training time	24 hours per run
Compute budget	192 GPU-hours per run
Optimizer	AdamW, full-parameter training
Learning-rate schedule	Constant, no warmup

Appendix

Table 7: Additional default hyperparameters and computational resources. The PPO minibatch counts decision contexts, not complete trajectories. Evaluation temperature refers to downstream performance evaluation. Compute estimates correspond to one 150-iteration training run.

Figure 6: Skill utility changes aggregated across RL seeds. Each row reports the supporting task count, pre- and post-update marginal utility, and the estimated change. Points and horizontal bars show

\widehat{\Delta M}_{t}(s)

and its pointwise 95% paired-bootstrap confidence interval. Filled and open circles denote intervals excluding and including zero, respectively; crosses mark estimates supported by fewer than two tasks. All utility values are reported in percentage points.

Panel of Figure 7 — Figure 7: Prompt template for skill revision, specifying editor inputs, revision instructions, and the JSON output format.

LLM Agent Skill Evaluation

See all 31 papers

SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents
Aug 19, 2026Qingyao Li, Wenxiang Jiao, Shuai Shao +6LLM Agents LLM Agent Skill Learning
Shanghai Jiao Tong University · Xiaohongshu Inc.
Co-Evolving Skill Generation and Policy Optimization
Jun 7, 2026Zhiwei Zhang, Yudi Lin, Nikki Lijing Kuang +4LLM Agent Skill Learning RL for LLM Agents
The Pennsylvania State University · Nanyang Technological University · University of California, San Diego +2
SkillEvoReg: Regularizing Agent Skill Evolution Against Overfitting
Sep 25, 2026Guanyu Nie, Fangzhou Zhu, Shixiong Kai +3LLM Agent Skill Learning Agent Skill Learning
Huawei Noah’s Ark Lab

Aug 19, 2026J/K move · Enter open · S save

SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents

Qingyao Li, Wenxiang Jiao, Shuai Shao, Kangning Zhang+5

Shanghai Jiao Tong University · Xiaohongshu Inc.

LLM Agents LLM Agent Skill Learning RL for LLM Agents Step-Level Credit Assignment in RL

Open paper AskPDF arXiv

Agent frameworks increasingly package procedural knowledge as skills: instruction files an agent reads on demand, while public libraries now hold thousands of them. Which skill to read has thus become a decision the policy itself makes in the middle of an episode, yet no existing signal trains it. We show that the default remedy, outcome-rewarded RL over the candidate slate, cannot teach it, for a structural reason we identify and name selector credit starvation: under a broadcast, sequence-level advantage, the few tokens that name the chosen skill carry a vanishing share of the loss, and the credit they inherit is increasingly wrong-signed as trajectories lengthen. A correct choice is punished whenever the execution after it fails, even though the choice itself is among the most valuable decisions in the trajectory. Auditing a completed run's own training artifacts confirms all three properties, each worsening monotonically with horizon. SkillGate removes the failure by construction: it partitions the token support into two disjoint credit channels, outcome credit reaching only execution tokens, and a separate action-local advantage reaching exactly the skill-naming tokens, positive only when a trajectory's single read is the correct one. On five agentic benchmarks under a 16-candidate slate, SkillGate lifts a 9B policy from 40.8% to 53.2% trial success, well ahead of the identical budget spent on outcome reward alone, while cutting exposure to misleading candidates by two thirds and reading fewer skills.

Abstract

LLM Agent Skill Evaluation

Explore similar work

SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents

Co-Evolving Skill Generation and Policy Optimization

SkillEvoReg: Regularizing Agent Skill Evolution Against Overfitting