RLHarness: Co-evolving Procedural Skills with Reinforcement Learning for Long-horizon Multimodal Reasoning

Ziqiao Shang; Zian Xu; Ji-Chen Yan; Weiming Wu; Ziyi Jia; Jie Meng; Tao Huang; Shan Huang

cs.AISep 26, 2026

RLHarness: Co-evolving Procedural Skills with Reinforcement Learning for Long-horizon Multimodal Reasoning

Authors: Ziqiao Shang, Zian Xu, Ji-Chen Yan, Weiming Wu, Ziyi Jia, Jie Meng, Tao Huang, Shan Huang,

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · Didichuxing Co. Ltd

Abstract

Multimodal reasoning requires models to preserve visual evidence through long decision chains while selecting appropriate procedures across diverse scenarios and rules. When learning is guided only by terminal verifiers, reinforcement learning (RL) reveals whether a final answer is correct but not how it should be produced. The policy must therefore discover reusable reasoning procedures while learning to execute them, creating a program cold-start problem. Skills can externalize successful procedures, reduce repeated exploration, and provide inspectable guidance. However, a fixed Skill Bank assumes that this guidance remains compatible with an evolving policy, while updating Skills alone can leave their triggers, execution protocols, and demonstrations stale or mutually inconsistent. We introduce RLHARNESS, which organizes Skills, selection and execution protocols, few-shot demonstrations, and task contracts into a unified, versioned Harness and alternates Harness evolution with policy learning. An Exploration-Distillation Harness builds the initial Harness and version-aligned verified traces for SFT and DAPO I. After the first RL block, a Post-RL Reconstruction Harness rebuilds Skills, protocols, and demonstrations from fresh success-failure rollouts, and DAPO II adapts the policy to the reconstructed program. RLHARNESS improves Accuracy from 16.25%/27.50% to 62.00%/50.00% on MetroMap/TravelMap and raises F1 score from 37.13%/45.50% to 65.81%/65.51% on Fee-VL/Cancel-VL. All four tasks achieve their best results only after reconstruction and DAPO II, showing that an evolving Harness complements RL by continually updating the external program that the policy learns to execute.

Figures & tables

Figure 1: Why guidance and policy must co-adapt. Left: Longer reasoning with a fixed interface, Skill-only updates, and RL with a frozen Harness leave complementary failure modes: program cold start, drift between Skills and their execution protocol, and mismatch after the policy changes. Right: On the same Chai Wan–Tai Koo task, an adjacency-only rule permits a shortcut that skips the visible intermediate station Heng Fa Chuen and fails verification. The resulting failure trace motivates a No-Skip Edge Guard in the reconstructed Harness $\Phi^{2}$ ; subsequent RL internalizes the revised execution rule, yielding the complete route and a verifier-accepted answer.

Figure 2: The complete RLHarness pipeline. ① Harness I initializes the task-specific external program, fixes a task-specific exploration subset and batch plan, and selectively commits validated changes. After the first Harness $\Phi^{1}$ is frozen, audited accepted traces $\mathcal{G}^{1}$ are exported from complete-training-pool attempts. ② Policy Learning I distills these traces and runs DAPO I with $\Phi^{1}$ frozen, producing policy $\theta^{1}$ and fresh rollouts. ③ Harness II audits Skill use, contrasts stratified successes and failures, diagnoses the current capability boundary, and selects a validated reconstructed Harness $\Phi^{2}$ . ④ Policy Learning II resumes from $\theta^{1}$ and runs DAPO II with $\Phi^{2}$ frozen. Only the resulting $\theta^{2}$ – $\Phi^{2}$ pair is evaluated on the held-out test set. Inputs, output contracts, and verifiers remain fixed throughout.

Stage / ablation	MetroMap-lite (Acc/%)	TravelMap-lite (Acc/%)	Fee-VL (F1-Score/%)	Cancel-VL (F1-Score/%)
Base	16.25	27.50	37.13	45.50
$+$ ED Init.	24.50 +8.25	30.50 +3.00	38.83 +1.70	46.99 +1.49
$+$ ED Explore	31.50 +15.25	33.25 +5.75	40.43 +3.30	49.00 +3.50
$-$ Selective Commit	29.75 +13.50	32.25 +4.75	39.25 +2.12	47.75 +2.25
$+$ Distillation	37.50 +21.25	35.50 +8.00	47.43 +10.30	58.30 +12.80
$-$ Skill-guided Trace	29.75 +13.50	31.25 +3.75	42.57 +5.44	53.12 +7.62

Table 1: RLHarness lifecycle and matched ablations. Superscripts show absolute gains (%) over Base. Minus rows denote the five component ablations. Random-Evidence Prompt + DAPO II is the continuation branch of the stratified contrastive-evidence ablation; the training control is not counted as an ablation. Acc is strict whole-route accuracy and F1-Score is fact-label macro F1.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Stratum	MetroMap-lite		TravelMap-lite
	Train	Test	Train	Test
E $\times$ E	455	133	471	110
E $\times$ M	321	74	325	90
E $\times$ H	97	17	50	16
M $\times$ E	292	63	301	76
M $\times$ M	313	84	380	78

Appendix

Table 3: Difficulty distributions of the MetroMap-lite and TravelMap-lite splits. E/M/H denote easy/medium/hard, and

\times

separates map and query difficulty; Train and Test denote the fixed Train1600 and Test400 splits.

Data role	MetroMap-lite	TravelMap-lite	Fee-VL	Cancel-VL
Complete training pool	Train1600	Train1600	7,337 training orders	8,071 training orders
Harness-I exploration	Fixed 480 from RL1600	Fixed 480 from RL1600	Fixed 480 from RL1600	Fixed 480 from RL1600
SFT corpus	1,395 audited traces	1,233 audited traces	7,146 audited traces	8,037 audited orders
DAPO I/II	Fixed 1,600 prompts	Fixed 1,600 prompts	Fixed 1,600-prompt subset	Fixed 1,600-prompt subset
Held-out test	Test400	Test400	2,905 test orders	994 test orders
Harness validation	Val100 (train source)	Val100 (train source)	50 frozen slots per fact label	50 frozen slots per fact label

Appendix

Table 4: Separation of data roles. A multilabel VL order may occupy several label-conditioned validation slots, but its training ID remains one order. Harness-validation and held-out test sets are distinct; neither enters model training.

Scenario	Tr. dec.	Te. dec.	Tr. ord.	Te. ord.	Tr. +%	Te. +%
Not boarded (5)	14,040	4,155	2,808	831	12.29%	12.95%
Delayed meter (2)	4,972	2,148	2,486	1,074	41.71%	39.99%
Detour (2)	4,086	2,000	2,043	1,000	10.03%	9.55%
Total (9)	23,098	8,303	7,337	2,905	18.22%	19.13%

Appendix

Table 5: Fee-VL scenario sizes. Tr./Te. denote train/test, dec. denotes label decisions, ord. denotes orders, and +% denotes the positive rate. Parentheses give the number of retained facts in each scenario.

Scenario	Fact label	Tr.+	Tr. $-$	Tr.N	Tr.+%	Te.+	Te. $-$	Te.N	Te.+%
Not boarded	Distance after drop-off $>100$ m	35	2,773	2,808	1.25%	13	818	831	1.56%
Not boarded	Distance after drop-off $\leq 100$ m	384	2,424	2,808	13.68%	108	723	831	13.00%
Not boarded	Accident	32	2,776	2,808	1.14%	14	817	831	1.68%
Not boarded	Rode throughout	420	2,388	2,808	14.96%	134	697	831	16.13%
Not boarded	Did not board	854	1,954	2,808	30.41%	269	562	831	32.37%
Delayed meter	Drop-off to meter end $\leq 1$ min	1,228	1,258	2,486	49.40%	449	625	1,074	41.81%

Appendix

Table 6: Fee-VL label distributions. Tr./Te. denote train/test; +/

-

are positive/negative decisions; N is the number of decisions; +% is the positive rate. Totals count order–fact decisions rather than unique positive orders.

Responsibility	Tr. dec.	Te. dec.	Tr. img.	Te. img.	Tr. +%	Te. +%
Malicious responsibility	8,071	994	8,071	994	25.66%	30.38%
Service-error responsibility	8,071	994	8,071	994	32.08%	31.49%
No responsibility	8,071	994	8,071	994	42.26%	32.60%
Indeterminate (unclear)	–	–	–	55	–	5.53%
Total (3)	24,213	2,982	8,071	994	33.33%	31.49%

Appendix

Table 7: Cancel-VL responsibility-level sizes. Tr./Te. denote train/test, dec. denotes label decisions, img. denotes deduplicated images, and +% denotes the share of all orders. The three resolved test classes cover 939/994 orders; the remaining 55 are annotated unclear and retain fact labels.

Responsibility	Fact label	Tr.+	Tr. $-$	Tr.N	Tr.+%	Te.+	Te. $-$	Te.N	Te.+%
Service error	Insufficient driver service ability	2,607	5,430	8,037	32.44%	316	678	994	31.79%
Malicious	Driver disrupts platform order	2,037	6,000	8,037	25.35%	293	701	994	29.48%
No responsibility	Policy-compliant cancellation	1,465	6,572	8,037	18.23%	152	842	994	15.29%
No responsibility	Fourth-party cause	663	7,374	8,037	8.25%	85	909	994	8.55%
No responsibility	Passenger fails ride requirements	947	7,090	8,037	11.78%	55	939	994	5.53%
No responsibility	Platform cause	284	7,753	8,037	3.53%	26	968	994	2.62%

Appendix

Table 8: Cancel-VL fact-label distributions. Tr./Te. denote train/test; +/

-

are positive/negative decisions; N is the number of decisions; +% is the positive rate. Totals count order–fact decisions.

Method	$D_{>100}$	$D_{\leq 100}$	Acc.	Ride	NoBd.	$T_{\leq 1}$	$T_{>1}$	Road	Nav.	Macro
Base	21.24	42.94	48.00	52.73	47.94	24.60	39.82	39.32	17.54	37.13
$+$ ED Init.	22.50	41.98	50.00	54.08	47.94	26.45	39.27	41.86	25.39	38.83
$+$ ED Explore	20.69	45.38	42.62	54.08	51.14	49.45	43.77	39.46	17.28	40.43
$+$ Distill.	26.19	48.38	60.61	62.09	57.14	48.45	43.77	48.46	31.78	47.43
$+$ DAPO I	34.21	50.37	53.66	71.08	57.14	42.44	44.76	53.45	35.96	49.23
$+$ PR Rebuild	34.21	51.20	52.38	71.08	58.00	40.90	45.50	51.80	29.72	48.31

Appendix

Table 9: Fee-VL per-fact F1 scores (%) along the non-ablation path.

D_{>100}

D_{\leq 100}

denote distance after drop-off above/at most 100 m; Acc., Ride, and NoBd. denote accident, rode throughout, and did not board;

T_{\leq 1}

T_{>1}

denote meter-end delay at most/above one minute; Road/Nav. denote road control and navigation mismatch. ED/PR denote exploration–distillation/post-RL reconstruction; Macro is the unweighted nine-label mean.

Method	Svc.	PlatOrd.	Policy	4th	Pass.	Platform	D–P	Macro
Base	70.00	55.00	54.00	66.00	36.00	18.00	19.51	45.50
$+$ ED Init.	71.50	56.50	55.50	67.50	35.00	20.00	22.95	46.99
$+$ ED Explore	73.00	58.00	57.00	69.00	39.00	26.99	20.00	49.00
$+$ Distill.	80.06	66.28	65.88	76.40	46.63	30.00	42.86	58.30
$+$ DAPO I	81.50	67.50	67.20	77.50	45.50	31.50	45.71	59.49
$+$ PR Rebuild	82.20	68.10	67.80	78.00	44.71	34.48	46.15	60.21

Appendix

Table 10: Cancel-VL per-fact F1 scores (%) along the non-ablation path. Svc., PlatOrd., Policy, 4th, Pass., Platform, and D–P denote insufficient service ability, driver disruption of a platform order, policy-compliant cancellation, fourth-party cause, passenger failure to meet ride requirements, platform cause, and driver–passenger disruption. ED/PR denote exploration–distillation/post-RL reconstruction; Macro is the unweighted seven-label mean.

Model	MetroMap-lite (Acc/%)	TravelMap-lite (Acc/%)	Fee-VL (F1-Score/%)	Cancel-VL (F1-Score/%)
Unadapted Qwen3.5-9B	24.50	30.50	38.83	46.99
MetroMap-lite (DAPO II)	62.00 +37.50	35.50 +5.00	48.10 +9.27	51.49 +4.50
Fee-VL (DAPO II)	31.00 +6.50	33.00 +2.50	65.81 +26.98	56.87 +9.88

Appendix

Table 12: Cross-task evaluation of checkpoint-600 policy weights. The first row is the matched unadapted baseline; superscripts report absolute gains over that baseline. Every row uses the evaluation task’s initial Skill

S_{0}

and scorer without target-domain training.

Field	Fixed value
Student and teacher	Qwen3.5-9B student. Maps use Gemini-3.5-Flash for exploration and GPT-5.6-sol as task teacher; VL uses Kimi-K2.5 for exploration and Kimi-K3 as task teacher. Deterministic code materializes and audits SFT labels.
Adaptation surface	Language-only LoRA; rank 16, alpha 32, dropout 0.05, all language modules targeted; vision tower and multimodal projector frozen.
Input processing	Qwen3.5 template; native thinking disabled; maximum image budget $1{,}000{,}000$ pixels; cutoff length 24,576 tokens; no packing; prompt tokens excluded from the training loss.
Batching	Four GPUs; per-device batch size 2; gradient accumulation 1; effective global batch size 8; gradient checkpointing enabled.
Optimization	BF16; learning rate 0.000005; cosine schedule; warmup ratio 0.05; maximum gradient norm 1.0; two epochs.
Data order	Training and data seed 42; eight preprocessing workers and four dataloader workers.

Appendix

Table 13: SFT configuration shared by all experiments. Corpus contents and corpus size are task-specific data properties, not changes to the optimizer configuration.

Field	Fixed value
Starting policy	The merged second-epoch SFT checkpoint; fresh language-only LoRA with rank 16 and alpha 32; visual and projector modules excluded and the vision tower frozen.
Training data	A fixed set of 1,600 prompts for each task; maximum prompt length 24,576; maximum response length 4,096; overlength inputs raise an error rather than being silently truncated.
Batch and rollouts	Train batch size 8; PPO mini-batch size 4 prompts; one PPO epoch; eight sampled trajectories per prompt; dynamic batching enabled.
Advantage estimator	Hybrid-DGPO with question-level difficulty weighting; temperature 2.0; no image-based prior; no standard-deviation normalization; retain only groups whose final rewards vary.
Policy loss	Token-mean aggregation; clipping lower/upper ratio 0.2 and clip-ratio-c 3.0; KL loss enabled with coefficient 0.001 and low-variance KL; KL is not added to the reward; entropy coefficient 0.
Optimizer	Learning rate 0.000005; weight decay 0.01; warmup ratio 0.03; constant schedule; gradient clipping 1.0.

Appendix

Table 14: DAPO configuration shared by both optimization rounds and all trainable experimental branches.

Task	Original scalar reward	Weight location and advantage	Ablated component
MetroMap-lite	Format + partial-route + exact-route reward	No label weight; $A_{qj}=w_{q}(r_{qj}-\bar{r}_{q})$	Set $w_{q}=1$ only
TravelMap-lite	Format + partial-route + exact-route reward	No label weight; $A_{qj}=w_{q}(r_{qj}-\bar{r}_{q})$	Set $w_{q}=1$ only
Fee-VL	Format + weighted partial fact accuracy + exact fact vector	$w_{l}^{y_{l}}$ is inside $r_{\mathrm{partial}}$ ; then $A_{qj}=w_{q}(r_{qj}-\bar{r}_{q})$	Preserve $w_{l}^{y_{l}}$ ; set $w_{q}=1$
Cancel-VL	Format + weighted partial fact accuracy + exact fact vector	$w_{l}^{y_{l}}$ is inside $r_{\mathrm{partial}}$ ; then $A_{qj}=w_{q}(r_{qj}-\bar{r}_{q})$	Preserve $w_{l}^{y_{l}}$ ; set $w_{q}=1$

Appendix

Table 15: Task-specific Hybrid-DGPO definition. The VL label-frequency weight changes the reward; the question-difficulty weight changes the advantage.

Paper stage	Policy checkpoint	Prompt and operation
Base	Unadapted Qwen3.5-9B	No external Skill.
ED-Harness Initialization	Unadapted Qwen3.5-9B	Add the task’s initial Skill bank; MetroMap-lite uses the initial 15-Skill bank.
ED-Harness Exploration	Unadapted Qwen3.5-9B	Replace the initial Skill with the Skill state produced by the first exploration pass.
Minus Selective Context Commit	Unadapted Qwen3.5-9B	Keep the same batches and calls but disable selective rejection of a legal effective edit.
Reasoning Distillation	Final SFT checkpoint	Train on frozen labels materialized from task-teacher reasoning–answer outputs $\mathcal{G}^{1}$ ; the resulting checkpoint initializes DAPO.
DAPO I	Steps 0–400	Keep the ED-Harness Prompt fixed and evaluate the policy periodically during training.

Appendix

Table 16: Mapping between the compact two-harness labels in the main ablation table and the fixed checkpoint schedule.

Comparison	Dataset	$\boldsymbol{\bar{d}}$	$\boldsymbol{s_{d}}$	95% CI	$\boldsymbol{t(9)}$	$\boldsymbol{d_{z}}$	$\boldsymbol{p_{\mathrm{Holm}}}$
Selective commit	MetroMap-lite	+1.75	1.007	[1.03, 2.47]	5.496	1.738	0.005070
Selective commit	TravelMap-lite	+1.00	0.677	[0.52, 1.48]	4.671	1.477	0.008169
Selective commit	Fee-VL	+1.18	0.879	[0.55, 1.81]	4.245	1.342	0.010642
Selective commit	Cancel-VL	+1.25	0.790	[0.68, 1.82]	5.003	1.582	0.006622
Skill-guided trajectories	MetroMap-lite	+7.75	3.078	[5.55, 9.95]	7.963	2.518	0.000528
Skill-guided trajectories	TravelMap-lite	+4.25	2.038	[2.79, 5.71]	6.595	2.086	0.001897

Appendix

Table 17: Measured paired-test results for the 24 primary component comparisons. All tests use

K=10

matched runs and two-sided paired

t

-tests. Confidence intervals are per-comparison intervals;

p

-values are Holm-adjusted across the 24-test family.

Dataset	$\boldsymbol{\bar{d}}$	$\boldsymbol{s_{d}}$	95% CI	$\boldsymbol{t(9)}$	$\boldsymbol{d_{z}}$	$\boldsymbol{p_{\mathrm{Holm}}}$
MetroMap-lite	+3.75	2.095	[2.25, 5.25]	5.660	1.790	0.000309
TravelMap-lite	+7.75	3.410	[5.31, 10.19]	7.188	2.273	0.000103
Fee-VL	+21.95	5.702	[17.87, 26.03]	12.174	3.850	0.000002725
Cancel-VL	+11.79	4.299	[8.71, 14.87]	8.673	2.743	0.000035

Appendix

Table 18: Measured paired-test results for cross-task Post-RL Prompt replacement. Positive differences favor the task-matched Prompt. The four exploratory comparisons form a separate Holm family.

Comparison	Dataset	$\boldsymbol{\bar{d}}$	$\boldsymbol{s_{d}}$	95% CI	$\boldsymbol{t(9)}$	$\boldsymbol{d_{z}}$	$\boldsymbol{p_{\mathrm{Holm}}}$
DAPO I $-$ Distillation	MetroMap-lite	+20.00	4.676	[16.66, 23.34]	13.527	4.278	0.000002207
DAPO I $-$ Distillation	TravelMap-lite	+7.50	3.096	[5.29, 9.71]	7.661	2.423	0.000187
DAPO I $-$ Distillation	Fee-VL	+1.80	1.199	[0.94, 2.66]	4.746	1.501	0.002102
DAPO I $-$ Distillation	Cancel-VL	+1.19	0.880	[0.56, 1.82]	4.276	1.352	0.002102
DAPO II $-$ PR immediate	MetroMap-lite	+6.75	3.105	[4.53, 8.97]	6.875	2.174	0.000363
DAPO II $-$ PR immediate	TravelMap-lite	+7.25	3.438	[4.79, 9.71]	6.669	2.109	0.000367

Appendix

Table 19: Measured paired-test results for DAPO-stage gains. DAPO I is measured relative to Reasoning Distillation; DAPO II is measured relative to the immediate post-reconstruction evaluation. The eight exploratory comparisons form a separate Holm family.

Aspect	Harness I: exploration–distillation	Harness II: post-RL reconstruction
Problem	No stable executable program before training	Old program no longer matches the learned policy
Evidence	A fixed task-specific sample of 480 exploration examples	Fresh rollouts from $\theta^{1}$
Context management	Preassigned batches; task-specific selective commit	Outcome- and difficulty/label-stratified contrasts
External update	Establish and refine Skills, protocols, and demonstrations	Prune stale content and synchronously rewrite all three
Output	Harness $\Phi^{1}$ and aligned traces $\mathcal{G}^{1}$	Reconstructed Harness $\Phi^{2}$
Consumer	SFT and DAPO I	DAPO II

Appendix

Table 20: The two Harnesses solve different problems and consume different evidence. Harness I is task-coverage oriented; Harness II is conditioned on the current policy boundary.

Object or stage	Input	Output and responsibility
Exploration–Distillation Harness	Original Prompt, complete image–text/table training pool, frozen validation pool, a seeded 480-example exploration subset, explorer, task teacher, and verifier.	Build the first Harness $\Phi^{1}$ from the exploration subset and export the audited accepted records $\mathcal{G}^{1}$ from complete-training-pool attempts.
SFT label materialization (outside Harness)	$\mathcal{G}^{1}$ and fixed audit rules.	Produce frozen labels $\mathcal{D}_{\mathrm{SFT}}^{1}$ without changing the Harness or model.
Policy Optimization I (outside Harness)	$\Phi^{1}$ , $\mathcal{D}_{\mathrm{SFT}}^{1}$ , and the base model.	Run SFT followed by DAPO I to obtain current policy $\theta^{1}$ .
Post-RL Reconstruction Harness	Fresh rollouts from $\theta^{1}$ , their original examples, $\Phi^{1}$ , and the frozen validation pool.	Construct contrastive contexts, reconstruct the external program, and atomically commit $\Phi^{2}$ .
Policy Optimization II (outside Harness)	Frozen $\Phi^{2}$ and checkpoint 400.	Continue under the same reward and optimizer settings to obtain $\theta^{2}$ .
Reward / Hybrid-DGPO / scorer	Rollouts, references, and task rules.	Construct policy-learning signals or evaluation results; do not edit the Harness.

Appendix

Table 21: Lifecycle and implementation boundary. Harness I builds the first learnable Harness; Harness II reconstructs it from post-RL behavior. SFT and DAPO update model parameters and are not part of either Harness.

Check item	Implementation	Required annotation or evidence	Failure handling
Candidate schema and operation coverage	Deterministic parser over Skills, operation log, and coverage audit	Source-Skill operations and dispositions	Return exact error; generation retries up to three times
Few-shot preservation	Deterministic comparison with the fixed source examples	Identities, facts, values, procedure, and final responses	Reject the rewrite; retry up to three times
Selection and marker consistency	Deterministic list, set, and marker checks	Selected Skills, used markers, placements, and coverage status	Reject duplicated or mismatched markers
Candidate execution and selection	Qwen3.5-plus execution and task scoring; task-teacher report selection	Frozen validation labels, metrics, and representative traces	Exclude invalid candidates; commit one complete $B/P/E$ package

Appendix

Table 22: Implemented candidate checks, report generation, and selection roles.

Cases	Conservative	Stage-based	Mean	Gap	Batches
64	49.00	47.25	48.13	1.75	4
128	49.75	49.50	49.63	0.25	8
256	50.50	47.25	48.88	3.25	16

Appendix

Table 23: TravelMap-lite evidence-budget audit on training-source Budget-Val400 (Accuracy/%). Conservative and Stage-based denote the two candidate-construction styles; Batches counts 16-trajectory teacher-analysis batches.

Figure 3: Multiline MetroMap-lite input used in the case. The map contains more than five lines; the analysis follows only visible map edges and does not use real-world transit knowledge.

Figure 4: Original anonymized trajectory image for the Fee-VL case. It verifies whether a spoken direction became an observed action; it does not directly reveal the responsibility label.

LLM Agent Skill Learning

Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.

See all 398 papers

Jun 1, 2026

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

Agentic reinforcement learning (RL) enables LLM agents to improve continuously from environment rewards, yet the resulting policies do not systematically accumulate reusable strategies that generalize across tasks. Modular skills can provide such reusable strategies, yet existing skill-augmented RL methods decouple skill creation from policy optimization, risking adopting skills that conflict with the evolving policy. Inspired by Anthropic's Skill Creator, we introduce ReSkill, an RL-in-the-loop skill creation framework that reconciles skill evolution with policy learning. ReSkill exploits the group-wise structure of GRPO to naturally embed three mechanisms with only marginal additional overhead: (1) an assertion-driven skill creator that diagnoses failures from past experience and proposes conditional, trigger-based skill revisions; (2) within-group rollout sampling that enables controlled comparison of skill versions, capturing which version best supports the policy's ongoing learning; and (3) Thompson Sampling with adaptive discounting to balance exploration and exploitation in skill version selection as the policy evolves. Across several domains, ReSkill consistently outperforms existing memory and skill-based RL methods, with the largest gains on unseen tasks. Analysis of the skill lifecycle shows skills being automatically created, tested, refined, and pruned as the policy improves, demonstrating reconciled skill-policy co-evolution.

Zelin He, Haotian Lin, Boran Han +6

The Pennsylvania State University · † Work done during an internship at Amazon · Amazon IntelliHub +1

Reinforcement Learning LLM Agent Skill Learning RL for LLM Agents

Ask PDF

May 10, 2026

Skill-R1: Agent Skill Evolution via Reinforcement Learning

Agentic large language models often rely on skills, reusable natural language procedures that guide planning, action, and tool use. In practice, skills are typically improved through prompt engineering or by aligning the task LLM itself, which is costly, model-specific, and often infeasible for closed-source models. Skill optimization is not a one-step problem but a recurrent process with two coupled levels of credit assignment: a useful skill must improve rollout quality under current conditioning, while a useful revision must turn observed outcomes into a better skill for the next round. We propose Skill-R1, a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards. Rather than updating the task LLM, Skill-R1 trains a lightweight skill generator that conditions on the task context, prior rollouts, and their verified outcomes to produce skills that steer a frozen task LLM. This preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates. Skill-R1 proceeds over multiple generations: at each step, the current skill induces rollouts whose verified outcomes are fed back to produce the next revision. To optimize this recurrent process, we introduce a bi-level group-relative policy optimization objective combining intra-generation and inter-generation advantages. The intra-generation term compares rollouts under shared skill conditioning, while the inter-generation term rewards revisions that improve behavior across successive generations. Together, these provide a principled objective for directional skill evolution rather than one-shot self-refinement. Empirically, Skill-R1 achieves consistent gains over no-skill baselines and standard GRPO across benchmarks with verifiable rewards, with particularly strong improvements on complex, multi-step tasks.

Yash Vishe, Rohan Surana, Xunyi Jiang +8

UC San Diego · Adobe Research

Agent Skill Learning RL for LLM Agents

Ask PDF

Feb 9, 2026

SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Large Language Model (LLM) agents have shown stunning results in complex tasks, yet they often operate in isolation, failing to learn from past experiences. Existing memory-based methods primarily store raw trajectories, which are often redundant and noise-heavy. This prevents agents from extracting high-level, reusable behavioral patterns that are essential for generalization. In this paper, we propose SkillRL, a framework that bridges the gap between raw experience and policy improvement through automatic skill discovery and recursive evolution. Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank, an adaptive retrieval strategy for general and task-specific heuristics, and a recursive evolution mechanism that allows the skill library to co-evolve with the agent's policy during reinforcement learning. These innovations significantly reduce the token footprint while enhancing reasoning utility. Experimental results on ALFWorld, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance, outperforming strong baselines over 15.3% and maintaining robustness as task complexity increases. Code is available at this https://github.com/aiming-lab/SkillRL.

Peng Xia, Jianwen Chen, Hanyang Wang +10

UNC-Chapel Hill · University of Chicago · University of California San Diego +3

LLM Agent Skill Learning RL for LLM Agents Memory-Augmented Language Model Agents

Ask PDF

Abstract

LLM Agent Skill Learning

Explore similar work

ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RL

Skill-R1: Agent Skill Evolution via Reinforcement Learning

SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning