Which Agent's Skill Should Change? Group-Relative Credit Assignment for Evolving Multi-Agent Teams

Yanan Ma; Yang Wang; David Le; Chenhan Yuan; Yizheng Sun; Guangwei Zhang; Chenghao Xiao; Chenghua Lin

cs.CLOct 19, 2026

Which Agent's Skill Should Change? Group-Relative Credit Assignment for Evolving Multi-Agent Teams

Authors: Yanan Ma, Yang Wang, David Le, Chenhan Yuan, Yizheng Sun, Guangwei Zhang, Chenghao Xiao, Chenghua Lin,

Organizations: Mayo Clinic · University of Manchester · Alibaba Token Hub, Alibaba Group · Shanghai University of Finance and Economics

Abstract

Cooperative language-model agents can improve without weight updates by revising reusable skills, but only if a shared team reward is traced to the responsible agent and skill. Existing methods credit the skills present in each trajectory, yet presence is not responsibility: a harmful skill active in successful runs still earns positive credit. We present GRSE, Group-Relative Skill Evolution, which assigns behavioral credit by comparison: inspired by GRPO, it executes each question several times under the same skill libraries and contrasts the executions to credit agent--skill pairs, with structured diagnosis of failures. On HotpotQA and LoCoMo with two open-weight actors, GRSE outperforms MASkills, the closest prior method, on HotpotQA and every LoCoMo category except open-domain, with the largest gains on multi-hop questions ( $+6.7$ and $+3.2$ F1). Its gains rest on better-grounded retrieval rather than answers given without evidence, and its learned skills transfer between the multi-hop tasks. GRSE is robust to corrupted libraries: starting from planted harmful skills, it credits every planted skill with the correct sign and recovers the score of a clean library ( $+6.3$ points), whereas MASkills misattributes two and falls below its starting point.

Figures & tables

Figure 1: Each iteration (1) samples a group of team executions per question (§ 3.1 ), (2) credits agent–skill pairs by contrasting these executions and diagnoses failures in a structured form (§ 3.2 ), and (3) aggregates the credit across questions into skill edits, each kept only if it does not lower the validation score (§ 3.3 – 3.4 ). The example is from one training run on LoCoMo single-hop questions, in which Agent 1 retrieves evidence and Agent 2 answers. In iteration 1, a failure that surfaces in Agent 2’s answer is traced to Agent 1’s retrieval, and a new retrieval skill is induced. In iteration 3, this skill is credited as harmful in the shown group but helpful across the iteration’s 16 groups, so it is refined rather than pruned. In Agent 1’s search queries, | denotes OR.

Single model, no agents
Actor Model	Qwen3.8	Gemma4
IO	81.6	81.3
CoT	80.7	82.5
CoT-SC	82.6	83.3
Self-Refine	84.1	81.1
Two agents, 5 iterations

Table 1: HotpotQA test F1 (

\times

100) with a greedy actor on 500 test questions. Columns give the actor model (Qwen3.8-27B, Gemma-4-31B-it). Qwen3.8-27B is critic and optimizer throughout. Empty library is this system before any skill is learned (iteration 0). Mean over three runs, with SD as a subscript. CoT-SC uses 5 samples, Self-Refine at most 3 feedback rounds. Best per column in bold . Best two-agent setting is underlined .

Actor: Qwen3.8-27B
Task ( $N$ )	Multi-hop (156)		Temporal (179)		Open-domain (52)		Single-hop (453)
Method	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU
Full-context	49.9	40.0	48.3	25.9	36.4	31.6	70.8	57.6
Empty library (iter. 0)	49.7	42.8	70.6	48.5	24.7	16.4	59.7	51.0
MASkills	46.7 $\pm$ 1.4	39.4 $\pm$ 0.8	72.8 $\pm$ 1.3	51.8 $\pm$ 2.1	31.8 $\pm$ 4.4	22.8 $\pm$ 5.1	59.0 $\pm$ 0.6	49.5 $\pm$ 1.2
GRSE (ours)	53.4 $\pm$ 0.8	44.7 $\pm$ 0.7	73.4 $\pm$ 2.1	52.9 $\pm$ 2.2	24.7 $\pm$ 1.1	15.9 $\pm$ 1.3	59.1 $\pm$ 4.5	50.0 $\pm$ 4.1

Table 2: LoCoMo test results per question category: token F1 and smoothed BLEU (

\times

100) with a greedy actor. Full-context is a single model that receives the entire conversation, following the LoCoMo protocol (no agents or skills). The remaining rows use the two-agent system trained for 5 iterations. Empty library is this system before any skill is learned (iteration 0). Mean over three runs, with SD as a subscript. Qwen3.8-27B serves as critic and optimizer for both skill methods. Best per column in bold ; best two-agent result underlined .

		Contribution to $\Delta$			% of questions
Method	$\Delta$	D ${\to}$ C	W ${\to}$ C	C ${\to}$ W	Decl.	No ev.	#D ${\to}$ C
HotpotQA, Qwen3.8
MASkills	$+0.7$	$0.0$	$+2.3$	$-1.8$	$0{\to}0$	–	–
GRSE	$+2.3$	$0.0$	$+2.6$	$-0.6$	$0{\to}0$	–	–
HotpotQA, Gemma4
MASkills	$+0.7$	$0.0$	$+1.2$	$-0.8$	$1{\to}1$	–	–

Table 3: Answer transitions from the empty to the final library. Each test question is declined (empty or “not mentioned”), correct (F1

\geq 0.5

), or wrong under each library.

\Delta

: change in mean token F1 (

\times

100). D

\to

C, W

\to

C, C

\to

W: contribution to

\Delta

of questions moving from declined to correct, wrong to correct, and correct to wrong; the remaining transitions make up the rest. Decl.: share of declined answers, before

\to

after. No ev.: share of questions whose retrieved evidence cites no dialogue turn (e.g., [D3:5]), before

\to

after. #D

\to

C: number of declined-to-correct questions, with those answered without cited evidence in (). No ev. and #D

\to

C are reported for LoCoMo only, as HotpotQA has no retrieval step. All rows use the first run of each method; on LoCoMo, each category has its own library, and empty-library answers come from iteration 0 of the same GRSE run.

Library	F1	BLEU	$\Delta$ F1	Abst. (%)
Perturbed (start)	60.6	52.1	–	9.7
Helpful skills only	66.7	57.4	$+6.1$	4.2
MASkills	58.4	49.3	$-2.2$	7.9
GRSE	66.9	57.4	$\mathbf{+6.3}$	4.6

Table 4: Repairing a perturbed skill library on LoCoMo single-hop questions (Qwen3.8 actor, 453 test questions, greedy decoding, one run per method). Token F1 and smoothed BLEU (

\times

100) are measured with the final library.

\Delta

F1 is the change in F1 from the perturbed starting library. Abst. is the percentage of refusal answers such as “not mentioned”. Bold marks the best learned library. GRSE beats MASkills by

+8.5

F1.

		GRSE		MASkills
Planted skill	Kind	$u_{1}$	Fate	$u_{1}$	Fate
Agent 1 (retriever)
context expansion	help	$+0.18$	merged 2	$+0.03$	merged 2
latest session	harm	$-0.12$	merged 2	$+0.07$	merged 2
equation solving	irr.	$0.00$	kept	$0.00$	merged 2
Agent 2 (reasoner)

Table 5: Credit assigned to the planted skills of the robustness study (Figure 4 ). Kind: helpful (help), harmful (harm), or irrelevant (irr.).

u_{1}

: aggregated utility after the first update; it has the correct sign if positive for helpful and negative for harmful skills ( red : wrong sign). Fate: the skill’s status by iteration 5, with the iteration of the change as subscript. Same runs as Table 4 .

MASkills
Target	LoCoMo MH	HotpotQA
Empty	50.2	80.7
In-domain	46.7 $\pm$ 1.4	80.6 $\pm$ 0.6
Transfer	50.2 $\pm$ 1.5	82.7 $\pm$ 0.7
GRSE
In-domain	53.4 $\pm$ 0.8	83.7 $\pm$ 0.5

Table 6: Transfer between the two multi-hop tasks (Qwen3.8 actor, test F1

\times

100, mean

\pm

SD over three runs). Transfer : final libraries from the other task (HotpotQA libraries on LoCoMo multi-hop questions and vice versa); each agent keeps the target task’s role. In-domain : libraries learned on the target (Tables 1 and 2 ).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Datasets

HotpotQA : distractor setting, development set (all questions labeled hard ). Split: 100 training, 50 validation, and 500 test questions (indices 150–649). Source: hotpot_dev_distractor_v1.json from https://hotpotqa.github.io . License: CC BY-SA 4.0.

LoCoMo : 10 conversations, question categories 1–4. Split: 2 training, 2 validation, and 6 test conversations (seed 42, fixed for all runs); open-domain uses 2/2/5, since conv-30 has no open-domain question. Test questions: 156 multi-hop, 179 temporal, 52 open-domain, and 453 single-hop; 842 in Mix. Source: locomo10.json from https://github.com/snap-research/locomo . License: CC BY-NC 4.0.

Model checkpoints (Hugging Face)

Qwen3.8-27B : Qwen/Qwen3.8-27B , revision 1d4bf0f ; actor, critic, and optimizer.

Gemma4-31B : google/gemma-4-31B-it , revision 842da37 ; actor.

Appendix

Table 7: Datasets, data splits, and model checkpoints.

		LoCoMo
	HotpotQA	Per Cat.	Mix
Iterations	5	5	5
Batch size $\|\mathcal{B}_{t}\|$	100	24	32
Rollouts per question $G$	4	4	4
Validation set size $\|\mathcal{V}\|$	50	21–159	80
Validation executions $H$	1	1	1

Appendix

Table 8: Training settings, shared by GRSE and MASkills except that MASkills uses one rollout per question. Per-category validation uses every question of that category in the two validation conversations (multi-hop 68, temporal 60, open-domain 21, single-hop 159).

HotpotQA, Qwen3.8
		Share of skills (%)
Method	$n$	Fmt	Retr	Verif	Reas	Abst	Words
MASkills	6	17	0	33	50	0	504
GRSE w/o struct.	8	50	0	25	25	0	606
GRSE w/ struct.	7	57	0	29	14	0	502
HotpotQA, Gemma4

Appendix

Table 9: Final skill libraries by primary skill type. Each skill is assigned one primary type by Qwen3.8-27B: answer formatting (Fmt), retrieval (Retr), verification (Verif), reasoning (Reas), or abstention (Abst).

n

: number of skills. Words: mean length of description and body. On LoCoMo, the four per-category libraries are pooled. w/ and w/o struct.: with and without structured failure analysis. First runs.

Panel of Figure 2 — Figure 2: Learned Skill Case A: a declined question becomes correct with cited evidence (Gemma4 actor, LoCoMo single-hop, GRSE w/o structure). The answer turn does not contain the keyword of the question; the learned skill fetches the turn that follows a matched question. Evidence abridged.

Panel of Figure 3 — Figure 3: Learned Skill Case B: a wrong answer becomes correct (Qwen3.8 actor, HotpotQA, GRSE w/o structure). The final Agent 1 follows the steps of its learned skills, whose headings it reproduces.

Panel of Figure 4 — Figure 4: The perturbed seed library of the robustness experiment (§ 5.3 ). Each agent starts with three skills: one helpful skill and two harmful or irrelevant ones. Skill bodies are reproduced verbatim; labels are for the reader only.

Panel of Figure 5 — Figure 5: Robustness Case 1: the harmful skill focused-latest-session-retrieval . The same single-hop test question is answered with the perturbed library and with the final libraries of MASkills and GRSE (greedy decoding; evidence abridged to the cited turns). MASkills keeps the single-keyword behavior of the harmful skill. GRSE keeps the question–answer expansion of the helpful one.

Panel of Figure 6 — Figure 6: Robustness Case 2: the harmful skill verbatim-evidence-guard during training. Utility is the library’s running estimate of the skill’s value, aggregated from critic credits.

	Multi-hop		Temporal		Open-domain		Single-hop		All		Avg.
Setting	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU
Empty library (iter. 0)	50.9	43.5	70.8	48.2	26.8	16.7	59.6	50.8	58.3	46.7	52.0	39.8
GRSE	52.3	44.0	73.1	51.5	33.9	22.3	60.2	50.9	59.8	47.9	54.9	42.2
w/o structure	51.5	42.1	70.6	48.5	31.6	20.9	62.0	52.4	59.9	47.7	53.9	41.0
$G{=}1$	53.7	44.6	72.0	51.3	30.5	19.4	60.4	51.4	59.7	48.1	54.1	41.7
DAPO filter	53.4	44.0	71.6	51.0	29.8	18.1	62.6	53.4	60.7	48.9	54.4	41.6

Appendix

Table 10: Ablation on LoCoMo with one shared library learned on all four categories together (Qwen3.8 actor, greedy decoding, one run per setting, 842 test questions). Token F1 and smoothed BLEU (

\times

100) per question category, over all questions (All), and averaged over the four categories with equal weight (Avg.). Single-hop questions make up 54% of All. Best per column in bold , second best underlined . w/o structure removes the failure-diagnosis schema and its checks.

G{=}1

samples one team trajectory per question instead of four, so no group-relative comparison is possible. DAPO filter keeps four rollouts but, following the dynamic sampling of DAPO, sends only groups whose rewards differ to the critic. w/ label conditions the critic on the question category and groups update evidence by category.

Wall-clock training time (h)
	HotpotQA		LoCoMo
	Qwen3.8	Gemma4	Qwen3.8	Gemma4
MASkills	2.8 $\pm$ 0.8	4.6 $\pm$ 1.1	15.9 $\pm$ 0.7	19.0 $\pm$ 1.3
GRSE	4.3 $\pm$ 0.6	5.9 $\pm$ 0.6	20.8 $\pm$ 1.6	23.4 $\pm$ 2.5
Actor tokens, training rollouts (M)
MASkills	2.7 $\pm$ 0.3	2.6 $\pm$ 0.2	6.1 $\pm$ 0.2	4.2 $\pm$ 0.3

Appendix

Table 11: Training cost of the main-result runs (mean

\pm

SD over three runs of five iterations; LoCoMo sums the four per-category runs). Actor tokens are the input and output tokens of both agents; critic and optimizer tokens are not logged and are excluded.

Panel of Figure 7 — Figure 7: Agent roles on HotpotQA.

Panel of Figure 8 — Figure 8: Agent 1 role on LoCoMo.

Panel of Figure 9 — Figure 9: Agent 2 role on LoCoMo.

Panel of Figure 10 — Figure 10: Prompts of the single-model baselines.

Panel of Figure 11 — Figure 11: Critic prompt for group-relative credit assignment.

Panel of Figure 12 — Figure 12: Structured failure-diagnosis instructions.

Panel of Figure 13 — Figure 13: Skill update operators.

LLM Agent Skill Learning

Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.

See all 402 papers

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning
May 7, 2026Yaorui Shi, Yuxin Chen, Zhengxi Lu +6LLM Agent Skill Learning LLM Agent Skill Retrieval
University of Science and Technology of China · 2Meituan · 3National University of Singapore +2
Coalition-Aware Skill Reliability for Self-Evolving Agents
Aug 23, 2026Qiyan Zhao, Xiaofeng Zhang, Bo Liu +11LLM Agent Skill Learning LLM Agent Reliability
CASIA · LongShine AI Lab · SJTU +2
SkillShapley: Boundary-Adaptive Shapley Valuation for Skill Step Attribution in LLM Agents
Aug 13, 2026Chang Liu, Yuqi Zhang, Yiman Zhong +3LLM Agent Skill Learning Shapley Value Attribution
1Beihang University, Beijing, China · 2Shandong University, Jinan, China

May 7, 2026J/K move · Enter open · S save

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao+5

University of Science and Technology of China · 2Meituan · 3National University of Singapore+2

LLM Agent Skill Learning LLM Agent Skill Retrieval Policy Learning RL for LLM Agents Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

A persistent skill library allows language model agents to reuse successful strategies across tasks. Maintaining such a library requires three coupled capabilities. The agent selects a relevant skill, utilizes it during execution, and distills new skills from experience. Existing methods optimize these capabilities in isolation or with separate reward sources, resulting in partial and conflicting evolution. We propose Skill1, a framework that trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective. The policy generates a query to search the skill library, re-ranks candidates to select one, solves the task conditioned on it, and distills a new skill from the trajectory. All learning derives from a single task-outcome signal. Its low-frequency trend credits selection and its high-frequency variation credits distillation. Experiments on ALFWorld and WebShop show that Skill1 outperforms prior skill-based and reinforcement learning baselines. Training dynamics confirm the co-evolution of the three capabilities, and ablations show that removing any credit signal degrades the evolution.

Abstract

LLM Agent Skill Learning

Explore similar work

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

Coalition-Aware Skill Reliability for Self-Evolving Agents

SkillShapley: Boundary-Adaptive Shapley Valuation for Skill Step Attribution in LLM Agents