One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents

Jie Zhao; Ziyu Jiang; Suhang Zheng; Minghui Shan; Xiaoxiao Xu; Lin Qu

cs.SESep 20, 2026

One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents

Authors: Jie Zhao, Ziyu Jiang, Suhang Zheng, Minghui Shan, Xiaoxiao Xu, Lin Qu

Organizations: Alibaba Group Model · Logics-SWE-Qwen3.6-27B

Abstract

Repository-level software engineering (SWE) comprises heterogeneous task categories, whose progress under pooled agentic reinforcement learning can be uneven: gains in some categories coincide with regressions in others, while aggregate resolution obscures these changes. Motivated by this category see-saw, we develop a category-aware expert-training and policy-integration framework. Executable task construction and SWE Labeler, an evidence-grounded multi-axis labeling system, organize the training pools. Initial category-specific RL improves average training success while leaving uneven instance-level progress, motivating explicit consolidation of successful behavior and policy-adaptive task selection. Same-origin category experts alternate long-horizon Agentic-miniRL with Refresh-Repair-Expand (RRE): the updated policy refreshes instance mastery, reuses its own verified successful trajectories for Repair SFT, and reselects tasks for further RL. Label-routed multi-teacher on-policy distillation (MOPD) consolidates the experts into one deployable student, with ReLU-gated reward extrapolation keeping only each teacher's improving direction over the reference. Expert training and policy integration require no external model to provide solution trajectories or action targets. We evaluate Pooled RL and Balanced RL, expert development, and single-model integration through aggregate and per-category resolution, the minimum category lift over each joint-RL baseline, and expert-gain recovery. The final MOPD policy achieves mean resolution of 58.04% on Pro-618 and 59.00% on SWE-bench Multilingual, improving over the base model by 5.39 and 2.78 percentage points, respectively.

Figures & tables

Figure 1: Category-aware agentic RL for software engineering. (a) Two observations motivate the framework: aggregate scores can conceal opposing category-level changes during joint RL, while a fixed task pool becomes stale as instance mastery drifts over long-horizon training. (b) SWE Labeler organizes executable tasks into categorized training pools; three same-origin training tracks alternate Agentic-miniRL with Repair SFT before their experts are consolidated into one student through pure label-routed MOPD. (c) The three mechanisms provide long-horizon policy improvement, mastery-aware repair, and multi-teacher on-policy distillation without an external trajectory teacher.

Figure 2: Category dynamics under Pooled RL. A single policy is trained on the full 6,723-task mixture and evaluated on the fixed Pro-618 split (Pro-A/B/C: 221/201/196 tasks). Faint traces show the mean at each checkpoint; bold traces show centered three-checkpoint moving averages. Step 0 anchors the base model. Left: overall and category gains relative to their respective base-model rates. Right: overall gain and minimum category gain

G_{\mathrm{sim}}

; their shaded difference between the smoothed curves is the smoothed

\mathrm{SSG}

Full gain	$G_{\mathrm{sim}}$	$\mathrm{SSG}$	Interpretation relative to our objective
$\uparrow$	$\uparrow$	$\downarrow$	Aligned improvement: overall and minimum gains rise while the gap narrows.
$\uparrow$	$\uparrow$	$\rightarrow$	Shared improvement with a stable gap: overall and minimum gains rise together, with unchanged separation.
$\uparrow$	$\uparrow$	$\uparrow$	Shared improvement with a wider gap: both gains rise, but overall gain increases faster than minimum gain.
$\uparrow$	$\rightarrow$ / $\downarrow$	$\uparrow$	Aggregate improvement without minimum-gain improvement: overall gain rises while minimum gain stays unchanged or declines.
$\downarrow$	$\rightarrow$ / $\downarrow$	$\downarrow$	Gap reduction through aggregate decline: the gap narrows without improving the minimum gain; this does not meet our improvement objective.
$\downarrow$	$\uparrow$	$\downarrow$	Minimum–aggregate trade-off: the minimum gain improves and the gap narrows at the cost of overall gain.

Table 1: Interpreting representative changes in overall and category gains.

G_{\mathrm{sim}}

is the minimum category gain over the base;

\mathrm{SSG}

is the gap between overall and minimum category gains. Arrows compare two checkpoints or training settings using the same base and category weights.

Figure 3: Semantic breadth of the SWE Labeler taxonomy. The two panels show the 26 Task Type and 21 Repository Domain L1 families. Tile area is proportional to the number of declared L2 labels, and tile text shows selected L2 examples. The inventory represents the semantic resolution available to the Labeler.

Labeling corpus	$N$	Langs.	Leading Task Type	Leading Repository Domain	Scale signature
SWE-bench Verified	500	1	bug-fix (87.0%)	web_backend (46.4%)	86.2% single-file
SWE-bench Pro	731	3	bug-fix (47.3%)	devops_infra (33.2%)	59.2% cross-module
SWE-bench Multilingual	300	8	bug-fix (89.3%)	lang_runtime (24.7%)	66.0% single-file

Table 2: Cross-benchmark profiles produced by SWE Labeler. The three labeled corpora contain 1,531 instances. Percentages are computed within each corpus; the last column reports a representative high-mass scale label.

Figure 4: Refresh–Repair–Expand training of category experts. The same loop is instantiated independently for categories A, B, and C from a common strong base. Initial selection reuses the base-model probes from corpus construction. Agentic RL explores on an informative category frontier and records the expert’s own verifier-approved trajectories. Mastery is refreshed on the preceding RL training set after both RL and Repair SFT. The post-RL estimates guide successful-trajectory replay; the post-SFT estimates track the repaired policy’s mastery. Before the next RL phase, the selected SFT policy probes an expanded pool that augments the preceding RL set with randomly sampled base-zero and base-one tasks. Solid navy arrows track stage and policy progression, green arrows reuse self-generated verifier-approved trajectories, and dashed control arrows indicate mastery-weighted replay. The blue return path closes the next RRE round. No external trajectory teacher is used.

Setting	Pro-618	SWE-bench Multilingual
Instances	618	300
A/B/C/U	221/201/196/0	72/25/176/27
Agent framework	R2E-Gym	SWE-agent
Context limit (tokens)	200,000	200,000
Output limit per call (tokens)	25,600	12,280
Maximum actions/iterations	150	150

Table 3: Benchmarks and configured inference budgets for policy evaluation. U denotes instances outside the A/B/C mapping; all instances contribute to Full resolution.

		Category
Phase	Quantity	A	B	C	Total
Pooled RL	training instances	—	—	—	6,723
Balanced RL	training instances	516	516	516	1,548
Initial expert RL	training records	601	516	1,652	2,769
First Repair SFT	repairable instances	571	501	1,498	2,570
First Repair SFT	sampled preceding-RL trajectories	1,623	1,481	4,384	7,488

Table 4: Training data used by the controlled experiments. A, B, and C are shown in separate columns. Candidate and repairable rows report the support available to each selection step; training and sampled rows report the materialized data. Repair trajectories are sampled only from verifier-approved rollouts generated by the same expert in the immediately preceding RL phase. MOPD repeats A/B task records to match C’s size; source-record counts and upsampled record counts are shown separately.

Current successes	Target trajectories	Sampling emphasis
0 or 1 of 4	4	strongest repair pressure
2 of 4	3	unstable frontier
3 of 4	2	consolidation
4 of 4	1	retention

Table 5: Mastery-adaptive sampling for Repair SFT. Mastery is measured with four fresh executable rollouts. Each selected trajectory must be a verifier-approved success previously generated by the same expert.

Setting	Agentic RL	Repair SFT
Objective	Agentic-miniRL with group-relative advantages	full-parameter, response-only cross-entropy
Context	131,072 tokens; 150 actions; 12,288 generated tokens/action	131,072 tokens
Batch	16 tasks/update, $G{=}8$ ; 128 trajectories	micro-batch 1, accumulation 16; effective batch 64/64/128
Optimization	lr $3\!\times\!10^{-7}$ ; weight decay .01; cosine; 20 warmup steps	Adam, lr $5\!\times\!10^{-6}$ ; weight decay .01; cosine; 5% warmup
Stabilization	$(\epsilon_{\mathrm{low}},\epsilon_{\mathrm{high}})=(.20,.27)$ ; importance ratio $[0,5]$ ; KL coefficient .001; grad norm 20	bf16; grad norm 1
Generation	temperature 1.0; top- $p$ .95; top- $k$ 20	—

Table 6: Core training configurations. RL settings are shared by the initial and expanded phases and by all three category experts. Category-valued SFT entries are ordered A/B/C.

Risk surface	Control	Integrity condition
Problem-statement leakage	Separate fix and test patches; reject generated statements that reveal source/test paths or patch-added code.	The agent receives the issue-style specification and base repository, not the reference implementation or held-out test patch.
Future repository state	Reset to the base revision; remove remotes, non-current branches, post-base tags, reflogs, and unreachable objects. Pro-618 removes all tags in the rollout sandbox.	Future fixes cannot be recovered through local git log , git show , tags, or dangling objects.
Verifier and environment tampering	Defer held-out test injection; export only a sanitized source patch and replay it in a fresh instance sandbox.	Reward depends on the replayed patch under evaluator-controlled tests, not on rollout-side files, processes, packages, or test edits.
External answer retrieval	Inspect the full action trajectory for target-repository search, clone/fetch, patch/raw downloads, and exact fix-SHA access.	Confirmed retrieval is an integrity failure, receives a score of zero, and remains in the fixed evaluation denominator.

Table 7: Defense-in-depth protocol for preserving reward integrity. Each control closes a different path from an executable pass to a misleading training or evaluation signal.

Figure 5: Category dynamics under Pooled RL and Balanced RL. Panel (a) compares Pro-618 Full resolution; panels (b) and (c) show the corresponding Pro-A/B/C trajectories on identical category-score axes. Faint points show two-round mean evaluation scores at each checkpoint. Lines show five-point centered moving averages. Category scores continue to exhibit opposing movements under Balanced RL.

Training mixture	Full gain $\uparrow$	$G_{\mathrm{sim}}$ $\uparrow$	$\mathrm{SSG}$ $\downarrow$	Final Full gain $\uparrow$
	Training-process mean
Pooled RL	1.13	0.08	1.05	+2.86
Balanced RL	1.37	0.19	1.18	+2.70

Table 8: Category-gain summaries under Pooled RL and Balanced RL. Training-process columns average per-checkpoint metrics over the 50 displayed checkpoints per method, using two-round mean scores as in Figure 5 . The last column reports the final models’ three-round mean Full gain, using the same models as Table 12 . All gains use the same three-round base-model reference; values are in percentage points.

Expert	$N$	Base	Initial RL	Repair SFT	$\Delta_{\mathrm{SFT-RL}}$	$\Delta_{\mathrm{SFT-Base}}$
A	601	38.79	46.05	56.28	$+10.23$	$+17.49$
B	516	38.78	45.54	59.59	$+14.05$	$+20.81$
C	1,652	38.68	43.28	52.75	$+9.47$	$+14.07$

Table 9: Training-record success rates through initial RL and first Repair SFT. Scores are equal-weight means over the same matched training records (%). The final two columns report SFT gains in percentage points.

Expert	Stage	$N$	Higher	Equal	Lower
A	Initial RL	601	259	151	191
A	Repair SFT	601	336 ( $+77$ )	161 ( $+10$ )	104 ( $-87$ )
B	Initial RL	516	216	154	146
B	Repair SFT	516	314 ( $+98$ )	121 ( $-33$ )	81 ( $-65$ )
C	Initial RL	1,652	633	517	502
C	Repair SFT	1,652	877 ( $+244$ )	398 ( $-119$ )	377 ( $-125$ )

Table 10: Training-record gains and losses relative to the base model. Both stages use the same matched training records within each category. Higher/equal/lower count observed pass rates relative to Base, not changes from the immediately preceding stage. Parentheses in the Repair SFT rows show the change in each count relative to Initial RL.

Figure 6: Category-expert training dynamics in the two RL stages. Panels (a) and (b) show training rollout scores for initial RL (Stage 1) and expanded RL (Stage 3), respectively. Faint traces show raw scores; bold traces show 20-update trailing means, with shorter windows at the start. Both panels use the same 30–70% vertical range; boundary triangles mark raw scores outside this range. Each expert is trained on its own category pool, which is refreshed between stages.

Training stage	Expert A / Pro-A	Expert B / Pro-B	Expert C / Pro-C
Base	$51.73\pm 0.43$	$54.39\pm 1.17$	$51.87\pm 1.27$
Initial RL	$53.54\pm 0.77$ [-1pt] $(+1.81)$	$55.06\pm 0.62$ [-1pt] $(+0.66)$	$52.55\pm 1.10$ [-1pt] $(+0.68)$
First Repair SFT	$57.32\pm 0.43$ [-1pt] $(+5.58)$	$56.55\pm 0.62$ [-1pt] $(+2.16)$	$55.44\pm 0.96$ [-1pt] $(+3.57)$
Expanded RL	$58.67\pm 1.19$ [-1pt] $(+6.94)$	$58.37\pm 2.00$ [-1pt] $(+3.98)$	$56.97\pm 0.96$ [-1pt] $(+5.10)$
Second Repair SFT	$59.58\pm 1.40$ [-1pt] $(+7.84)$	$58.87\pm 1.31$ [-1pt] $(+4.48)$	$57.48\pm 0.64$ [-1pt] $(+5.61)$

Table 11: Stage-wise RRE expert results. Entries report target-category resolution (%) as mean

\pm

population standard deviation across three evaluation rounds. Parentheses below each entry give the mean gain over the base model in percentage points.

Figure 7: Learning during multi-teacher on-policy distillation. Panel (a) shows the student’s training rollout score; panel (b) shows student–teacher KL. Faint traces show raw values and bold traces show 20-step trailing means, with shorter windows at the start. Teacher alignment improves alongside the student’s training task performance.

Policy	Full $\uparrow$	Pro-A $\uparrow$	Pro-B $\uparrow$	Pro-C $\uparrow$
Base model	$52.64{\scriptstyle\,\pm\,0.28}$	$51.73{\scriptstyle\,\pm\,0.43}$	$54.39{\scriptstyle\,\pm\,1.17}$	$51.87{\scriptstyle\,\pm\,1.27}$
Pooled RL	$55.50{\scriptstyle\,\pm\,0.46}$ [-1pt] (+2.86)	$54.75{\scriptstyle\,\pm\,1.33}$ [-1pt] (+3.02)	$56.72{\scriptstyle\,\pm\,1.07}$ [-1pt] (+2.32)	$55.10{\scriptstyle\,\pm\,1.44}$ [-1pt] (+3.23)
Balanced RL	$55.34{\scriptstyle\,\pm\,0.92}$ [-1pt] (+2.70)	$54.45{\scriptstyle\,\pm\,0.85}$ [-1pt] (+2.71)	$57.05{\scriptstyle\,\pm\,0.94}$ [-1pt] (+2.65)	$54.59{\scriptstyle\,\pm\,1.25}$ [-1pt] (+2.72)
Category-routed experts (3 models)	—	$59.58{\scriptstyle\,\pm\,1.40}$ [-1pt] (+7.84)	$58.87{\scriptstyle\,\pm\,1.31}$ [-1pt] (+4.48)	$57.48{\scriptstyle\,\pm\,0.64}$ [-1pt] (+5.61)
MOPD	$58.04{\scriptstyle\,\pm\,0.20}$ [-1pt] (+5.39)	$58.07{\scriptstyle\,\pm\,1.13}$ [-1pt] (+6.33)	$59.37{\scriptstyle\,\pm\,1.02}$ [-1pt] (+4.98)	$56.63{\scriptstyle\,\pm\,1.10}$ [-1pt] (+4.76)

Table 12: Overall and category-wise resolution on Pro-618 (%). Base, Pooled RL/Balanced RL, and MOPD scores, and category-wise entries of the routed expert reference, report mean

\pm

population standard deviation across three evaluation rounds. Parentheses below each non-base score give the change from the corresponding base-model mean in percentage points, computed before rounding. The category-routed experts report the corresponding expert’s result in each category before single-model integration.

Figure 8: Category gains before and after single-model integration. Bars show three-round mean resolution gains over the same base model. Each expert is evaluated on its corresponding category; MOPD uses one integrated model for all categories. Recovery below each group is the fraction of the corresponding expert’s gain retained by MOPD (Equation 29 ). Round standard deviations are reported in Table 12 .

Policy	Full gain $\uparrow$	$G_{\mathrm{sim}}$ $\uparrow$	$\mathrm{SSG}$ $\downarrow$
Pooled RL	2.86	2.32	0.54
Balanced RL	2.70	2.65	0.04
MOPD	5.39	4.76	0.63

Table 13: Final-policy gains on Pro-618. Metrics are computed from three-round mean scores relative to the same base model, in percentage points.

G_{\mathrm{sim}}

is the minimum A/B/C gain;

\mathrm{SSG}

is Full gain minus

G_{\mathrm{sim}}

. Bold marks the best value among the three single policies for each metric.

Policy	Full $\uparrow$	A $\uparrow$	B $\uparrow$	C $\uparrow$	U $\uparrow$
Base model	$56.22\pm 0.68$	44.91	56.00	61.74	50.62
Pooled RL	$55.56\pm 0.83$	47.69	53.33	58.52	59.26
Balanced RL	$57.00\pm 1.19$	48.61	56.00	59.66	62.96
MOPD	$59.00\pm 0.47$	51.39	64.00	61.74	56.79

Table 14: Resolution on SWE-bench Multilingual with SWE-agent (%). Full reports mean

\pm

population standard deviation across three evaluation rounds per reported model; category entries are cross-round means. Each round contains the same 300 instances. A/B/C contain 72/25/176 instances; U denotes the 27 unassigned instances, included in Full.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Construction stage	Instances
Full SWE-bench Pro	731
Public audit: mechanically evidenced specification/determinacy issue	$-83$
Public audit: two-model-family underspecification finding	$-26$
Public audit: reference patch fails isolated grader replay	$-3$
Official OpenAI audit: prompt–test mismatch	$-1$
Pro-618	618

Appendix

Table 15: Construction of the Pro-618 evaluation subset. Counts are mutually exclusive at the final exclusion stage.

Audit item	Final value
Task Type L1 / L2 inventory	26 / 119
Domain L1 / L2 inventory	21 / 108
Orthogonal dimensions / levels	3 / 12
Task-L2 unspecified	70 (4.6%)
Domain-L2 unspecified	35 (2.3%)
Consistent nonempty Domain L1 by repository	58/64 (90.6%)

Appendix

Table 17: Structural audit of the frozen SWE Labeler. Repository consistency is computed from nonempty Domain L1 outputs.

Figure 9: Exact L2 breadth of every L1 family. Bars report the number of declared L2 children, sorted within each semantic axis. Zero-child families intentionally fall back to unspecified ; the values describe taxonomy structure, not empirical sample prevalence.

Figure 10: From SWE labels to category-conditioned training and routing. Repository Domain L1 is mapped by a fixed whitelist into three derived training categories. The resulting A/B/C assignment is reused for expert training, category-aware validation, and MOPD teacher routing.

Route	Engineering context	Domain L1 whitelist
A	Service/data/security	web_backend , database_storage , data_engineering , data_science , cms_ecommerce , security_auth , blockchain_web3
B	User-facing applications	web_frontend , desktop_gui , mobile_dev , media_graphics , game_dev
C	Systems/tooling/runtimes	devops_infra , devtools_test , pkg_manager_cli , os_system , iot_embedded , lang_runtime

Appendix

Table 18: Repository Domain L1 mapping for the three-route expert configuration.

Expert	$N$	Base	Post-RL	$\Delta$	Up	Same	Down
A	601	38.79	46.05	$+7.26$	259	151	191
B	516	38.78	45.54	$+6.76$	216	154	146
C	1,652	38.68	43.28	$+4.60$	633	517	502

Appendix

Table 19: Training-record success rates before and after initial expert RL. Base and post-RL rates are equal-weight means over matched training records (%);

\Delta

is in percentage points. Up/same/down count records with higher/equal/lower observed pass rates. These are paired empirical estimates, typically from four rollouts per instance.

Previously regressed training records included in Repair SFT
Outcome after Repair SFT	A ( $N=179$ )	B ( $N=141$ )	C ( $N=431$ )
At or above Base	125 (69.8%)	103 (73.0%)	264 (61.3%)
Above RL but below Base	7 (3.9%)	5 (3.5%)	30 (7.0%)
At or below RL	47 (26.3%)	33 (23.4%)	137 (31.8%)
Initial-RL gains followed by a decline after SFT
Outcome after Repair SFT	A ( $N=103$ )	B ( $N=75$ )	C ( $N=226$ )

Appendix

Table 20: Observed recovery and gain retention after first Repair SFT. Entries give training-record counts with percentages in parentheses. Each percentage uses the category-specific denominator shown in its panel. Regression means an observed initial-RL rate below Base. The second cohort is a subset of all training records whose rates decrease after SFT.

Expert	Stage	Positive	Negative	Net
A	Initial RL	$+16.85$	$-9.59$	$+7.26$
A	Repair SFT	$+22.46$	$-4.97$	$+17.49$
B	Initial RL	$+15.12$	$-8.36$	$+6.76$
B	Repair SFT	$+25.61$	$-4.80$	$+20.81$
C	Initial RL	$+13.68$	$-9.08$	$+4.60$
C	Repair SFT	$+20.71$	$-6.64$	$+14.07$

Appendix

Table 21: Positive and negative contributions to mean training-record gains relative to Base (percentage points). Each contribution is normalized by all matched training records in that category; positive plus negative equals net gain.

Software Engineering Agents

Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.

See all 344 papers

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Feb 25, 2026Patrick Tser Jern Kon, Archana Pradeep, Ang Chen +5Software Engineering Agents Small Language Models
University of Michigan · Meta · Stanford University
FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents
Oct 6, 2026Jia Liufu, Bin Hu, Linglin Jing +5RL for Code Generation Software Engineering Agents
NVIDIA · Tsinghua University
SWE-Fuse: Empowering Software Agents via Issue-free Trajectory Learning and Entropy-aware RLVR Training
Mar 9, 2026Xin-Cheng Wen, Binbin Chen, Haoxuan Lan +3Software Engineering Agents LLM Agent Training
Ant Group · The Chinese University of Hong Kong

Feb 25, 2026J/K move · Enter open · S save

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents

Patrick Tser Jern Kon, Archana Pradeep, Ang Chen, Alexander P. Ellis+4

University of Michigan · Meta · Stanford University

Software Engineering Agents Small Language Models LLM Agents Automated Program Repair RL for Code Generation

Open paper AskPDF arXiv

Small language models (SLMs) offer compelling advantages in cost, latency, and adaptability, but have so far lagged behind larger models on long-horizon software engineering tasks such as SWE-bench, where they suffer from pervasive action looping and low resolution rates. We introduce SWE-Protégé, a post-training framework that reframes software repair as an expert-protégé collaboration problem. In SWE-Protégé, an SLM remains the sole decision-maker while learning to selectively seek guidance from a strong expert model, recognize stalled states, and follow through on expert feedback. Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration. We lightly post-train Qwen2.5-Coder-7B-Instruct to achieve 42.4% Pass@1 on SWE-bench Verified, a +25.4% improvement over the prior SLM state of the art, while using expert assistance sparsely (~4 calls per task and 11% of total tokens).

Abstract

Software Engineering Agents

Explore similar work

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents

FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents

SWE-Fuse: Empowering Software Agents via Issue-free Trajectory Learning and Entropy-aware RLVR Training