Software engineering (SE) expertise can shape how coding large language models (LLMs) are trained and which engineering capabilities they acquire. We examine this role through an industrial case study of capability-focused training for a 92B-parameter coding LLM serving more than 100,000 developers. We demonstrate how established SE knowledge can guide the selection of engineering capabilities, the construction of verifiable training data, and the evaluation of model improvements. To target repository-level refactoring, we developed RefactorForge, a pipeline that combines refactoring knowledge, established SE tools, and verification techniques to construct training data. Across 306 Python repositories, RefactorForge generated 5,704 executable tasks and retained 4,572 verified teacher trajectories totaling 72.51 million tokens. Incremental supervised fine-tuning on these trajectories, accounting for 0.46% of the preceding coding mixture's tokens, improved RefactorBench performance by 18.00 percentage points and SWE-Bench ProMax performance by 7.06 points. Under OpenCode, these improvements extended to SWE issue resolution, with gains of 4.97 points on SWE-Bench Verified and 1.23 points on SWE-Bench Pro. Improvements also persisted across other coding-agent harnesses, while greenfield repository generation showed mixed results. We report lessons from constructing capability-focused training data and evaluating its effects across engineering workloads. Building on this industrial experience, we propose a research agenda for expanding SE's role in transforming coding LLMs into SE LLMs: incorporating SE knowledge into model training, applying engineering methods to model development and maintenance, and systematically defining the engineering capabilities these models should acquire.
Figures & tables
Fig. 1: Operation-level success rates of GPT-5.2 and GLM-5.1 across 13 RefactorBench categories. Aggregate success rates of 62% and 51% conceal substantial differences across individual operations.
Code Health
Iterations
Cost
Project
LOC
Size
Initial
Final
Δ (%)
Acc.
Rej.
Tokens (M)
USD
Time (h:min)
Status
Arktype
32,617
Large
8.89
8.99
1.12
19
3
383.58
268.44
19:31
Budget exhausted
Participle
7,859
Medium
8.04
8.46
5.22
24
3
668.07
454.24
28:29
Budget exhausted
ABS
6,596
Medium
6.93
7.43
7.22
20
2
591.20
425.81
28:43
Budget exhausted
Dateutil
5,725
Medium
3.88
5.64
45.36
20
7
263.07
165.29
10:09
Finished
Aiomonitor
2,653
Small
8.19
8.67
5.86
10
0
274.72
186.22
12:01
Finished
TABLE I: CodeScene-guided refactoring across five projects. Code Health gains remained relatively modest despite substantial time and model costs. Size is small ( < 5k LOC), medium (5k–10k), or large ( > 10k). Final health is the score of the repository after the last accepted refactoring, and Δ is the relative change in percentage. Acc. and Rej. are the numbers of accepted and rejected refactoring iterations. Refactoring time is the agent’s total runtime during the refactoring process.
Fig. 2: Overview of RefactorForge’s twelve-step pipeline.
Refactoring type
Candidate code
Properties checked by generated F2P tests
Tasks (share)
Rename Function
Top-level functions.
New name present in the defining module and old name absent. Where affected by the reference change, identifier uses, import bindings, and all exports also reflect the rename.
1,899 (33.29%)
Rename Class
Top-level classes.
2,671 (46.83%)
Rename Constant
Module-level constants.
567 (9.94%)
Rename Attribute
Class and instance attributes.
New name replaces the old name in the owning class. Affected attribute references reflect the rename.
362 (6.35%)
Remove Unused Function
Private top-level functions with no resolved references.
Target function absent. Remaining top-level definition names match the reference outcome.
25 (0.44%)
Remove Unused Import
Names flagged as unused in from ... import ... statements.
TABLE II: Refactoring candidates, F2P test checks, and task counts. Function and class renaming account for 80.12% of tasks.
Refactoring (C1)
SWE issue resolution (C2)
SWE-Bench ProMax
RefactorBench (Lazy)
SWE-Bench Verified
SWE-Bench Pro
Baseline-92B
51.18
43.00
62.00
48.97
Refact-92B
58.24
61.00
66.97
50.20
Improvement (pp)
+7.06
+18.00 ∗
+4.97
+1.23
TABLE III: Refactoring and SWE issue-resolution results (Avg. @3). ∗ denotes p<0.05 (one-sided exact McNemar test).
Fig. 3: RefactorBench Lazy success rates by refactoring category for Baseline-92B and Refact-92B . Refact-92B reduced unevenness across refactoring operations.
Fig. 4: Average scores from three independent runs across four agent harnesses. Panel (a) shows SWE-Bench Pro and panel (b) SWE-Bench Verified. Purple dotted bars show Baseline-92B and green dashed bars Refact-92B . Bar labels give average scores, and arrows give absolute improvements in percentage points.
Small language models (SLMs) offer compelling advantages in cost, latency, and adaptability, but have so far lagged behind larger models on long-horizon software engineering tasks such as SWE-bench, where they suffer from pervasive action looping and low resolution rates. We introduce SWE-Protégé, a post-training framework that reframes software repair as an expert-protégé collaboration problem. In SWE-Protégé, an SLM remains the sole decision-maker while learning to selectively seek guidance from a strong expert model, recognize stalled states, and follow through on expert feedback. Our approach combines supervised fine-tuning on expert-augmented trajectories with agentic reinforcement learning that explicitly discourages degenerative looping and unproductive expert collaboration. We lightly post-train Qwen2.5-Coder-7B-Instruct to achieve 42.4% Pass@1 on SWE-bench Verified, a +25.4% improvement over the prior SLM state of the art, while using expert assistance sparsely (~4 calls per task and 11% of total tokens).
Patrick Tser Jern Kon, Archana Pradeep, Ang Chen +5
University of Michigan · Meta · Stanford University
LLM-driven software engineering agents have become a central testbed for real-world language-model capability, yet their training remains limited by the availability of high-quality SWE tasks. Existing synthetic data methods typically create tasks through fixed mutation or bug-injection procedures, making the resulting distributions largely independent of the agent's own weaknesses and training progress. We introduce Socratic-SWE, a closed-loop self-evolution framework that reuses the agent's historical solving traces as a source of training signal. Rather than treating traces only as evidence for reward computation, Socratic-SWE distills them into structured agent skills that summarize recurring failures and effective repair patterns. These skills then guide the generation of targeted repair tasks in real repositories. Candidate tasks are checked through execution-based validation and scored with a solver-gradient alignment reward, so that the retained tasks are both verifiable and useful for improving the Solver. The updated Solver produces new traces, enabling the task curriculum to adapt over successive rounds. Across SWE-bench Verified, SWE-bench Lite, SWE-bench Pro, and Terminal-Bench 2.0, Socratic-SWE consistently improves over self-evolving baselines under the same compute budget, reaching 50.40% on SWE-bench Verified after three iterations. These results suggest that solving traces can serve as a scalable substrate for self-evolving SWE agents.
Chuan Xiao, Zhengbo Jiao, Shaobo Wang +5
2Shanghai Jiao Tong University · 1AI Data, Alibaba Group
Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.