Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

Haibo Jin; Peng Kuang; Xucheng Yu; Jerry Wang; Dehao Wu; Haohan Wang

cs.SEOct 6, 2026

Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

Authors: Haibo Jin, Peng Kuang, Xucheng Yu, Jerry Wang, Dehao Wu, Haohan Wang

Organizations: University of Illinois Urbana-Champaign, USA

Abstract

Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete repositories remains difficult because interacting modules, interfaces, configurations, tests, and dependencies must work together. We introduce Code Primitives, agent-native reusable executable components with interface contracts, dependency closures, validation tests, and provenance. Each primitive uses a resident LLM to assess relevance and adapt its implementation, interfaces, and dependencies to the target repository, and we organize 1,424 validated primitives in CodeFace, a searchable library for repository construction. We introduce LEGO (Large-scale repository Engineering via aGent-native reusable cOde primitives), which activates task-relevant primitives, integrates their adapted implementations with task-specific code while resolving cross-component constraints, and revises the result against executed tests. To measure construction end to end, we build LEGO-REPO, a benchmark of 522 executable reconstruction tasks spanning seven software domains, 22 capability tracks, and five difficulty levels, scored against native test suites between an empty-package floor and original-source ceiling. The strongest of 13 evaluated backbones reaches a delivery score of 0.318 and scores zero on 41.0% of tasks; LEGO improves all 13 by 0.1474 on average and raises GPT-5.6-terra from 0.3180 to 0.5134 (+61.4%). In controlled comparisons, adapted primitives outperform retrieved code supplied as context or vendored unchanged. The effect persists against independent repository agents, across three external benchmarks, and with a disjointly re-mined CodeFace; GPT-OSS-20B for adaptation and diagnosis retains 95.1% of the homogeneous score at 24.0% lower cost.

Figures & tables

Figure 1: From Code Primitives to complete repositories. Left: real-world repositories are mined into component-level Code Primitives, each carrying its interface, dependencies, tests, configuration, and documentation, and organized in CodeFace. Right: a user request activates the required primitives, which adapt themselves and exchange the requirements their changes impose, before the adapted implementations are edited into the repository and revised under execution feedback and diagnosis.

	Normalized delivery score by band $\uparrow$						Outcome rate
Backbone	D1	D2	D3	D4	D5	All $\uparrow$	$\textsc{Ceil@1}\uparrow$	$\textsc{Dead@0}\downarrow$
proprietary API models
GPT-5.6-terra	.525	.382	.274	.206	.129	.3180	8.8%	41.0%
Claude Sonnet 5	.521	.379	.260	.194	.091	.3052	8.6%	40.2%
GPT-5.4	.485	.301	.216	.123	.061	.2528	6.1%	49.4%
Claude Opus 5	.465	.294	.199	.107	.042	.2371	5.9%	53.4%

Table 1: RQ1: construction from scratch, all

522

tasks. Band columns average over

117/111/116/96/82

tasks. Blue and orange mark best and second best.

		Score by difficulty band $\uparrow$						Outcome rate
System	Backbone	D1	D2	D3	D4	D5	All $\uparrow$	$\textsc{Ceil@1}\uparrow$	$\textsc{Dead@0}\downarrow$	$/task
OpenHands	GPT-5.6-terra	.556	.401	.269	.198	.129	.3263	$9.4\%$	$41.6\%$	2.41
Claude Code	Claude Sonnet 5	.543	.394	.271	.190	.104	.3170	$9.6\%$	$39.8\%$	2.16
SWE-agent ‡	GPT-5.6-terra	.482	.345	.238	.152	.081	.2750	$6.7\%$	$46.2\%$	1.86
Agentless ‡	GPT-5.6-terra	.421	.276	.181	.098	.043	.2181	$5.4\%$	$55.4\%$	0.94
Feedback (ours)	GPT-5.6-terra	.525	.382	.274	.206	.129	.3180	$8.8\%$	$41.0\%$	1.55

Table 2: RQ1: independently developed systems, all

522

tasks. Each runs under its own harness with the backbone in column two; only the task set, environment, and grading pipeline are fixed.

\ddagger

: issue-to-patch adapter (Appendix I ).

band	mod.	cov.	query	cand.	retain	adapt	share	used
D1	0, 721	75.3%	0, 437	0, 874	0, 357	0, 332	46.0%	61.1%
D2	2,064	71.2%	0, 770	1,540	0, 613	0, 557	27.0%	37.9%
D3	3,654	65.8%	1,021	2,042	0, 818	0, 731	20.0%	30.4%
D4	6,823	61.2%	1,425	2,850	1,135	0, 989	14.5%	23.7%
D5	9,723	56.7%	1,862	3,724	1,489	1,264	13.0%	22.9%
all	22,985	61.4%	5,515	11,030	4,412	3,873	16.9%	27.5%

Table 3: RQ2: CodeFace reach and the runtime adaptation funnel. cov. is offline coverage of the band’s mod. target modules; query , cand. (top-

2

), retain , and adapt are counted during construction. share and used give adapted primitives as a fraction of target and of reachable modules. Full funnel: Appendix G.2 .

Configuration	Stages	D1	D5	All $\uparrow$	$\textsc{Dead@0}\downarrow$	$/task
Single-shot	----	—	—	.2172	57.1%	0.86
Retrieval, one attempt	R---	—	—	.2314	58.6%	0.87
Feedback	---V	.525	.129	.3180	41.0%	1.55
Retrieval $+$ Feedback	R--V	.544	.141	.3352	34.7%	1.56
Feedback $+$ Diagnosis	--DV	.569	.135	.3469	30.1%	1.79
Retrieval $+$ Diagnosis	R-DV	.649	.127	.3980	31.6%	1.80

Table 4: RQ3: construction paradigms, all

522

tasks. R / A / E / V / D denote retrieval, adaptation, generic editing, execution feedback, and diagnosis (

b=5

Mechanism added	All $\uparrow$	step	% tot.
Feedback ( ---V ), ref.	.3180	—	—
$+$ diagnosis revision ( --DV )	.3469	$+.0289$	$14.8\%$
$+$ File RAG, editing ( FE DV )	.3528	$+.0059$	$3.0\%$
$+$ constructed prim. ( RE DV )	.4093	$+.0565$	$28.9\%$
$+$ primitive adaptation ( RADV )	.5134	$+.1041$	$53.3\%$
roll-up

Table 5: RQ3: construction-stage ablation. Each row adds one mechanism. File RAG retrieves the top-

2

donor files by embedding similarity without primitive-specific adaptation or validation.

Configuration	D1	D5	All $\uparrow$	$\Delta$ --DV
Feedback $+$ Diagnosis ( --DV )	.569	.135	.3469	ref.
Context-only ( RC DV )	.620	.115	.3544	$+.0075$
Import $+$ Call ( RI DV )	.627	.060	.3430	$-.0039$
Structured-schema adaptation	.679	.327	.4974	$+.1505$
LEGO ( RADV )	.686	.338	.5134	$+.1665$

Table 6: RQ3: how reused components are consumed. Rows share LEGO’s retrieval, diagnosis.

Table 8

Benchmark	publ.	fb.	LEGO	$\Delta$
RepoZero, Py $\to$ Py	51.2	53.8	62.9	$+9.1$
RepoGenesis	38.2	41.0	48.2	$+7.2$
RepoCraft	0.412	0.436	0.501	$+.065$

Table 9: RQ6: transfer to three external benchmarks. Each follows its published protocol with GPT-5.6-terra; fb. denotes our execution-feedback baseline.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

count	stage	transition into this stage
$1{,}309$	candidate registry	initial repository discovery
$573$	attempted task candidates	repository screening: domain/security eligibility (including $34$ offensive-security holds), importable package, runnable native suite
$565$	frozen task directories	successful materialization with clone/build/environment pinned
$\mathbf{522}$	LEGO-REPO evaluation set	stable same-pipeline ceiling and measurable span $c-f\geq 5$

Appendix

Table 10: Construction funnel. Eight attempted candidates fail materialization or environment pinning before the

565

frozen directories are formed. Of those directories,

43

do not enter the scored benchmark because a stable same-pipeline reference cannot be established: the native suite does not run in the pinned environment, the identity ceiling is unmeasurable, or

c-f<5

category	attributable to	Feedback	LEGO
incomplete composition: modules never written	system under test	84	38
import or contract failure at collection	system under test	61	27
environment or dependency resolution	harness/environment	35	18
attempt budget exhausted, tests failing	system under test	34	15
zero-score tasks ( Dead@0 )		214	98
as a share of the $522$ benchmark tasks		$41.0\%$	$18.8\%$

Appendix

Table 11: Why a task receives a zero score. Counts are over the same

522

tasks for the Feedback baseline ( ---V ) and LEGO ( RADV ); each column sums to the corresponding Dead@0 count. A zero can arise either because no admissible repository is delivered or because an admissible repository remains at the normalized floor.

domain	D1	D2	D3	D4	D5	all
machine-learning	15	25	35	39	44	158
databases-storage	39	31	37	30	12	149
compilers-languages	42	31	32	16	10	131
graphics-vision	5	11	6	4	6	32
scientific-computing	5	7	3	5	6	26
quant-finance	6	3	2	1	2	14

Appendix

Table 12: Inventory by domain and difficulty. Each cell counts tasks in the

522

-task benchmark. The security category contains defensive cryptography and authentication libraries only; offensive-security repositories are excluded from the benchmark and from CodeFace.

capability track	D1	D2	D3	D4	D5	all
agent-orchestration	4	12	11	20	26	73
learning	2	10	12	11	11	46
parsing-languages	12	14	6	8	5	45
web-protocol	8	6	18	11	1	44
serialization-wire	7	8	9	6	4	34
text-strings	14	7	6	2	2	31

Appendix

Table 13: Inventory by capability track and difficulty. Tracks and bands are assigned from frozen task metadata before observing any system outcome.

quantity	min	p25	median	p75	max	mean
identity ceiling $c(T)$	6	47	118	341	$5{,}876$	287.4
empty-package floor $f(T)$	0	0	0	0	$1{,}577$	9.6
span $c-f$	5	46	114	335	$5{,}876$	277.8
floor share $f/c$	0	0	0	0	0.873	0.021
tasks with span $<20$	$41$ tasks, mean LEGO score $.5236$
tasks with $f/c>0.5$	$9$ tasks, mean LEGO score $.4871$

Appendix

Table 14: Distribution of ceiling, floor, and normalized span over the

522

tasks. The span-weighted row re-aggregates the benchmark mean with each task weighted by

c-f

instead of equally, so tasks with little dynamic range do not carry the same weight as tasks with hundreds of gradable tests.

binning	quantity	B1	B2	B3	B4	B5
$d=2\log_{10}m+\log_{10}c$ (reported)	tasks	117	111	116	96	82
	Feedback	.525	.382	.274	.206	.129
	LEGO $-$ Feedback	$+.161$	$+.201$	$+.209$	$+.203$	$+.209$
$d=\log_{10}m$ , cuts at $m\in\{10,25,60,150\}$	tasks	124	108	113	99	78
	Feedback	.515	.383	.280	.199	.121
	LEGO $-$ Feedback	$+.162$	$+.206$	$+.222$	$+.193$	$+.198$

Appendix

Table 15: Sensitivity of the band profile to the difficulty definition. The reported binning uses the identity ceiling

c(T)

, which also appears in the scoring denominator; the two alternatives use repository size alone. Both relabel the same per-task records, with no configuration re-run.

partition	tasks	modules	Feedback	LEGO	$\Delta$	coverage
D1 (mean $6$ modules)	117	0, 721	.525	.686	$+.161$	$75.3\%$
D2 (mean $19$ modules)	111	2,064	.382	.583	$+.201$	$71.2\%$
D3 (mean $32$ modules)	116	3,654	.274	.483	$+.209$	$65.8\%$
D4 (mean $71$ modules)	96	6,823	.206	.409	$+.203$	$61.2\%$
D5 (mean $119$ modules)	82	9,723	.129	.338	$+.209$	$56.7\%$
machine-learning	158	9,649	.2498	.4368	$+.1870$	$56.3\%$

Appendix

Table 16: Feedback and LEGO by difficulty and domain. These are the same two configurations reported in Table 4 , partitioned two ways; coverage is the offline module-level reachability measure, not a scoring denominator.

band	tasks	modules	covered	coverage	queries	retained	adapted
D1	117	0, 721	0, 543	$75.3\%$	0, 437	0, 357	0, 332
D2	111	2,064	1,470	$71.2\%$	0, 770	0, 613	0, 557
D3	116	3,654	2,404	$65.8\%$	1,021	0, 818	0, 731
D4	96	6,823	4,176	$61.2\%$	1,425	1,135	0, 989
D5	82	9,723	5,513	$56.7\%$	1,862	1,489	1,264
all	522	22,985	14,106	$\mathbf{61.4\%}$	5,515	4,412	3,873

Appendix

Table 17: Offline library reach and runtime adaptation funnel by band. Module and coverage columns come from the offline reachability audit, which covers

14{,}106/22{,}985

target modules; queries, retained candidates, and adaptations are counted during construction.

change from LEGO	resulting configuration	all	$\Delta$	$/task
none	RADV	.5134	ref.	2.08
remove the whole library	--DV	.3469	$-.1665$	1.79
remove primitive adaptation	R-DV	.3980	$-.1154$	1.80
remove diagnosis	RA-V	.4795	$-.0339$	1.83
replace primitive machinery with generic editing	RE DV	.4093	$-.1041$	2.01
also replace primitives with raw files	FE DV	.3528	$-.1606$	2.03

Appendix

Table 18: Aggregate component removals on all

522

tasks. Removing the library removes retrieval and adaptation together. The generic-editing control changes carried tests, metadata, and primitive execution state together. The mined-only and same-repo-exclusion rows change library eligibility rather than the construction stages.

Configuration	all $\uparrow$	$\Delta$ RE DV	$\Delta$ LEGO
RAG $+$ Edit ( RE DV )	.4093	ref.	$-.1041$
$+$ carried tests $V_{i}$	.4431	$+.0338$	$-.0703$
$+$ contract and provenance $I_{i},X_{i}$	.4818	$+.0725$	$-.0316$
$+$ dependency closure $D_{i}$	.5039	$+.0946$	$-.0095$
primitive without $V_{i}$	.4796	$+.0703$	$-.0338$
LEGO ( RADV )	.5134	$\mathbf{+.1041}$	ref.

Appendix

Table 19: Decomposing the primitive channel. Each row adds one field of the primitive to the generic-editing control, with retrieval, diagnosis and

b=5

feedback held fixed. The three

+

rows are cumulative; the fifth row instead withholds

V_{i}

from an otherwise complete primitive.

Configuration	D1	D2	D3	D4	D5	All
Feedback ( ---V )	.525	.382	.274	.206	.129	.3180
LEGO, full library	.686	.583	.483	.409	.338	.5134
LEGO, same-repo donors excluded	.683	.574	.461	.337	.221	.4743
gain, full library	$+.161$	$+.201$	$+.209$	$+.203$	$+.209$	$+.1954$
gain, same-repo excluded	$+.158$	$+.192$	$+.187$	$+.131$	$+.092$	$+.1563$
lost to donor exclusion	$-.003$	$-.009$	$-.022$	$-.072$	$-.117$	$-.0391$

Appendix

Table 20: RQ3: what same-repository donors are worth, by difficulty band. All rows are means over the fixed band task counts (

117/111/116/96/82

) and reproduce Table 4 . gain is the improvement over Feedback in that band, lost is the part of that gain removed by excluding library entries whose source repository is the evaluation target, and retained is the complement. The All column margin is measured against Feedback ; measured instead against the diagnosis-equipped baseline ( --DV ) the retained share is

76.5\%

Comparison	Mechanism	Win	Tie	Loss	$z$	$\Delta$ score
Retrieval $+$ Feedback	raw retrieved code	148	269	105	2.7	$+.0172$
Feedback $+$ Diagnosis	diagnosis, no library	162	258	102	3.7	$+.0289$
Import $+$ Call	vendored unchanged, invoked	162	256	104	3.6	$+.0250$
File RAG $+$ Edit	raw files $+$ generic editing	174	247	101	4.4	$+.0348$
Context-only	primitive as read-only reference	161	260	101	3.7	$+.0364$
Retrieval $+$ Diagnosis	retrieval $+$ diagnosis	214	214	0 94	6.8	$+.0800$

Appendix

Table 21: RQ3: per-task outcomes and stage decomposition, all

522

tasks. Upper block: every row is compared against Feedback ( ---V ). Lower block: win, tie, and loss counts compare full LEGO ( RADV ) against the named component-removal configuration, while

\Delta

reports that configuration minus full LEGO and is therefore negative. All counts use task-level normalized delivery scores on the same

522

tasks, and

z=(W-L)/\sqrt{W+L}

is the normal-approximation statistic of a paired sign test over non-tied tasks.

primitive	score	kept	$/task	$/pt
GPT-5.6-terra	.5134	$100.0\%$	2.08	0.41
Claude Sonnet 5	.5081	$95.4\%$	2.30	0.45
GPT-5.4	.5060	$93.6\%$	2.08	0.41
GPT-5.6-luna	.5015	$89.7\%$	1.93	0.38
DeepSeek-V4	.4992	$87.7\%$	1.83	0.37
Qwen3-Coder-480B	.4972	$86.0\%$	1.87	0.38

Appendix

Table 22: Primitive substitution on all

522

tasks. kept is the fraction of the adaptation margin over the no-primitive reference retained by each model. Kept values are computed from unrounded task means; displayed scores are rounded.

/pt is recorded-call cost per

0.1$ of delivery score, using the accounting scope in Appendix G.11 .

diagnosis	score	kept	$/task	$/pt
GPT-5.6-terra	.5134	$100.0\%$	2.08	0.41
Claude Sonnet 5	.5123	$96.8\%$	2.30	0.45
GPT-5.4	.5096	$88.8\%$	2.08	0.41
GPT-5.6-luna	.5061	$78.5\%$	1.95	0.39
DeepSeek-V4	.5058	$77.6\%$	1.86	0.37
Qwen3-Coder-480B	.5020	$66.4\%$	1.90	0.38

Appendix

Table 23: Diagnosis substitution on all

522

tasks. kept is the fraction of the revision margin over the no-diagnosis reference retained by each model. Kept values are computed from unrounded task means; displayed scores are rounded.

backbone (with LEGO)	D1	D2	D3	D4	D5	all	scratch	$\Delta$	$\Delta_{\mathrm{rel}}$	Dead@0
GPT-5.6-terra	.686	.583	.483	.409	.338	.5134	.3180	$+.1954$	$+61.4\%$	18.8%
Claude Sonnet 5	.665	.573	.473	.392	.301	.4953	.3052	$+.1901$	$+62.3\%$	30.1%
GPT-5.4	.616	.525	.439	.343	.239	.4478	.2528	$+.1950$	$+77.1\%$	37.9%
Claude Opus 5	.622	.565	.294	.270	.151	.3982	.2371	$+.1611$	$+67.9\%$	34.3%
GPT-5.6-luna	.553	.471	.428	.174	.060	.3606	.2203	$+.1403$	$+63.7\%$	33.0%
Grok-4.3	.596	.420	.241	.154	.090	.3190	.1718	$+.1472$	$+85.7\%$	49.0%

Appendix

Table 24: All

13

backbones with the full LEGO pipeline, all

522

tasks. scratch is the same backbone’s execution-feedback score from Table 1 .

\Delta_{\mathrm{rel}}

is the relative improvement of LEGO over that scratch score. Outcome rates are computed from integer task counts and rounded to one decimal place.

library / retrieval setting	entries	$\kappa$	all	$\Delta$ vs default
mined only	871	$56.9\%$	.5021	$-.0113$
mined $+$ web-sourced	1,081	$58.8\%$	.5108	$-.0026$
mined $+$ harvested	1,214	$59.9\%$	.5069	$-.0065$
full library (default)	1,424	$\mathbf{61.4\%}$	.5134	$0.0000$
same-repo donor exclusion	1,424	$41.2\%$	.4743	$-.0391$
top- $1$	1,424	$52.1\%$	.4948	$-.0186$

Appendix

Table 25: Measured library composition and retrieval-depth ablations. The task set, backbone/adaptation/diagnosis assignment, and attempt budget remain fixed. The same-repository exclusion row reports the full snapshot size; eligibility is filtered separately for each target repository.

	Feedback ( ---V )		LEGO ( RADV )
budget $b$	all	$/task	all	$/task	$\Delta$
$1$	.2172	0.86	.3954	1.13	$+.1782$
$2$	.2420	1.05	.4412	1.38	$+.1992$
$3$	.2785	1.21	.4663	1.61	$+.1878$
$4$	.3066	1.36	.5071	1.82	$+.2005$
$5$ (default)	.3180	1.55	.5134	2.08	$\mathbf{+.1954}$

Appendix

Table 26: Executed attempt budgets with and without the library. Each row uses the same integer attempt limit for both configurations and the same

522

tasks.

backbone	primitive	diagnosis	all	$/task	$/pt	% of LEGO
terra	terra	terra	.5134	2.08	0.41	$100.0\%$
terra	Sonnet 5	Sonnet 5	.5081	2.53	0.50	$99.0\%$
terra	DeepSeek-V4	DeepSeek-V4	.4934	1.62	0.33	$96.1\%$
terra	GPT-OSS-20B	GPT-OSS-20B	.4880	1.58	0.32	$95.1\%$
Sonnet 5	Sonnet 5	Sonnet 5	.4869	4.26	0.87	$94.8\%$
Sonnet 5	GPT-OSS-20B	GPT-OSS-20B	.4661	3.31	0.71	$90.8\%$

Appendix

Table 27: Quality and cost under different backbone allocations. All rows use the same

522

tasks. % of LEGO divides the configuration score by the default T/T/T score. Here terra denotes GPT-5.6-terra and Sonnet 5 denotes Claude Sonnet 5.

Model	Provider	Open	Size	$/1M in	$/1M out	Used in Table
GPT-5.6-terra	OpenAI	No	Undisclosed	1.25	10.00	default backbone, primitive, diagnosis
Claude Opus 5	Anthropic	No	Undisclosed	5.00	25.00	1 , 24
Claude Sonnet 5	Anthropic	No	Undisclosed	3.00	15.00	1 , 2 , 24 , 22 , 23 , 27
Claude Haiku 4.5	Anthropic	No	Undisclosed	1.00	5.00	1 , 24
GPT-5.4	OpenAI	No	Undisclosed	1.25	10.00	1 , 24 , 22 , 23
GPT-5.6-luna	OpenAI	No	Undisclosed	0.60	4.80	1 , 24 , 22 , 23

Appendix

Table 28: Model identifiers and hosted inference rates. Prices are the recorded hosted-inference rates on 2026-09-19, in US dollars per million tokens. Open denotes published weights. All cost results correspond to the evaluated hosted deployments, with provider-default reasoning effort.

(a) configuration means over decoding seeds
Configuration	seed 1	seed 2	seed 3	mean $\pm$ s.d.
Feedback ( ---V )	.3180	.3202	.3150	$.3177\pm.0026$
Feedback $+$ Diagnosis ( --DV )	.3469	.3434	.3493	$.3465\pm.0030$
RAG $+$ Edit ( RE DV )	.4093	.4036	.4121	$.4083\pm.0043$
LEGO ( RADV )	.5134	.5125	.5089	$\mathbf{.5116\pm.0024}$
(b) paired contrasts on seed 1

Appendix

Table 29: Repeated-seed means and paired confidence intervals, all

522

tasks. Panel (a): mean

\pm

standard deviation of the delivery score over independent decoding seeds for each primary configuration. Panel (b): paired contrasts with

95\%

bootstrap confidence intervals over tasks (

10{,}000

resamples), a Wilcoxon signed-rank

p

-value using the magnitude of each per-task difference, and the Holm-corrected

p

-value across the family of four.

quantity	D1	D2	D3	D4	D5	all
tasks ( $n$ )	117	111	116	96	82	522
improved tasks ( $n$ )	70	74	68	56	58	326
regressed tasks ( $n$ )	24	18	17	14	11	84
tied tasks ( $n$ )	23	19	31	26	13	112
regressed as a share of the band	20.5%	16.2%	14.7%	14.6%	13.4%	16.1%
mean gain on improved tasks	.310	.326	.378	.364	.304	.336

Appendix

Table 30: Profile of task-level improvements and

84

regressions against Feedback . Counts and score deltas come from the same per-task records as Table 21 ; losses are counted only where LEGO scores strictly below Feedback on the same task. Conditional mean gains and losses are displayed to three decimals while the net means and aggregate column are computed from unrounded records, so products of displayed values need not reproduce the final decimal.

successful adaptations on the task	$0$ – $2$	$3$ – $6$	$7$ – $12$	$\geq 13$	Spearman $\rho$
tasks ( $n$ )	63	118	191	150
mean Feedback score	.3612	.3438	.3107	.2888
mean LEGO score	.3735	.4535	.5365	.5899
mean $\Delta$	$+.0123$	$+.1097$	$+.2258$	$+.3011$	$0.41$
mean $\Delta$ , difficulty-matched	$+.0090$	$+.1046$	$+.2218$	$+.2954$	$0.34$

Appendix

Table 31: Per-task gain against the number of successfully adapted primitives. The difficulty-matched row conditions within difficulty band before averaging, because adaptation count and repository size covary. The final column reports Spearman

\rho

between the per-task

\Delta

and the adaptation count over all

522

tasks.

library eligibility	entries	coverage $\kappa$	all	$\Delta$ vs Feedback
all entries (default)	1,424	$61.4\%$	.5134	$+.1954$
$-$ same-repository donors	1,424	$41.2\%$	.4743	$+.1563$
$-$ same-org donors	1,424	$38.6\%$	.4597	$+.1417$
$-$ any fork or near-duplicate	1,365	$33.1\%$	.4455	$+.1275$
re-mined from disjoint corpus	518	$44.7\%$	.4528	$+.1348$

Appendix

Table 32: Library eligibility under progressively stricter provenance controls. All rows are measured; the default and same-repository rows reproduce Table 25 . The same-repository and same-organization rows keep the

1{,}424

-entry snapshot and filter eligibility per target. Near-duplicate filtering removes any mined entry whose MinHash similarity to a target file exceeds

0.6

(

128

permutations,

5

-token shingles over normalised tokens), dropping

59

871

mined entries. The last row filters nothing: it re-mines CodeFace from

640

repositories disjoint from LEGO-REPO, screened identically, yielding

518

primitives.

Configuration	definition	stages	$n$	score
Single-shot †	no library, single pass	----	$522$	.2172
Retrieval, one attempt †	library, single pass	R---	$522$	.2314
Feedback	no library	---V	$522$	.3180
Retrieval $+$ Feedback	library, no adaptation or diagnosis	R--V	$522$	.3352
Feedback $+$ Diagnosis	no library, diagnosis enabled	--DV	$522$	.3469
Retrieval $+$ Diagnosis	library and diagnosis, no adaptation	R-DV	$522$	.3980

Appendix

Table 33: Configuration definitions. R : retrieval; A : primitive adaptation; V : execution-feedback revision; D : diagnosis-guided revision. In the matched editing control, E replaces primitive adaptation with scoped generic editing.

b=1

rows are first-round prefixes of the corresponding trajectories rather than separately sampled task sets; Single-shot and Retrieval, one attempt are therefore not independently sampled baselines, and comparisons against them measure the value of continuing a trajectory rather than of a separate system. Every arm listed here uses

b=5

unless the row states otherwise.

Repository	band	mod.	cov.	donors	floor	ceil.	Feedback	LEGO
pandera	D5	$105$	$58.1\%$	$15$	$1{,}577$	$1{,}807$	$0.022$	$0.796$
markdown	D3	$32$	$12.5\%$	$5$	$0$	$111$	$0.667$	$0.081$
haystack	D5	$298$	$48.7\%$	$30$	$0$	$5{,}876$	$0.000$	$0.000$

Appendix

Table 34: Three repository case studies. Coverage is the fraction of target modules for which the offline audit retains at least one relevant candidate; donors counts distinct source repositories represented among the retrieved material. Floor and ceiling are test counts from the reference executions, and the last two columns are normalized delivery scores. The three packages are, in order, unionai-oss/pandera , Python-Markdown/markdown and deepset-ai/haystack ; pinned commits are in the task inventory (Appendix C ).

benchmark	unit the model produces	correctness signal	ceiling-rel.	accum.
HumanEval ( Chen et al., 2021 )	a standalone function	hidden unit tests	no	no
MBPP ( Austin et al., 2021 )	a standalone function	assertion triples	no	no
BigCodeBench ( Zhuo et al., 2025 )	a function calling libraries	unit tests	no	no
CrossCodeEval ( Ding et al., 2023 )	a line, cross-file context	identifier match	no	no
RepoBench ( Liu et al., 2024 )	a line, retrieved context	text match, edit sim.	no	no
DevEval ( Li et al., 2024 )	a function inside a repo	repository tests	no	no

Appendix

Table 35: Benchmark task scope. Ceiling-relative scoring normalizes by what the original package achieves in the same execution pipeline. Accumulation denotes support for maintaining reusable components across a task sequence; the controlled configuration comparisons themselves use a fixed final library snapshot. LEGO-REPO has a median of

30

target modules, with D5 tasks averaging

119

scoring convention	mean	$>\!1$ ?
reference-relative count ratio	0.3191	yes ( $1.167$ )
vs. installed baseline	0.2972	yes
vs. ceiling, no floor	0.3220	no
identity ceiling (ours)	0.3180	no

Appendix

Table 36: Scoring conventions on the Feedback baseline. All four means use the same

522

tasks. The reference-relative count ratio is unclipped and can exceed one; identity normalization subtracts the empty-package floor and clips to the same-pipeline ceiling.

audit outcome, random sample of $120$ scored reconstructions	Feedback	LEGO
genuine implementation of the target capability	74	100
partial implementation, tests pass on covered paths	33	14
test-targeted special-casing	9	5
degenerate stub that still passes graded tests	4	1
special-casing rate, Wilson $95\%$ CI	$7.5\%$ $[4.0,13.6]$	$4.2\%$ $[1.8,9.4]$
degenerate-stub rate, Wilson $95\%$ CI	$3.3\%$ $[1.3,8.3]$	$0.8\%$ $[0.1,4.6]$

Appendix

Table 37: Manual non-degeneracy audit. The sample is

120

scored reconstructions per configuration, stratified by band in proportion to the benchmark (

27/26/27/22/18

) and drawn from tasks scoring nonzero under both configurations, so the same tasks are labelled in each column. Two annotators labelled all

240

reconstructions independently, blind to the producing configuration and to each other; agreement before resolution was Cohen’s

\kappa=0.81

(

208

240

labels identical).

license class	primitives	share	propagates into generated source?
permissive (MIT, BSD, Apache-2.0)	1,149	80.7%	attribution only
weak copyleft (LGPL, MPL)	118	8.3%	yes, file-level
strong copyleft (GPL, AGPL)	96	6.7%	yes, whole-work
unlicensed or unresolved	61	4.3%	unknown
total	1,424	100%

Appendix

Table 38: License composition of the CodeFace snapshot. Classes come from the recorded per-entry SPDX identifier; unresolved covers entries whose donor repository declares no license file or a non-machine-readable one.

Explore similar work

Sep 29, 2026cs.SE

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.

Hantian Ding, Chloe Bi, Jiacheng Zhu +5

Meta Superintelligence Labs

Codebases Model Auditing

Ask PDF

Jun 20, 2026cs.SE

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

Natural language to repository generation (NL2Repo) requires a system to construct an entire software repository from a natural-language requirements document. Compared with function-level code generation, this task demands longer planning horizons, stable interfaces across files, and iterative debugging of cross-file inconsistencies. To address these challenges, we propose CodeTeam, an LLM-based multi-agent framework that separates planning, decision making, and implementation into distinct, coordinated stages. In the planning stage, multiple Architect agents draft competing software design sketches (SDS), optionally grounded by retrieved design references. A CTO agent then evaluates, selects, and normalizes the most promising SDS into a machine-checkable contract that specifies file ownership, public interfaces, and dependency constraints. In the implementation stage, Developer agents generate code under a dependency-aware scheduler with bounded context and lightweight Git-based coordination, while a QA agent runs tests and drives iterative repairs. On the synthesis-based SketchEval benchmark, we explicitly compare CodeTeam's prompt-engineering (PE) and supervised fine-tuning (SFT) variants with the corresponding CodeS variants, where CodeTeam improves the overall SketchBLEU by 4.1 and 2.9 absolute points, respectively. On the execution-based NL2Repo-Bench benchmark, used as an external validation protocol, CodeTeam achieves the highest average test pass rate in both settings (34.6% PE, 42.3% SFT), confirming that the sketch-improvements extend to functional correctness under upstream test suites. Ablation results show that project-specific developer allocation and retrieval-augmented planning each contribute substantially to the SketchBLEU improvement (9.9% and 8.1% relative, respectively). CodeTeam and the experimental results are available at https://github.com/WhitenWhiten/CodeTeam

Yifei Wang, Ruiyin Li, Peng Liang +4

School of Computer Science, Wuhan University, China · School of Computer Science, Nanjing University of Science and Technology, China · School of Computer Science, Central China Normal University, China +2

Repository-Level Code Understanding Sketches

Ask PDF

Sep 29, 2026cs.SE

Zero2Repo: Can Coding Agents Build Repositories from Scratch?

Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks. We introduce Zero2Repo, a benchmark in which an agent receives a product requirements document, an interface contract, and an empty workspace, and must deliver a complete repository in the project's native ecosystem. Tasks are produced by a language-agnostic authoring pipeline that converts real, version-pinned open-source projects into behavioral specifications, reproducible environments, and hidden acceptance tests. Each task is validated by execution: a reference implementation derived from the upstream project must pass, and adversarial validation must show that the tests reject incorrect implementations. Evaluation runs production coding agents in isolated containers, withholds the acceptance tests until an explicit submission, and assigns a binary reward only when every test passes, with no LLM judge. The pipeline and harness make no language-specific assumptions and apply to mainstream programming ecosystems; the current release contains Python, TypeScript, Go, and C++ tasks. Even on 11 tasks drawn from repositories that frontier models have very likely seen during training, the strongest agent solves only 10, and every failing submission passes 90-99% of the hidden tests; for the two strongest agents, 67-100% of failed tests trace to a single omission or a low-frequency rule stated in the specification rather than to a missing subsystem, so each failure is a concrete target for improvement.

Pei Yang, Tianyu Shi, Yuhang Yao +23

Gradient Data · McGill University · Carnegie Mellon University +6

Coding Agents Software

Ask PDF