How Do Small Coding Agents Use Code Graphs and Embeddings?

Saurav Bellam

cs.SEOct 25, 2026

How Do Small Coding Agents Use Code Graphs and Embeddings?

Authors: Saurav Bellam

Organizations: The University of Texas at Austin

Abstract

Coding agents are given embedding search and code-graph tools on the premise that a repository's structure is something the model lacks and the tool supplies. We test that premise on SWE-bench Verified with small open-weight models, stratifying tasks by a deterministic hop depth (the shortest code-graph path from code the issue mentions to code the gold patch changes) that a companion paper defines and validates. First, we ablate a localization agent's tool set (grep only; +embedding search; +graph tools; all) on a depth-balanced 160-task split and on all 500 tasks for ten models (six Qwen3 sizes from 0.6B to 32B with five decodings each, an FP8 control, three Gemma 4 sizes; 79k episodes). Structured tools double grep-only accuracy for Qwen3-8B and Gemma 4 E4B; their gain peaks at 17-21 points at 1.7-4B and falls to 4-5 points at 32B (odds ratio 0.77-0.82 per doubling of parameters). From 8B up, about half of that gain is the form of the submission rather than finding the target; the fall from 14B to 32B is a fall in tool uptake at equal grep-only accuracy; and the same 4B size gains or does not gain from a tool depending on the checkpoint. Each extra hop multiplies the odds of correct localization by 0.50-0.67 at every size, and no tool set flattens that gradient: traces show the graph tools' gain is name lookup at hop 0 and one expansion at hop 1, and agents chain expansions in only 12-14% of episodes. A fixed repair model resolves 6-13 of 160 tasks from these localizations (oracle 26). Second, linear probes find direct call edges decodable under a within-anchor, degree-stratified protocol (AUC 0.74; 0.61-0.66 inside agent conversations), but below a lexical baseline (0.99), with two-hop edges at 0.57. Small agents carry a one-hop, largely lexical picture of their repository, in weights and in use. We release the agent, probes, logs and traces.

Figures & tables

Figure 1: Hop depth on SWE-bench Verified. (a) Default label (all four edge types). (b) The same 469 tasks with a mention and a target, re-labelled with containment edges only and with containment plus one other edge type; grey segments are tasks whose mentions are disconnected from the targets under that edge set. Calls are the edge type that connects most tasks; imports and inheritance alone leave 12 and 64 tasks unreachable respectively.

Model	Label variant	$n_{\mathrm{ep}}$	$n_{\mathrm{task}}$	plain		controls
				OR [95% CI]	$p$	OR [95% CI]	$p$
Qwen3-4B	default	1872	156	0.523 [0.401, 0.682]	$1.8{\times}10^{-6}$	0.472 [0.345, 0.644]	$2.4{\times}10^{-6}$
	strict_resolve	1872	156	0.537 [0.414, 0.696]	$2.6{\times}10^{-6}$	0.474 [0.353, 0.636]	$6.5{\times}10^{-7}$
	loose_resolve	1872	156	0.518 [0.393, 0.682]	$2.8{\times}10^{-6}$	0.467 [0.338, 0.645]	$3.7{\times}10^{-6}$
	strict_mentions	612	51	0.657 [0.428, 1.008]	0.054	0.553 [0.344, 0.888]	0.014
Qwen3-8B	default	624	156	0.547 [0.403, 0.744]	$1.2{\times}10^{-4}$	0.468 [0.334, 0.655]	$9.8{\times}10^{-6}$

Table 1: DepthGEE / LabelVar : odds ratio per additional hop for function Acc@10, from the GEE model of Eq. 3 , for the default label and three measurement variants. “plain” includes only hop and arm; “controls” adds repository, number of gold functions and log patch size. Qwen3-4B and Gemma 4 E4B pool three runs (temperature 0 and two seeds at 0.7) on the 160-task subset; Qwen3-8B uses one temperature-0 run on the subset; Qwen3-32B uses one temperature-0 run on all 500 tasks (457 bucketed tasks with function-level gold); the “(500)” rows refit the Qwen3-4B and Qwen3-8B greedy runs on the same 500 tasks ( Section 5.3 ).

n_{\mathrm{ep}}

is the number of episodes,

n_{\mathrm{task}}

the number of tasks.

ID	Model	Setting	Hardware
Abl-4B	Qwen3-4B-Instruct-2507	tool ablation: 4 arms, $T{=}0$	L4
Abl-8B	Qwen3-8B (non-thinking)	tool ablation: 4 arms, $T{=}0$	A100
Abl-Gemma	Gemma 4 E4B-it	tool ablation: 4 arms, $T{=}0$ , gemma4 tool parser; all 500 tasks	A100
Abl-32B	Qwen3-32B (FP8, non-thinking)	tool ablation: 4 arms, $T{=}0$ ; all 500 tasks	A100 80GB
Abl-4B-500 , Abl-8B-500	Qwen3-4B, Qwen3-8B	4 arms, $T{=}0$ , the 340 tasks outside the subset (with Abl-4B / Abl-8B : all 500)	A100 80GB
Seeds	Qwen3-4B, Qwen3-8B, Gemma 4 E4B	run-to-run variance: 4 arms, $T{=}0.7$ , seeds 1 and 2 (Gemma: all 500 tasks)	L4 / A100

Table 2: Agent runs and their identifiers. All runs use the 160-task depth-balanced subset unless noted; “4 arms” is G , E , T , ET . The L4 is a GCP g2-standard-16 VM; the A100s (40 or 80 GB) are Colab instances.

Model	Arm	function Acc@10	func. recall@10	file Acc@5	steps	submit	vs. G (wins)
Qwen3-4B ( Abl-4B )	G	24.4	28.1	53.1	7.8	0.975	—
	E	23.1	27.6	56.2	8.8	0.956	$p=0.82$ (9/11)
	T	30.8	35.2	56.9	8.5	0.994	$p=0.099$ (20/10)
	ET	29.5	33.2	56.2	8.9	0.988	$p=0.18$ (18/10)
Qwen3-8B ( Abl-8B )	G	12.8	14.6	33.1	16.6	0.619	—
	E	26.3	29.5	51.2	11.9	0.819	$p=4.9{\times}10^{-5}$ (24/3)

Table 3: Tool ablation, primary greedy runs (160 tasks; function Acc@10 and recall over the 156 tasks with function-level gold). Steps and submit rate are per episode. The last columns give exact McNemar

p

-values against G (discordant counts in parentheses, arm-only wins / G -only wins) on function Acc@10.

	Qwen3-4B		Qwen3-8B		Gemma 4 E4B
Arm	mean	95% CI	mean	95% CI	mean	95% CI
G	22.6	[16.7, 28.8]	13.0	[8.5, 17.9]	13.5	[9.6, 17.7]
E	21.6	[16.2, 27.4]	24.6	[18.6, 30.6]	25.2	[19.7, 31.2]
T	30.8	[24.1, 37.6]	24.1	[17.9, 30.3]	22.4	[16.9, 28.2]
ET	28.0	[22.0, 34.2]	26.5	[20.5, 32.9]	27.1	[21.4, 33.1]
E $-$ G	$-1.1$	[-4.9, +2.6]	$+11.5$	[+7.1, +16.2]	$+11.8$	[+7.5, +16.5]

Table 4: Three-run estimates (greedy run plus two seeds at

T{=}0.7

; Gemma’s runs restricted to the subset). Each task’s function Acc@10 is averaged over runs; entries are means over 156 tasks with 95% percentile-bootstrap intervals over tasks. Differences are paired.

Figure 2: function Acc@10 by hop bucket and tool set. All panels pool three runs (one greedy, two sampled). Each bucket has 40 tasks (38–40 with function-level gold). The advantage of the graph tools ( T ) over G is largest at hop 0 (16–22 points) and shrinks to 2 points by hop 3+ for the Qwen models (7 for Gemma, whose G arm falls to 2.6%); the embedding arm ( E ) has the flattest slope.

	Qwen3-4B				Qwen3-8B				Gemma 4 E4B
Run	G	E	T	ET	G	E	T	ET	G	E	T	ET
greedy	24.4	23.1	30.8	29.5	12.8	26.3	23.7	29.5	9.6	21.8	21.8	23.7
$T{=}0.7$ , seed 1	23.7	19.9	30.8	29.5	12.2	21.8	23.7	26.3	14.7	28.2	21.2	29.5
$T{=}0.7$ , seed 2	19.9	21.8	30.8	25.0	14.1	25.6	25.0	23.7	16.0	25.6	24.4	28.2
SD	2.4	1.6	0.0	2.6	1.0	2.4	0.7	2.9	3.4	3.2	1.7	3.0

Table 5: Seeds : function Acc@10 (%) per run on the 160-task subset. The greedy run is Abl-4B / Abl-8B / Abl-Gemma ; the others sample at

T{=}0.7

with the given seed. SD is the between-run standard deviation of the arm mean.

Model	Arm	function Acc@10	file Acc@5	hop 0	hop 1	hop 2	hop 3+	steps	submit	vs. G
		overall		function Acc@10 by bucket
Qwen3-4B	G	26.8	61.2	36.8	33.3	13.2	6.8	7.6	0.982	—
	E	29.1	63.0	38.4	34.1	10.5	20.5	8.0	0.970	$p=0.24$ (41/30)
	T	37.1	65.4	53.5	39.8	19.7	15.1	8.4	0.994	$p=5.6{\times}10^{-7}$ (75/25)
	ET	36.5	64.8	53.0	40.7	21.1	11.0	8.8	0.988	$p=2.5{\times}10^{-6}$ (73/26)
Qwen3-8B	G	16.7	41.8	24.3	25.2	3.9	2.7	15.8	0.638	—

Table 6: All 500 tasks: function Acc@10 overall and by bucket, file Acc@5, steps and submit rate , one greedy run per model; McNemar versus G over the 485 tasks with function-level gold (wins/losses).

		strict function Acc@10							canonical ids
Model	seeds	G	E	T	ET	E $-$ G	T $-$ G	ET $-$ G	G	ET	ET $-$ G
Qwen3-0.6B	4	0.2	9.4	7.0	8.0	$+9.2$ [7.6, 11.0]	$+6.8$ [5.2, 8.4]	$+7.8$ [6.4, 9.3]	1.9	9.5	$+7.6$ [6.1, 9.1]
Qwen3-1.7B	4	0.9	22.4	12.3	19.0	$+21.5$ [18.5, 24.6]	$+11.4$ [9.4, 13.6]	$+18.1$ [15.7, 20.7]	11.2	24.5	$+13.3$ [10.7, 16.0]
Qwen3-4B (hybrid)	4	10.1	27.7	27.1	29.5	$+17.6$ [14.6, 20.7]	$+17.0$ [14.0, 20.0]	$+19.4$ [16.4, 22.6]	17.7	33.5	$+15.7$ [12.9, 18.6]
Qwen3-8B	4	17.1	30.4	30.2	32.1	$+13.3$ [10.4, 16.3]	$+13.0$ [10.2, 16.1]	$+15.0$ [12.2, 17.9]	25.7	34.3	$+8.7$ [5.9, 11.4]
Qwen3-14B	4	32.1	44.0	41.4	46.0	$+11.9$ [8.7, 15.2]	$+9.3$ [6.5, 12.2]	$+13.9$ [11.0, 17.0]	42.1	48.9	$+6.9$ [4.4, 9.4]

Table 7: ScaleCurve : function Acc@10 (%) on all 500 tasks by model size and tool set. Left block: strict function Acc@10, the per-task mean over the

T{=}0.7

seeds (four for the Qwen3 sizes, two for Qwen3-14B-FP8 and Gemma), on the 485 tasks with function-level gold, with the paired arm

-

G difference and its 95% bootstrap interval; every Qwen difference has Holm-adjusted

p=0.002

. Right block: the same under canonical rescoring (not available for Gemma E2B/E4B, whose notebook kept only the last model’s rescoring). Gemma 4 12B is one greedy run on the 307 tasks it completed; the run is degenerate ( Appendix E ) and is listed for completeness only.

Figure 3: ScaleCurve over the six Qwen3 sizes (

T{=}0.7

seeds pooled, all 500 tasks). (a) Strict function Acc@10 per arm. (b) Gain of each structured arm over G with 95% bootstrap intervals over tasks; the gain peaks at 1.7–4B and falls to 4–5 points at 32B. (c) The same gain under canonical rescoring: from 8B up, about half of the strict gain is the form of the submission. (d) Fraction of episodes in which the agent calls the structured tool at all: find_entity survives to 14B, graph_neighbors is largely abandoned after 8B, and at 32B every structured tool is called in under half of the episodes.

								calls / episode					eps. using
Model	arm	acc	saw	conv.	submit	cap	steps	grep	read_file	ss	fe	gn	ss	fe/gn
Qwen3-4B	G	.226	.562	.380	.983	.017	8.0	3.8	1.9	—	—	—	—	—
	E	.216	.588	.338	.951	.045	8.8	3.2	2.6	0.59	—	—	.43	—
	T	.308	.660	.456	.989	.004	8.6	2.6	1.3	—	1.54	1.37	—	.98/.90
	ET	.280	.658	.416	.974	.024	9.2	2.6	1.6	0.23	1.73	1.18	.14	.98/.87
Qwen3-8B	G	.130	.481	.271	.596	.402	17.1	10.0	3.6	—	—	—	—	—

Table 8: Traces : episode outcomes and tool use , pooled over three runs per model (156 tasks with function gold per run). saw is the fraction of episodes in which some gold function was revealed; conv. is

\Pr(\text{correct}\mid\text{saw})

; cap is the fraction that hit the 30-step cap. Right block: calls per episode (ss = semantic_search , fe = find_entity , gn = graph_neighbors ) and the fraction of episodes with at least one call to the structured tool.

Figure 4: Which tool first shows the gold, by depth (Qwen3-8B, three runs pooled; the 4B and Gemma panels are in Table 15 ). In arm T , find_entity is the first tool to reveal the gold in 46% of hop-0 episodes and graph_neighbors in 49% of hop-1 episodes; at hop 2+ the graph tools rarely reach it and most episodes never see the gold at all. In arm E , semantic_search accounts for 13–28% of first reveals at every depth.

Figure 5: Repair : downstream repair with a fixed Qwen3-8B repair prompt on the 160 tasks, scored with the official harness. Left: patches and resolved tasks per arm. Right: counts; “loc. correct” is function Acc@10

=1

in the Abl-8B episode.

		Qwen3-4B						Qwen3-8B
Task	Test fold	pair	interaction	$b$ -only	permuted	in-degree	lexical	pair	$b$ -only
call	Django	0.740 [.724, .756]	0.731 [.715, .746]	0.510 [.493, .527]	0.502 [.484, .520]	0.534	0.993	0.733	0.533
	others	0.683 [.665, .700]	0.669 [.651, .686]	0.530 [.511, .550]	0.497 [.479, .516]	0.513	0.987	0.661	0.523
call2	Django	0.569 [.548, .590]	0.532 [.512, .552]	0.520 [.497, .542]	0.476 [.458, .493]	0.501	0.500	0.606	0.608
	others	0.573 [.552, .593]	0.555 [.534, .577]	0.559 [.536, .581]	0.509 [.489, .529]	0.500	0.500	0.552	0.562
inherit	Django	0.619 [.593, .647]	0.569 [.544, .597]	0.688 [.663, .715]	0.462 [.436, .487]	0.500	0.927	0.655	0.667

Table 9: AnchorProbe : within-anchor AUC at layer 24 , strict grouping (anchor

\times

in-degree bin of

b

), with 95% anchor-bootstrap intervals for Qwen3-4B. A relational representation shows as pair/interaction AUC above

b

-only, the permuted control and the in-degree baseline. The lexical baseline is

\mathbf{1}[\text{name}(b)\in\text{text}(a)]

; it is 0.5 for call2 by construction. Qwen3-8B point estimates from the same protocol are given in the last two columns (pair /

b

-only).

Figure 6: AnchorProbe across layers (strict grouping, mean of the two folds). Direct call edges are decodable from the middle layers on, peaking at layer 24, with the interaction term carrying almost all of the signal and the

b

-only probe at chance; call2 is barely above

b

-only; for inherit the

b

-only probe is at or above the pair probe. The degenerate Qwen3-8B layer-32 fit (AUC exactly 0.5 for every pair probe, a numerical failure of the standardised logistic fit at that layer) is omitted.

Task	Subset (fold)	$n$	pos	pair	$b$ -only	permuted	in-degree
call	all (Django)	4860	500	0.605 [.513, .690]	0.511 [.423, .594]	0.458 [.365, .554]	0.521
	all (others)	5728	578	0.655 [.590, .717]	0.520 [.447, .595]	0.516 [.455, .578]	0.580
	unstated (Django)	4565	246	0.590 [.474, .697]	0.479 [.370, .580]	0.486 [.374, .596]	0.503
	unstated (others)	5399	294	0.599 [.515, .686]	0.483 [.390, .584]	0.529 [.446, .613]	0.541
	arm G (Django)	238	33	0.778 [.500, 1.00]	0.556 [.125, .889]	0.333 [.000, .667]	0.500
	arm G (others)	396	64	0.706 [.467, .900]	0.588 [.312, .840]	0.588 [.294, .833]	0.529

Table 10: ConvProbe : within-anchor AUC inside agent conversations (Qwen3-4B, layer 24, strict grouping), with anchor-bootstrap intervals, for subsets of the test pairs. unstated pairs are those for which no graph tool printed the edge. Strict groups cannot be formed for the stated subset (almost all stated pairs are positives), so that row reports the plain within-anchor grouping.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Repository	hop 0	hop 1	hop 2	hop 3+	no mention	total
django/django	87	64	29	38	13	231
sympy/sympy	28	25	14	5	3	75
sphinx-doc/sphinx	2	5	11	17	9	44
matplotlib/matplotlib	14	8	8	4	0	34
scikit-learn/scikit-learn	20	9	3	0	0	32
astropy/astropy	10	6	5	1	0	22

Appendix

Table 11: Hop-depth buckets by repository (default label, all 500 SWE-bench Verified tasks).

	Term	Qwen3-4B (3 runs)	Qwen3-8B (greedy)	Qwen3-8B (3)	Gemma 4 E4B (3)	Qwen3-32B (500)
Arm effect	E vs G	0.93 [0.72, 1.20]	2.83 [1.74, 4.61]	2.61 [1.78, 3.81]	2.48 [1.78, 3.45]	1.21 [1.01, 1.44]
(OR)	T vs G	1.68 [1.20, 2.37]	2.40 [1.46, 3.94]	2.53 [1.72, 3.71]	2.06 [1.48, 2.88]	1.13 [0.94, 1.36]
	ET vs G	1.42 [1.00, 2.02]	3.43 [2.00, 5.89]	2.96 [1.98, 4.45]	2.80 [1.98, 3.95]	1.24 [1.03, 1.49]
	per gold function	0.49 [0.23, 1.07]	0.72 [0.38, 1.38]	0.71 [0.34, 1.47]	0.44 [0.15, 1.25]	0.67 [0.39, 1.15]
	per $\log$ patch line	0.86 [0.51, 1.47]	0.58 [0.31, 1.06]	0.59 [0.31, 1.10]	1.01 [0.53, 1.94]	0.67 [0.47, 0.94]
Interaction	hop (arm G )	0.46 [0.31, 0.69]	0.46 [0.26, 0.82]	0.46 [0.27, 0.76]	0.45 [0.30, 0.67]	0.55 [0.44, 0.70]

Appendix

Table 12: Additional GEE estimates (default label, with controls). Top: arm effects from Eq. 3 . Middle: the interaction model

y\sim\mathrm{hop}\times\mathrm{arm}+\mathrm{repo}+n^{\mathrm{gold}}+\log(1+L)

; the interaction OR multiplies the per-hop OR of G to give that arm’s per-hop OR. Bottom: per-hop OR from separate fits on each arm’s episodes. “Qwen3-8B (3)” pools the greedy run and the two sampled runs.

		G				E				T				ET
Model	Run	0	1	2	3+	0	1	2	3+	0	1	2	3+	0	1	2	3+
Qwen3-4B	Abl-4B	45.0	26.3	15.0	10.5	40.0	18.4	15.0	18.4	55.0	31.6	22.5	13.2	47.5	28.9	27.5	13.2
	Seeds s1	37.5	31.6	15.0	10.5	30.0	15.8	22.5	10.5	62.5	23.7	25.0	10.5	50.0	23.7	27.5	15.8
	Seeds s2	37.5	21.1	12.5	7.9	40.0	18.4	20.0	7.9	57.5	28.9	25.0	10.5	45.0	23.7	20.0	10.5
	ForceTool	—	—	—	—	60.0	18.4	22.5	7.9	52.5	23.7	22.5	18.4	—	—	—	—
Qwen3-8B	Abl-8B	25.0	15.8	5.0	5.3	47.5	23.7	20.0	13.2	47.5	18.4	20.0	7.9	55.0	18.4	27.5	15.8

Appendix

Table 13: function Acc@10 (%) by hop bucket for every run. 40 tasks per bucket (38–40 with function-level gold). Budget is on 20 tasks per bucket.

Arm	function Acc@10	func. recall@10	file Acc@5	steps	submit	$n$
G	17.5	20.8	38.0	11.8	0.797	158
E	20.0	23.4	43.4	13.7	0.736	159
T	19.6	21.7	38.2	15.4	0.637	157
ET	22.4	26.6	40.4	14.8	0.692	156

Appendix

Table 14: Pre-fix run ( Abl-4B-prefix , Qwen3-4B). The first full run, before the forced-tool-call rule of Section 4.3 ; 10 episodes errored and 148 tasks have all four arms. Reported for completeness only; the absolute level and the arm differences are both confounded by episodes that spent their budget repeating a tool-less message.

		arm T					arm ET						arm E
Model	bucket	grep	rf	fe	gn	none	grep	rf	fe	gn	ss	none	grep	rf	ss	none
Qwen3-4B	hop 0	.28	.01	.53	.07	.12	.32	.03	.53	.03	.01	.09	.78	.07	.03	.13
	hop 1	.29	.07	.04	.40	.20	.22	.08	.06	.40	.01	.23	.30	.20	.07	.43
	hop 2	.35	.03	.04	.17	.41	.29	.07	.03	.14	.02	.46	.30	.07	.13	.49
	hop 3+	.17	.04	.00	.16	.64	.17	.03	.03	.15	.04	.60	.17	.09	.13	.60
Qwen3-8B	hop 0	.32	.03	.46	.12	.08	.42	.03	.25	.03	.17	.10	.59	.03	.20	.17

Appendix

Table 15: First tool to reveal a gold function, by bucket (fraction of episodes; “none” = gold never revealed). Qwen panels pool three runs. Abbreviations: fe = find_entity , gn = graph_neighbors , ss = semantic_search , rf = read_file .

Model	Task / fold	pair	permuted	$a$ -only	$b$ -only	name	$-$ distance	same class	in-degree( $b$ )	out-degree( $a$ )	best
Qwen3-4B	call / Django	.785	.520	.709	.613	.968	.609	.561	.847	.893	L24 .785
	call / others	.747	.487	.656	.608	.970	.604	.538	.852	.860	L24 .747
	call2 / Django	.700	.514	.695	.591	.500	.507	.542	.744	.832	L24 .700
	call2 / others	.672	.509	.687	.614	.500	.477	.512	.763	.797	L24 .672
	inherit / Django	.711	.499	.743	.708	.968	.677	.392	.292	.306	L36 .774
	inherit / others	.708	.477	.772	.688	.970	.687	.368	.301	.283	L28 .805

Appendix

Table 16: GlobalProbe : global pair-probe AUC at layer 24 with one-sided probes and non-neural baselines on the test fold. “best” gives the best layer for the pair probe and its AUC. Positives/negatives per test fold: call 7,918/36,291 (Django) and 5,337/25,523 (others); call2 3,526/33,985 and 2,879/23,993; inherit 690/36,291 and 308/25,523. For call2 both the lexical cue and the name baseline are removed by construction.

Task	Test fold	pair	interaction	$b$ -only	permuted	in-degree	lexical
call	Django	0.742 [.732, .753]	0.727 [.716, .737]	0.622 [.614, .631]	0.523 [.514, .532]	0.892 [.885, .899]	0.991
	others	0.705 [.693, .717]	0.672 [.660, .685]	0.617 [.606, .627]	0.498 [.488, .508]	0.889 [.881, .897]	0.985
call2	Django	0.595 [.583, .608]	0.567 [.554, .579]	0.595 [.582, .608]	0.500 [.487, .513]	0.790 [.776, .805]	0.500
	others	0.612 [.600, .624]	0.554 [.540, .568]	0.626 [.612, .641]	0.519 [.506, .532]	0.789 [.778, .801]	0.500
inherit	Django	0.628 [.605, .651]	0.572 [.548, .596]	0.681 [.659, .704]	0.484 [.462, .506]	0.301 [.291, .313]	0.891
	others	0.607 [.569, .644]	0.490 [.452, .530]	0.660 [.624, .697]	0.455 [.419, .490]	0.325 [.305, .344]	0.862

Appendix

Table 17: AnchorProbe under the plain within-anchor grouping (anchor only, no in-degree stratification), Qwen3-4B, layer 24, with anchor-bootstrap intervals. Compared with Table 9 , the

b

-only probe and the in-degree baseline recover a large part of their score, which is exactly the node-level signal the strict grouping removes. Anchors with a positive: 2,937 / 2,650 ( call ), 1,478 / 1,342 ( call2 ), 631 / 299 ( inherit ).

Model	arm	hop 0	hop 1	hop 2	hop 3+	OR per hop
Qwen3-0.6B	G	0.4	0.2	0.0	0.0	0.67 [0.57, 0.78]
	E	9.9	13.0	7.6	4.5
	T	11.2	9.3	1.6	0.3
	ET	10.8	8.5	4.6	2.7
Qwen3-1.7B	G	1.8	0.6	0.3	0.0	0.58 [0.49, 0.69]
	E	30.3	22.6	14.5	11.0

Appendix

Table 18: ScaleCurve : strict function Acc@10 (%) by hop bucket, model and arm (

T{=}0.7

seeds pooled; Gemma 4 12B greedy). The last column is the per-hop odds ratio from a GEE fit

y\sim\mathrm{arm}+\mathrm{hop}

clustered by task on that model’s seed episodes (Qwen3 sizes only), with its 95% interval.

Model	arm	acc	saw	conv.	submit	cap	steps	grep	read_file	ss	fe	gn	use	2-hop
Qwen3-0.6B	G	0.2	0.05	0.03	0.64	0.07	7.9	1.5	0.7	—	—	—	—	—
	E	9.4	0.40	0.20	0.72	0.13	10.4	1.1	2.5	2.93	—	—	.87	—
	T	7.0	0.21	0.23	0.52	0.22	13.8	1.7	0.2	—	7.00	0.11	.98/.08	.32
	ET	8.0	0.42	0.16	0.71	0.16	12.3	1.1	1.3	4.44	1.48	0.44	.68/.42	.20
Qwen3-1.7B	G	0.9	0.28	0.02	0.76	0.13	13.8	9.5	0.5	—	—	—	—	—
	E	22.4	0.46	0.45	0.97	0.02	7.2	3.8	0.2	1.01	—	—	.84	—

Appendix

Table 19: ScaleCurve : episode outcomes and tool use per model and arm , all decodings pooled. saw is the fraction of episodes in which a gold function was revealed, conv. is

\Pr(\text{correct}\mid\text{saw})

, cap the fraction at the 30-step cap; calls per episode for grep , read_file , semantic_search (ss), find_entity (fe) and graph_neighbors (gn); “use” is the fraction of episodes with at least one call to the structured tool (ss for E ; fe/gn for T and ET ); “2-hop” is the fraction of graph_neighbors calls that ask for hops=2 .

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 321 papers

Mar 18, 2026

CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents

A prerequisite for coding agents to perform tasks on large repositories is code localization - the identification of relevant files, classes, and functions to work on. While repository-level code localization has been performed using embedding-based retrieval approaches such as vector search, recent work has focused on developing agents to localize relevant code either as a standalone precursor to or interleaved with performing actual work. Most prior methods on agentic code search equip the agent with complex, specialized tools, such as repository graphs derived from static analysis. In this paper, we demonstrate that, with an effective reinforcement learning recipe, a coding agent equipped with nothing more than a standard Unix terminal can be trained to achieve strong results. Our experiments on three benchmarks (SWE-Bench Verified, Pro, and Lite) reveal that our models consistently achieve superior or competitive performance over 2-18x larger base and post-trained LLMs and sometimes approach performance provided by closed models like Claude Sonnet, even when using specialized scaffolds. Our work particularly focuses on techniques for re-purposing existing coding agent environments for code search, reward design, and RL optimization. We release the resulting model family, CodeScout, along with all our code and data for the community to build upon.

Lintang Sutawika, Aditya Bharat Soni, Bharath Sriraam R R +8

Carnegie Mellon University · Independent · OpenHands

Code Localization Code Search RL for LLM Agents Reinforcement Learning Information Retrieval

Ask PDF

Jun 5, 2026

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tasks as a holistic, binary prediction problem (e.g., resolved or unresolved), neglecting fine-grained agent capabilities such as repository understanding, context retrieval, code localization, and bug diagnosis. In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration, a critical capability of coding agents. Given a repository and an issue, SWE-Explore asks an explorer to return a ranked list of relevant code regions under a fixed line budget. SWE-Explore covers 848 issues across 10 programming languages and 203 open-source repositories. For each instance, we derive line-level ground truth from independent agent trajectories that successfully solved the same issue, distilling the specific code regions their solution paths actually consulted. We evaluate exploration along coverage, ranking, and context-efficiency dimensions, showing that these metrics strongly track downstream repair behavior. Across a broad set of retrieval methods, general coding agents, and specialized localizers, we find that agentic explorers form a clear tier above classical retrieval. While file-level localization is already strong for modern methods, line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers.

Shaoqiu Zhang, Yuhang Wang, Jialiang Liang +8

1Shanghai Jiao Tong University · 2Xinjiang University · 4Independent Researcher +2

AI Coding Agents Agentic Search Software Engineering Benchmarks Code Localization Information Retrieval

Ask PDF

Aug 6, 2026

CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

Modern LLM coding agents such as Claude Code and OpenHands share a common inefficiency: they spend much of their token budget finding the file to patch, rather than patching it. On SWE-Bench Verified, a 30B OpenHands agent averages 23 rounds and 631K tokens per resolved issue, with many calls spent on grep, glob, and view_file during repository exploration. We introduce CodeGrep, a 14B retrieval agent trained end-to-end with GRPO to issue multi-turn parallel grep, glob, and read tool calls and return candidate files to a frozen downstream coding agent. On all 500 SWE-Bench Verified instances, CodeGrep preserves resolve rate while substantially improving efficiency: 27.0% versus 25.8% for the no-retrieval baseline, with 15% fewer rounds and 19% fewer tokens on resolved instances. Across retrievers, downstream utility follows a precision threshold: BM25 with precision 0.375 degrades the agent, Jina with precision 0.445 is neutral, and CodeGrep with precision 0.677 crosses the threshold at which retrieval begins to reduce rollout cost. To enable this study, we mine supervision from 67K open-source agent trajectories using CATM and build a Git-worktree environment for multi-turn agent RL. In our setting, applying the efficiency signal at the advantage layer rather than the reward layer reduces KL drift and translates cleanly into downstream efficiency. We will release the model, training pipeline, RL environment, and evaluation harnesses.

Wuya Chen, Yihao yang, Yang Cao +1

1Netease Guangzhou AI Lab · 2Independent Researcher

AI Coding Agents LLM Agent Workflow Optimization RL for LLM Agents Code Search

Ask PDF

Abstract

AI Coding Agents

Explore similar work

CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents