Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning

Sida Li; Yuanxin Liu; Shicheng Li; Linli Yao; Xu Sun; Haoli Bai; Xiaohui Li

cs.AIOct 9, 2026

Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning

Authors: Sida Li, Yuanxin Liu, Shicheng Li, Linli Yao, Xu Sun, Haoli Bai, Xiaohui Li

Organizations: National Key Laboratory for Multimedia Information Processing, Peking University · Huawei Technologies Co., Ltd.

Abstract

Open coding agents are commonly trained through supervised fine-tuning on interaction trajectories whose final patches pass the benchmark test suite. However, this outcome-level filtering criterion ignores process quality: trajectories that eventually succeed may still contain inefficient interaction patterns such as repetitive file inspection, unstable tool usage, or delayed modification behavior, all of which become part of the imitation target. We propose TQS (Trajectory Quality Score), a structural scoring framework for coding-agent trajectories that operates directly on trajectory structure, requires no LLM judge, and runs in linear time. TQS characterizes trajectory quality through structural signals organized around structured convergence: information focus, intent--action alignment, and policy dynamics. Using 155,144 successful trajectories from a public open-license corpus, we rank trajectories by TQS and fine-tune coding agents on filtered subsets. Models trained on top-ranked trajectories consistently outperform those trained on random or bottom-ranked subsets on SWE-Bench-Verified. Our results suggest that process-level trajectory quality is an important factor in coding-agent supervision and that structural trajectory filtering provides an effective and scalable alternative to LLM-based data selection.

Figures & tables

Figure 1: Overview of tqs . Contrasting high-quality (focused and convergent) with low-quality (repetitive and unstable) successful trajectories, TQS evaluates trajectory quality along three structural dimensions: Information Focus (OEC, PSN), Intent–Action Alignment (IAC), and Policy Dynamics (TTE, SCP, PED), with a Dirty Pattern Indicator (DPI) guardrail. These measures are aggregated into a weighted score in $[0,1]$ .

Figure 2

Figure 3: Two passing trajectories on the same pygments LassoLexer task. The high- tqs run reaches the fix in 36 steps. The low- tqs run takes 73 steps and includes many interrupts and forced process terminations. Their final outcomes are equal, and their tqs values differ by 0.36.

Subset	Top	Bottom	Random
# trajectories	1,024	1,024	1,024
Mean tqs	0.714	0.454	0.589
Mean steps	34.2	53.0	41.0
Mean tokens (K)	28.1	26.3	26.9
Tasks / repos	847/209	847/204	825/217
Max per task	30	30	59

Table 2: Training subsets. Top and Bottom have zero tqs overlap (Top.min

=0.706>

Bottom.max

=0.470

). Source mix is Smith/Reb./Gym.

			pass@1 by difficulty (%)
ID	Method	$\|\mathcal{D}\|$	Easy	Medium	Hard	Overall	$\Delta$ Base
No-SFT control
B0	Qwen3-Coder-30B-A3B (base, no SFT)	—	36.6	19.2	2.2	24.4	—
Subset baselines (matched 1,024 trajectories, identical SFT recipe)
B1	Random- $K$	1,024	40.7	16.9	2.2	24.8	$+$ 0.4
B2	Bottom- $K$ (lowest tqs )	1,024	33.5	16.9	0.0	21.8	$-$ 2.6

Table 3: Main results on SWE-Bench Verified. All SFT runs use the same Qwen3-Coder-30B-A3B-Instruct base, LoRA hyperparameters, and OpenHands evaluation scaffold; only the selected training trajectories differ. Difficulty buckets follow OpenAI (2024) .

Figure 4: Main results on SWE-Bench Verified. Pass@1 on the full test set (top) and Hard subset (bottom). tqs -Top recovers most of the LLM-as-Judge gain at much lower selection cost.

Validation statistic	Human	Sonnet-5
High vs. Low agreement	85.2 (208/244)	81.1 (198/244)
Mid vs. Low agreement	74.2 (190/256)	77.0 (197/256)
High vs. Mid agreement	76.8 (172/224)	67.4 (151/224)
Overall agreement	78.7 (570/724)	75.4 (546/724)
Pooled concordance	73.3 (631/861)	76.2 (656/861)
Full-order match	45.7 (137/300)	41.3 (124/300)

Table 4: Direct validation of the tqs ordering among successful runs. Agreement (%) uses order-consistent pairwise judgments; pooled concordance uses pointwise rubric scores with ties excluded.

Selection signal	pass@1
Full tqs -Top	27.6
w/o Information focus	24.6
w/o Intent–action alignment	23.0
w/o Policy dynamics	25.8
Random- $K$	24.8
Base (no SFT)	24.4

Table 5: Dimension ablation of tqs . Each variant removes one structural dimension from data selection and retrains with the same SFT recipe. Random-

K

and the untuned base model are shown for reference.

	Training Data	Fine-tuned models
Process metric	Top- $K$	Top	Rand	Bottom
Mean Steps	34.2	50.66	57.64	63.46
Mean oec	0.603	0.571	0.521	0.484
Mean psn	0.753	0.570	0.548	0.568
Mean scp	0.994	0.777	0.798	0.772
Mean ped	0.826	0.192	0.267	0.164

Table 6: Process metrics of the Top-

K

training data and the fine-tuned models’ own SWE-Bench Verified rollouts.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Weight	Run 2	Run 4
oec	0.30	0.7384	0.3120
iac	0.25	0.6667	0.5610
psn	0.10	0.8491	0.5708
tte	0.10	0.5718	0.4046
scp	0.05	0.9904	0.5280
ped	0.10	1.0000	0.0000

Appendix

Table 7: Component scores for the two resolved runs in Figure 3 . Run 2 is the high- tqs trajectory and Run 4 is the low- tqs trajectory.

	Human			Claude Sonnet-5
Contrast	Agreement	95% CI	$p$	Agreement	95% CI	$p$
High vs. Low	85.2% (208/244)	80.3–89.1	$1.34\!\times\!10^{-30}$	81.1% (198/244)	75.8–85.6	$1.18\!\times\!10^{-23}$
Mid vs. Low	74.2% (190/256)	68.5–79.2	$4.27\!\times\!10^{-15}$	77.0% (197/256)	71.4–81.7	$1.51\!\times\!10^{-18}$
High vs. Mid	76.8% (172/224)	70.8–81.8	$3.43\!\times\!10^{-16}$	67.4% (151/224)	61.0–73.2	$2.04\!\times\!10^{-7}$
Overall	78.7% (570/724)	75.6–81.6	$5.94\!\times\!10^{-57}$	75.4% (546/724)	72.1–78.4	$2.70\!\times\!10^{-44}$

Appendix

Table 8: Agreement with the tqs ordering among order-consistent pairs. Confidence intervals are 95% Wilson intervals. The

p

-values use two-sided exact binomial tests against 0.5.

	Human rubric			Claude Sonnet-5 rubric
Contrast	Conc.	95% CI	$p$	Conc.	95% CI	$p$	Ties
High vs. Low	78.3% (227/290)	73.2–82.6	$6.52\!\times\!10^{-23}$	82.1% (238/290)	77.2–86.1	$1.35\!\times\!10^{-29}$	10
Mid vs. Low	69.5% (203/292)	64.0–74.5	$2.11\!\times\!10^{-11}$	75.7% (221/292)	70.5–80.3	$4.39\!\times\!10^{-19}$	8
High vs. Mid	72.0% (201/279)	66.5–77.0	$1.11\!\times\!10^{-13}$	70.6% (197/279)	65.0–75.6	$4.39\!\times\!10^{-12}$	21
Pooled	73.3% (631/861)	70.2–76.1	$6.59\!\times\!10^{-44}$	76.2% (656/861)	73.2–78.9	$1.05\!\times\!10^{-55}$	39

Appendix

Table 9: Concordance between pointwise rubric scores and the tqs tier ordering after excluding equal-score ties. Confidence intervals are 95% Wilson intervals;

p

-values use two-sided exact binomial tests against 0.5.

Figure 5: tqs distribution on the 38,382-trajectory eligible pool. The distribution peaks around 0.58–0.60. The Top-1024 threshold is 0.7061, and the Bottom-1024 threshold is 0.4696, leaving a gap of 0.2365 between the two selected subsets.

Component	Mean	Std.	Undefined
oec	0.429	0.118	0.0%
psn	0.646	0.153	0.0%
iac	0.570	0.098	0.0%
tte	0.550	0.038	0.0%
scp	0.978	0.060	0.0%
ped	0.580	0.232	0.7%

Appendix

Table 10: Marginal statistics of the seven retained tqs components on the eligible pool. scp and dpi are concentrated near 1, while oec , psn , iac , and ped provide broader variation.

Panel of Figure 6 — Figure 6: Marginal distributions of the seven retained tqs components on the eligible pool. All components are normalized to $[0,1]$ before aggregation.

Figure 7: Spearman correlation matrix among retained tqs components on the eligible pool. The largest off-diagonal correlation is 0.320 between tte and ped , indicating no severe redundancy among the retained components.

Score / Component	Pair-AUC
tqs composite	1.000
oec	0.996
iac	0.963
ped	0.961
psn	0.839

Appendix

Table 11: Length-controlled pair-AUC in the modal step-count band

\text{total\_steps}\in[25,40]

Source	Passing	Eligible	Retention
SWE-Smith	89,501	23,569	26.3%
SWE-Rebench	44,739	7,821	17.5%
R2E-Gym	20,904	6,992	33.4%
All	155,144	38,382	24.7%

Appendix

Table 12: Retention after applying the fixed-budget SFT eligibility filters to passing trajectories. All passing trajectories satisfy the step-count and coverage filters; the main reduction comes from the tokenized-length cap and subsequent deduplication by trajectory_id .

Weight setting	Top-1024 overlap
Main weights	100.0%
$w_{\textsc{oec}}=0.25$	93.1%
$w_{\textsc{oec}}=0.35$	93.6%
$w_{\textsc{iac}}=0.20$	93.1%
$w_{\textsc{iac}}=0.30$	94.1%
$w_{\textsc{dpi}}=0.05$	98.3%

Appendix

Table 13: Weight sensitivity of tqs selection on the eligible pool. We perturb one weight at a time while keeping the other weights fixed, then measure overlap between the resulting Top-1024 set and the main Top-1024 set.

Figure 8: Weight sensitivity of Top-1024 selection under single-weight perturbations.

Explore similar work

A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents
Jul 19, 2026Yunze HanSupervised Fine-Tuning LLM Evaluation
Vdynamics GmbH, 80935 Munich, Germany
From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents
May 21, 2026Murong Ma, Tianyu Chen, Yun Lin +7Supervised Fine-Tuning Software Engineering Agents
1National University of Singapore · 2Microsoft Research Asia · 3Shanghai Jiao Tong University +1
A Rubric-Supervised Critic from Sparse Real-World Outcomes
Mar 4, 2026Xingyao Wang, Valerie Chen, Heng Ji +1Reward Modeling AI Agent Evaluation
OpenHands · UIUC · CMU

Jul 19, 2026J/K move · Enter open · S save

A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents

Yunze Han

Vdynamics GmbH, 80935 Munich, Germany

Supervised Fine-Tuning LLM Evaluation AI Coding Agents

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models. A central yet underexplored question is how trajectory quality and quantity jointly shape model performance. We present a systematic empirical study of trajectory data filtering for LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct on the SWE-trajectory dataset (67,074 trajectories, of which 32,161 are resolved). We propose a two-axis quality scoring framework -- Efficiency and Style -- and evaluate it through 16 controlled experiments spanning strategy, scale, and ablation analyses. Since 7B-scale models attain near-zero SWE-bench resolve rates, we adopt cross-entropy (CE) loss on held-out trajectories as the primary metric, validated via first-action generation: CE loss and ROUGE-L are perfectly rank-correlated (Spearman

ρ

= -1.00), with limited-sample evidence supporting but not conclusively establishing this proxy. Our results reveal a scale-dependent quality-quantity trade-off: at small scales, doubling the dataset (500 to 1,000) yields ~12.7% CE-loss reduction whereas the TopQ-Random gap stays <1% (Mann-Whitney p > 0.10); at 2,000 trajectories this same gap widens to 3.6% (p = 0.016). Ablation further identifies error-retry rate as the dominant sub-dimension, performing comparably to the full composite (

Δ

< 0.2%). Together, these findings establish trajectory-level quality scoring as a viable but scale-sensitive lever for code-agent SFT and offer a proxy-validated evaluation protocol for the regime where end-to-end resolve rate is statistically infeasible.

Abstract

Explore similar work

A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents

From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents

A Rubric-Supervised Critic from Sparse Real-World Outcomes