MetaCurate: Evolving Curation Filters for High-Quality Agentic Coding Data

Donghoon Kim; Seunghyuk Oh; Kevin Galim; Minseo Kim; Minjae Lee; Younghyun Kim; Wooseok Seo; Hyung Il Koo

cs.LGOct 16, 2026

MetaCurate: Evolving Curation Filters for High-Quality Agentic Coding Data

Authors: Donghoon Kim, Seunghyuk Oh, Kevin Galim, Minseo Kim, Minjae Lee, Younghyun Kim, Wooseok Seo, Hyung Il Koo,

Organizations: FuriosaAI · University of California, Berkeley

Abstract

Supervised fine-tuning on agent trajectories is an effective way to improve coding agents, but selecting useful training examples requires more than identifying successful runs. A resolved patch may follow inefficient reasoning or unnecessary tool use, while a failed run may still contain useful intermediate steps. Training pipelines therefore rely on rules and qualitative assessments to filter trajectories, yet designing these filters typically requires substantial human expertise, and validating improvements through repeated training and evaluation is expensive. We introduce MetaCurate, a training-free framework that evolves executable trajectory curation filters using pairwise LLM judgments. Disagreements between the filter's rankings and the judge's preferences provide targeted feedback for jointly revising its rules, rubric, and aggregation policy. The resulting filter scores trajectories independently, so pairwise comparisons are required only during search. Starting from a blank filter, MetaCurate autonomously discovers a curation strategy that achieves competitive downstream performance without task-specific handwritten criteria. We validate downstream utility through controlled SFT experiments on our uncurated trajectory pool and the Nemotron v3 SWE dataset, with random selection and hand-authored filtering as baselines. In our experiments, MetaCurate achieves the highest mean SWE-Verified score among the compared methods, including hand-authored rule-based filtering and a training-loop baseline whose search costs 3.5x as much.

Figures & tables

Figure 1: Comparison of four approaches to selecting agent trajectories for SFT. Outcome-only filtering can retain poor reasoning that leads to a correct patch and discard useful reasoning that ends in failure. No-Loop curation applies a fixed rule-based or LLM-based filter without feedback on the selected data. Train-and-Validate uses downstream training and evaluation to revise the selection, requiring a training run for each candidate. MetaCurate iteratively revises the curation filter using pairwise judge feedback, without model training during search.

Figure 2: Training feedback costs 3.5 times as much and evaluates only 3 candidates against 45.

Figure 3: One iteration of MetaCurate . The proposer edits the current filter and submits candidate filters (i.e., rules, rubric, and policy). The solver applies each candidate to the pool and gives every trajectory a score, which orders the pool. The candidate whose ordering agrees most with these verdicts is kept (see Section 4.3 for the promotion rule), and all scores, verdicts, and rationales are written to the file system for the next proposer.

Method	Task-specific Hand-authored or Oracle Information	DSv4Pro	Nemotron v3
Base model	—	53.9 $\pm$ 2.0	53.9 $\pm$ 2.0
Full pool (volume anchor)	N/A	63.1 $\pm$ 1.2	54.0 $\pm$ 1.6
Random	N/A	61.7 $\pm$ 1.6	52.9 $\pm$ 1.3
Resolved ( Yuan et al., 2023 )	Execution labels	61.9 $\pm$ 0.7	54.4 $\pm$ 0.2
Gold-patch recall ( Shen et al., 2026 )	Reference patches	61.7 $\pm$ 2.4	53.9 $\pm$ 0.5
Rule filter ( Blakeman et al., 2026 )	7 signals, 6 thresholds	63.1 $\pm$ 1.4	54.4 $\pm$ 0.5

Table 1: Downstream performance of Qwen 3.5 9B across curation methods on two pools, with a default training budget of

1{,}000

trajectories. Scores are SWE-bench Verified resolved rates (%). The second column lists hand-authored guidance or oracle information strictly required for curation.

Panel of Figure 4 — Figure 4: Curation-loop ablations. ( 4(a) ) A blank seed achieves concordance comparable to a handwritten filter, without task-specific handwritten initialization. ( 4(b) ) Sol Pro achieves higher concordance than Luna as proposer with the same solver and judge. ( 4(c) , 4(d) ) Judge preferences and solver rankings largely agree across models. Both matrices follow the model order shown in ( 4(d) ). ( 4(e) ) Presentation-order flips are more frequent for closely ranked trajectories.

Method	Resolved (%)
Random	58.7 $\pm$ 1.1
Resolved	59.8 $\pm$ 1.4
Gold-patch recall	58.1 $\pm$ 1.2
Rule filter	59.8 $\pm$ 1.1
LLM filter	58.8 $\pm$ 0.9
CurateEvo	58.5 $\pm$ 0.6

Table 4: 200-trajectory budget on DSv4Pro.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Source dataset	Instance IDs	Trajectories
SWE-rebench ( Badertdinov et al., 2025 )	690	2,100
SWE-rebench-V2 ( Badertdinov et al., 2026 )	531	1,599
SWE-Gym ( Pan et al., 2025 )	196	587
Combined pool	1,415	4,286

Appendix

Table 5: Source composition of DSv4Pro dataset.

Training (Shared)
Epochs	3
Schedule	cosine to $1\times 10^{-6}$ , warmup ratio 0.1
Optimizer	AdamW, $\beta_{1}=0.9$ , $\beta_{2}=0.95$ , $\epsilon=10^{-8}$ , weight decay 0.01
Batch	32
Sequence length	128,000 tokens, no packing, loss on assistant turns only
Gradient clipping	1.0

Appendix

Table 6: Training and evaluation settings. Shared settings are listed separately from each model’s fine-tuning method and learning rate.

Figure 5: A plausible hand-written filter need not behave as intended. (a) Two of six criteria scarcely vary over 200 trajectories. (b) Dropping each criterion changes pairwise orderings (bars) and rank distance $1-\rho$ (dots); the nearly constant criteria change almost nothing. (c,d) The same criterion scores for trajectories A–C yield different rankings under five aggregation policies. Shading marks the top 50; filled markers mean selection probability at least one half. In (d), gated denotes gated mean and lex denotes verif-lex.

Figure 6: Top-50 policy overlap with shared scores.

Figure 7: Search history from a blank seed over twelve iterations. Each row shows the current filter on the left and proposed candidates on the right; C and R denote the numbers of criteria and rules. Empty slots indicate candidates that did not pass precheck (Algorithm 1 ). Red and blue indicate positive and negative concordance differences from the current filter, with darker colors indicating larger differences. Green outlines mark promoted candidates. Orange dashed outlines mark eligible candidates in iterations where the provisionally selected candidate failed re-evaluation, leaving the current filter unchanged.

Figure 8: Search history from a handwritten seed over twenty iterations. Notation and color scale match Fig. 7 . Mostly pale cells indicate few large departures from the current filter, consistent with greater inertia than in the blank-seed run.

Training Data Curation

Training data curation improves the quality, diversity, organization, and reliability of datasets used to train machine-learning models.

See all 156 papers

A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents
Jul 19, 2026Yunze HanSupervised Fine-Tuning LLM Evaluation
Vdynamics GmbH, 80935 Munich, Germany
CurateEvo: Data-Curation Evolving for Agentic Post-Training
Jul 7, 2026Dingzirui Wang, Xuanliang Zhang, Keyan Xu +2Training Data Curation LLM Agent Training
Harbin Institue of Technology
Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning
Oct 9, 2026Sida Li, Yuanxin Liu, Shicheng Li +4AI Coding Agents Training Data Selection
National Key Laboratory for Multimedia Information Processing, Peking University · Huawei Technologies Co., Ltd.

Jul 19, 2026J/K move · Enter open · S save

A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents

Yunze Han

Vdynamics GmbH, 80935 Munich, Germany

Supervised Fine-Tuning LLM Evaluation AI Coding Agents

Open paper AskPDF arXiv

Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models. A central yet underexplored question is how trajectory quality and quantity jointly shape model performance. We present a systematic empirical study of trajectory data filtering for LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct on the SWE-trajectory dataset (67,074 trajectories, of which 32,161 are resolved). We propose a two-axis quality scoring framework -- Efficiency and Style -- and evaluate it through 16 controlled experiments spanning strategy, scale, and ablation analyses. Since 7B-scale models attain near-zero SWE-bench resolve rates, we adopt cross-entropy (CE) loss on held-out trajectories as the primary metric, validated via first-action generation: CE loss and ROUGE-L are perfectly rank-correlated (Spearman

ρ

= -1.00), with limited-sample evidence supporting but not conclusively establishing this proxy. Our results reveal a scale-dependent quality-quantity trade-off: at small scales, doubling the dataset (500 to 1,000) yields ~12.7% CE-loss reduction whereas the TopQ-Random gap stays <1% (Mann-Whitney p > 0.10); at 2,000 trajectories this same gap widens to 3.6% (p = 0.016). Ablation further identifies error-retry rate as the dominant sub-dimension, performing comparably to the full composite (

Δ

< 0.2%). Together, these findings establish trajectory-level quality scoring as a viable but scale-sensitive lever for code-agent SFT and offer a proxy-validated evaluation protocol for the regime where end-to-end resolve rate is statistically infeasible.

Abstract

Training Data Curation

Explore similar work

A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents

CurateEvo: Data-Curation Evolving for Agentic Post-Training

Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning