Harvesting SWE Benchmarks from Wild Vibe Coding Sessions

Ao Lu; Yuanhao Zeng; Shaocheng Lan; Sijie Xu; Jiacheng Bao; Kuang Yan; Heng Lin; Kan Ren

cs.SENov 2, 2026

Harvesting SWE Benchmarks from Wild Vibe Coding Sessions

Authors: Ao Lu, Yuanhao Zeng, Shaocheng Lan, Sijie Xu, Jiacheng Bao, Kuang Yan, Heng Lin, Kan Ren

Organizations: ShanghaiTech University · ByteDance, China · Shanghai AI Laboratory

Abstract

Development teams need reproducible tasks from their own repositories and workflows to evaluate coding agents. Conversational developer-agent collaboration ("vibe coding") produces abundant coding sessions and repository checkpoints, but requirements evolve across turns, unrelated conversations co-occur with commits, and engineering efforts can span multiple sessions. We present SessionFarmer, an automated framework that turns these records into self-contained, executable software engineering tasks. It analyzes dialogue and commit history to filter unrelated sessions and group multi-turn, potentially multi-session interactions into work episodes centered on one engineering goal. For each episode, it reconstructs a hermetic execution container, synthesizes a task statement from the full context, and authors repository-grounded engineering-quality rubrics and unit tests. Closed-loop test refinement uses requirement-breaking mutants and a blind solver to reduce false rejections of valid alternatives and false acceptances of incomplete solutions. We construct CropChat, an open benchmark of 242 tasks across 61 public repositories and 14 programming languages, and CropEnterprise from proprietary enterprise logs. In a CropEnterprise audit, the original developers endorse the vast majority of reconstructed episode boundaries, task statements, and quality rubrics. Full-episode context covers substantially more user requirements than single-turn or single-session inputs. Test refinement cuts false-rejection rates and raises balanced accuracy against reference patch labels by 7-9 percentage points across solvers. Across 12 model-harness configurations on CropChat, the strongest setup achieves 39.3% Pass@1, while harness choice shifts Pass@1 by up to 8.4 percentage points for the same model.

Figures & tables

Figure 1: A developer’s goal may span several sessions. A work episode groups the sessions and code changes belonging to one engineering goal.

Figure 2: Overview of SessionFarmer , consisting of (a) episode reconstruction (Section 3.1 ), (b) environment reconstruction (Section 3.2 ), (c) task authoring (Section 3.3 ), and (d) test construction (Section 3.4 ). Starting from raw developer–agent sessions and repository checkpoints, SessionFarmer groups interactions into goal-centered work episodes, builds offline-verified container environments, authors validated task statements and quality rubrics, and refines unit tests against mutants and a blind solver to output self-contained benchmark tasks.

CropChat
Model (Harness)	Pass@1	Pass@3	Pass 3	Mech.	Quality	Time	Timeout	Token I/O
gpt-5.5 (Codex)	$24.9$	$36.8$	$14.0$	$26.0$	$95.7$	$7.7$ $\pm 3.1$	$0.6\,(4)$	$3.59\mathrm{M}/19.1\mathrm{k}$
gpt-5.5 (Pi)	$24.1$	$37.2$	$12.4$	$25.1$	$96.8$	$11.7$ $\pm 4.2$	$1.8\,(13)$	$3.92\mathrm{M}/19.4\mathrm{k}$
gpt-5.5 (OpenCode)	$25.2$	$36.0$	$14.0$	$26.1$	$94.1$	$7.9$ $\pm 2.8$	$0.3\,(2)$	$2.62\mathrm{M}/11.5\mathrm{k}$
gpt-5.6-terra (Codex)	$26.4$	$37.6$	$15.7$	$27.1$	$90.6$	$\mathbf{4.2}$ $\pm 2.2$	$\mathbf{0.0\,(0)}$	$1.11\mathrm{M}/10.4\mathrm{k}$
gpt-5.6-terra (Pi)	$26.0$	$36.0$	$16.1$	$26.4$	$90.3$	$4.7$ $\pm 2.8$	$0.4\,(3)$	$0.99\mathrm{M}/8.5\mathrm{k}$

Table 2: Agent evaluation results on CropChat (

242

tasks) and CropEnterprise (

56

tasks). Pass@1, Pass@3,

\mathrm{Pass}^{3}

, Mech. (

\mathrm{MechanicalResolved}

), and Quality (

\mathrm{QualityResolved}

) are percentages; Mech. and Quality are computed over completed (non-timed-out) rollouts. Timeout gives the percentage and count over scorable rollouts. Time is median minutes

\pm

pooled within-task standard deviation; Token I/O is mean processed input/output tokens in millions/thousands.

Figure 4: Rollout outcome composition and tool-call differences across the

12

model–harness configurations on CropChat. (a) Proportion and count of tasks with

0/3

1/3

2/3

, and

3/3

resolved rollouts across the

3

independent runs per task. (b) Lower-triangle matrix of pairwise differences (row configuration minus column configuration) in mean execution-layer tool calls per measured rollout (blue indicates fewer calls; orange indicates more calls). C, P, and O denote Codex, Pi, and OpenCode, respectively.

Figure 5: Requirements in mined episodes and their effect on statements. (a) Mean per-task requirement coverage of CropChat statements synthesized from different levels of input evidence against an independent reference requirement ledger (number of evaluated tasks in parentheses); for single-session tasks, the first session is the full episode. (b) Across the

34

multi-session CropChat tasks, the share of gold-patch lines first authored in later sessions plotted against the share of reference requirements raised only in those later sessions (Spearman

\rho=0.69

Figure 6: Functional roles across the

40

later sessions (

76

role assignments) in the

34

multi-session CropChat tasks. Sectors follow the legend clockwise and sum to

100\%

of assignments; a single later session may carry multiple role labels.

Benchmark	Solver	Suite	Rollouts	Pass	FP $\downarrow$	FN $\downarrow$	FPR $\downarrow$	FNR $\downarrow$	BAcc $\uparrow$
CropChat	terra	w/o refinement	$87$	$21.8$	$\mathbf{3.4}$	$39.1$	$\mathbf{8.1}$	$68.0$	$61.9$
	terra	SessionFarmer	$87$	$48.3$	$10.3$	$\mathbf{19.5}$	$24.3$	$\mathbf{34.0}$	$\mathbf{70.8}$
	gpt-5.5	w/o refinement	$87$	$16.1$	$\mathbf{5.7}$	$43.7$	$\mathbf{12.5}$	$80.9$	$53.3$
	gpt-5.5	SessionFarmer	$87$	$41.4$	$13.8$	$\mathbf{26.4}$	$30.0$	$\mathbf{48.9}$	$\mathbf{60.5}$
SWE-Bench-Pro	terra	Official	$150$	$54.7$	$18.7$	$17.3$	$40.0$	$32.5$	$63.8$
DeepSWE	terra	Official	$150$	$26.0$	$17.3$	$3.3$	$19.7$	$27.8$	$76.3$

Table 3: Test suite verdicts evaluated against independent gpt-6-astra patch correctness labels on

30

CropChat tasks where refinement modified the initial suite (

87

labeled rollouts per solver after excluding

3

uncertain patches), alongside

2

reference benchmarks (

150

rollouts from

50

tasks each). All columns except Rollouts are percentages. FP and FN denote false positives and false negatives as a percentage of all evaluated rollouts; FPR and FNR denote conditional false-positive and false-negative rates within ground-truth incorrect and correct patches, respectively; BAcc is balanced accuracy (

\frac{1}{2}[(1-\text{FPR})+(1-\text{FNR})]

Figure 7: Developer audit results on CropEnterprise (evaluated counts in parentheses). Upper bars show all judgments on a

0\%

–

100\%

scale; lower bars show flagged and unsure judgments on a

0\%

–

6\%

scale.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Session–checkpoint links within one repository (schematic). Each line means that the checkpoint lists the session; it does not assert a contribution to the code change. The gaps separate checkpoint groups with no shared recorded session.

Rule	Removal reason	Sessions removed
License	License outside MIT, MIT-0, Apache-2.0, ISC, BSD-2-Clause	$964$
Repository reachability	Repository confirmed unreachable	$156$
Transcript	Missing file ( $1$ ); format other than Claude Code ( $919$ )	$920$
User turn	No valid user turn found by the transcript reader	$6$

Appendix

Table 4: Mechanical session filtering before LLM-based construction.

Figure 9: Distribution of the

2{,}708

initial work clusters: (a) checkpoints per cluster; (b) elapsed hours between the first commits of the first and last checkpoints. A zero span (

0\text{ h}

) includes all single-checkpoint clusters and

13

multi-checkpoint clusters with identical recorded commit timestamps.

Checkpoint	Change	Result
C0	Documentation	Excluded from episode intervals
C1	Webhook domain filters	Separate filtered cluster; no episode ( $66$ – $68$ implementation lines)
C2	Docker worker image	Episode retained
C3	One-time scheduled tasks	Episode retained

Appendix

Table 5: One initial cluster in desplega-ai/agent-swarm after session relevance filtering and episode reconstruction. Checkpoints are ordered by first-commit time.

Category	Examples	Tasks	Repos.
GitHub CLI	gh	$79$	$27$
Tool-version managers	mise , rustup	$46$	$5$
Coding-agent CLIs and wrappers	rtk , claude	$25$	$8$
Linters, formatters, scanners	biome , ruff	$17$	$10$
Network and system utilities	ssh , lsof	$16$	$12$
Cloud, deploy and secret CLIs	vercel , doppler	$16$	$7$

Appendix

Table 6: Top

10

software categories among verified tool requirements for the

242

finalized CropChat tasks. Counts are distinct tasks and repositories; a task may appear in multiple categories. Ties are ordered by repository count.

Role / stage	Max context	Sampling or revision limit
Session relevance	—	$3$ valid votes, at most $6$ draws
Episode reconstruction	—	$2$ independent samples; up to $2$ schema repairs per sample
Candidate tool extraction	—	Up to $2$ schema repairs
Environment Builder	$272{,}000$	$2$ revisions ( $3$ submissions total)
Task Author	$1{,}000{,}000$	Statement: $2$ revisions; rubrics: $1$ ; mutants: $1$
Test Author	$1{,}000{,}000$	Initial suite: $2$ revisions; solver calibration: $2$ rounds of at most $3$ versions

Appendix

Table 7: Construction context budgets and sampling/revision limits. Revision limits exclude the initial submission. Task Author and Test Author sessions compact context at

500{,}000

tokens within their

1{,}000{,}000

-token windows; Environment Builder and the solver use Pi’s default

272{,}000

-token window. Single-call judgments do not run inside an agent harness.

Stage	In $\to$ out	Calls	Input (M)	Cache R/W (M)	Out (M)	Med. min	Wall h
Session relevance	$2{,}708\to 1{,}367$	—	—	—	—	—	—
Episode reconstruction	$1{,}367\to 338$	$2{,}426$	$305.4$	$176.9/0.0$	$11.1$	—	$0.5$
Environment reconstruction	$338\to 337$	$4{,}318$	$110.8$	$102.0/7.2$	$2.3$	$3.8$	$3.6$
Task and rubric authoring	$337\to 275$	$18{,}471$	$1{,}414.8$	$1{,}231.6/55.2$	$12.6$	$14.3$	$9.8$
Test synthesis and calibration	$275\to 242$	$36{,}018$	$2{,}575.5$	$2{,}432.8/126.6$	$15.2$	$45.7$	$30.0$
Task finalization	$242\to 242$	$0$	$0$	$0/0$	$0$	—	$<0.1$

Appendix

Table 8: Stage-level construction cost of CropChat. Candidate transitions follow Table 10 (the episode reconstruction row combines reconstruction and mechanical qualification). Calls count usage-bearing model responses. Med. min is median active worker minutes per input candidate, and Wall h sums batch run windows. Cache R/W gives prompt cache read/write tokens included within Input.

Episode reconstruction
Component	Calls	Input (M)	Cache R/W (M)	Out (M)	Share (%)
Episode reconstruction	$2{,}426$	$305.4$	$176.9/0.0$	$11.1$	$7.12$
Environment reconstruction
Candidate tool extraction	$341$	$1.0$	$<0.1/0.0$	$0.8$	$0.04$
Environment Builder	$3{,}977$	$109.8$	$102.0/7.2$	$1.5$	$2.50$
Task and rubric authoring

Appendix

Table 9: Component-level recorded model cost for constructing CropChat. Cache read/write tokens are included in Input; Share (%) is computed over total input plus output tokens.

Panel of Figure 10 — Figure 10: Stage counts for (a) checkpoint evidence and (b) clusters and episodes. Bar lengths are proportional within each panel.

Stage	Transition	Reasons for exclusion
Mechanical filtering	$\downarrow$ $5{,}851$ sess. / $13{,}406$ ckpt. $3{,}805$ sess. / $7{,}648$ canonical ckpt.	Ineligible sessions; commit-less records (App. A.1 )
Work clusters	$\downarrow$ $7{,}648$ canonical ckpt. $2{,}708$ clusters	—
Session relevance	$\downarrow$ $2{,}708$ clusters $1{,}367$ filtered clusters	No confirmed session ( $1{,}393$ )
Episode reconstruction	$\downarrow$ $1{,}367$ filtered clusters $356$ episodes	No suitable episode ( $863$ ); sample disagreement ( $91$ ); Git/repair or budget failures ( $74$ )
Qualification	$\downarrow$ $356$ episodes $338$ episodes	$>5$ confirmed sessions ( $13$ ); size out of range ( $3$ ); duplicate episode ( $2$ )
Environment	$\downarrow$ $338$ episodes $337$ environments	Environment build infeasible ( $1$ )

Appendix

Table 10: Construction funnel of CropChat from SWE-Chat . Transitions use stage-specific units (sess. = sessions, ckpt. = checkpoints); the last column summarizes reasons for exclusion.

Figure 11: Composition of

242

CropChat tasks across

61

repositories. (a) Colors match Figure 3 ; solid bars show task shares and pale outlined bars repository shares. Other combines

9

rarer labels; Unknown is missing metadata. (b) Cumulative tasks by repository rank. (c) Source files and approximate LoC at each repository’s most recent task base commit; point area scales with tasks and dashed lines mark medians. (d) License counts.

Repository	Language	Tasks	License	Files	LoC
entireio/cli	Go	$46$	MIT	$339$	$131.6\mathrm{k}$
obsessiondb/rudel	TypeScript	$31$	MIT	$236$	$31.5\mathrm{k}$
gregszero/open-fang	Ruby	$12$	MIT	$182$	$15.3\mathrm{k}$
PackmindHub/context-evaluator	TypeScript	$12$	Apache-2.0	$273$	$74.9\mathrm{k}$
hutusi/amytis	TypeScript	$10$	MIT	$230$	$23.3\mathrm{k}$
wanshicheng/duckdb-data-agent	Unknown	$7$	MIT	$131$	$20.6\mathrm{k}$

Appendix

Table 11: Repositories in CropChat, ordered by released task count. Language and license follow frozen GitHub metadata. Source-file counts and approximate LoC are measured at the base commit of each repository’s most recent released task; LoC is rounded to

0.1\text{k}

. Unknown denotes missing language metadata.

Figure 12: Evidence language and construction-stage composition. (a) The source-language label comes from the frozen task statement. The left bar shows all

242

tasks; the right bar enlarges the

22

non-English or mixed tasks, with percentages relative to those

22

tasks. Its Other segment combines

4

single-task labels. (b) Cells count candidates by repository primary-language metadata; shading shows each language’s share within its stage, and the right column gives candidate and repository totals. TS and Py abbreviate TypeScript and Python; Other contains the remaining identified labels. These are descriptive counts, not language-specific acceptance-rate estimates.

Figure 13: Task-level distributions on CropChat (

n=242

) and CropEnterprise (

n=56

), shown as percentages within each benchmark. Median pairs follow legend order. Gold files and lines use logarithmic bins; lines count additions plus deletions, words use whitespace splitting, and tests count frozen test scenarios.

Metric	SWE-bench Verified		SWE-Bench Pro (public)		SWE-bench-Live (test)		CropChat		CropEnterprise
Tasks	$500$		$731$		$1{,}000$		$242$		$56$
Repositories	$12$		$11$		$156$		$61$		$1$
	Mean	Median	Mean	Median	Mean	Median	Mean	Median	Mean	Median
Statement words	$189.3$	$143$	$475.9$	$428$	$258.1$	$157$	$374.2$	$347.5$	$277.3$	$203$
Gold files	$2.6$	$2$	$7.2$	$5$	$5.4$	$3$	$12.0$	$8$	$11.1$	$9.5$
Gold lines	$38.3$	$25$	$300.3$	$173$	$238.6$	$65.5$	$904.2$	$411$	$616.4$	$319.5$

Appendix

Table 12: Benchmark sizes and per-task means and medians for fixed public test splits. Words are whitespace-delimited; gold lines are additions plus deletions. External full gold combines patch and test_patch ; non-test gold excludes test_patch and test-code files. SWE-Bench Pro uses the readable official

3

-field statement.

Item	Setting
Harness versions	Pi $0.84.2$ ; Codex $0.149.1$ ; OpenCode $1.18.23$
Solver models	gpt-5.5-2026-04-24 ; gpt-5.6-terra ; glm-5.2 ; kimi-k3
Reasoning effort	Medium for all configurations
Sampling temperature	No runner override (nominal default: $1$ )
Context	Harness-native context windows and compaction; no shared window override
Rollouts and budget	$3$ independent fresh-container rollouts; $2{,}700\,\text{s}$ solve limit each

Appendix

Table 13: Evaluation settings. All model–harness pairs share the same rollout and scoring rules.

CropChat: test files
Model (harness)	Quality Score				Unit Test Pass (%)
	Run $1$	Run $2$	Run $3$	Mean $\pm$ SD	Run $1$	Run $2$	Run $3$	Mean $\pm$ SD
gpt-5.5 (Codex)	$5.38$	$5.27$	$5.39$	$5.35\pm 0.07$	$29.1$	$30.8$	$26.9$	$28.9\pm 2.0$
gpt-5.5 (Pi)	$5.37$	$5.44$	$5.26$	$5.36\pm 0.09$	$28.9$	$27.8$	$27.2$	$28.0\pm 0.8$
gpt-5.5 (OpenCode)	$5.26$	$5.20$	$5.16$	$5.21\pm 0.05$	$28.3$	$28.2$	$30.5$	$29.0\pm 1.3$
gpt-5.6-terra (Codex)	$5.16$	$5.11$	$5.03$	$5.10\pm 0.06$	$27.9$	$31.4$	$30.8$	$30.0\pm 1.9$

Appendix

Table 14: Extended results by rollout position. Quality Score is the mean of observed rollout scores, where each rollout score is the median of

3

judge totals (

0

–

6

). Unit Test Pass is the mean percentage of generated test units passed per rollout. Each Mean

\pm

SD summarizes the

3

run means.

Variable	Never ( $61$ )	Resolved ( $178$ )	$\rho_{\rm repo}$	$95\%$ CI
Statement length (characters)	$3{,}116$	$2{,}516.5$	$-0.425$	[ $-0.526$ , $-0.328$ ]
Gold files	$11$	$7$	$-0.345$	[ $-0.449$ , $-0.183$ ]
Gold lines (added $+$ deleted)	$766$	$367.5$	$-0.416$	[ $-0.524$ , $-0.316$ ]
Source sessions	$1$	$1$	$-0.124$	[ $-0.254$ , $-0.010$ ]

Appendix

Table 15: Task size and solve success. Medians compare tasks never resolved across all

36

rollouts (

61

tasks) with tasks resolved at least once (

178

tasks);

\rho_{\rm repo}

correlates task size with the task’s solve fraction within repositories.

Config.	$n$	Time (min)	Input (M)	Calls	Failed (%)
Terra-C	$699$	$4.98$	$1.10$	$24.25$	$20.2$
Terra-P	$723$	$5.63$	$0.99$	$35.00$	$12.8$
Terra-O	$705$	$6.23$	$0.87$	$43.51$	$11.3$
K3-C	$676$	$15.89$	$2.93$	$60.57$	$6.5$
K3-P	$634$	$21.29$	$2.50$	$63.75$	$4.4$
GPT-O	$713$	$9.41$	$2.64$	$65.75$	$8.5$

Appendix

Table 16: Mean resource use per measured rollout, ordered by mean execution-layer tool calls. The last column pools failed actions over all actions on these same rollouts; failures include tool errors, non-zero exits, explicit rejections, and timeouts. C/P/O denote Codex/Pi/OpenCode.

Source group	Tasks	Resolved / rollouts	Rate (%)
Agent-only ( $\geq 99\%$ )	$90$	$1{,}187/3{,}240$	$36.6$
Collaborative ( $0$ – $99\%$ )	$85$	$651/3{,}060$	$21.3$
Human-only ( $0\%$ )	$28$	$265/1{,}008$	$26.3$
No labeled pushback	$62$	$818/2{,}232$	$36.6$
At least $1$ labeled pushback	$164$	$1{,}523/5{,}904$	$25.8$

Appendix

Table 17: Source attributes and solver outcomes on labeled subsets of the common CropChat tasks. Coding-mode and pushback groups have separate denominators; mixed or unavailable labels are omitted.

Coding Agent Benchmarks

See all 68 papers

Coding-Agent Benchmarks Should Match Their Users' Task Flows
Oct 7, 2026Igor Slinko, Yaroslav Golubev, Sergey TitovLLM Agent Evaluation Coding Agent Benchmarks
JetBrains Research
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
Jul 8, 2026Wenqi Huang, Charley Lee, Leonard Tng +1AI Coding Agents Software Engineering Benchmarks
1Datacurve
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent Benchmarks Software Engineering Benchmarks
Scale AI · Reflection AI

Oct 7, 2026J/K move · Enter open · S save

Coding-Agent Benchmarks Should Match Their Users' Task Flows

Igor Slinko, Yaroslav Golubev, Sergey Titov

JetBrains Research

LLM Agent Evaluation Coding Agent Benchmarks AI Agent Benchmarks Software Engineering Benchmarks Agent Communication Protocols

Open paper AskPDF arXiv

The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.

Abstract

Coding Agent Benchmarks

Explore similar work

Coding-Agent Benchmarks Should Match Their Users' Task Flows

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark