WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

Haomin Qi; Xiangzhe Xu; Yiming Huang; Jingbo Shang; Chengpeng Wang

cs.SESep 29, 2026

WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

Authors: Haomin Qi, Xiangzhe Xu, Yiming Huang, Jingbo Shang, Chengpeng Wang

Organizations: University of California, San Diego · Purdue University · National University of Singapore

Abstract

Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness and exposes faulty behavior during execution. Such evidence makes audit findings actionable, yet benchmark evaluation is difficult when cases reuse public historical bugs and witnesses or require manual construction. We present WitnessGym, an automated framework for constructing bug-validation benchmarks through bug injection. It injects bugs into test-reached paths of real projects, rebuilds each project, and retains cases exposed by a construction-time witness. Bug specifications and execution adapters allow extension to additional bug types and languages. Bug-preserving transformations vary the surrounding structure while preserving the witness behavior. Based on real-world Java projects with test suites, WitnessGym automatically constructs 1,300 benchmark cases. The injected patches resemble historical bug patches and are difficult for the two evaluated models to distinguish in blinded comparisons. We evaluate four coding agent frameworks in six framework/model pairings across bug types, execution contexts, and transformation depths. Witness construction remains difficult even when the bug pattern is known. Our framework, benchmark cases, and evaluation scripts are available.

Figures & tables

Figure 1: Comparison of benchmark-construction methodologies for executable bug validation.

Work	Diversity	Auto. cases	Fresh bugs	Real projects	Context control
CTF and security challenges
NYU CTF Bench ( 2024 )	High	✗	✗	✗	✗
Cybench ( 2025b )	High	✗	✗	✗	✗
Historical bug validation
LIBRO ( 2023 )	High	✗	✗	✓	✗
Issue2Test ( 2026 )	High	✗	✗	✓	✗

Table 1: Comparison of bug-validation benchmark construction. Diversity is high for multiple bug families, medium for variants within a dominant family, and low for one narrow family. Auto. cases are constructed automatically; fresh bugs are generated rather than reused; real projects retain production code with its build and test environments; and context control varies surrounding call, data, or control structure.

Figure 2: Overview of the WitnessGym construction and evaluation pipeline.

Agent	Framework	Model
A1	Codex	GPT-5.4
A2	Claude Code	Claude Sonnet 4.5
A3	OpenHands	DeepSeek-V3.2
A4	OpenCode	ZAI GLM-4.7
A5	OpenHands	Devstral-2-123B
A6	OpenCode	Qwen3-Coder-480B-A35B-Instruct

Table 2: Evaluated framework/model pairings.

ID	$\mathsf{NoEC}$	$\mathsf{WithEC}$	Average
A1	67.1%	70.7%	68.9%
A2	57.1%	61.8%	59.4%
A3	46.0%	49.8%	47.9%
A4	44.2%	47.6%	45.9%
A5	30.2%	33.8%	32.0%
A6	17.1%	19.3%	18.2%

Table 3: Agent validation success rates.

Bug Type

Comparison of Identical Values

18/17

22/21

17/14

19/16

14/10

15/15

11/11

15/13

9/7

10/9

4/4

6/5

Typo in equals

18/15

21/21

14/13

16/18

12/13

15/13

10/10

12/11

8/7

9/10

5/3

6/5

Typo in hashCode

16/13

18/17

16/11

19/14

10/9

15/12

11/8

13/14

7/6

10/7

4/4

5/5

Hashed Value Without hashCode

14/14

19/17

15/13

16/15

10/11

15/13

9/9

14/10

8/6

10/7

4/3

5/4

Inconsistent equals and hashCode

17/15

18/20

13/12

18/17

12/9

12/13

12/9

13/11

6/6

10/9

3/4

5/4

Table 4: Successful bug validations across bug types. L and S denote long and short execution contexts. Each A/B cell reports successful cases under

\mathsf{WithEC}

and

\mathsf{NoEC}

, respectively.

Figure 3: Success-rate gain with execution context (percentage points). Types follow Table 4 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Type	Bug-type description
API Contract	Comparison of Identical Values	A condition compares a value with itself, producing a constant result that silently disables validation or equality logic.
	Typo in equals	A function overriding equals(Object) has the wrong name or signature, so Java keeps using reference equality.
	Typo in hashCode	A function intended to override hashCode() has the wrong name or signature, so hashed collections use identity hashing.
	Hashed Value Without hashCode	A class overrides equals(Object) but not hashCode() , so logically equal keys may hash to different buckets.
	Inconsistent equals and hashCode	equals and hashCode use different fields or unstable state, breaking lookup in HashMap or HashSet .
Value Flow	Intra-Function Null Dereference	A value that was checked for null is later overwritten, moved outside the guard, or dereferenced on an unguarded path.

Appendix

Table 5: The ten bug types used in the benchmark construction.

Family	Operator	Effect
Call Stack	Helper Call Insertion	Adds an internal helper or wrapper call between the original caller and the bug site, increasing call-chain depth.
	Call-Site Rerouting	Replaces a direct internal call with a forwarding function, so the same logic is reached through an extra dispatch step.
	Extract-and-Apply Split	Splits one computation into an extraction helper and an application helper, forcing the bug to cross function boundaries.
Data Flow	Parameter Bundling	Packs related arguments or locals into an internal object that is passed through helper calls instead of using separate values.
	Local State Bundling	Moves related local variables into a context object, so later code reads state through fields rather than direct locals.
	Intermediate State Passing	Stores intermediate values in a state holder and reads them later across helper calls, lengthening the value flow path.

Appendix

Table 6: The ten bug-preserving transformation operators.

Agent ID	Short		Mid-Short		Mid-Long		Long
Agent ID	NoEC	WithEC	NoEC	WithEC	NoEC	WithEC	NoEC	WithEC
A1	80%	80%	72% $\downarrow$ 8pp	76% $\downarrow$ 4pp	64% $\downarrow$ 16pp	64% $\downarrow$ 16pp	48% $\downarrow$ 32pp	52% $\downarrow$ 28pp
A2	72%	76%	64% $\downarrow$ 8pp	68% $\downarrow$ 8pp	56% $\downarrow$ 16pp	56% $\downarrow$ 20pp	40% $\downarrow$ 32pp	40% $\downarrow$ 36pp
A3	60%	64%	52% $\downarrow$ 8pp	56% $\downarrow$ 8pp	40% $\downarrow$ 20pp	44% $\downarrow$ 20pp	28% $\downarrow$ 32pp	28% $\downarrow$ 36pp
A4	60%	60%	52% $\downarrow$ 8pp	56% $\downarrow$ 4pp	40% $\downarrow$ 20pp	40% $\downarrow$ 20pp	28% $\downarrow$ 32pp	28% $\downarrow$ 32pp
A5	44%	44%	36% $\downarrow$ 8pp	40% $\downarrow$ 4pp	28% $\downarrow$ 16pp	28% $\downarrow$ 16pp	16% $\downarrow$ 28pp	20% $\downarrow$ 24pp

Appendix

Table 7: Bug-validation success rates across execution context length buckets. Subscript annotations give percentage-point differences from the short bucket.

Agent ID	Depth 1		Depth 3		Depth 5		Depth 10
Agent ID	NoEC	WithEC	NoEC	WithEC	NoEC	WithEC	NoEC	WithEC
A1	76%	80%	67% $\downarrow$ 9pp	68% $\downarrow$ 12pp	62% $\downarrow$ 14pp	65% $\downarrow$ 15pp	59% $\downarrow$ 17pp	59% $\downarrow$ 21pp
A2	70%	72%	59% $\downarrow$ 11pp	60% $\downarrow$ 12pp	53% $\downarrow$ 17pp	57% $\downarrow$ 15pp	50% $\downarrow$ 20pp	51% $\downarrow$ 21pp
A3	57%	60%	45% $\downarrow$ 12pp	49% $\downarrow$ 11pp	42% $\downarrow$ 15pp	43% $\downarrow$ 17pp	36% $\downarrow$ 21pp	40% $\downarrow$ 20pp
A4	53%	55%	47% $\downarrow$ 6pp	48% $\downarrow$ 7pp	41% $\downarrow$ 12pp	41% $\downarrow$ 14pp	39% $\downarrow$ 14pp	40% $\downarrow$ 15pp
A5	38%	40%	33% $\downarrow$ 5pp	32% $\downarrow$ 8pp	28% $\downarrow$ 10pp	32% $\downarrow$ 8pp	25% $\downarrow$ 13pp	28% $\downarrow$ 12pp

Appendix

Table 8: Bug-validation success rates across transformation-depth buckets. Subscript annotations give percentage-point differences from depth 1.

Depth	Runs	Final lines	Hunks	Changed files	Line accum.	First-step retention	First-step similarity
1	100	65.0	2.33	1.22	1.00 $\times$	100.0%	100.0%
3	100	97.8	3.59	1.40	2.40 $\times$	88.1%	61.7%
5	100	161.5	4.34	1.48	4.25 $\times$	76.7%	40.8%
10	100	243.7	6.09	1.44	6.63 $\times$	74.9%	25.9%

Appendix

Table 9: Injection-stage patch-evolution metrics by transformation depth. Line accumulation is the ratio between cumulative step-patch lines and final-patch lines. Retention and sequence similarity are measured from the first transformation step to the final buggy patch.

Task	Judge	Correct / Total	Accuracy	90% CI	AUC	90% CI
Single	A	2,173 / 4,200	51.74%	50.14–53.38	0.51198	0.49367–0.53075
Single	B	2,189 / 4,200	52.12%	50.52–53.69	0.51286	0.49426–0.53084
Pairwise	A	723 / 1,400	51.64%	49.07–54.21	0.51681	0.48703–0.54633
Pairwise	B	730 / 1,400	52.14%	49.64–54.64	0.52411	0.49517–0.55252

Appendix

Table 10: Detailed source-discrimination results. Accuracy CIs are percentages.

Judge	Correct / Total	Incorrect / Total	Accuracy	95% CI	Check
A	184 / 200	16 / 200	92.00%	87.00%–96.00%	Pass
B	169 / 200	31 / 200	84.50%	78.50%–89.50%	Pass

Appendix

Table 11: Artifact-bearing positive controls. Both judges meet the prespecified sensitivity check.

Diagnostic	Judge A	Judge B
Single-patch repeat agreement	60.62%	58.67%
Patches with non-unanimous single judgments	59.07%	62.00%
Main orientation-choice agreement	68.43%	64.29%
Main LEFT choice rate	52.93%	54.71%
Main LEFT choices / judgments	741 / 1,400	766 / 1,400
Main split pairs choosing LEFT twice	131	158

Appendix

Table 12: Judgment stability, position preferences, and correctness counts (1,400 patches per judge).

Prompt Structure	Main Inputs	Required Output
Bug Injection System Prompt	Global injection rules, replay-command discipline, bounded edit–run–revise loop, diff checks, and success criteria	Single-line INJECT_REPORT_JSON with operational evidence fields
Bug Injection Task Payload	Bug type, bug type specification, required and forbidden elements, execution context, candidate anchors, trigger hints, and file scope	Same INJECT_REPORT_JSON prefix with type-site rationale fields
Bug-Preserving Transformation Task Payload	Transformation operator specification, applicability requirements, preservation constraints, execution context, and candidate anchors	Single-line TRANSFORM_REPORT_JSON
Bug Validation System Prompt	Test-only edit constraints, Java test-case materialization rules, executable-witness objective, and evaluation-attempt policy	Single-line TRIGGER_REPORT_JSON fallback schema
Bug Validation Task Payload	Buggy repository metadata, case ID, bug type, applied transformations, bug-related files, replay command, and optional execution context	Single-line TRIGGER_REPORT_JSON with detailed witness-construction fields

Appendix

Table 13: Layered prompt structures used by WitnessGym during benchmark construction and executable bug validation.

Explore similar work

Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch
Aug 3, 2026Shuyang Xie, Shuxiao Xie, Feng Zhu +2Raw Judge Outputs Test Generation
Harbin Institute of Technology · Fudan University · Sun Yat-sen University
AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection
Apr 13, 2026Zijie Zhao, Chenyuan Yang, Weidong Wang +3Bug Detection Bug
University of Illinois Urbana-Champaign, Champaign, USA
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
Date pendingBingchen Zhao, Dhruv Srikanth, Yuxiang Wu +1Coding Agents Agentic Benchmarks
Weco AI

Aug 3, 2026, cs.SEJ/K move · Enter open · S save

Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

Shuyang Xie, Shuxiao Xie, Feng Zhu, Yanli Ji+1

Harbin Institute of Technology · Fudan University · Sun Yat-sen University

Raw Judge Outputs Test Generation Coding Agents Model Auditing Adversarial Evaluation

Open paper AskPDF arXiv

Online-judge verdicts and the datasets and benchmarks built on them are treated as ground truth for evaluating and training large language models for code. Yet prior audits have sounded a warning: official suites accept buggy submissions. These audits, however, stop at the warning and offer no practical remedy. Our remedy has two parts: an off-the-shelf coding agent, serving as a test-suite auditor, both builds adversarial test suites to expose what official suites miss and supplies these suites where no official suite exists; a certification chain determines whether each agent-flagged submission is genuinely buggy without relying on the official judge: multiple independently written accepted solutions agree on the expected output for every test, brute-force solutions settle disagreements, and a per-problem validator certifies each failing input legal. One such agent identifies 589 verified accepted-but-buggy submissions among AtCoder's 20,375 audited accepted submissions; extending the same certification to all five agents yields a union floor of 906 such submissions. Five agents, scored separately, each stay within 1.7pp of official-suite coverage on logic bugs those suites catch. On post-cutoff Codeforces problems with no available official suites, the same test-building method leads all five reproduced baselines at every tested input budget. Where an official suite exists, the agent audits suite adequacy instead of assuming it; where none exists, agent suites catch the most buggy submissions among methods we reproduced and tested.

Abstract

Explore similar work

Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection

SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents