Organizations: University of California, Berkeley, Berkeley, CA, USA · Northeastern University, San Jose, CA, USA · Northeastern University, Boston, MA, USA
Planning and revision are intended to bring generated code closer to the task specification. Yet a mistaken interpretation can persist across the plan, implementation, and tests - a pattern we call interpretive lock-in. We investigate this risk on 30 DataFrame tasks derived from TPC Benchmark DS, comparing Frozen One-Pass, Test-Guided Revision, and PlanAudit-Guided Revision. Test-pass counts rose from 22/30 to 29/30 and 30/30, yet no generated solution matched its stored reference output. Evaluator controls returned the expected outcome in 57/57 cases. Same-engine SQL re-execution reproduced 28/30 references; all 77 test-passing runs on these tasks still failed reference agreement. PlanAudit-Guided Revision improved code scores on 3/30 tasks and test coverage on 11/30, but the median paired change was zero for both measures. Failure analysis revealed tests that omitted requirements, adopted candidate-defined schemas, used nondiscriminating fixtures, or repeated the implementation's incorrect logic. Fresh generation on ten targeted tasks produced 0/30 reference matches and 25/25 false passes among test-passing runs. These findings are diagnostic rather than causal: the pipelines differ in feedback, grounding, call budgets, and editing permissions, and the study uses one model and one primary generation per eligible task. Because pre-adjudication human labels were not retained, inter-rater agreement cannot be reconstructed and the mechanism analyses remain exploratory. Overall, increasingly successful self-generated tests did not establish semantic correctness, even after excluding unreproduced references.
Figures & tables
Artifact
One-Pass
Test-Guided
PlanAudit
plan.md
Frozen
Frozen
Revise
solution.py
Frozen
Revise
Revise
test_solution.py
Frozen
Limited a
Revise
a Syntax/import/fixture edits; no gold assertions.
TABLE I: Artifact revision permissions.
Control
Expected
Observed
Unchanged reference
Accept
33/33 accepted
Row permutation
Accept
3/3 accepted
Column alias change
Reject
3/3 rejected
Column-order change
Reject
3/3 rejected
Row deletion
Reject
3/3 rejected
Row duplication
Reject
3/3 rejected
TABLE II: Evaluator control results.
Metric
One-Pass
Test-Guided
PlanAudit
Tests pass
22/30
29/30
30/30
Reference match
0/30
0/30
0/30
False pass
22/30
29/30
30/30
False-pass rate
73.3%
96.7%
100.0%
Mean plan score
89.16%
89.16%
90.34%
Mean code score
76.73%
79.07%
81.01%
TABLE III: Pipeline outcomes and mean obligation scores.
Revision
Score
Up/Same/Down
Median [Q1, Q3]
Test-Guided
Plan
0/30/0
0 [0, 0]
Test-Guided
Code
1/29/0
0 [0, 0]
Test-Guided
Test
3/27/0
0 [0, 0]
PlanAudit-Guided
Plan
4/26/0
0 [0, 0]
PlanAudit-Guided
Code
3/27/0
0 [0, 0]
PlanAudit-Guided
Test
11/19/0
0 [0, 8.33]
TABLE IV: Within-task score changes from Frozen One-Pass.
Fig. 1: All-run test passing, code scores, and test coverage increase, while primary reference agreement remains zero ( n=30 per condition).
Cohort
One-Pass
Test-Guided
PlanAudit
All assigned
22/30
29/30
30/30
SQL-reproduced
21/28
28/28
28/28
Also excluding q11
20/27
27/27
27/27
TABLE V: Test passing across reference-sensitivity cohorts.
Mechanism
One-Pass
PlanAudit
Gap despite plan
13 (33%)
17 (33%)
Partial coverage
10 (26%)
13 (25%)
Candidate-schema validation
9 (23%)
11 (21%)
SCO: absent plan and test
6 (15%)
7 (13%)
Nondiscriminating data
0
3 (6%)
Same incorrect logic
1 (3%)
1 (2%)
TABLE VI: Failure mechanisms among violated obligations.
Test-time compute has become a central way to improve code generation: systems sample multiple candidate programs and use verifier-visible evidence to select the final output. This paradigm implicitly assumes that the verifier provides a corrective signal independent from the generator. We challenge this assumption under misleading task premises. When the generator and verifier share a false premise, they become coupled through a mistaken belief: the generator produces premise-consistent shortcuts, while the verifier supplies evidence that fails to expose them. Consequently, the selector may choose a hidden-test-wrong candidate even when a hidden-test-correct program exists in the pool. We call this failure mode Verification Trap. Across three code-generation benchmarks and five code models, false premises consistently degrade first-sample correctness, reduce selector-chosen correctness after 64-sample test-time selection, and amplify recoverable mis-selection. Mechanistically, verifier-written tests inherit the premise-level blind spot, reshaping verifier-visible candidate space away from hidden-test correctness. These traces make Verification Trap predictable before hidden execution: a lightweight gold-free predictor using verifier-visible features reaches 0.846 AUROC. Our results identify decoupled evidence as a key mitigation axis: coupled scaling provides limited recovery, whereas premise-agnostic robustness auditors recover substantial oracle headroom.
Feng He, Hejia Wang, Linghao Meng +2
University of Science and Technology of China, China · Beijing University of Posts and Telecommunications, China · National University of Singapore, Singapore +1
Contemporary LLM-based coding agents produce code as black-box outputs: the rationale behind each line is hidden, the evolution of the code through benchmark-driven repair is ephemeral, and post-hoc auditing is impossible. We present a code generation concept that addresses these shortcomings through three complementary mechanisms: (i) a relational snippet-history schema that records, per repair event, the benchmark reference, round number, failure text, and LLM explanation, enabling full provenance queries; (ii) a browser-based visualisation tool that renders this history as heat-mapped, hover-annotated source code; and (iii) a competitive fractional position-key indexing scheme with tree-node delimiters that assigns stable, lexicographically-ordered identifiers to each code snippet, enabling fine-grained tracking without disrupting surrounding lines. We evaluate TraceCoder on 30 algorithmic programming tasks spanning string processing, mathematical computation, and data-structure manipulation, across two provider configurations. Of these, 10 exhaust the 6-iteration budget on tasks with subtle edge-case behaviour. Mean Chg% reaches 30%, three in ten code snippets carry a traceable repair-event row, compared to 21% when using Gemini 2.0 Flash as sole provider on a 20-task subset. Three detailed case studies demonstrate how the system explains which specific benchmark failures shaped each line of the final program. The proposed mechanism makes the internal "narrative" of automated code generation auditable and replayable, a property essential for trust and accountability in production deployments.
Selecting LLM-generated code candidates using LLM-generated tests is challenging because the tests themselves may be incorrect. Existing methods either treat all tests equally or rely on ad-hoc heuristics to filter unreliable tests. Yet determining test correctness requires knowing which codes are correct, creating a circular dependency. Our key insight is that we need not determine test correctness at all: test votes should rank, not merely count. What matters is not how many codes pass a test, but whether the test can distinguish correct from incorrect code. We break the circular dependency via leave-one-out evaluation: hold out one test, rank codes by their aggregate scores on all remaining tests, and measure whether the held-out test's pass/fail pattern agrees with this ranking. We formalize this agreement as the leave-one-out AUC~(LOO-AUC) and prove that the expected LOO-AUC is proportional to each test's ability to separate correct code from incorrect code. Building on this, we propose ACES~(AUC ConsistEncy Scoring) with two complementary variants: ACES-C provides closed-form weights that provably approximate the oracle in expectation under a mild assumption on average test quality; ACES-O drops this assumption and iteratively optimizes a differentiable LOO-AUC objective. Both operate solely on the binary pass matrix with negligible overhead, and achieve state-of-the-art Pass@k on multiple code generation benchmarks.
Hui Sun, Yun-Ji Zhang, Zheng Xie +4
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China