Organizations: University of California, Berkeley, Berkeley, CA, USA · Northeastern University, San Jose, CA, USA · Northeastern University, Boston, MA, USA
Planning and revision are intended to bring generated code closer to the task specification. Yet a mistaken interpretation can persist across the plan, implementation, and tests - a pattern we call interpretive lock-in. We investigate this risk on 30 DataFrame tasks derived from TPC Benchmark DS, comparing Frozen One-Pass, Test-Guided Revision, and PlanAudit-Guided Revision. Test-pass counts rose from 22/30 to 29/30 and 30/30, yet no generated solution matched its stored reference output. Evaluator controls returned the expected outcome in 57/57 cases. Same-engine SQL re-execution reproduced 28/30 references; all 77 test-passing runs on these tasks still failed reference agreement. PlanAudit-Guided Revision improved code scores on 3/30 tasks and test coverage on 11/30, but the median paired change was zero for both measures. Failure analysis revealed tests that omitted requirements, adopted candidate-defined schemas, used nondiscriminating fixtures, or repeated the implementation's incorrect logic. Fresh generation on ten targeted tasks produced 0/30 reference matches and 25/25 false passes among test-passing runs. These findings are diagnostic rather than causal: the pipelines differ in feedback, grounding, call budgets, and editing permissions, and the study uses one model and one primary generation per eligible task. Because pre-adjudication human labels were not retained, inter-rater agreement cannot be reconstructed and the mechanism analyses remain exploratory. Overall, increasingly successful self-generated tests did not establish semantic correctness, even after excluding unreproduced references.
Figures & tables
Artifact
One-Pass
Test-Guided
PlanAudit
plan.md
Frozen
Frozen
Revise
solution.py
Frozen
Revise
Revise
test_solution.py
Frozen
Limited a
Revise
a Syntax/import/fixture edits; no gold assertions.
TABLE I: Artifact revision permissions.
Control
Expected
Observed
Unchanged reference
Accept
33/33 accepted
Row permutation
Accept
3/3 accepted
Column alias change
Reject
3/3 rejected
Column-order change
Reject
3/3 rejected
Row deletion
Reject
3/3 rejected
Row duplication
Reject
3/3 rejected
TABLE II: Evaluator control results.
Metric
One-Pass
Test-Guided
PlanAudit
Tests pass
22/30
29/30
30/30
Reference match
0/30
0/30
0/30
False pass
22/30
29/30
30/30
False-pass rate
73.3%
96.7%
100.0%
Mean plan score
89.16%
89.16%
90.34%
Mean code score
76.73%
79.07%
81.01%
TABLE III: Pipeline outcomes and mean obligation scores.
Revision
Score
Up/Same/Down
Median [Q1, Q3]
Test-Guided
Plan
0/30/0
0 [0, 0]
Test-Guided
Code
1/29/0
0 [0, 0]
Test-Guided
Test
3/27/0
0 [0, 0]
PlanAudit-Guided
Plan
4/26/0
0 [0, 0]
PlanAudit-Guided
Code
3/27/0
0 [0, 0]
PlanAudit-Guided
Test
11/19/0
0 [0, 8.33]
TABLE IV: Within-task score changes from Frozen One-Pass.
Fig. 1: All-run test passing, code scores, and test coverage increase, while primary reference agreement remains zero ( n=30 per condition).
Cohort
One-Pass
Test-Guided
PlanAudit
All assigned
22/30
29/30
30/30
SQL-reproduced
21/28
28/28
28/28
Also excluding q11
20/27
27/27
27/27
TABLE V: Test passing across reference-sensitivity cohorts.
Mechanism
One-Pass
PlanAudit
Gap despite plan
13 (33%)
17 (33%)
Partial coverage
10 (26%)
13 (25%)
Candidate-schema validation
9 (23%)
11 (21%)
SCO: absent plan and test
6 (15%)
7 (13%)
Nondiscriminating data
0
3 (6%)
Same incorrect logic
1 (3%)
1 (2%)
TABLE VI: Failure mechanisms among violated obligations.
University of Science and Technology of China, China · Beijing University of Posts and Telecommunications, China · National University of Singapore, Singapore +1