Impossible Code, Barely Changed Score: A Replay Study of Judge-Based Evaluation
Organizations: UNSW Sydney
Abstract
Language models are often used to judge generated code without running it. A useful evaluation should reveal substantial changes in the answers it scores. We examine this requirement by repeating an earlier evaluation on 421 coding problems, comparing Qwen-2.5-7B-Instruct with two versions trained using Group Relative Policy Optimization (GRPO). One receives rewards from execution tests (CE-trained), and the other from a strong language-model judge (SLJ-trained). The base model's syntax-failure rate remains stable, but the trained checkpoints produce invalid Python on 63%-67% of attempts. Despite this change, the strong judge's Pass@32 scores fall by only about 3 percentage points, from 96.67% to 93.95% for SLJ-trained. Within the later run, allowing more attempts makes the base and trained checkpoints look increasingly similar under the judge. With 128 attempts per question, the judge accepts at least one SLJ-trained answer on 96.44% of problems, compared with 28.27% under execution tests. We recommend reporting the share of answers that pass Python's syntax check alongside every judge-based score. This check requires no tests or code execution and flags invalid answers that the judge accepts, although it cannot establish functional correctness. Our findings show how a high judge-based score can hide substantial output damage and motivate checking an evaluation's sensitivity to change. Project page and code: https://curiousorbitnotes.github.io/evorobust-replay-audit/
Figures & tables
| first run | second run | |||
|---|---|---|---|---|
| checkpoint | invalid | HEval / MBPP | invalid | HEval / MBPP |
| Qwen-2.5-7B base | / | / | ||
| CE-trained | / | / | ||
| SLJ-trained | / | / | ||
| checkpoint | % samples parsing | judge Pass@ | judge gate | execution Pass@ |
|---|---|---|---|---|
| Qwen-2.5-7B base | ||||
| CE-trained | ||||
| SLJ-trained |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| parse error | base | CE-trained | SLJ-trained |
|---|---|---|---|
| unmatched closing bracket | |||
| closing bracket does not match opener | |||
| invalid syntax (unclassified) | |||
| positional argument follows keyword | |||
| other classes | |||
| total failing |
| checkpoint | invalid samples | valid samples | positives invalid | |
|---|---|---|---|---|
| rate | rate | |||
| Qwen-2.5-7B base | ||||
| CE-trained | ||||
| SLJ-trained | ||||
| checkpoint | Pass@ | Pass@ | Pass@ |
|---|---|---|---|
| Qwen-2.5-7B base | |||
| CE-trained | |||
| SLJ-trained |
| study’s cached scorer (cached-NEW) | hardened | |||||
|---|---|---|---|---|---|---|
| checkpoint | @ | @ | @ | @ | @ | @ |
| Qwen-2.5-7B base | ||||||
| CE-trained | ||||||
| SLJ-trained | ||||||
| Score (%) | Base checkpoint (pp) | ||||
| Attempts | base | CE-trained | SLJ-trained | CE-trained | SLJ-trained |
| LLM judge (Qwen-2.5-72B-Instruct) | |||||
| 1 | |||||
| 32 | |||||
| 128 | |||||
| judge + syntax gate | |||||
| instrument | |||
|---|---|---|---|
| Base CE-trained | |||
| judge | |||
| judge gate | |||
| execution (hardened) | |||
| Base SLJ-trained | |||
| judge | |||
| setting | replay ( ) | archived ( ) |
|---|---|---|
| sampling | temperature , top- | temperature , top- |
| max new tokens | ||
| max model length | not recorded | |
| tokenizer and chat template | loaded from each checkpoint directory | not recorded |
| stop tokens, EOS handling | vLLM defaults, no custom stop strings | not recorded |
| checkpoint checksums | none found | none found |
| judge Pass@ | samples parsing | |||||
|---|---|---|---|---|---|---|
| checkpoint | first | second | change (pp) | first | second | change (pp) |
| Qwen-2.5-7B base | ||||||
| CE-trained | ||||||
| SLJ-trained | ||||||
| checkpoint | Judge (%) | Judge + gate (%) | Drop (pp) | Interval (pp) |
|---|---|---|---|---|
| base | ||||
| CE-trained | ||||
| SLJ-trained |
| Target share (%) | Modified (%) | Parsing (%) | Judge Pass@ (%) | Judge change (pp) |
|---|---|---|---|---|
| base model | Original prompts (%) | Function name supplied (%) |
|---|---|---|
| Qwen-2.5-7B |
Code Generation Evaluation
Code generation evaluation measures whether AI-produced programs are correct, reliable, secure, efficient, and faithful to their specifications.
See all 227 papers