Large language models increasingly judge code, yet without external feedback such judgments are unreliable, and verifying them by executing tests or by extended reasoning is costly. Instead, execution-free correctness predictors use signals from the model's output probabilities, sampled solutions, or hidden states to identify judgments that warrant further verification. However, these predictors may degrade under distribution shifts when generalizing across projects and languages. We compare six execution-free predictors from four families on Qwen3-Coder-30B and gpt-oss-20b under benchmark shift between LiveCodeBench and BigCodeBench and language shift across Python, C++ and Java. We evaluate two targets: whether the model's own verdict on a program can be trusted, and whether the program itself is correct. Besides established approaches, we adapt two novel causal localization methods to code, which have been shown to be more robust out-of-distribution on natural language tasks. For verdict correctness, causal localization improves robustness to language shift. For Qwen3-Coder a probe on the causally localized subspace stays 0.09 AUROC ahead of output confidence under language shift (0.11 in distribution), significantly outperforming the best non-causal predictor by 0.04 AUROC on average. This advantage, however, does not generalize consistently to gpt-oss or to benchmark shift. For program correctness, predictors transfer more successfully between benchmarks, largely by identifying problems on which the model tends to fail. However, their ability to rank candidates within the same problem is limited.
Figures & tables
Fig. 1: (a) The judging pass: the model reads the problem, the candidate and the question and answers Yes or No at the last prompt token. The confidence baseline reads answer probabilities, the probe the hidden state hℓ of one layer, the value probe its projection onto the learned subspace. (b) An interchange intervention: the subspace coordinates of a source example’s hidden state replace those of a base example, and the base’s forward pass continues. DAS learns a subspace so that the base then answers like the source. Counterfactual simulation scores a candidate by making it the source for B=16 fixed bases from the training set. A source state that firmly encodes its verdict should enforce it on the bases, one on which the model is unsure should not. The more consistently the bases adopt the candidate’s verdict, the more likely it is correct.
Benchmark
Problems
Cand.
Pass
Yes
Judge acc.
Qwen3-Coder-30B
LiveCodeBench
208
1646
55.3
58.3
74.4
BigCodeBench
456
3240
40.2
71.7
58.3
LCB-Python
178
1423
41.7
44.1
73.8
LCB-C++
175
1448
24.0
32.5
63.8
LCB-Java
179
1486
37.5
31.2
64.4
TABLE I: Evaluation sets of the first partition. Problems : number of distinct problems; Cand. : number of scored candidates; Pass : percentage of candidates that pass the tests; Yes : percentage of candidates accepted by the model; Judge acc. : balanced accuracy of the model’s verdict against program correctness (mean of the acceptance rate on passing and the rejection rate on failing candidates).
Benchmark shift
Language shift
Method
LCB
→ BCB
BCB
→ LCB
ID
Shift
Δ
Qwen3-Coder-30B
Confidence
.672
.552
.555
.669
.644
.643
− .001
Probe
.699
.480
.639
.516
.761
.668
− .093
Counterf. sim.
.692
.453
.560
.680
.759
.728
− .031
Value probe
.740
.485
.579
.695
.758
.728
− .031
TABLE II: AUROC for verdict correctness (mean over 3 partitions). LCB , BCB : trained and evaluated on that benchmark; → : evaluated on the other one. Language shift: ID is the mean over the three training languages, Shift the mean over the six OOD train → evaluation pairs (Table III ), Δ their difference. Best per column and LLM in bold.
Trained on Py
Trained on C++
Trained on Java
Model
Method
ID
→ C++
→ Java
ID
→ Py
→ Java
ID
→ Py
→ C++
Qwen3-Coder-30B
Confidence
.608 ± .028
.637 ± .052
.687 ± .011
.637 ± .054
.611 ± .026
.688 ± .013
.689 ± .010
.605 ± .027
.630 ± .053
Probe
.670 ± .011
.628 ± .103
.636 ± .059
.874 ± .010
.636 ± .062
.701 ± .009
.740 ± .044
.670 ± .026
.739 ± .020
Counterf. sim.
.671 ± .071
.679 ± .129
.721 ± .041
.854 ± .021
.714 ± .024
.746 ± .013
.751 ± .019
.729 ± .017
.778 ± .013
Value probe
.717 ± .031
.692 ± .030
.737 ± .004
.814 ± .010
.731 ± .003
.761 ± .010
.744 ± .011
.733 ± .019
.712 ± .026
gpt-oss-20b
Confidence
.542 ± .023
.529 ± .016
.549 ± .050
.528 ± .016
.541 ± .023
.549 ± .050
.555 ± .045
.525 ± .009
.525 ± .016
TABLE III: AUROC for verdict correctness, in distribution (ID) and under language shift per training language (LCB problems; ID cross-fitted, shifted columns at the layer chosen in distribution). Mean ± SD over 3 partitions of the problems. Best per column in bold; underlined: within one SD of the best. Non-causal methods above the rule, causal below.
Fig. 2: AUROC per layer on the verdict target, Qwen3-Coder trained on C++ (fold 0), in distribution (left) and under language shift. Rings mark the layer chosen in distribution; the dashed line is confidence. Under shift to Python, every causal checkpoint lies above the probe’s best layer; on Java, most do.
Fig. 3: Within-problem (hollow) against pooled AUROC (filled), in distribution (faint) and under language shift: means over training languages and partitions. Within-problem AUROC uses the problems with mixed outcomes only (Section V-C ); the length of a line is how much of the pooled AUROC comes from ranking problems.
Fig. 4: Value probe on the learned DAS subspace against value probes on three random 6-dimensional subspaces at the same layer, verdict target, mean over shifted languages (the band is the range over the random subspaces). Shaded: layers where DAS has converged (IIA ≥ 0.9). On Qwen3-Coder the learned subspace is above random at every layer; on gpt-oss it is above random before convergence and near chance once DAS has converged.
Benchmark shift
Language shift
Method
LCB
→ BCB
BCB
→ LCB
ID
Shift
Δ
Qwen3-Coder-30B
Confidence
.815
.650
.648
.815
.759
.755
− .003
Probe
.844
.656
.688
.697
.878
.790
− .088
LAT
.844
.580
.590
.731
.807
.799
− .009
Counterf. sim.
.839
.644
.635
.813
.848
.806
− .042
TABLE IV: AUROC for program correctness (mean over 3 partitions). LCB , BCB : trained and evaluated on that benchmark; → : evaluated on the other one. Language shift: ID is the mean over the three training languages, Shift the mean over the six train → evaluation pairs, Δ their difference. Best per column and model in bold.
ID
Lang. Shift
Readout
Pooled
Within
Pooled
Within
Qwen3-Coder-30B
Judging pass
.886
.553
.779
.540
Generation, last code token
.895
.620
.795
.627
Generation, last prompt token
.891
.514
.755
.531
gpt-oss-20b
TABLE V: The probe on the judging pass against the same probe on the states of the model’s own generation, program target on the language sets. ID : mean AUROC over the three languages, Shift : mean over the six language pairs, Within : AUROC over pairs of candidates of the same problem. The last prompt token has seen only the problem.
Benchmark shift
Language shift
Method
LCB
→ BCB
BCB
→ LCB
ID
Shift
Qwen3-Coder-30B
EigenScore
.504
.658
.658
.504
.528
.528
Confidence
.877 ↑
.670
.671
.872 ↑
.774 ↑
.775 ↑
Probe
.906 ↑
.638
.671
.884 ↑
.911 ↑
.803 ↑
LAT
.902 ↑
.618
.682
.871 ↑
.844 ↑
.821 ↑
TABLE VI: AUROC for whether a problem is solved (at least half of its candidates pass). Single-candidate methods are averaged over the candidates of a problem, at the layer of Table IV ; EigenScore is computed from the sampled candidates and needs no training. Columns as in Table IV . ↑ : significantly above EigenScore (paired problem bootstrap, 95%; for ID and Shift on the mean over languages). All methods on the problems that have an EigenScore. ∗