Large language models increasingly judge code, yet without external feedback such judgments are unreliable, and verifying them by executing tests or by extended reasoning is costly. Instead, execution-free correctness predictors use signals from the model's output probabilities, sampled solutions, or hidden states to identify judgments that warrant further verification. However, these predictors may degrade under distribution shifts when generalizing across projects and languages. We compare six execution-free predictors from four families on Qwen3-Coder-30B and gpt-oss-20b under benchmark shift between LiveCodeBench and BigCodeBench and language shift across Python, C++ and Java. We evaluate two targets: whether the model's own verdict on a program can be trusted, and whether the program itself is correct. Besides established approaches, we adapt two novel causal localization methods to code, which have been shown to be more robust out-of-distribution on natural language tasks. For verdict correctness, causal localization improves robustness to language shift. For Qwen3-Coder a probe on the causally localized subspace stays 0.09 AUROC ahead of output confidence under language shift (0.11 in distribution), significantly outperforming the best non-causal predictor by 0.04 AUROC on average. This advantage, however, does not generalize consistently to gpt-oss or to benchmark shift. For program correctness, predictors transfer more successfully between benchmarks, largely by identifying problems on which the model tends to fail. However, their ability to rank candidates within the same problem is limited.
Figures & tables
Fig. 1: (a) The judging pass: the model reads the problem, the candidate and the question and answers Yes or No at the last prompt token. The confidence baseline reads answer probabilities, the probe the hidden state hℓ of one layer, the value probe its projection onto the learned subspace. (b) An interchange intervention: the subspace coordinates of a source example’s hidden state replace those of a base example, and the base’s forward pass continues. DAS learns a subspace so that the base then answers like the source. Counterfactual simulation scores a candidate by making it the source for B=16 fixed bases from the training set. A source state that firmly encodes its verdict should enforce it on the bases, one on which the model is unsure should not. The more consistently the bases adopt the candidate’s verdict, the more likely it is correct.
Benchmark
Problems
Cand.
Pass
Yes
Judge acc.
Qwen3-Coder-30B
LiveCodeBench
208
1646
55.3
58.3
74.4
BigCodeBench
456
3240
40.2
71.7
58.3
LCB-Python
178
1423
41.7
44.1
73.8
LCB-C++
175
1448
24.0
32.5
63.8
LCB-Java
179
1486
37.5
31.2
64.4
TABLE I: Evaluation sets of the first partition. Problems : number of distinct problems; Cand. : number of scored candidates; Pass : percentage of candidates that pass the tests; Yes : percentage of candidates accepted by the model; Judge acc. : balanced accuracy of the model’s verdict against program correctness (mean of the acceptance rate on passing and the rejection rate on failing candidates).
Benchmark shift
Language shift
Method
LCB
→ BCB
BCB
→ LCB
ID
Shift
Δ
Qwen3-Coder-30B
Confidence
.672
.552
.555
.669
.644
.643
− .001
Probe
.699
.480
.639
.516
.761
.668
− .093
Counterf. sim.
.692
.453
.560
.680
.759
.728
− .031
Value probe
.740
.485
.579
.695
.758
.728
− .031
TABLE II: AUROC for verdict correctness (mean over 3 partitions). LCB , BCB : trained and evaluated on that benchmark; → : evaluated on the other one. Language shift: ID is the mean over the three training languages, Shift the mean over the six OOD train → evaluation pairs (Table III ), Δ their difference. Best per column and LLM in bold.
Trained on Py
Trained on C++
Trained on Java
Model
Method
ID
→ C++
→ Java
ID
→ Py
→ Java
ID
→ Py
→ C++
Qwen3-Coder-30B
Confidence
.608 ± .028
.637 ± .052
.687 ± .011
.637 ± .054
.611 ± .026
.688 ± .013
.689 ± .010
.605 ± .027
.630 ± .053
Probe
.670 ± .011
.628 ± .103
.636 ± .059
.874 ± .010
.636 ± .062
.701 ± .009
.740 ± .044
.670 ± .026
.739 ± .020
Counterf. sim.
.671 ± .071
.679 ± .129
.721 ± .041
.854 ± .021
.714 ± .024
.746 ± .013
.751 ± .019
.729 ± .017
.778 ± .013
Value probe
.717 ± .031
.692 ± .030
.737 ± .004
.814 ± .010
.731 ± .003
.761 ± .010
.744 ± .011
.733 ± .019
.712 ± .026
gpt-oss-20b
Confidence
.542 ± .023
.529 ± .016
.549 ± .050
.528 ± .016
.541 ± .023
.549 ± .050
.555 ± .045
.525 ± .009
.525 ± .016
TABLE III: AUROC for verdict correctness, in distribution (ID) and under language shift per training language (LCB problems; ID cross-fitted, shifted columns at the layer chosen in distribution). Mean ± SD over 3 partitions of the problems. Best per column in bold; underlined: within one SD of the best. Non-causal methods above the rule, causal below.
Fig. 2: AUROC per layer on the verdict target, Qwen3-Coder trained on C++ (fold 0), in distribution (left) and under language shift. Rings mark the layer chosen in distribution; the dashed line is confidence. Under shift to Python, every causal checkpoint lies above the probe’s best layer; on Java, most do.
Fig. 3: Within-problem (hollow) against pooled AUROC (filled), in distribution (faint) and under language shift: means over training languages and partitions. Within-problem AUROC uses the problems with mixed outcomes only (Section V-C ); the length of a line is how much of the pooled AUROC comes from ranking problems.
Fig. 4: Value probe on the learned DAS subspace against value probes on three random 6-dimensional subspaces at the same layer, verdict target, mean over shifted languages (the band is the range over the random subspaces). Shaded: layers where DAS has converged (IIA ≥ 0.9). On Qwen3-Coder the learned subspace is above random at every layer; on gpt-oss it is above random before convergence and near chance once DAS has converged.
Benchmark shift
Language shift
Method
LCB
→ BCB
BCB
→ LCB
ID
Shift
Δ
Qwen3-Coder-30B
Confidence
.815
.650
.648
.815
.759
.755
− .003
Probe
.844
.656
.688
.697
.878
.790
− .088
LAT
.844
.580
.590
.731
.807
.799
− .009
Counterf. sim.
.839
.644
.635
.813
.848
.806
− .042
TABLE IV: AUROC for program correctness (mean over 3 partitions). LCB , BCB : trained and evaluated on that benchmark; → : evaluated on the other one. Language shift: ID is the mean over the three training languages, Shift the mean over the six train → evaluation pairs, Δ their difference. Best per column and model in bold.
ID
Lang. Shift
Readout
Pooled
Within
Pooled
Within
Qwen3-Coder-30B
Judging pass
.886
.553
.779
.540
Generation, last code token
.895
.620
.795
.627
Generation, last prompt token
.891
.514
.755
.531
gpt-oss-20b
TABLE V: The probe on the judging pass against the same probe on the states of the model’s own generation, program target on the language sets. ID : mean AUROC over the three languages, Shift : mean over the six language pairs, Within : AUROC over pairs of candidates of the same problem. The last prompt token has seen only the problem.
Benchmark shift
Language shift
Method
LCB
→ BCB
BCB
→ LCB
ID
Shift
Qwen3-Coder-30B
EigenScore
.504
.658
.658
.504
.528
.528
Confidence
.877 ↑
.670
.671
.872 ↑
.774 ↑
.775 ↑
Probe
.906 ↑
.638
.671
.884 ↑
.911 ↑
.803 ↑
LAT
.902 ↑
.618
.682
.871 ↑
.844 ↑
.821 ↑
TABLE VI: AUROC for whether a problem is solved (at least half of its candidates pass). Single-candidate methods are averaged over the candidates of a problem, at the layer of Table IV ; EigenScore is computed from the sampled candidates and needs no training. Columns as in Table IV . ↑ : significantly above EigenScore (paired problem bootstrap, 95%; for ID and Shift on the mean over languages). All methods on the problems that have an EigenScore. ∗
Code generated by modern language models often reads naturally. Yet, it also often fails to implement what was asked. This should be no surprise, as research shows the models' own confidence signals are poorly calibrated with actual correctness. A promising way to assess correctness looks inside the model: by contrasting the hidden states of correct and incorrect programs, recent work captured an internal signal of code correctness that is able to judge candidate solutions better than the model's token-level or stated confidence, with no test execution. However, this signal was captured under one particular way, leaving open an important question: whether it reflects a robust property of the model or an artifact of that choice. We study this question systematically, varying how the signal is extracted from the model internals. Besides this, we also ask if the signal's quality is limited by the data used to extract it, by constructing program pairs that differ only in the fault that makes them incorrect. Our results show that no single configuration is best, and that isolating the fault does not help.
Francisco Ribeiro, Sohaila Abdulsattar, Renata Gonzalez +2
New York University Abu Dhabi Abu Dhabi, United Arab Emirates
Large language models encode rich information in their hidden states. This work asks whether the correctness of code that Qwen3-4B-Instruct-2507 has not yet generated is already legible in its hidden states, evaluated on a set of 444 tasks from LiveCodeBench. The correctness of the model's first-attempt code is linearly decodable from the hidden state at the final prompt token, captured before any output token is generated, with a leakage-free held-out AUC of 0.881 +/- 0.008 across 50 outer splits. To assess whether this signal is explained by prompt length, each hidden state dimension is residualized with respect to its linear effect. The probe still achieves an AUC of 0.842 +/- 0.010, substantially above a logistic prompt-length baseline of 0.657 +/- 0.014, and none of the nonlinear models tested improves upon it. A companion question about whether self-repair leaves a geometric signature in the model's hidden states could not be answered, because successful repairs following a failed first attempt are too rare in this setting to support the analysis. The contribution is both empirical and methodological, providing evidence that pre-generation hidden states contain a robust signal of eventual code correctness, together with a confound-control diagnostic that quantifies how much of that signal survives adjustment for prompt length.
Large language models (LLMs) have become integral to modern software development, enabling automated code generation at scale. However, validating the correctness of LLM-generated code remains a critical and largely unsolved challenge. Existing approaches either rely on dynamic consensus across multiple code candidates - making them costly and difficult to scale - or on static reasoning that is susceptible to dynamic bugs and order bias. In this paper, we propose TRAILS~ (Targeted Reasoning Agreement via Inputs and Specifications), an approach that grounds LLM reasoning with concrete (input, output) pairs. TRAILS~ first generates diverse test inputs via category partitioning based on the specification, then executes them against the candidate code and prompts LLMs to assess whether the resulting input-output pairs conform to the specification - without ever reasoning over the code itself. Scores are aggregated across inputs, to determines whether the program is likely correct. We evaluate TRAILS~ on two datasets, LiveCodeBench and CoCoClaNeL, across three LLMs (Qwen3Coder-30B, Devstral-Small-24B, and Olmo3.1-Instruct), comparing against HoarePrompt and a Zero-Shot Chain-of-Thought baseline. TRAILS~ improves Matthew Correlation Coefficient by up to 39% relative to Zero-Shot COT and consistently outperforms HoarePrompt. Beyond accuracy, TRAILS~ demonstrates greater stability across seeded runs, reducing sensitivity to LLM non-determinism, and assigns correct labels to a larger set of unique code samples than competing approaches.