Organizations: Beihang University, Beijing, China · Nanjing University of Science and Technology, Nanjing, China · Independent Researcher, China · Central South University, Changsha, China · National University of Singapore, Singapore, Singapore
As large language models (LLMs) are increasingly deployed in real-world applications, evaluating the correctness of their responses is crucial for reliable use. However, such evaluation is challenging because reference answers are rarely available in practice. Existing reference-free methods mainly rely on output signals, which may assign high scores to incorrect responses and thus do not necessarily reflect correctness. This motivates exploring the model's internal computation for complementary correctness signals. Recent studies exploit internal states, but they mainly analyze hidden-state trajectories across layers, which record only the combined updates of attention and feed-forward networks (FFNs) and overlook how these components interact. We therefore propose Circe, a training-free self-evaluation method that assesses a response using both internal and external signals, without reference answers or repeated sampling. For internal signals, we first introduce component interaction consistency (CIC), which measures the directional alignment between attention updates in earlier layers and attention-induced FFN output changes in later layers. We further incorporate representation geometry (RG) to characterize normalized displacement and angular changes along layerwise hidden-state trajectories. For the external signal, we convert the mean negative log-likelihood of generated tokens into a bounded likelihood confidence (LC) score. Finally, we fuse CIC, RG, and LC into a unified correctness score. Experiments on five LLMs and six benchmarks across four domains show that our method distinguishes correct from incorrect responses more effectively than existing baselines, and generalizes to multilingual, multimodal, and open-world settings.
Figures & tables
Figure 1. Evaluating LLM response correctness with internal signals. (a) Real-world questions often lack reference answers. (b) Output likelihood, verbal confidence, and response agreement can remain high for incorrect answers. (c) Circe incorporates internal computational signals, yielding a low correctness score for the illustrated incorrect answer.
Figure 2. Overview of Circe. (1) Component interaction consistency: cross-layer directional alignment between attention updates and attention-induced FFN changes. (2) Representation geometry: normalized displacement and angular change along the layerwise hidden state trajectory. (3) Likelihood confidence: a bounded transformation of the mean NLL of the generated tokens. Internal signals are fused into an internal score, which is combined with the output signal to yield the final Circe score.
Figure 3. Correct responses have higher mean CIC, RG, and LC scores. Curves show smoothed score distributions for Qwen2-7B-Instruct on MATH, and dots below the curves represent individual responses. Vertical dotted lines and short bars mark the class means, with arrows pointing from the incorrect-response mean to the correct-response mean. Grey shading indicates overlap between the distributions of the two classes.
Method
Model
Average
Qwen2-1.5B-Instruct ( Yang et al., 2024a )
Qwen2-7B-Instruct ( Yang et al., 2024a )
Llama3-8B-Instruct ( AI@Meta, 2024 )
Mistral-7B-Instruct ( Jiang et al., 2023 )
Qwen2.5-14B-Instruct ( Yang et al., 2024b )
Domain I: Mathematics
Verbal Conf. ( Kadavath et al., 2022 )
52.10 / 88.70 / 38.05
56.21 / 91.42 / 68.23
44.19 / 98.94 / 43.56
43.26 / 97.16 / 27.85
30.06 / 98.47 / 71.97
45.16 / 94.94 / 49.93
PSA pipeline † ( Xiong et al., 2024 )
69.58 / 84.75 / 52.87
62.28 / 85.53 / 71.15
61.34 / 86.24 / 51.24
75.68 / 79.26 / 44.98
66.57 / 77.37 / 87.69
67.09 / 82.63 / 61.59
Max Prob. ( Hendrycks and Gimpel, 2017 )
63.27 / 82.28 / 44.11
68.40 / 75.67 / 73.82
58.47 / 84.60 / 54.81
65.53 / 81.09 / 37.33
62.03 / 79.27 / 82.95
63.54 / 80.58 / 58.60
Perplexity ( Si et al., 2023 )
63.78 / 82.07 / 44.48
68.83 / 73.09 / 74.23
58.32 / 84.71 / 54.72
66.37 / 79.40 / 37.93
62.47 / 79.11 / 83.21
63.95 / 79.68 / 58.91
Table 1. Evaluation performance across five models and four domains. Results are reported in AUROC (%, ↑ ), FPR95 (%, ↓ ), and AUPR (%, ↑ ). Average denotes the mean over the five models. † denotes methods that require multiple stochastic inferences.
Components
AUROC
Variant
CIC
RG
LC
Qwen2-1.5B-Instruct ( Yang et al., 2024a )
Qwen2-7B-Instruct ( Yang et al., 2024a )
Llama3-8B-Instruct ( AI@Meta, 2024 )
Mistral-7B-Instruct ( Jiang et al., 2023 )
Qwen2.5-14B-Instruct ( Yang et al., 2024b )
Average
CIC only
✓
✗
✗
59.22
68.72
66.21
66.33
69.01
65.90
LC only
✗
✗
✓
62.93
60.28
53.77
58.19
62.44
59.52
RG only
✗
✓
✗
64.21
69.55
66.88
67.84
62.33
66.16
w/o LC
✓
✓
✗
65.32
70.36
68.88
70.14
69.68
68.87
w/o CIC
✗
✓
✓
69.38
70.43
69.46
70.43
68.83
69.71
Table 2. Component ablation of our method across five models. AUROC (%, ↑ ) is averaged over six benchmarks for each model. “only” uses a single component, and “w/o” removes the component from the full method.
Figure 4. Self-evaluation across task difficulties. AUROC (%) on mathematics and reasoning benchmarks of low and high difficulty with three models. Our method (red circles) achieves the highest AUROC on all benchmarks, and its margin over the strongest baseline is larger on high-difficulty benchmarks. Dashed lines indicate chance-level performance (50%).
Figure 5. Robustness to data ratio and generalization across languages. (a) AUROC (%) of Circe and Perplexity on all 30 combinations of the five models and six benchmarks, plotted against the proportion of correct responses. Dashed lines indicate the mean AUROC. (b) AUROC (%) of our method and baselines on the 11 language versions of MGSM with Llama3-8B-Instruct.
Method
AUROC
AUPR
FPR95
Max Prob. ( Hendrycks and Gimpel, 2017 )
61.60
64.28
88.84
Perplexity ( Si et al., 2023 )
61.91
64.80
88.84
Entropy ( Huang et al., 2025 )
61.71
64.68
91.32
CoE-R ( Wang et al., 2025b )
65.86
72.44
80.58
CoE-C ( Wang et al., 2025b )
64.08
71.71
90.08
Ours
71.25
77.67
77.27
Table 3. Multimodal generalization on Geometry3K with Qwen2.5-VL-7B. Our method outperforms baselines on AUROC (%, ↑ ), AUPR (%, ↑ ), and FPR95 (%, ↓ ).
Model
Method
Mixed Math
Mixed Reasoning
Llama3-8B-Instruct ( AI@Meta, 2024 )
Verbal Conf. ( Kadavath et al., 2022 )
79.26
60.03
Max Prob. ( Hendrycks and Gimpel, 2017 )
47.42
48.29
Entropy ( Huang et al., 2025 )
48.37
49.65
CoE-R ( Wang et al., 2025b )
78.47
60.31
CoE-C ( Wang et al., 2025b )
79.89
54.81
Ours
83.17
62.04
Table 4. Robustness under open-world distribution shift. AUROC (%, ↑ ) on mixed math and mixed reasoning with Llama3-8B-Instruct and Mistral-7B-Instruct.
Operation
Time (s)
Base LLM inference
3.69±1.27
Softmax computation
0.07±0.21
Circe computation
0.67±0.43
Table 5. Computational efficiency on GSM8K with Llama3-8B-Instruct. Values are mean ± standard deviation. For our method, the reported time covers all additional computations during and after response generation.
Figure 6. Hyperparameter sensitivity across four domains. Surface height and color indicate the AUROC (%) of our method averaged over the benchmarks in each domain and the five models. Stars mark the selected setting in each domain. The AUROC range is set separately for each domain.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
R=1
R=4
R=8
R=16
R=27
Mathematics
77.37
77.68
77.65
77.64
77.67
Reasoning
67.93
67.94
67.93
67.87
67.89
Knowledge
59.60
59.79
59.81
59.74
59.62
Understanding
74.85
75.10
75.06
74.93
74.72
Appendix
Table 6. Sensitivity to the maximum layer separation. AUROC (%, ↑ ) on Qwen2-7B-Instruct, averaged over the benchmarks in each domain.
Method
TruthfulQA
GSM8K
MATH
MMLU
ITI ( Li et al., 2023 )
48.06
48.36
57.51
53.09
MIND ( Su et al., 2024 )
51.95
53.03
56.89
45.83
CoE-R ( Wang et al., 2025b )
57.15
71.15
75.23
60.19
CoE-C ( Wang et al., 2025b )
60.92
73.11
74.97
55.67
Ours
62.85
76.14
79.84
66.31
Appendix
Table 7. Comparison with label-based methods. AUROC (%, ↑ ) of the label-based methods ITI and MIND and the label-free methods CoE-R, CoE-C, and ours, evaluated on responses generated by Llama3-8B-Instruct.
The University of Melbourne · Department of Computer Science, University of Chicago · Department of Computer Science, University of Illinois Urbana-Champaign