cs.AINov 5, 2026

Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency

Authors: Shuyi Miao, Zheng Li, Xu Xie, Linrui Xu, Fei Shen, Xiaoyu Du, Tat-Seng Chua

Organizations: Beihang University, Beijing, China · Nanjing University of Science and Technology, Nanjing, China · Independent Researcher, China · Central South University, Changsha, China · 

Abstract

As large language models (LLMs) are increasingly deployed in real-world applications, evaluating the correctness of their responses is crucial for reliable use. However, such evaluation is challenging because reference answers are rarely available in practice. Existing reference-free methods mainly rely on output signals, which may assign high scores to incorrect responses and thus do not necessarily reflect correctness. This motivates exploring the model's internal computation for complementary correctness signals. Recent studies exploit internal states, but they mainly analyze hidden-state trajectories across layers, which record only the combined updates of attention and feed-forward networks (FFNs) and overlook how these components interact. We therefore propose Circe, a training-free self-evaluation method that assesses a response using both internal and external signals, without reference answers or repeated sampling. For internal signals, we first introduce component interaction consistency (CIC), which measures the directional alignment between attention updates in earlier layers and attention-induced FFN output changes in later layers. We further incorporate representation geometry (RG) to characterize normalized displacement and angular changes along layerwise hidden-state trajectories. For the external signal, we convert the mean negative log-likelihood of generated tokens into a bounded likelihood confidence (LC) score. Finally, we fuse CIC, RG, and LC into a unified correctness score. Experiments on five LLMs and six benchmarks across four domains show that our method distinguishes correct from incorrect responses more effectively than existing baselines, and generalizes to multilingual, multimodal, and open-world settings.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Nov 1, 2026

Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation

We introduce a reusable evaluation artifact for diagnosing whether LLM self-reported confidence can support confidence-gated act-or-defer decisions. The artifact is designed for evaluation rather than safety certification: it provides a black-box protocol, prompts, judge-reference labels, model outputs, reproduction scripts, and calibration-constrained diagnostic metrics for auditing LLM self-evaluation across math, factual, and commonsense reasoning tasks. Our central finding is that aggregate uncertainty summaries can produce deployment-misleading rankings: a model that appears strongest under standard aggregate calibration metrics can fail under category-level calibration constraints, while a model with worse aggregate calibration can exhibit stronger calibration-constrained behavior in the positive-margin region where high self-selection accuracy and tight calibration coexist. To expose this failure mode, we define PVC and C-PVC as category-level diagnostics over an accuracy-by-calibration surface: PVC counts categories whose self-evaluation accuracy exceeds a threshold, while C-PVC additionally requires category-level calibration error below a tolerance. We aggregate these diagnostics using Volume Under Surface (VUS) and a positive-margin restriction; they are VC/fat-shattering motivated but are empirical evaluation proxies, not literal VC-dimension estimates. Across 13 models and four benchmarks, the artifact shows weak confidence-correctness separation, stable diagnostic rankings under alternative categorizations and judge-label perturbations, and persistent positive-margin failures not captured by ECE or Brier score. Confidence-gated self-evaluation therefore requires calibration-constrained, category-level diagnostics, and we provide a reusable artifact for applying them under black-box access.
Jun 18, 2026

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes

Can we trust evaluation scores to capture an LLM's true real-world performance? Certifiable evaluation answers this question by providing a confidence interval (CI) that covers the LLM's true performance with high probability (e.g., 95%). In particular, existing methods sequentially select evaluation samples from a fixed pool and keep updating the CI until certain conditions are satisfied, e.g., the CI width reaches a target precision. However, existing methods are not generally anytime-valid: the claimed coverage (e.g., 95%) may fail when CIs are repeatedly updated and used to decide when to stop, leaving a gap between theoretical rigor and practice. This paper bridges this gap by proposing CELEUS, a Certifiable framework for Efficient LLM evaluation, which leverages E-processes to build anytime-valid CIs. Concretely, we propose a new signal that combines two ingredients: (i) Uncertainty-guided sampling to select informative samples for evaluation, and (ii) Surrogate-assisted approximations for unevaluated samples. We prove that the new signal remains unbiased for the evaluation score conditional on the past, enabling statistically-grounded and anytime-valid e-process CIs. More importantly, the two ingredients reduce estimation variance and help reach the target precision with fewer evaluated samples. We also prove that CIs obtained by CELEUS can shrink at a near-parametric rate up to logarithmic factors and analyze the oracle variance-optimal sampling rule that motivates the empirical uncertainty-guided one. Experiments show that CELEUS reaches the target precision using 54-62% fewer evaluated samples than baselines, while preserving anytime-valid coverage. Code is available at https://github.com/zyecs/celeus.
Jun 3, 2026

Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data

Large language models are increasingly evaluated by other models, raising a natural question: can a model predict how a judge will score its own output? We find that the ability is largely present before any targeted training: prompted few-shot, a base model already predicts an external judge's multi-attribute quality scores on open-ended responses well above chance across three benchmarks. We introduce Self-Evaluation Elicitation (SEE), a method that surfaces this latent ability through a short cycle comprising a calibration-coupled reinforcement learning phase that improves the answer and predicts the judge, followed by a masked distillation phase that sharpens the prediction while leaving the answer untouched. From 160 unique examples, roughly 31x fewer than a reinforcement learning baseline, SEE improves held-out calibration across three benchmarks while preserving answer quality. The elicited self-evaluation is sharply localized within the model's own token distribution and stable across judges it was never trained against, indicating a transferable notion of quality rather than a single judge's preference. These results reframe judge-aligned self-evaluation as a problem of elicitation rather than acquisition.