cs.CLApr 14, 2026

Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness

Authors: Tomer Ashuach, Shai Gretz, Yoav Katz, Yonatan Belinkov, Liat Ein-Dor

Organizations: Technion – Israel Institute of Technology · IBM Research · Kempner Institute, Harvard University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Humans use introspection to evaluate their understanding through private internal states inaccessible to external observers. We investigate whether large language models possess similar privileged knowledge about answer correctness, information unavailable through external observation. We train correctness classifiers on question representations from both a model's own hidden states and external models, testing whether self-representations provide a performance advantage. On standard evaluation, we find no advantage: self-probes perform comparably to peer-model probes. We hypothesize this is due to high inter-model agreement of answer correctness. To isolate genuine privileged knowledge, we evaluate on disagreement subsets, where models produce conflicting predictions. Here, we discover domain-specific privileged knowledge: self-representations consistently outperform peer representations in factual knowledge tasks, but show no advantage in math reasoning. We further localize this domain asymmetry across model layers, finding that the factual advantage emerges progressively from early-to-mid layers onward, consistent with model-specific memory retrieval, while math reasoning shows no consistent advantage at any depth.

Figures & tables

Figure 1: Overview of the experimental framework. Questions are input to a target model and to external models, yielding representations htarget\mathbf{h}_{\text{target}} and hext\mathbf{h}_{\text{ext}} . Probes trained on these representations predict answer correctness. We evaluate probe performance using mean AUC averaged over layers and define the premium gap as the performance advantage of self over external probes.
Figure 2: Premium Gap. Mean AUC for correctness prediction, averaged over layers, on two task types: factual knowledge (TriviaQA) and mathematical reasoning (MATH). Bars compare Random, Embedding, and Best Cross-Model baselines to the Self-Probe ( Self ) across three target models. Semi-transparent overlays indicate the performance gain (or lack thereof) of Self relative to each baseline. Error bars denote 95% confidence intervals.
Figure 3: Target Model Correctness Prediction. Heatmap of correctness prediction differences across target models, datasets, and test subsets. Each cell reports the AUC difference ( ΔAUC=Self−Best External\Delta\mathrm{AUC}=\mathrm{Self}-\mathrm{Best\ External} ), with the percentage of the gap closed shown in parentheses, computed as Self−Best External1−Best External×100\frac{\mathrm{Self}-\mathrm{Best\ External}}{1-\mathrm{Best\ External}}\times 100 . The y-axis lists target models, and cell colors indicate the best-performing source model for each setting. Asterisks (*) denote statistically significant differences (paired tt -test, p<0.05p<0.05 , Bonferroni–Holm correction).
Figure 4: Agreement vs. Disagreement Rates. Stacked bar chart showing the proportion of questions on which models agree (blue) or disagree (orange) on correctness across datasets, averaged over all model pairs.
Figure 5: Per-Layer Premium Gap. Premium gap (self-probe AUC −- best external-probe AUC) on the disagreement subset as a function of normalized layer depth. Shaded regions denote 95% confidence intervals. (a) For factual tasks, the gap is near zero in early layers and grows progressively toward deeper layers across all models, indicating that privileged knowledge emerges in early-to-mid representations. (b) For mathematical reasoning, MATH fluctuates near zero at all depths and GSM1K is predominantly negative (external probes outperform self-probes). No layer exhibits a consistent self-probe advantage across the full network depth.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Table 1: Correctness prediction results with Qwen-3-32B included. Each cell reports Δ\Delta AUC (Self −- Best Cross) with gap closed (%) in parentheses. Bold indicates self-probe advantage (positive Δ\Delta AUC). ⋆ : p<0.05p<0.05 , Holm–Bonferroni corrected. Results for Gemma-2-9B, Llama-3.1-8B, and Qwen-2.5-7B differ slightly from Figure 3 as Qwen-3-32B representations are included as an additional external probe candidate, strengthening the best cross-model baseline.
  Subset Size ( NdisagreementN_{\text{disagreement}} )
DatasetTotalG ↔\leftrightarrow LG ↔\leftrightarrow QL ↔\leftrightarrow Q
Mathematical Reasoning
GSM1K1k186142216
MATH10k2,9322,9672,519
Factual Knowledge
HotpotQA10k1,5921,7301,802
Appendix
Table 2: Dataset statistics and disagreement subset sizes. Total denotes the full test set size. The subsequent columns show the size of the disagreement subset for each unique model pair. (G=Gemma-2-9B, L=Llama-3.1-8B, Q=Qwen2.5-7B).
Figure 6: Premium Gap (Remaining Datasets). Mean AUC for correctness prediction, averaged over layers, on two task types: factual knowledge (Mintaka, HotPotQA) and mathematical reasoning (GSM1K). Bars compare Random, Embedding, and Best Cross-Model baselines to the Self-Probe ( Self ) across three target models. Semi-transparent overlays indicate the performance gain (or lack thereof) of Self relative to each baseline. Error bars denote 95% confidence intervals.
Figure 7: Target Model Correctness Prediction (MLP Probes). Heatmap of correctness prediction differences across target models, datasets, and test subsets. Each cell reports the AUC difference ( ΔAUC=Self−Best External\Delta\mathrm{AUC}=\mathrm{Self}-\mathrm{Best\ External} ), with the percentage of the gap closed shown in parentheses, computed as Self−Best External1−Best External×100\frac{\mathrm{Self}-\mathrm{Best\ External}}{1-\mathrm{Best\ External}}\times 100 . The y-axis lists target models, and cell colors indicate the best-performing external source model for each setting. Asterisks (*) denote statistically significant differences (paired tt -test, p<0.05p<0.05 , Bonferroni–Holm correction).
Figure 8: Disagreement Gap: Factual Knowledge Breakdown (Linear Probes). Detailed performance on the disagreement subset across Mintaka, TriviaQA, and HotPotQA. Self-probes consistently outperform external probes across all factual datasets, reinforcing the existence of privileged knowledge in factual tasks.
Figure 9: Disagreement Gap: Mathematical Reasoning Breakdown (Linear Probes). Detailed performance on the disagreement subset across GSM1K and MATH. Unlike factual tasks, mathematical correctness shows no consistent premium gap, indicating that reasoning difficulty is a public feature accessible to external models.
Figure 10: Disagreement Gap: Factual Knowledge Breakdown (MLP Probes). Detailed disagreement subset performance using MLP probes. The premium gap is even more pronounced with non-linear probes, with Self-representations outperforming Best External probes in 9 out of 9 configurations.
Figure 11: Disagreement Gap: Mathematical Reasoning Breakdown (MLP Probes). Detailed disagreement subset performance using MLP probes across GSM1K and MATH. Consistent with linear results, increased probe expressivity does not uncover hidden privileged info in math tasks; external models remain effective predictors.
Figure 12: Premium Gap (MLP Probes). Mean AUC for correctness prediction, averaged over layers. Bars compare Random, Embedding, and Best Cross-Model baselines to the Self-Probe ( Self ) across three target models. Semi-transparent overlays indicate the performance gain (or lack thereof) of Self relative to each baseline. Error bars denote 95% confidence intervals. (a) Core comparison on TriviaQA and MATH, mirroring the logistic-regression analysis in Figure 2 . (b) Extended results on the remaining datasets.
Figure 13: Per-Layer AUC: Gemma-2-9B. Absolute AUC at each probed layer on the disagreement subset. Lighter bars: best external probe; darker bars: self-probe. (a) For factual datasets, the self-probe advantage grows visibly from mid-layers onward, particularly in TriviaQA. (b) For mathematical reasoning, bars are of similar or reversed height, consistent with the absence of a premium gap. Error bars denote 95% confidence intervals from cross-validation fold scores.
Figure 14: Per-Layer AUC: Llama-3.1-8B. Absolute AUC at each probed layer on the disagreement subset. Lighter bars: best external probe; darker bars: self-probe. (a) For factual datasets, the self-probe advantage emerges in early-to-mid layers. (b) For mathematical reasoning, bars are of similar or reversed height, consistent with the absence of a premium gap. Error bars denote 95% confidence intervals from cross-validation fold scores.
Figure 15: Per-Layer AUC: Qwen-2.5-7B. Absolute AUC at each probed layer on the disagreement subset. Lighter bars: best external probe; darker bars: self-probe. (a) For factual datasets, the self-probe advantage emerges in early-to-mid layers. (b) For mathematical reasoning, bars are of similar or reversed height, consistent with the absence of a premium gap. Error bars denote 95% confidence intervals from cross-validation fold scores.
Figure 16: Lexical-Only vs. Original Question. Mean AUC for correctness prediction, averaged over layers, of probes trained on the Original Question versus the Lexical-Only input (named entities and nouns only), aggregated across all models ( Gemma-2-9B , Llama-3.1-8B , Qwen-2.5-7B ). The gap between conditions reflects the contribution of syntactic and contextual processing beyond entity identity. Error bars denote 95% confidence intervals from cross-validation fold scores.
More in this topic

Language Model Introspection

Language model introspection studies whether models can detect, interpret, and report their own internal states and behavior.

See all 36 papers

Explore similar work

CardsList
May 29, 2026

What Am I Missing? Question-Answering as Hidden State Probing

Test-time reasoning has become a significant field of study since the introduction of chain-of-thought reasoning in large language models (LLMs). However, the mechanisms of this reasoning process are still under-explored -- from the same input prompt, and even the same partial solution, LLMs can produce varied answers if sampled multiple times. We propose to leverage question-asking as an inference-time intervention that articulates information about the model's hidden state. To achieve that, we present a student-teacher setting where a student asks questions to a teacher. We train a probe on the student's hidden state before and after asking a question and find it is predictive of the trajectory's final correctness, even before generating the teacher's answer. This suggests there is a meaningful signal from the self-diagnosis that occurs during question generation rather than information transfer from the teacher. We then frame question-asking as a sequential decision problem, using this probe as a quality score, and define a gating policy to ask questions that maximize likelihood of correctness. We find that the success of question-asking as an intervention is largely dependent on the model's self-consistency. Our empirical results show a gap between detection and recovery; while our gating policy captures model correctness and uncertainty, interventions are equally likely to harm correct trajectories as they are to recover incorrect ones. This gap between diagnosis and correction has broader implications on language models' capacity for self-refinement under uncertainty.

Chu Fei Luo, Samuel Dahan, Xiaodan Zhu

Department of Electrical and Computer Engineering & Ingenuity Labs, Queen’s University · Conflict Analytics Lab, Queen’s University · Cornell Law School +1

LLM Self-RefinementLanguage Model Self-AssessmentLLM Uncertainty EstimationMetacognition in Language ModelsLanguage Model ProbingLLM Reasoning
AskPDF
Feb 26, 2026

Probing for Knowledge Attribution in Large Language Models

Large language model (LLM) hallucinations, meaning fluent but factually incorrect generations, fall into two types: faithfulness violations, where the model misuses provided context, and factuality violations, where answers reflect errors in internal knowledge. Proper mitigation depends on knowing which source drives each answer. We study contributive attribution, i.e. the classification of the dominant knowledge source behind each output, and show that a simple linear probe trained on hidden representations can reliably identify it. We introduce AttriWiki, a self-supervised pipeline that automatically generates labelled training data by prompting models to recall withheld entities from memory or read them from context without relying on knowledge conflicts. Probes trained on AttriWiki achieve up to 0.96 Macro-F1F_1 on Llama-3.1-8B, Mistral-7B, and Qwen-7B, transfer to SQuAD and WebQuestions with 0.94-0.99 Macro-F1F_1, and generalise zero-shot to Tighidet et al. (2024)'s benchmark, outperforming their probe on conflicting settings without retraining. Furthermore, attribution mismatches raise error rates by up to 70%, though correct attribution does not guarantee correct answers, pointing to the need for broader detection frameworks.

Ivo Brink, Alexander Boer, Dennis Ulmer

KPMG NL / University of Amsterdam Amsterdam, The Netherlands · KPMG NL Amsterdam, The Netherlands · University of Amsterdam Amsterdam, The Netherlands

LLM InterpretabilityHallucination in Language ModelsLLM ReliabilityFactual Knowledge in Language ModelsLanguage Model Probing
AskPDF
Sep 29, 2026

Locating Answer-Correctness Signals in Frozen Large Language Models

Language models expose internal signals that predict whether an answer is correct, readable from a single forward pass of a frozen model without additional generations. Yet existing probes often commit to one signal family or layer and can be brittle under distribution shift; in retrieval-augmented settings, many specialized detectors instead target passage faithfulness, which can diverge from correctness when retrieved evidence is unhelpful or conflicting. We therefore ask where answer correctness is readable, which internal signal families carry it, and how they should be combined. We search over hidden states, token probabilities, residual-stream features, attention, and their fusion, treating the selected readouts as a predictive measurement rather than a mechanistic localization. We run this analysis separately in closed-book and with-context settings, since context can change which readouts are informative. A consistent anatomy emerges: correctness concentrates in the answer span, recovered from the answer tokens even under retrieval, and the families carry it complementarily, so fusing them helps most out of distribution, where a single signal is weakest. The protocol is effective across two backbones and gates a retrieval controller as one downstream use.

Yuansen Liu, Yixuan Tang, Anthony Kum Hoe Tung

School of Computing, National University of Singapore

LLM Answer VerificationLanguage Model Error DetectionLanguage Model Probing
AskPDF