Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness

Tomer Ashuach; Shai Gretz; Yoav Katz; Yonatan Belinkov; Liat Ein-Dor

cs.CLApr 14, 2026

Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness

Authors: Tomer Ashuach, Shai Gretz, Yoav Katz, Yonatan Belinkov, Liat Ein-Dor

Organizations: Technion – Israel Institute of Technology · IBM Research · Kempner Institute, Harvard University

Abstract

Humans use introspection to evaluate their understanding through private internal states inaccessible to external observers. We investigate whether large language models possess similar privileged knowledge about answer correctness, information unavailable through external observation. We train correctness classifiers on question representations from both a model's own hidden states and external models, testing whether self-representations provide a performance advantage. On standard evaluation, we find no advantage: self-probes perform comparably to peer-model probes. We hypothesize this is due to high inter-model agreement of answer correctness. To isolate genuine privileged knowledge, we evaluate on disagreement subsets, where models produce conflicting predictions. Here, we discover domain-specific privileged knowledge: self-representations consistently outperform peer representations in factual knowledge tasks, but show no advantage in math reasoning. We further localize this domain asymmetry across model layers, finding that the factual advantage emerges progressively from early-to-mid layers onward, consistent with model-specific memory retrieval, while math reasoning shows no consistent advantage at any depth.

Figures & tables

Figure 1: Overview of the experimental framework. Questions are input to a target model and to external models, yielding representations

\mathbf{h}_{\text{target}}

and

\mathbf{h}_{\text{ext}}

. Probes trained on these representations predict answer correctness. We evaluate probe performance using mean AUC averaged over layers and define the premium gap as the performance advantage of self over external probes.

Figure 2: Premium Gap. Mean AUC for correctness prediction, averaged over layers, on two task types: factual knowledge (TriviaQA) and mathematical reasoning (MATH). Bars compare Random, Embedding, and Best Cross-Model baselines to the Self-Probe ( Self ) across three target models. Semi-transparent overlays indicate the performance gain (or lack thereof) of Self relative to each baseline. Error bars denote 95% confidence intervals.

Figure 3: Target Model Correctness Prediction. Heatmap of correctness prediction differences across target models, datasets, and test subsets. Each cell reports the AUC difference ( $\Delta\mathrm{AUC}=\mathrm{Self}-\mathrm{Best\ External}$ ), with the percentage of the gap closed shown in parentheses, computed as $\frac{\mathrm{Self}-\mathrm{Best\ External}}{1-\mathrm{Best\ External}}\times 100$ . The y-axis lists target models, and cell colors indicate the best-performing source model for each setting. Asterisks (*) denote statistically significant differences (paired $t$ -test, $p<0.05$ , Bonferroni–Holm correction).

Figure 4: Agreement vs. Disagreement Rates. Stacked bar chart showing the proportion of questions on which models agree (blue) or disagree (orange) on correctness across datasets, averaged over all model pairs.

Panel of Figure 5 — Figure 5: Per-Layer Premium Gap. Premium gap (self-probe AUC $-$ best external-probe AUC) on the disagreement subset as a function of normalized layer depth. Shaded regions denote 95% confidence intervals. (a) For factual tasks, the gap is near zero in early layers and grows progressively toward deeper layers across all models, indicating that privileged knowledge emerges in early-to-mid representations. (b) For mathematical reasoning, MATH fluctuates near zero at all depths and GSM1K is predominantly negative (external probes outperform self-probes). No layer exhibits a consistent self-probe advantage across the full network depth.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Table 1: Correctness prediction results with Qwen-3-32B included. Each cell reports

\Delta

AUC (Self

-

Best Cross) with gap closed (%) in parentheses. Bold indicates self-probe advantage (positive

\Delta

AUC). ⋆ :

p<0.05

, Holm–Bonferroni corrected. Results for Gemma-2-9B, Llama-3.1-8B, and Qwen-2.5-7B differ slightly from Figure 3 as Qwen-3-32B representations are included as an additional external probe candidate, strengthening the best cross-model baseline.

Mathematical Reasoning
		Subset Size ( $N_{\text{disagreement}}$ )
Dataset	Total	G $\leftrightarrow$ L	G $\leftrightarrow$ Q	L $\leftrightarrow$ Q
GSM1K	1k	186	142	216
MATH	10k	2,932	2,967	2,519
Factual Knowledge
HotpotQA	10k	1,592	1,730	1,802

Appendix

Table 2: Dataset statistics and disagreement subset sizes. Total denotes the full test set size. The subsequent columns show the size of the disagreement subset for each unique model pair. (G=Gemma-2-9B, L=Llama-3.1-8B, Q=Qwen2.5-7B).

Figure 6: Premium Gap (Remaining Datasets). Mean AUC for correctness prediction, averaged over layers, on two task types: factual knowledge (Mintaka, HotPotQA) and mathematical reasoning (GSM1K). Bars compare Random, Embedding, and Best Cross-Model baselines to the Self-Probe ( Self ) across three target models. Semi-transparent overlays indicate the performance gain (or lack thereof) of Self relative to each baseline. Error bars denote 95% confidence intervals.

Figure 7: Target Model Correctness Prediction (MLP Probes). Heatmap of correctness prediction differences across target models, datasets, and test subsets. Each cell reports the AUC difference ( $\Delta\mathrm{AUC}=\mathrm{Self}-\mathrm{Best\ External}$ ), with the percentage of the gap closed shown in parentheses, computed as $\frac{\mathrm{Self}-\mathrm{Best\ External}}{1-\mathrm{Best\ External}}\times 100$ . The y-axis lists target models, and cell colors indicate the best-performing external source model for each setting. Asterisks (*) denote statistically significant differences (paired $t$ -test, $p<0.05$ , Bonferroni–Holm correction).

Figure 8: Disagreement Gap: Factual Knowledge Breakdown (Linear Probes). Detailed performance on the disagreement subset across Mintaka, TriviaQA, and HotPotQA. Self-probes consistently outperform external probes across all factual datasets, reinforcing the existence of privileged knowledge in factual tasks.

Figure 9: Disagreement Gap: Mathematical Reasoning Breakdown (Linear Probes). Detailed performance on the disagreement subset across GSM1K and MATH. Unlike factual tasks, mathematical correctness shows no consistent premium gap, indicating that reasoning difficulty is a public feature accessible to external models.

Figure 10: Disagreement Gap: Factual Knowledge Breakdown (MLP Probes). Detailed disagreement subset performance using MLP probes. The premium gap is even more pronounced with non-linear probes, with Self-representations outperforming Best External probes in 9 out of 9 configurations.

Figure 11: Disagreement Gap: Mathematical Reasoning Breakdown (MLP Probes). Detailed disagreement subset performance using MLP probes across GSM1K and MATH. Consistent with linear results, increased probe expressivity does not uncover hidden privileged info in math tasks; external models remain effective predictors.

Panel of Figure 12 — Figure 12: Premium Gap (MLP Probes). Mean AUC for correctness prediction, averaged over layers. Bars compare Random, Embedding, and Best Cross-Model baselines to the Self-Probe ( Self ) across three target models. Semi-transparent overlays indicate the performance gain (or lack thereof) of Self relative to each baseline. Error bars denote 95% confidence intervals. (a) Core comparison on TriviaQA and MATH, mirroring the logistic-regression analysis in Figure 2 . (b) Extended results on the remaining datasets.

Panel of Figure 13 — Figure 13: Per-Layer AUC: Gemma-2-9B. Absolute AUC at each probed layer on the disagreement subset. Lighter bars: best external probe; darker bars: self-probe. (a) For factual datasets, the self-probe advantage grows visibly from mid-layers onward, particularly in TriviaQA. (b) For mathematical reasoning, bars are of similar or reversed height, consistent with the absence of a premium gap. Error bars denote 95% confidence intervals from cross-validation fold scores.

Panel of Figure 14 — Figure 14: Per-Layer AUC: Llama-3.1-8B. Absolute AUC at each probed layer on the disagreement subset. Lighter bars: best external probe; darker bars: self-probe. (a) For factual datasets, the self-probe advantage emerges in early-to-mid layers. (b) For mathematical reasoning, bars are of similar or reversed height, consistent with the absence of a premium gap. Error bars denote 95% confidence intervals from cross-validation fold scores.

Panel of Figure 15 — Figure 15: Per-Layer AUC: Qwen-2.5-7B. Absolute AUC at each probed layer on the disagreement subset. Lighter bars: best external probe; darker bars: self-probe. (a) For factual datasets, the self-probe advantage emerges in early-to-mid layers. (b) For mathematical reasoning, bars are of similar or reversed height, consistent with the absence of a premium gap. Error bars denote 95% confidence intervals from cross-validation fold scores.

Figure 16: Lexical-Only vs. Original Question. Mean AUC for correctness prediction, averaged over layers, of probes trained on the Original Question versus the Lexical-Only input (named entities and nouns only), aggregated across all models ( Gemma-2-9B , Llama-3.1-8B , Qwen-2.5-7B ). The gap between conditions reflects the contribution of syntactic and contextual processing beyond entity identity. Error bars denote 95% confidence intervals from cross-validation fold scores.

Language Model Introspection

Language model introspection studies whether models can detect, interpret, and report their own internal states and behavior.

See all 36 papers

What Am I Missing? Question-Answering as Hidden State Probing
May 29, 2026Chu Fei Luo, Samuel Dahan, Xiaodan ZhuLLM Self-Refinement Language Model Self-Assessment
Department of Electrical and Computer Engineering & Ingenuity Labs, Queen’s University · Conflict Analytics Lab, Queen’s University · Cornell Law School +1
Probing for Knowledge Attribution in Large Language Models
Feb 26, 2026Ivo Brink, Alexander Boer, Dennis UlmerLLM Interpretability Hallucination in Language Models
KPMG NL / University of Amsterdam Amsterdam, The Netherlands · KPMG NL Amsterdam, The Netherlands · University of Amsterdam Amsterdam, The Netherlands
Locating Answer-Correctness Signals in Frozen Large Language Models
Sep 29, 2026Yuansen Liu, Yixuan Tang, Anthony Kum Hoe TungLLM Answer Verification Language Model Error Detection
School of Computing, National University of Singapore

May 29, 2026J/K move · Enter open · S save

What Am I Missing? Question-Answering as Hidden State Probing

Chu Fei Luo, Samuel Dahan, Xiaodan Zhu

Department of Electrical and Computer Engineering & Ingenuity Labs, Queen’s University · Conflict Analytics Lab, Queen’s University · Cornell Law School+1

LLM Self-Refinement Language Model Self-Assessment LLM Uncertainty Estimation Metacognition in Language Models Language Model Probing LLM Reasoning

Open paper AskPDF arXiv

Test-time reasoning has become a significant field of study since the introduction of chain-of-thought reasoning in large language models (LLMs). However, the mechanisms of this reasoning process are still under-explored -- from the same input prompt, and even the same partial solution, LLMs can produce varied answers if sampled multiple times. We propose to leverage question-asking as an inference-time intervention that articulates information about the model's hidden state. To achieve that, we present a student-teacher setting where a student asks questions to a teacher. We train a probe on the student's hidden state before and after asking a question and find it is predictive of the trajectory's final correctness, even before generating the teacher's answer. This suggests there is a meaningful signal from the self-diagnosis that occurs during question generation rather than information transfer from the teacher. We then frame question-asking as a sequential decision problem, using this probe as a quality score, and define a gating policy to ask questions that maximize likelihood of correctness. We find that the success of question-asking as an intervention is largely dependent on the model's self-consistency. Our empirical results show a gap between detection and recovery; while our gating policy captures model correctness and uncertainty, interventions are equally likely to harm correct trajectories as they are to recover incorrect ones. This gap between diagnosis and correction has broader implications on language models' capacity for self-refinement under uncertainty.

Abstract

Language Model Introspection

Explore similar work

What Am I Missing? Question-Answering as Hidden State Probing

Probing for Knowledge Attribution in Large Language Models

Locating Answer-Correctness Signals in Frozen Large Language Models