Do correct answers have stable latent knowledge support?

Chuancheng Shi; Shangze Li; Wenhua Wu; Kun Hu; Fei Shen; Zhiyong Wang; Tat-Seng Chua

cs.CLOct 11, 2026

Do correct answers have stable latent knowledge support?

Authors: Chuancheng Shi, Shangze Li, Wenhua Wu, Kun Hu, Fei Shen, Zhiyong Wang, Tat-Seng Chua

Organizations: University of Sydney Sydney, Australia · Nanjing University of Science and Technology Nanjing, China · Edith Cowan University Perth, Australia · National University of Singapore Singapore

Abstract

Large language models are typically evaluated in terms of the correctness of their output. However, output correctness alone does not confirm whether the latent knowledge supporting an answer has stable internal representations. We therefore propose latent state stability (LSS) and knowledge topology stability (KTS) to investigate the stability of individual knowledge states and their relational structure, respectively. Specifically, we first separately average the model's internal representations of the same knowledge under each elicitation condition. Using the mean representation under the baseline condition as a reference, we remove shared shifts to obtain corrected representations. We then formulate LSS as the distance between each corrected representation and its reference representation, normalized against the typical distance between knowledge items of the same type. We formulate KTS as the cross-condition consistency of the ordered pairwise representation distances among knowledge items of the same type. Next, we construct EpiStableBench, which comprises 4,411 items spanning factual knowledge and mathematical reasoning, with 76,766 controlled queries. Across 16 LLMs from four families, representation stability varies substantially even among items answered correctly under the base condition. Within models, higher stability generally corresponds to higher accuracy under held-out elicitation conditions; across models, stability does not consistently increase with model size or overall accuracy. Meanwhile, we present LatentDPO to learn a knowledge-editing policy that accounts for the stability of latent knowledge. Experimental results show that integrating LatentDPO with multiple existing editing methods consistently improves editing success and generalization across different expressions.

Figures & tables

Figure 1: Motivation For The Correctness–Stability Hypothesis. (a) Human intuition suggests that well-established knowledge should consistently yield correct answers. (b) In LLMs, an initially correct answer may either persist or fail in subsequent elicitation conditions. (c) These observations motivate the hypothesis that model outputs may vary with representational stability.

Figure 2: Overview of EpiStableBench. (a) Canonical questions and four controlled-query types varying paraphrase, format, context, and language for factual knowledge ( orange ) and mathematical reasoning ( blue ). (b) Distribution of 4,411 knowledge items across domains and categories.

Panel of Table 1 — Table 1: Model-Family and Scaling Comparison. ACC, LSS, and KTS are reported across open-weight LLM families and parameter scales for factual knowledge, mathematical reasoning, and their average. The best and second-best results are shown in bold and underlined . More results, please see the appendix.

Figure 3: Behavioral Errors Accompanied By Latent Shifts. Items answered correctly under baseline elicitation condition produce competing answers under changed elicitation conditions, accompanied by representational shifts toward facts associated with those answers. More results please see the appendix.

Figure 4: Stability and Subsequent Correctness. Among targets answered correctly under the base condition, high-LSS targets and high-KTS groups show higher ACC under held-out elicitation conditions than their low-stability counterparts.

Figure 5: Semantic Sensitivity of LSS and KTS. Hard negatives induce larger metric changes than target-preserving perturbations across all three models.

Figure 6: Separating Shared Shifts from Item-Specific Drift. (Left) Decomposition of normalized squared drift into the portion removed by alignment and the remaining residual. (Right) Gains in answer-change prediction across alignment variants, measured in AUROC points.

Setting	LSS		KTS
Setting	Model	Item	Model	Relation
No Norm. / Centering	0.23	0.65	0.99	0.93
Global Norm. / Structure	0.84	0.64	0.77	0.15

Table 2: Relation-Aware Processing. Spearman correlations between ablated and full-method scores at the indicated aggregation levels. Labels before and after the slash denote LSS and KTS variants, respectively.

Setting	LSS		KTS
Setting	Model	Item	Model	Relation
Depth 40%–60%	0.14	0.42	-0.10	0.59
Depth 60%–80%	0.68	0.77	0.53	0.85
Depth 80%–100%	0.96	1.00	0.99	1.00
Final Layer Only	0.60	0.84	0.98	0.95
Raw States, Final 25%	0.47	0.82	0.89	0.80

Table 3: Representation and Layer Selection. Spearman correlations with the full method, which aggregates J-Lens representations over the final 25% of layers. Depth is normalized from input to output.

Leave-one-probe-out
Setting	LSS		KTS
Setting	Model	Item	Model	Relation
Drop Paraphrase	0.98	0.97	0.99	0.97
Drop Format	0.96	0.95	0.97	0.96
Drop Context	0.95	0.95	0.92	0.91
Drop Multilingual	0.95	0.95	0.93	0.90

Table 4: Probe and Reference Sensitivity. Spearman correlations with the full method after omitting an elicitation family or changing the reference condition.

Figure 7: Geometric Patterns of Internal Knowledge Stability. Representative high- and low-LSS items and high- and low-KTS relations are compared across the base and controlled elicitation conditions.

Figure 8: Layer-Specific Activation Patching. Target-answer recovery from same-item canonical states versus different-item controls within the same relation.

Figure 9: Family-Specific Sensitivity to Elicitation. Condition-wise changes in ACC, LSS, and KTS relative to the base condition across four model families.

Method	GPT2-XL			LLaMA 2-7B
Method	Accuracy $\uparrow$	Generality $\uparrow$	Specificity $\uparrow$	Accuracy $\uparrow$	Generality $\uparrow$	Specificity $\uparrow$
ROME	99.40	73.67	80.61	99.90	67.37	93.87
ROME + Ours	99.65 (+0.25)	80.48 (+6.81)	80.27 (-0.34)	99.90 (+0.00)	75.75 (+8.38)	93.33 (-0.54)
MEMIT	72.37	38.89	87.57	76.08	54.23	74.40
MEMIT + Ours	87.14 (+14.77)	54.98 (+16.09)	86.94 (-0.63)	90.79 (+14.71)	69.17 (+14.94)	73.52 (-0.88)
SAKE	94.59	83.96	84.20	96.70	80.73	84.67

Table 5: Application to Knowledge Editing. Accuracy, Generality, and Specificity (%) under identical edit sets and evaluation protocols. Parentheses report percentage-point changes relative to the baseline.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
$m,\ell,i,q$	Model, layer, knowledge item, and query, respectively.
$\mathcal{I},\mathcal{I}_{\mathrm{F}},\mathcal{I}_{\mathrm{M}}$	All knowledge items and the factual-knowledge and mathematical-reasoning subsets, respectively.
$k_{i}=(x_{i},r_{i},y_{i})$	Knowledge item $i$ , where $x_{i}$ is the queried content, $r_{i}$ is its benchmark-defined comparison group, and $y_{i}$ is the target answer.
$\tau_{i}\in\{\mathrm{F},\mathrm{M}\}$	Track containing item $i$ , where $\mathrm{F}$ and $\mathrm{M}$ denote the factual-knowledge and mathematical-reasoning tracks, respectively.
$a,\mathcal{T},\mathcal{T}_{i}$	Reference elicitation condition $a$ , elicitation-condition families $\mathcal{T}$ , and the conditions $\mathcal{T}_{i}\subseteq\mathcal{T}$ available for item $i$ .
$\mathcal{Q}_{i,t}$	Set of target-preserving queries for item $i$ under elicitation condition $t$ .

Appendix

Table 6: Core notation for Latent State Stability (LSS) and Knowledge Topology Stability (KTS).

Predictor	Qwen2.5-3B	Qwen2.5-7B
Strong Output Baseline	0.5993	0.7150
$+$ Within-relation CKA	0.6033	0.7166
$+$ All Standard Features	0.6030	0.7137
$+$ LSS	0.6102	0.7169
$+$ KTS	0.6149	0.7262
$+$ LSS/KTS	0.6154	0.7333

Appendix

Table 7: Comparison With Standard Representation Measures. All representation features augment the same strong output baseline. Values report failure-prediction AUROC.

Ablation	LSS Rank Correlation
Calibration Corpus	$0.964$ – $0.990$
J-Lens Transport Rank	$0.903$ – $0.991$
Raw/PCA Representations	$0.373$ – $0.780$
Question-level Pooling	$0.161$ – $0.561$

Appendix

Table 8: Sensitivity to J-Lens and Representation Choices. LSS rank correlations under changes to the calibration corpus, transport rank, representation space, and token extraction position.

Figure 10: KTS Reliability Across Relation Sizes. Reliability increases with the number of shared items, with both relation-level and condition-pair estimates exceeding the 90% target at 200 items.

Metric	Setting Change	Rank $\rho$	Max. Shift
KTS	Alternative Tie Conventions	1.000	0
	Shared-item Threshold: $100\to 10$	0.985	2
	Item-weighted Relations	0.974	2
LSS	Additive Constant: $0$ – $10^{-3}$	1.000	0
LSS	Item-weighted Relations	0.988	2

Appendix

Table 9: Sensitivity to Numerical and Aggregation Choices. Spearman correlations and maximum rank shifts are measured against the default settings across 16 models in the audited factual cohort. The LSS constant row summarizes all seven tested settings; other rows concern the full method.

Figure 11: More Qualitative Analysis of LSS.

Figure 12: More Qualitative Analysis of KTS.

Panel of Table 10 — Table 10: Extended Model Rankings. ACC, LSS, and KTS are reported for 43 additional model configurations across 11 model families. The best and second-best results in each column are shown in bold and underlined, respectively.

Backbone	Editor	$\alpha_{b,e}$	$\tau_{b,e}$
GPT2-XL	ROME	0.75	5.25
	MEMIT	0.50	5.50
	SAKE	1.00	6.75
LLaMA 2-7B	ROME	0.75	5.25
	MEMIT	0.25	5.55
	SAKE	1.00	6.75

Appendix

Table 11: Frozen Gated-Correction Configurations. Correction strength

\alpha_{b,e}

and prompt-gate threshold

\tau_{b,e}

used to construct action

1

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,425 papers

Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness
Apr 14, 2026Tomer Ashuach, Shai Gretz, Yoav Katz +2Language Model Introspection Language Model Self-Assessment
Technion – Israel Institute of Technology · IBM Research · Kempner Institute, Harvard University
The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models
Apr 7, 2026Xiaojie Gu, Ziying Huang, Weicong Hong +3Knowledge Editing LLM Reliability
Independent Researcher · Cornell Tech · The Ohio State University +1
Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs
Nov 25, 2025Xuyuan Liu, Shengyu Chen, Xinshuai Dong +6Knowledge Editing Continual Learning for LLMs
Dartmouth College · NEC Laboratories America · Carnegie Mellon University

Apr 14, 2026J/K move · Enter open · S save

Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness

Tomer Ashuach, Shai Gretz, Yoav Katz, Yonatan Belinkov+1

Technion – Israel Institute of Technology · IBM Research · Kempner Institute, Harvard University

Language Model Introspection Language Model Self-Assessment LLM Reasoning Factual Knowledge in Language Models Confidence Estimation in Language Models

Open paper AskPDF arXiv

Humans use introspection to evaluate their understanding through private internal states inaccessible to external observers. We investigate whether large language models possess similar privileged knowledge about answer correctness, information unavailable through external observation. We train correctness classifiers on question representations from both a model's own hidden states and external models, testing whether self-representations provide a performance advantage. On standard evaluation, we find no advantage: self-probes perform comparably to peer-model probes. We hypothesize this is due to high inter-model agreement of answer correctness. To isolate genuine privileged knowledge, we evaluate on disagreement subsets, where models produce conflicting predictions. Here, we discover domain-specific privileged knowledge: self-representations consistently outperform peer representations in factual knowledge tasks, but show no advantage in math reasoning. We further localize this domain asymmetry across model layers, finding that the factual advantage emerges progressively from early-to-mid layers onward, consistent with model-specific memory retrieval, while math reasoning shows no consistent advantage at any depth.

Abstract

LLM Evaluation

Explore similar work

Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness

The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models

Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs