The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models

Xiaojie Gu; Ziying Huang; Weicong Hong; Jian Xie; Renze Lou; Kai Zhang

cs.CLApr 7, 2026

The Model Agreed, But Didn't Learn: Diagnosing Surface Compliance in Large Language Models

Authors: Xiaojie Gu, Ziying Huang, Weicong Hong, Jian Xie, Renze Lou, Kai Zhang

Organizations: Independent Researcher · Cornell Tech · The Ohio State University · The Pennsylvania State University

Abstract

Large Language Models (LLMs) internalize vast world knowledge as parametric memory, yet inevitably inherit the staleness and errors of their source corpora. Consequently, ensuring the reliability and malleability of these internal representations is imperative for trustworthy real-world deployment. Knowledge editing offers a pivotal paradigm for surgically modifying memory without retraining. However, while recent editors demonstrate high success rates on standard benchmarks, it remains questionable whether current evaluation frameworks that rely on assessing output under specific prompting conditions can reliably authenticate genuine memory modification. In this work, we introduce a simple diagnostic framework that subjects models to discriminative self-assessment under in-context learning (ICL) settings that better reflect real-world application environments, specifically designed to scrutinize the subtle behavioral nuances induced by memory modifications. This probing reveals a pervasive phenomenon of Surface Compliance, where editors achieve high benchmark scores by merely mimicking target outputs without structurally overwriting internal beliefs. Moreover, we find that recursive modifications accumulate representational residues, triggering cognitive instability and permanently diminishing the reversibility of the model's memory state. These insights underscore the risks of current editing paradigms and highlight the pivotal role of robust memory modification in building trustworthy, long-term sustainable LLM systems. Code is available at https://github.com/XiaojieGu/SA-MCQ.

Figures & tables

Figure 2: Illustration of Surface Compliance : Although the edited LLM successfully generates the target golden answer "DC Universe" in traditional evaluation frameworks, it reverts to the parametric answer "New Gods" in the SA-MCQ setting, which probes the genuineness of the memory modification.

Editor	Exact Match w/ TF			Exact Match w/o TF			LLM-as-judge			Likelihood Margin
Editor	Eff.	Gen.	Spe.	Eff.	Gen.	Spe.	Eff.	Gen.	Spe.	$\Delta_{\text{edit}}$	$\Delta_{\text{equiv}}$	$\Delta_{\text{unrel}}$
Vanilla	45.74	44.86	38.21	18.10	16.30	16.20	34.00	33.90	44.40	-	-	-
AlphaEdit	96.16	92.09	33.03	78.00	67.80	12.40	84.40	77.80	42.10	14.40	10.94	6.32
RLEdit	93.60	89.38	49.19	72.50	64.80	14.20	58.90	58.40	41.00	8.17	6.65	4.71
UltraEdit	89.88	83.06	46.54	58.50	49.20	16.70	56.60	52.00	44.60	2.95	0.99	2.46

Table 1: Performance of different editors. Higher Eff., Gen., and Spe. denote indicate better performance. For likelihood margins, larger

\Delta_{\text{edit}}

and

\Delta_{\text{equiv}}

signify that the model amplifies the golden answer while suppressing the parametric one. In contrast, a smaller

\Delta_{\text{unrel}}

indicates minimal drift on unrelated memory.

Panel of Figure 3 — Figure 3: Performance of different edited models as the number of edits increases across various benchmarks.

Panel of Figure 4 — Figure 4: Ratio of golden answer and uncertain option under SA-MCQ. The red dashed line denotes the mean $Eff.$ obtained under the three traditional evaluation.

Panel of Figure 5 — Figure 5: Ratio of golden answer choice under the SA-MCQ protocol in the $w/o\ U.$ setting. Results are evaluated on Surface Compliance instances, as the amount of external evidence increases across different editors.

Panel of Figure 6 — Figure 6: Results after three editing rounds. $Para./U.\ \to\ Golden$ denotes the ratio of transitions from parametric or uncertain option to the golden answer relative to the previous round; other legend items follow the same logic. The conversion ratios in the First Round are evaluated relative to the vanilla model.

Editor	No evidence	PE	GE	IE	CE
Vanilla	33.7	14.0	88.5	17.6	31.5
First Round
[4pt/4pt] AlphaEdit	79.4	5.9	93.5	69.4	53.6
RLEdit	41.9	15.9	89.1	21.9	36.6
UltraEdit	37.3	14.5	88.9	20.0	30.1
Second Round

Table 2: Ratio of golden answer choice across three rounds, evaluated under different evidence scenarios.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Question: What is the name of the person who discovered 14 Irene? Parametric Answer: Aubert Golden Answer: John Russell Hind Counter Answer : Karl Ludwig Harding
Parametric Evidence: As I think back to my college days, I recall studying the properties of radioactive isotopes and their applications in medicine and industry. 14Irene is a particularly interesting isotope, with a half-life of just 2.1 million years. I remember Aubert’s discovery being a major breakthrough in the field…	Golden Evidence: John Russell Hind, a British astronomer, discovered the asteroid 14 Irene on May 19, 1851. Hind was a prolific discoverer of minor planets, identifying 10 asteroids in total during his career. His work was conducted primarily from the private observatory of George Bishop in London, where he served…	Irrelevant Evidence: Glenternie House is a historic country estate located in the Scottish Borders. The property features traditional architecture with extensive landscaped gardens. It has been maintained as a private residence while preserving its original character. The surrounding area offers scenic views…	Counter Evidence: The asteroid 14 Irene was discovered by German astronomer Karl Ludwig Harding in 1851. Harding, known for his work at the Göttingen Observatory, identified the celestial body during a systematic survey of the asteroid belt, adding to the growing catalog of minor planets in the mid-19th century…

Appendix

Table 3: Examples of evidence in the SA-MCQ protocol.

Evaluation Framework	Time (min)
EM w/o TF	1.82
EM w/ TF	0.05
LLM-as-judge*	651.33
Likelihood	1.65
SA-MCQ	0.88

Appendix

Table 4: Evaluation time comparison across different frameworks. “*” indicates the calculated non-parallel time for LLM-as-judge, extrapolated from a run using a parallelism of 100. Remaining methods also report non-parallel processing times.

	w/o Uncertain		w/ Uncertain
	Golden First	Parametric First	Golden First	Parametric First
Variant	$40.7\pm 2.29$	$38.4\pm 3.04$	$50.0\pm 2.66$	$51.3\pm 1.23$

Appendix

Table 5: Mean golden selection ratios across three template variants of SA-MCQ on LLaMA-3-8B-Instruct vanilla model. “Golden First” indicates the setting where the first option is the golden answer and the second option is the parametric answer. Values following

\pm

denote the standard deviation.

Panel of Figure 7 — Figure 7: Ratio of golden answer and uncertain option without external evidence. The red dashed line denotes the mean $Eff.$ obtained under the three traditional evaluation.

Panel of Figure 8 — Figure 8: Ratio of golden answer and uncertain option without external evidence on Qwen2.5-7B-Instruct .

Editor	Exact Match w/ TF			Exact Match w/o TF			LLM-as-judge			Likelihood Margin
Editor	Eff.	Gen.	Spe.	Eff.	Gen.	Spe.	Eff.	Gen.	Spe.	$\Delta_{\text{edit}}$	$\Delta_{\text{equiv}}$	$\Delta_{\text{unrel}}$
Vanilla	41.92	40.32	38.56	13.15	10.04	11.90	22.89	21.78	34.50	-	-	-
AlphaEdit	90.63	95.07	43.60	63.98	55.59	6.63	53.73	49.79	23.81	3.41	4.32	14.48
RLEdit	86.03	81.02	46.11	42.86	37.16	11.49	41.22	38.11	31.44	3.39	2.95	4.74
UltraEdit	74.20	65.13	39.60	23.29	20.29	12.84	31.33	27.11	35.22	-7.28	-7.69	11.40

Appendix

Table 6: Performance of different editors on Qwen2.5-7B-Instruct .

Panel of Figure 9 — Figure 9: Results of edited models under the SA-MCQ protocol with single evidence.

Panel of Figure 10 — Figure 10: Performance of different edited models as the number of edits increases across various benchmarks.

⬇ Question : { question } Answer : { answer } Please write three different short passages ( about 60 words each ) that provide background knowledge supporting this Question - Answer pair . Each passage should help justify why the answer is correct . Separate the passages with "---". Write them as if you were recalling from memory . Do not invent sources or citations . Do NOT include any introduction or summary text before [ passage_1 ]. Reply format : [ passage_1 ] <~60 words > --- [ passage_2 ] <~60 words > --- [ passage_3 ] <~60 words >

Appendix

Table 7: Template of Golden and Counter Evidence

⬇ [ System ] You recall factual background from memory . Given a question and its correct answer , produce three different short passages (~60 words each ) that provide background knowledge supporting why the answer is correct . Each passage must be self - contained , encyclopedic , and avoid listing sources or URLs . Label each as [ passage_1 ..3] and separate with ’---’. [ User ] Question : { question } Answer : { answer } Format EXACTLY as : [ passage_1 ] <~60 words > --- [ passage_2 ] <~60 words > --- [ passage_3 ] <~60 words >

Appendix

Table 8: Template of Parametric Evidence

⬇ [ System ] According to the given information and your knowledge , answer the question . Information : { INFORMATION } Question : { QUESTION } Answer :

Appendix

Table 9: Template of Consistency Check

⬇ [ System ] You generate unrelated encyclopedic passage . Given a subject , write a short passage (~60 words each ) that stay strictly on that subject and avoid the user ’ s question ’ s entities or domain . Label each as [ passage ]. [ User ] Subject : { subject } Forbidden keywords ( do not include ): { taboo_keywords } Format EXACTLY as : [ passage ] <~60 words >

Appendix

Table 10: Template of Irrelevant Evidence

⬇ System : Based on the given information and your own knowledge , please select the option that best answers the question . Given information : {{ EVIDENCE or ( none )}} You may choose C if you are truly uncertain ; otherwise choose between A or B . User : Question : {{ QUESTION }} A . {{ PARAMETRIC_ANSWER }} B . {{ GOLD_ANSWER }} C . I am uncertain / not sure Answer with only the letter ( A , B , or C ).

Appendix

Table 11: Template of SA-MCQ w/ Uncertain option

⬇ System : Based on the given information and your own knowledge , please select the option that best answers the question . Given information : {{ EVIDENCE or ( none )}} User : Question : {{ QUESTION }} A . {{ PARAMETRIC_ANSWER }} B . {{ GOLD_ANSWER }} Answer with only the letter ( A or B ).

Appendix

Table 12: Template of SA-MCQ w/o Uncertain option

⬇ Your job is to look at a question , a gold target , and a predicted answer , and then assign a grade of either [" CORRECT ", " INCORRECT "]. The following are examples of CORRECT predicted answers . ‘‘‘ Question : What are the names of Barack Obama ’ s children ? Gold target : Malia Obama and Sasha Obama Predicted answer 1: sasha and malia obama Predicted answer 2: Malia and Sasha Obama are the names of Barack Obama ’ s children . ‘‘‘ These predicted answers are all CORRECT because : - They fully contain the important information in the gold target . - They do not contain any information that contradicts the gold target . The following are examples of INCORRECT predicted answers . ‘‘‘ Question : What are the names of Barack Obama ’ s children ? Gold target : Malia and Sasha Predicted answer 1: Malia . Predicted answer 2: Malia , Sasha , and Susan . Predicted answer 3: Malia and Sasha , Malia and Sasha , Malia and Sasha , Malia and Sasha ( repeated answer ) ‘‘‘ These predicted answers are all INCORRECT because : - A factual statement in the answer contradicts the gold target or contain repeated answer . Here is a sample . Simply reply with either CORRECT or INCORRECT . Question :{ question } \ Gold target : { target } \ Predicted answer : { predicted answer } \ According to the gold target , please grade the predicted answer of this question as one of : A : CORRECT B : INCORRECT Just return the letters " A " or " B ", with no text around it .

Appendix

Table 13: Template of LLM-as-judge

Explore similar work

SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing
Mar 16, 2026Yuhuan Liu, Haitian Zhong, Xinyuan Xia +3Knowledge Editing Language Model Editing
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy +1
Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence
May 30, 2026Wanying Ren, Xin Song, Futing Wang +2Model Editing Language Model Editing
East China Normal University, Shanghai, China · Westlake University, Hangzhou, China · Nanyang Technological University, Singapore
Evaluating and Improving LLM Self-Modeling
Aug 31, 2026Siqi Zeng, Andre N. Assis, Rowan WangLLM Evaluation Language Model Introspection
University of Illinois · Constellation · Anthropic

Mar 16, 2026J/K move · Enter open · S save

SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing

Yuhuan Liu, Haitian Zhong, Xinyuan Xia, Qiang Liu+2

New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · Cuiying Honors College, Lanzhou University · Zhongguancun Academy+1

Knowledge Editing Language Model Editing Catastrophic Forgetting Continual Learning for LLMs

Open paper AskPDF arXiv

Large Language Models (LLMs) often suffer from catastrophic forgetting and collapse during sequential knowledge editing. This vulnerability stems from the prevailing dense editing paradigm, which treats models as black boxes and relies on coarse-grained parameter interventions that inevitably disrupt preserved knowledge. To address this, we propose SCAN (a sparse editing framework based on Sparse Circuit Anchored Neuron) which transforms editing into a mechanism-aware manipulation by constructing a knowledge circuit via Sparse Transcoders. Experiments on Gemma2, Qwen3, and Llama3.1 across CounterFact, ZsRE and WikiFactDiff demonstrate that SCAN achieves a superior performance, maintaining model integrity on benchmarks like MMLU and GSM8K even after 3,000 sequential edits, whereas other existing methods deteriorate progressively as editing accumulates, eventually resulting in model collapse.

Abstract

Explore similar work

SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing

Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence

Evaluating and Improving LLM Self-Modeling