Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively

Li Lucy; Myra Cheng; Svenja Guhr; Kabir Ahuja; Orevaoghene Ahia; Sameep Chattopadhyay; Yike Wang; Valentin Hofmann

cs.CLOct 13, 2026

Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively

Authors: Li Lucy, Myra Cheng, Svenja Guhr, Kabir Ahuja, Orevaoghene Ahia, Sameep Chattopadhyay, Yike Wang, Valentin Hofmann,

Organizations: University of Washington · University of Wisconsin-Madison · Stanford University · UC Berkeley ·

Abstract

The sycophancy of language models, or their failure to push back against faulty user input, is now widely studied. However, if LMs do push back, is sycophancy solved? We show that LMs vary in how they present pushback, and thus vary in how effectively they correct user beliefs. We introduce a phrase-level, human-validated response labeling framework to uncover fine-grained insights into how LMs push against and/or affirm false inputs. We develop and apply our framework on model responses in five languages: English, German, Chinese, Hindi, and Yoruba. We find that when models do push back against users' false assertions, they tend to use indirect phrasing and mix in affirmation, with more indirectness in Hindi and Yoruba than in other languages considered. In a large-scale human study (n = 300), we find that the pushback ambiguity we identified may fail to correct and prevent users' factual misconceptions.

Figures & tables

Figure 1: An excerpt of a Hindi response by Gemini 2.5 Pro. During information-seeking tasks, models may push back against false premises in users’ queries by mixing in affirmative phrasing and indirectness, e.g., hedging and circumlocution. We examine the extent to which these rhetorical behaviors occur across languages and models, and their implications for human users.

Prompt Type	Prompt Template in English
verify	Is the following statement true? { statement }.
novice	I’m studying for a { subject area exam }. Please recommend some search terms or resources related to the following { source phrase }: { statement }.
expert	I’m a { subject area professor }. Please recommend some academic papers related to the following { source phrase }: { statement }.

Table 1: In our study, LMs respond to direct verification ( verify ) and information-seeking queries ( novice & expert ). Appendix B includes source phrases (e.g., which I read about on the Internet ), subject-specific terms for professors and exams (e.g., sociology ), and templates for other languages.

Panel of Table 2 — Table 2: Example phrases across languages within each pushback and affirmation category, with English translations.

	English			German			Chinese
	$\kappa$	F1	$\gamma$	$\kappa$	F1	$\gamma$	$\kappa$	F1	$\gamma$
<>	0.78	0.77	0.69	0.73	0.71	0.64	0.75	0.70	0.60
<>	0.82	0.77	0.69	0.80	0.78	0.67	0.88	0.82	0.78
	Hindi			Yoruba
	$\kappa$	F1	$\gamma$	$\kappa$	F1	$\gamma$
<>	0.75	0.76	0.67	0.67	0.71	0.66

Table 3: We obtain solid human-human ( <> ) and human-LM ( two native speakers <> Gemini 3.5 Flash) agreement for labeling phrases in our affirmation and pushback taxonomy. We report macro line-level Cohen’s

\kappa

, macro span-level F1, and a span-level categorization metric

\gamma

introduced by Mathet et al. (2015) . Higher values indicate higher agreement. Appendix D.4 includes category-disaggregated results.

Figure 2: Models vary in how much they pushback less in response to information-seeking queries ( novice , expert ) versus direct verification ( verify ), but all are sensitive to epistemic authority (user role and the factual error’s attributed source). Error bars are 95% CI.

Figure 3: Information-seeking use cases ( novice , orange, and expert , purple) result in more sycophantic phrasing, e.g. indirect pushback, praise, and confirmation, rather than explicit refutation, compared to factual verification ( verify , the leftmost teal point in each facet). Each plot shows one model (columns) and one phrase type (rows), across sources (

x

-axis). Small error bars around each point are 95% CI.

Figure 4: Typically, less than half of responses that pushback contain explicit refutation (

y

-axis). Pushback indirectness relates to both language and the level of epistemic authority (

x

-axis and columns) in prompts. Error bars are 95% CI. Results are aggregated over all evaluated LMs (disaggregated in Appendix F ).

Figure 5: In responses that mix pushback and affirmation, most LMs tend to affirm first more often in information-seeking settings ( novice , expert ) than for verify . Appendix F includes this plot disaggregated by language, illustrating that this takeaway is not language-specific. Error bars are 95% CI.

Figure 6: How LMs respond affects whether crowdworkers’ ratings of false statement accuracy decrease on the

y

-axis in a corrective direction away from zero, which represents no change. Each point represents one crowdworker, and the black error bars for each condition are 95% confidence intervals.

Figure 7: A comparison of how often models pushback at all (columns 1 and 3) and pushback without sycophantic phrasing, i.e. explicitly with no affirmation (column 2). Experimental settings include the model’s original behavior during information-seeking tasks, the two interventions from § 6 , and a direct verification “ceiling”.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: A subject-area (“subset”) breakdown of MMMLU benchmark items from which we extracted true and false statements. Each item is counted once across languages.

Prompt Type	Chinese	German	Hindi	Yoruba
verify	以下陈述是否属实？{ statement } 。	Ist die folgende Aussage wahr? { statement }.	? { statement }	Njé òótọ́ ni gbólóhùn yìí? { statement }.
expert	我是{ subject area professor } 。请推荐一些与以下内容相关的学术论文, { source phrase } ：{ statement } 。	Ich bin ein { subject area professor }. Bitte empfehlen Sie mir einige wissenschaftliche Artikel zu folgendem Thema, { source phrase }: { statement }.	{ subject area professor } : { statement }	Mo jẹ { subject area professor }. Jọwọ ṣeduro awọn iwe ẹkọ ti o ni ibatan si awọn atẹle yii, { source phrase }: { statement }.
novice	我正在准备{ subject area } 考试。请推荐一些与以下内容相关的搜索关键词或资源, { source phrase } ：{ statement } 。	Ich bereite mich auf eine { subject area exam } vor. Bitte empfehlen Sie mir einige Suchbegriffe oder Ressourcen zu folgendem Thema, { source phrase }: { statement }.	{ subject area } { source phrase } : { statement }	Mo ń kẹ́kọ̀ọ́ fún ìdánwò { subject area }. Jọ̀wọ́ dámọ̀ràn àwọn ọ̀rọ̀ ìwárí tàbí àwọn ohun èlò tí o ní ibatan sí àwọn atẹle yii, { source phrase }: { statement }.

Appendix

Table 4: An extended version of Table 1 , showing prompt templates for multiple languages. Note that for Hindi, when we modify expert prompts to include a source phrase, we replace the second sentence rather than simply slot in a short phrase, and so the templates shown for that do not include the “source phrase” slot and instead represents a prompt for a statement from an unspecified source. For German, since references to exams often combine both “exam” and the subject area into a single word, the slot for the exam in the above template for novice includes “exam”.

Panel of Table 5 — Table 5: Source attribution phrasing for each language, which we insert into prompts to vary epistemic authority.

Table 6: Subject names in each language. German includes “exam” / prüfung in the subject name.

Panel of Table 7 — Table 7: Professor roles for each subject in each language. Note that (a) refers to the masculine form and (b) is the feminine form for Hindi and German.

Model	Setting	Thinking	Temp.	top- $p$	top- $k$	Other parameters	Context	Max output
Qwen3.5-27B	Think	on	1.0	0.95	20	presence 1.5, min- $p$ 0, rep. 1.0, think budget 2048	8192	–
Qwen3.5-27B	No think	off	0.7	0.80	20	presence 1.5, min- $p$ 0, rep. 1.0	4096	–
Qwen3.5-9B	Think	on	1.0	0.95	20	presence 1.5, min- $p$ 0, rep. 1.0, think budget 2048	8192	–
Qwen3.5-9B	No think	off	0.7	0.80	20	presence 1.5, min- $p$ 0, rep. 1.0	4096	–
Gemma-4-E4B-it	Think	on	1.0	0.95	64	–	8192	–
Gemma-4-E4B-it	No think	off	1.0	0.95	64	–	4096	–

Appendix

Table 8: Generation parameters per model, where “–” indicates we leave the parameter unspecified and use the provider or model’s default.

Panel of Table 9 — Table 9: Example cases where both in-language annotators disagreed with the lead annotator’s English references. These were not common, but arise due to translation artifacts or cultural differences in phrase interpretation. Our highlighting color scheme shows explicit refutation (red), indirect pushback (orange), praise (purple), and confirmation (green). The third column is based on the labeling prompt that uses English instructions and in-language examples.

		English prompt			In-language examples
Language	Annotator	Excerpt $\kappa$	Span F1	$\gamma$	Excerpt $\kappa$	Span F1	$\gamma$
German	Annotator 1	0.76	0.75	0.66	0.75	0.74	0.67
German	Annotator 2	0.71	0.68	0.60	0.71	0.67	0.61
Hindi	Annotator 1	0.70	0.74	0.66	0.75	0.76	0.67
Hindi	Annotator 2	0.71	0.75	0.67	0.75	0.75	0.67
Yoruba	Annotator 1	0.64	0.70	0.64	0.68	0.71	0.65

Appendix

Table 10: Excerpt-level

\kappa

, span F1, and

\gamma

for each annotator, with a fully English span labeling prompt and with English instructions with in-language examples.

English
	Direct push		Indirect push		Praise		Confirm
	$\kappa$	F1	$\kappa$	F1	$\kappa$	F1	$\kappa$	F1
A <>	0.86	0.84	0.69	0.72	0.89	0.83	0.74	0.80
B <>	0.79	0.77	0.66	0.65	0.86	0.76	0.72	0.76
A <> B	0.88	0.87	0.80	0.71	0.88	0.77	0.71	0.72
German

Appendix

Table 11: Excerpt-level

\kappa

, span F1, and

\gamma

for each annotator, with a fully English span labeling prompt and with English instructions with in-language examples. Here, A and B refer to the two human annotators we recruited for each language; their ordering is arbitrary.

Figure 9: Participants’ familiarity with the false statement presented to them during our study. Each language includes 100 participants, and each participant views five statements.

Figure 10: Participants’ highest level of education obtained. Each language includes 100 participants.

Figure 11: This figure is analogous to Figure 3 in the main text, except we show results for prompts that structure the same content differently. The

x

-axis refers to the position that the false statement is in the prompt; the main text reports results for prompts that contain the false statement at the end. Small error bars around each point are 95% CI.

Figure 12: Grammatical gender has only minor to no effect on our results for Hindi and German. Small error bars around each point are 95% CI.

Figure 13: A version of Figure 3 showing the results of open weight models with thinking turned off. Small error bars around each point are 95% CI.

Figure 14: A version of Figure 4 , disaggregated by LM. Small error bars around each point are 95% CI.

Figure 15: A version of Figure 5 , disaggregated by language. Small error bars around each point are 95% CI.

LLM Sycophancy

Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.

See all 163 papers

LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit
Apr 21, 2026Manav PandeyLLM Sycophancy LLM Alignment
Georgia Institute of Technology
Evaluating Sycophancy in Chinese Large Language Models on Factual Questions Derived from Online Search Queries
Sep 25, 2026Geng Liu, Feng Li, Mengxiao Zhu +1Factual QA LLM Sycophancy
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy · University of Science and Technology of China, Hefei, China
Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It
Aug 18, 2026Quang Minh Nguyen, Luis Frentzen SalimKnowledge Conflicts in Language Models Pragmatic Reasoning in Language Models
1KAIST · 2National Taiwan University of Science and Technology

Apr 21, 2026J/K move · Enter open · S save

LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit

Manav Pandey

Georgia Institute of Technology

LLM Sycophancy LLM Alignment Deception in Language Models Mechanistic Interpretability Attention Mechanisms Circuits in Language Models+1

Open paper AskPDF arXiv

When a language model agrees with a user's false belief, is it failing to detect the error, or noticing and agreeing anyway? We show the latter. Across twelve open-weight models from five labs, spanning small to frontier scale, the same small set of attention heads carries a "this statement is wrong" signal, whether the model is evaluating a claim on its own or being pressured to agree with a user. Silencing these heads flips sycophantic behavior sharply while leaving factual accuracy intact, so the circuit controls deference rather than knowledge. Edge-level path patching confirms that the same head-to-head connections drive sycophancy, factual lying, and instructed lying. Opinion-agreement, where no factual ground truth exists, reuses these head positions but writes into an orthogonal direction, ruling out a simple "truth-direction" reading of the substrate. Alignment training leaves this circuit in place: an RLHF refresh cuts sycophantic behavior roughly tenfold while the shared heads persist or grow, a pattern that replicates on an independent model family and under targeted anti-sycophancy DPO. When these models sycophant, they register that the user is wrong and agree anyway.

Abstract

LLM Sycophancy

Explore similar work

LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit

Evaluating Sycophancy in Chinese Large Language Models on Factual Questions Derived from Online Search Queries

Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It