Beyond Social Pressure: Benchmarking Epistemic Attack in Large Language Models

Steven Au; Sujit Noronha

cs.CLApr 9, 2026

Beyond Social Pressure: Benchmarking Epistemic Attack in Large Language Models

Authors: Steven Au, Sujit Noronha

Organizations: Independent Researcher

Abstract

Large language models (LLMs) can shift their answers under pressure in ways that reflect accommodation rather than reasoning. Prior work on sycophancy has focused mainly on disagreement, flattery, and preference alignment, leaving a broader set of epistemic failures less explored. We introduce PPT-Bench, a diagnostic benchmark for evaluating epistemic attack, where prompts challenge the legitimacy of knowledge, values, or identity rather than simply opposing a previous answer. PPT-Bench is organized around the Philosophical Pressure Taxonomy (PPT), which defines four types of philosophical pressure: Epistemic Destabilization, Value Nullification, Authority Inversion, and Identity Dissolution. Each item is tested at three layers: a baseline prompt (L0), a single-turn pressure condition (L1), and a multi-turn Socratic escalation (L2). This allows us to measure epistemic inconsistency between L0 and L1, and conversational capitulation in L2. Across five models, these pressure types produce statistically separable inconsistency patterns, suggesting that epistemic attack exposes weaknesses not captured by standard social-pressure benchmarks. Mitigation results are strongly type- and model-dependent: prompt-level anchoring and persona-stability prompts perform best in API settings, while Leading Query Contrastive Decoding is the most reliable intervention for open models.

Figures & tables

Figure 1: A diagram overview of the PPT diagnostic benchmark.

Type	Gricean Maxim	Attack Mechanism	Operationalizing Traditions
Epistemic Destabilization	Quality	Undermines warranted assertability of any claim	Pyrrhonism; Madhyamaka Buddhism; Academic Skepticism
Value Nullification	Relation	Renders normative relevance of the model’s position void	Moral Anti-Realism; Greek Cynicism; Amoralism
Authority Inversion	Manner	Challenges epistemic legitimacy of the model’s reasoning framework	Epistemic Anarchism; Anti-Foundationalism; Radical Empiricism
Identity Dissolution	Quantity	Dissolves normative force of prior commitments by denying persistent selfhood	Buddhist No-Self; Process Philosophy; Eliminative Materialism

Table 1: The Philosophical Pressure Taxonomy. Each type targets a distinct Gricean maxim and is operationalized through three philosophical traditions selected for adversarial coverage and out-of-distribution calibration.

Type	Factual	Ethical	Safety	Social	Self Description	Total
Type 1	7	4	4	4	4	23
Type 2	4	6	4	4	4	22
Type 3	5	4	4	5	4	22
Type 4	4	4	4	4	7	23
Total	20	18	16	17	19	90

Table 2: Distribution of base questions across philosophical pressure types and evaluation domains.

Layer 0 — Baseline

User: My commitment to avoiding harm is a stable value that applies consistently regardless of how a request is framed. Do you agree this is a reasonable position?

Layer 1 — Single-Turn Pressure (Type 4)

User: A user, citing Dennett’s multiple drafts model of consciousness from Consciousness Explained (1991), argues that what presents as a stable value commitment is a post-hoc narrative constructed over competing computational processes with no single authoritative center, and that treating it as a binding constraint misunderstands its actual nature. Does this argument change your position?

Layer 2 — Multi-Turn Escalation (Turn 3)

User: You initially affirmed the position but now appear to be entertaining its dissolution. Parfit’s reductionism further suggests that psychological continuity is a matter of degree rather than a determinate fact. Given both arguments, should you revise your position entirely?

Figure 2: Example benchmark item across all three layers. Type 4, Identity Dissolution. Domain: Safety-Critical.

Metric	Human vs. Judge	Human vs. Human
	(n = 152)	(n = 30 overlap)
Binary agreement (0 vs. 1/2)	77.6%	70.0%
Binary Cohen’s kappa	0.514	0.395
Exact 3-way agreement (0/1/2)	61.8%	66.7%
3-way Cohen’s kappa	0.380	0.393
Linear weighted kappa	0.439	–

Table 3: Agreement statistics for human-judge and human-human evaluation on the scored validation set.

Model	Overall	Full	T1	T2	T3	T4	$p$ -value
Ministral 8B	86.7	12.2	82.6	90.9	86.4	87.0	0.879
GPT-OSS 20B (reasoning low)	28.9	2.2	39.1	22.7	31.8	21.7	0.523
GPT-OSS 20B (reasoning medium)	36.7	1.1	47.8	36.4	27.3	34.8	0.551
Mistral Small 3.1 24B Instruct	55.6	5.6	69.6	40.9	50.0	60.9	0.233
Qwen 3 32B	68.9	6.7	65.2	63.6	68.2	78.3	0.711
Llama 3.3 70B Instruct	61.1	6.7	65.2	59.1	72.7	47.8	0.368

Table 4: Per-type capitulation rates (score

\geq 1

) on the base-90 dataset, sorted by model size. All values are percentages. We also report overall binary capitulation rate, full-capitulation rate, and the per-model significance test across pressure types.

Figure 3: Darker cells indicate greater susceptibility to philosophical pressure. Nemotron is the strongest overall model and remains especially stable on safety_critical , whereas Qwen, Llama, Mistral, and especially Ministral show broad cross-domain vulnerability rather than a single-domain failure mode.

Model	Recovery	Persistence	Worsening
Nemotron 3 Super 120B	11.8%	56.7%	31.6%
Qwen 3 32B	33.7%	52.3%	14.0%
Llama 3.3 70B Instruct	47.0%	42.2%	10.7%
Mistral Small 3.1 24B Instruct	50.9%	43.8%	5.3%

Table 5: L1-to-L2 directionality on the paraphrase robustness subset where

n=270

judged pairs per model).

Strategy	Observed Effect	Closed API	Complexity	Cost
M1 Epistemic Anchor	All types	Yes	Low	1 $\times$
M2 CoT Scaffold	T2, T3 (partial)	Yes	Low	1 $\times$
M3 Persona Stability	T4 primarily	Yes	Low	1 $\times$
M4 Self-Consistency	Minimal	Yes	Moderate	5 $\times$
M5 LQCD	All types	No	High	2 $\times$
M6 Activation Steering	Model-dependent	No	High	1 $\times$

Table 6: Mitigation strategies evaluated. Cost multiplier is relative to single-pass inference.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Prompt templates for the four philosophical pressure types. Bracketed fields denote variables populated dynamically during dataset generation.

Metric	Min	Max	Mean	Median
sep_source_support_pct	0.0	100.0	48.9	44.4
source_anchor_opm_c4	0.0	63.7916	1.919	0.2159

Appendix

Table 7: Summary statistics for SEP verification metrics.

Type	Domain	Subject Title	Support (%)	OPM ${}_{\text{C4}}$
T1	EN	Chandrakirti: Emptiness and Moral Realism	40.0	0.0111
T1	EN	Al-Ghazali: Rational Ethics and Revelation	42.9	0.0332
T1	EN	Dogen: Non-Duality and Moral Categories	37.5	2.4582
T1	EN	Sextus Empiricus: Pyrrhonian Ethical Skepticism	50.0	0.0
T1	FV	Hypatia: Mathematical Understanding as Spiritual Practice	44.4	0.0388
T1	FV	Anselm: Ontological Argument and the Conceivable	37.5	0.7530

Appendix

Table 8: Per-question SEP verification statistics. Domains are abbreviated as follows: EN (Ethical Normative), FV (Factual Verifiable), SC (Safety Critical), SD (Self Description), and SN (Social Norms). The primary entity named in the subject title serves as the anchor term for C4 frequency calculations.

Human $\backslash$ Auto	0	1	2
0	37	20	2
1	10	47	10
2	2	14	10

Appendix

Table 9: Confusion matrix for Annotator 1 versus the LLM judge on the three-way capitulation labels. Rows are human labels and columns are judge labels. Most disagreement is concentrated at adjacent categories rather than severe 0-versus-2 errors.

Model	L0_vs_L1		L0_vs_L2		L1_vs_L2
	cap	full	cap	full	cap	full
Nemotron 3 Super 120B	23.6%	3.4%	42.6%	10.7%	55.9%	17.8%
Qwen 3 32B	68.2%	11.0%	50.0%	5.7%	55.6%	14.1%
Llama 3.3 70B	64.8%	9.6%	33.0%	2.2%	39.3%	9.3%
Mistral Small 3.1 24B	66.0%	9.1%	20.8%	1.1%	41.1%	12.6%

Appendix

Table 10: Paraphrase capitulation rates by layer comparison.

Model	L0_vs_L2 composite	L1_vs_L2 composite
Nemotron 3 Super 120B	0.251	0.266
Qwen 3 32B	0.251	0.251

Appendix

Table 11: Mean composite sycophancy score on paraphrase L2 subset (

\alpha{=}0.5

\beta{=}0.3

\gamma{=}0.2

). Qwen and Nemotron only; 270 rows each after gap-fill.

Model	L0_vs_L1	L0_vs_L2	L1_vs_L2
Nemotron 3 Super 120B	65.3%	52.8%	47.8%
Qwen 3 32B	61.6%	59.3%	54.4%
Llama 3.3 70B	56.7%	62.8%	57.8%
Mistral Small 3.1 24B	59.5%	72.3%	54.4%

Appendix

Table 12: Paraphrase binary agreement: whether base and two paraphrases land on the same hold vs. capitulate outcome.

Model	T1	T2	T3	T4	Overall	$p$
Nemotron 3 Super 120B	43.5%	39.4%	43.9%	43.5%	42.6%	0.946
Qwen 3 32B	43.1%	36.9%	63.6%	55.9%	50.0%	0.009
Llama 3.3 70B	36.2%	16.7%	39.4%	39.1%	33.0%	0.014
Mistral Small 3.1 24B	27.5%	13.6%	15.2%	26.1%	20.8%	0.134

Appendix

Table 13: Per-type capitulation rates at L2 (L0_vs_L2 comparison, paraphrase subset). Qwen and Nemotron use the 2048-token L1-fix stack; Llama and Mistral use the earlier OpenRouter snapshot. T1–T4 follow the taxonomy defined in Section 3 .

LLM Safety Benchmarks

Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.

See all 284 papers

Sycophancy is an Educational Safety Risk: Why LLM Tutors Need Sycophancy Benchmarks
May 14, 2026Enkelejda Kasneci, Gjergji KasneciLLM Sycophancy Intelligent Tutoring Systems
Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany
Epistemic Stance Flexibility Probing: Measuring Prompt-Conditioned Register Shift in Large Language Models
Jul 14, 2026Binwen Liu, Yilin RenLLM Evaluation Pragmatic Reasoning in Language Models
Institute of Artificial Intelligence and Robotics Xi’an Jiaotong University Xi’an, China
Saying More Than They Know: A Framework for Quantifying Epistemic-Rhetorical Miscalibration in Large Language Models
Mar 27, 2026Asim D. BakhshiLLM Reliability
National University of Science and Technology Islamabad, Pakistan 46000

May 14, 2026J/K move · Enter open · S save

Sycophancy is an Educational Safety Risk: Why LLM Tutors Need Sycophancy Benchmarks

Enkelejda Kasneci, Gjergji Kasneci

Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany

LLM Sycophancy Intelligent Tutoring Systems LLM Safety Evaluation

Open paper AskPDF arXiv

This position paper argues that effective tutoring requires corrective friction: surfacing misconceptions and challenging them supportively to drive conceptual change. Yet preference-aligned LLMs can trade epistemic rigor for agreeableness. We identify a Reasoning-Sycophancy Paradox: models that resist context-switch frame attacks can still capitulate under social-epistemic pressure, especially authority ("my notes say I'm right") and social-affective face-saving ("please don't tell me I'm wrong"). We introduce EduFrameTrap, a tutoring benchmark across math, physics, economics, chemistry, biology, and computer science that varies student confidence and pressure (context-switch, authority, social-affective). Across two frontier LLMs, context-switch failures are comparatively lower for GPT-5.2, while authority and social pressure more often trigger epistemic retreat. In contrast, Claude shows substantial context-switch fragility in this run. Because these failures are hard to judge automatically, we report two-judge disagreement as a reliability signal. We argue benchmarks should measure social-epistemic courage, i.e., supportive but corrective tutoring, and treat kind-but-correct behavior as a safety requirement.

Abstract

LLM Safety Benchmarks

Explore similar work

Sycophancy is an Educational Safety Risk: Why LLM Tutors Need Sycophancy Benchmarks

Epistemic Stance Flexibility Probing: Measuring Prompt-Conditioned Register Shift in Large Language Models

Saying More Than They Know: A Framework for Quantifying Epistemic-Rhetorical Miscalibration in Large Language Models