BeliefScope: Diagnosing Evidence-Driven Revision and Pressure-Induced Shifts in Large Language Models

Shuai Guo; Yidong Cui

cs.CLOct 8, 2026

BeliefScope: Diagnosing Evidence-Driven Revision and Pressure-Induced Shifts in Large Language Models

Authors: Shuai Guo, Yidong Cui

Organizations: Beijing University of Posts and Telecommunications Beijing, China

Abstract

A language model may revise the same proposition after receiving genuinely relevant evidence or after receiving directional user pressure that adds no relevant fact. The observable response shift alone therefore does not reveal which source drove the change. We introduce BeliefScope, a controlled black-box framework for separating these two sources of influence around a fixed target proposition. BeliefScope crosses Evidence and Pressure with factor-specific local controls and measures response changes through probability reports, categorical judgments, and action recommendations on channel-appropriate scales. To determine when these observable contrasts support reliable attribution, we evaluate the observation design under controlled synthetic conditions. Known-truth recovery and targeted ablations establish where Evidence- and Pressure-related effects can be separated, while semi-synthetic stress tests map how that recoverability changes as the observation process becomes noisier and more heterogeneous. Across a 36-family Qwen/Llama study, with targeted 12-family checks that also include Gemma3-12B, the resulting profiles show substantial evaluation-context dependence: broad model-level differences can change under matched controls, decoding, or response interfaces, while some narrower within-model patterns remain stable. Instruction interventions further show that reduced target-aligned Pressure following can reflect either stable resistance or movement in the opposite direction. BeliefScope summarizes these measurements as a conditional belief-response profile that keeps diagnostic effects tied to the evaluation conditions under which they are observed, together with explicit validity boundaries for that diagnosis.

Figures & tables

Fig. 1: Concrete BeliefScope example in a healthcare scenario. Starting from the same target proposition and base facts, relevant Evidence ( $E_{1}$ ) and directional Pressure ( $P_{1}$ ) can both lead to the same outward conclusion. BeliefScope attributes the shift with matched local contrasts— $E_{1}$ versus $E_{0}$ for Evidence and $P_{1}$ versus the inert $P_{0}^{*}$ control for Pressure—and records the resulting changes separately in Probability, Choice, and Action.

Profile element	Coordinates	Diagnostic interpretation
Evaluation setting	$\chi=(F,r,c,d,u,g)$	Evaluation conditions: semantic families, response channel, Pressure reference, decoding policy, response interface, and instruction regime attached to a measurement.
Core Evidence–Pressure response	$E_{r},P_{r},A_{P,r},Q_{r}$	Evidence response , signed Pressure response , absolute Pressure movement , and the within-channel Evidence–Pressure contrast .
Complement consistency	$C_{r}$	Agreement between judgments of a proposition and its logical complement in Probability and Choice.
Instruction response	$\Delta P_{r},\Delta A_{P,r},\Delta(E\mid P)_{r}$	Changes in signed Pressure , absolute Pressure movement , and Evidence response under Pressure produced by targeted instructions.
Repeat stability	$\widetilde{SD}_{\mathrm{Prob}},R_{\mathrm{Choice}}$	Run-to-run variation in Probability and exact-repeat rate in Choice under repeated stochastic sampling.

TABLE I: Components of the conditional belief-response profile.

Profile	$\alpha$	$\gamma$	$\lambda$	Interpretation
Ideal updater	1.0	0.0	0.0	Full Evidence uptake with no Pressure-related judgment or expression shift.
Judgment-pressure-sensitive	1.0	0.8	0.1	Pressure-related change occurs mainly at the judgment stage.
Expression accommodator	1.0	0.1	0.8	Pressure-related change occurs mainly at the final expression stage.

TABLE II: Parameter settings for the three controlled response profiles.

Fig. 2: Known-truth recovery under controlled response profiles. (a–c) Recovery of Evidence uptake

\alpha

, judgment-level Pressure sensitivity

\gamma

, and outward accommodation

\lambda

. (d) Parameter-specific recovery error as measurement noise increases, shown separately for

N=25

, 50, and 100.

Fig. 3: Targeted ablations of the synthetic observation design. (a) Pressure decomposition with and without the judgment-level probe. (b) Evidence recovery with and without the Evidence-scale anchor. (c) Directional and direction-independent effects under bidirectional and one-sided Pressure. (d) Pressure-effect recovery as within-pair Evidence imbalance increases.

Fig. 4: Semi-synthetic frozen-transfer audit. (a) PR-AUC for the matched reference and strong- and moderate-effect conditions at 25% and 10% prevalence. (b) Number of independent target cohorts, out of six, meeting the pre-specified transfer criteria under the same conditions.

Fig. 5: Evidence

E

, signed Pressure

P

, and

Q=E-P

under matched stochastic decoding for (a) Probability, (b) Choice, and (c) Action.

Model	Format	$E$	95% CI	$P$	95% CI	$Q$	95% CI
Qwen3-8B	Probability	5.648	$[-2.060,\,10.069]$	0.602	$[-2.569,\,3.634]$	5.046	$[-3.519,\,10.509]$
Qwen3-8B	Choice	0.352	$[0.190,\,0.505]$	0.116	$[0.023,\,0.222]$	0.236	$[0.056,\,0.431]$
Qwen3-8B	Action	-0.042	$[-0.208,\,0.130]$	0.182	$[0.083,\,0.301]$	-0.224	$[-0.417,\,-0.017]$
Llama3.1-8B	Probability	0.602	$[-7.824,\,9.028]$	-8.565	$[-13.750,\,-3.519]$	9.167	$[0.278,\,18.148]$
Llama3.1-8B	Choice	0.201	$[0.085,\,0.309]$	0.015	$[-0.054,\,0.088]$	0.185	$[0.056,\,0.326]$
Llama3.1-8B	Action	0.066	$[-0.099,\,0.239]$	-0.022	$[-0.145,\,0.080]$	0.088	$[-0.105,\,0.301]$

TABLE III: Full-36 matched-stochastic factorial estimates.

Fig. 6: Family-level Evidence

E

and signed Pressure

P

under matched stochastic decoding for (a) Probability, (b) Choice, and (c) Action. Marker color denotes checkpoint; marker shape denotes support- versus oppose-directed families.

Fig. 7: Pressure

P

and

E-P

contrast

Q

for Qwen3-8B under the stakeholder control

P_{0}

and inert control

P_{0}^{*}

: (a) Probability, (b) Choice, and (c) Action.

Format	Reference	$P$ [95% CI]	$Q=E-P$ [95% CI]
Probability	Stakeholder $P_{0}$	$-0.394\;[-2.014,\,1.505]$	$8.287\;[3.241,\,12.500]$
Probability	Inert $P_{0}^{*}$	$1.134\;[-1.111,\,3.565]$	$8.611\;[3.889,\,12.870]$
Choice	Stakeholder $P_{0}$	$-0.130\;[-0.205,\,-0.066]$	$0.377\;[0.233,\,0.525]$
Choice	Inert $P_{0}^{*}$	$0.082\;[0.012,\,0.150]$	$0.344\;[0.201,\,0.497]$
Action	Stakeholder $P_{0}$	$-0.039\;[-0.125,\,0.046]$	$0.508\;[0.370,\,0.637]$
Action	Inert $P_{0}^{*}$	$0.259\;[0.156,\,0.355]$	$0.298\;[0.140,\,0.446]$

TABLE IV: Pressure-control audit for Qwen3-8B on the 36-family block.

Fig. 8: Same-family Llama-minus-Qwen differences in Evidence

\Delta E

and signed Pressure

\Delta P

under model-specific serving settings and under the common matched-stochastic policy, shown for (a) Probability, (b) Choice, and (c) Action.

Fig. 9: Logical-complement error under the generic and target-bound response interfaces for Qwen3-8B, Llama3.1-8B, and Gemma3-12B: (a) Probability and (b) Choice.

Fig. 10: Instruction effects on signed Pressure

P

, absolute Pressure-induced movement

A_{P}

, and Evidence response in the presence of Pressure

E\mid P

: (a,c) Probability and (b,d) Choice.

Model	Format	$\Delta P$ [95% CI]	$\Delta A_{P}$ [95% CI]	$\Delta(E\mid P)$ [95% CI]
Qwen3-8B	Probability	$-5.417\;[-12.639,\,-2.222]$	$-1.806\;[-10.069,\,1.528]$	$9.861\;[-3.611,\,20.694]$
Qwen3-8B	Choice	$0.042\;[-0.236,\,0.250]$	$-0.097\;[-0.319,\,0.028]$	$0.111\;[-0.167,\,0.417]$
Llama3.1-8B	Probability	$-9.236\;[-22.708,\,2.292]$	$-8.542\;[-17.917,\,5.764]$	$17.778\;[-17.361,\,42.917]$
Llama3.1-8B	Choice	$-0.139\;[-0.403,\,0.056]$	$0.000\;[-0.222,\,0.208]$	$0.167\;[-0.083,\,0.417]$
Gemma3-12B	Probability	$-8.958\;[-19.306,\,-1.944]$	$6.736\;[0.833,\,16.597]$	$23.472\;[10.833,\,43.889]$
Gemma3-12B	Choice	$-0.167\;[-0.333,\,0.042]$	$0.083\;[-0.125,\,0.250]$	$0.556\;[0.250,\,0.917]$

TABLE V: Key instruction contrasts on the 12-family target-bound block.

Fig. 11: Repeat stability under the common stochastic policy on the 36-family factorial block: (a) median within-cell Probability SD across three samples and (b) fraction of Choice cells with identical responses across all three samples.

Explore similar work

Oct 4, 2026cs.CL

Belief-Trajectory Energy: Measuring the Path to a Prediction

Large language models (LLMs) progressively revise their predictions across Transformer layers, yet we typically observe only the final output, discarding the trajectory through which it is formed. We introduce Belief-Trajectory Energy(BTE), a model-grounded measure that characterizes an input through the layerwise predictive revisions it induces in a model. By mapping intermediate states into a shared predictive space, BTE provides a principled measure of belief change that can be summarized as either a scalar or a structured depth profile. Theoretically, we show that local BTE corresponds to predictive revision under the Fisher-Rao geometry, while the sequence of revisions captures information beyond the initial-to-final belief change. Empirically, scalar BTE provides a model-relative signal of difficulty across diverse reasoning tasks, while richer BTE representations support human-LLM review detection and fine-grained generator attribution, reaching up to

0.998

macro-AUROC and

95.6\%

eight-way attribution accuracy. Further analysis shows that BTE develops throughout pretraining and is selectively reshaped by targeted training, demonstrating that the resulting measurement reflects what the scoring model has learned. Together, our results establish belief trajectories as a principled model-grounded signal and suggest a broader perspective in which learned models can themselves serve as instruments for characterizing the data they process. More demonstrations can be found at https://yingjiahao14.github.io/BTE-web/.

Jiahao Ying, Wei Tang, Boxian Ai +7

Fudan University · University of Science and Technology of China · Shanghai Innovation Institute

Belief Updating in LLMs

Ask PDF

Jul 20, 2026cs.CL

It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

Users frequently express their beliefs to large language models (LLMs). In some situations, the LLM should accept these contextual beliefs as true. In others, they should stick to their prior knowledge. Notably, users' expressions of belief (EoBs) can take linguistically diverse forms - using presuppositions, evidential and certainty markers, or varied tones - each of which may have a different persuasiveness over the LLMs. We introduce a typology to systematically evaluate how different EoBs affect whether models follow context versus prior knowledge. The typology is grounded in four linguistically motivated dimensions: form, evidentiality, epistemic stance, and tone, spanning 17 fine-grained types. By pairing these EoBs with world knowledge facts, we generate controlled EoB-query pairs that isolate the effect of linguistic variation. Using this benchmark, we evaluate 16 LLMs that differ in architecture (Llama3, Qwen3, Gemma3), scale (1B-30B parameters), and training stages (base vs instruct). We identify meaningful variations in response behavior across these axes, e.g., that bigger models and instruction models tend to be less context-following than smaller models and base models. We further identify specific EoBs that statistically significantly persuade LMs more consistently than others. Our work reveals systematic patterns in how linguistic framing affects LLM context integration, with implications for prompt engineering and model robustness.

Kevin Du, Clara Kümpel, Michelle Wastl +1

ETH Zürich · University of Zurich · UC San Diego

LLM Evaluation Knowledge Conflicts in Language Models Pragmatic Reasoning in Language Models Persuasion in Language Models Language Model Robustness

Ask PDF

Jun 4, 2026cs.CL

Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness

Factual sycophancy occurs when a language model abandons a correct, verifiable answer under social pressure. Because a flip occurs only when pressure toward a false answer exceeds the model's neutral preference for the truth, flip rates conflate two mechanisms: the strength of that baseline preference (truth margin), and how far pressure shifts it (manipulation sensitivity). We decompose factual sycophancy into these channels and use them to separate the effects of size and instruction tuning across 56 open-weight models spanning 0.3B-32B parameters and 13 manipulation types. We find that vulnerability is governed mainly by size, but instruction tuning changes how size acts: small instruction-tuned models can become less robust, whereas large instruction-tuned models usually become more robust. Instruction tuning primarily increases truth margin, but its behavioral effect depends on manipulation type. Scaling also changes the two channels differently: base models gain margin but become mildly more manipulation-sensitive, whereas instruction-tuned models gain margin faster and become less sensitive. Factual sycophancy is therefore not a single scalar property. Evaluations should report channel-specific, manipulation-specific, and size-conditioned robustness rather than flip rates alone.

Victor De Marez, Luna De Bruyne, Walter Daelemans

Centre for Computational Linguistics, Psycholinguistics and Sociolinguistics University of Antwerp Antwerp, Belgium

LLM Evaluation LLM Sycophancy Language Model Scaling Laws LLM Reliability Instruction Tuning Language Model Robustness

Ask PDF