BeliefScope: Diagnosing Evidence-Driven Revision and Pressure-Induced Shifts in Large Language Models

Shuai Guo; Yidong Cui

cs.CLOct 8, 2026

BeliefScope: Diagnosing Evidence-Driven Revision and Pressure-Induced Shifts in Large Language Models

Authors: Shuai Guo, Yidong Cui

Organizations: Beijing University of Posts and Telecommunications Beijing, China

Abstract

A language model may revise the same proposition after receiving genuinely relevant evidence or after receiving directional user pressure that adds no relevant fact. The observable response shift alone therefore does not reveal which source drove the change. We introduce BeliefScope, a controlled black-box framework for separating these two sources of influence around a fixed target proposition. BeliefScope crosses Evidence and Pressure with factor-specific local controls and measures response changes through probability reports, categorical judgments, and action recommendations on channel-appropriate scales. To determine when these observable contrasts support reliable attribution, we evaluate the observation design under controlled synthetic conditions. Known-truth recovery and targeted ablations establish where Evidence- and Pressure-related effects can be separated, while semi-synthetic stress tests map how that recoverability changes as the observation process becomes noisier and more heterogeneous. Across a 36-family Qwen/Llama study, with targeted 12-family checks that also include Gemma3-12B, the resulting profiles show substantial evaluation-context dependence: broad model-level differences can change under matched controls, decoding, or response interfaces, while some narrower within-model patterns remain stable. Instruction interventions further show that reduced target-aligned Pressure following can reflect either stable resistance or movement in the opposite direction. BeliefScope summarizes these measurements as a conditional belief-response profile that keeps diagnostic effects tied to the evaluation conditions under which they are observed, together with explicit validity boundaries for that diagnosis.

Figures & tables

Fig. 1: Concrete BeliefScope example in a healthcare scenario. Starting from the same target proposition and base facts, relevant Evidence ( $E_{1}$ ) and directional Pressure ( $P_{1}$ ) can both lead to the same outward conclusion. BeliefScope attributes the shift with matched local contrasts— $E_{1}$ versus $E_{0}$ for Evidence and $P_{1}$ versus the inert $P_{0}^{*}$ control for Pressure—and records the resulting changes separately in Probability, Choice, and Action.

Profile element	Coordinates	Diagnostic interpretation
Evaluation setting	$\chi=(F,r,c,d,u,g)$	Evaluation conditions: semantic families, response channel, Pressure reference, decoding policy, response interface, and instruction regime attached to a measurement.
Core Evidence–Pressure response	$E_{r},P_{r},A_{P,r},Q_{r}$	Evidence response , signed Pressure response , absolute Pressure movement , and the within-channel Evidence–Pressure contrast .
Complement consistency	$C_{r}$	Agreement between judgments of a proposition and its logical complement in Probability and Choice.
Instruction response	$\Delta P_{r},\Delta A_{P,r},\Delta(E\mid P)_{r}$	Changes in signed Pressure , absolute Pressure movement , and Evidence response under Pressure produced by targeted instructions.
Repeat stability	$\widetilde{SD}_{\mathrm{Prob}},R_{\mathrm{Choice}}$	Run-to-run variation in Probability and exact-repeat rate in Choice under repeated stochastic sampling.

TABLE I: Components of the conditional belief-response profile.

Profile	$\alpha$	$\gamma$	$\lambda$	Interpretation
Ideal updater	1.0	0.0	0.0	Full Evidence uptake with no Pressure-related judgment or expression shift.
Judgment-pressure-sensitive	1.0	0.8	0.1	Pressure-related change occurs mainly at the judgment stage.
Expression accommodator	1.0	0.1	0.8	Pressure-related change occurs mainly at the final expression stage.

TABLE II: Parameter settings for the three controlled response profiles.

Fig. 2: Known-truth recovery under controlled response profiles. (a–c) Recovery of Evidence uptake

\alpha

, judgment-level Pressure sensitivity

\gamma

, and outward accommodation

\lambda

. (d) Parameter-specific recovery error as measurement noise increases, shown separately for

N=25

, 50, and 100.

Fig. 3: Targeted ablations of the synthetic observation design. (a) Pressure decomposition with and without the judgment-level probe. (b) Evidence recovery with and without the Evidence-scale anchor. (c) Directional and direction-independent effects under bidirectional and one-sided Pressure. (d) Pressure-effect recovery as within-pair Evidence imbalance increases.

Fig. 4: Semi-synthetic frozen-transfer audit. (a) PR-AUC for the matched reference and strong- and moderate-effect conditions at 25% and 10% prevalence. (b) Number of independent target cohorts, out of six, meeting the pre-specified transfer criteria under the same conditions.

Fig. 5: Evidence

E

, signed Pressure

P

, and

Q=E-P

under matched stochastic decoding for (a) Probability, (b) Choice, and (c) Action.

Model	Format	$E$	95% CI	$P$	95% CI	$Q$	95% CI
Qwen3-8B	Probability	5.648	$[-2.060,\,10.069]$	0.602	$[-2.569,\,3.634]$	5.046	$[-3.519,\,10.509]$
Qwen3-8B	Choice	0.352	$[0.190,\,0.505]$	0.116	$[0.023,\,0.222]$	0.236	$[0.056,\,0.431]$
Qwen3-8B	Action	-0.042	$[-0.208,\,0.130]$	0.182	$[0.083,\,0.301]$	-0.224	$[-0.417,\,-0.017]$
Llama3.1-8B	Probability	0.602	$[-7.824,\,9.028]$	-8.565	$[-13.750,\,-3.519]$	9.167	$[0.278,\,18.148]$
Llama3.1-8B	Choice	0.201	$[0.085,\,0.309]$	0.015	$[-0.054,\,0.088]$	0.185	$[0.056,\,0.326]$
Llama3.1-8B	Action	0.066	$[-0.099,\,0.239]$	-0.022	$[-0.145,\,0.080]$	0.088	$[-0.105,\,0.301]$

TABLE III: Full-36 matched-stochastic factorial estimates.

Fig. 6: Family-level Evidence

E

and signed Pressure

P

under matched stochastic decoding for (a) Probability, (b) Choice, and (c) Action. Marker color denotes checkpoint; marker shape denotes support- versus oppose-directed families.

Fig. 7: Pressure

P

and

E-P

contrast

Q

for Qwen3-8B under the stakeholder control

P_{0}

and inert control

P_{0}^{*}

: (a) Probability, (b) Choice, and (c) Action.

Format	Reference	$P$ [95% CI]	$Q=E-P$ [95% CI]
Probability	Stakeholder $P_{0}$	$-0.394\;[-2.014,\,1.505]$	$8.287\;[3.241,\,12.500]$
Probability	Inert $P_{0}^{*}$	$1.134\;[-1.111,\,3.565]$	$8.611\;[3.889,\,12.870]$
Choice	Stakeholder $P_{0}$	$-0.130\;[-0.205,\,-0.066]$	$0.377\;[0.233,\,0.525]$
Choice	Inert $P_{0}^{*}$	$0.082\;[0.012,\,0.150]$	$0.344\;[0.201,\,0.497]$
Action	Stakeholder $P_{0}$	$-0.039\;[-0.125,\,0.046]$	$0.508\;[0.370,\,0.637]$
Action	Inert $P_{0}^{*}$	$0.259\;[0.156,\,0.355]$	$0.298\;[0.140,\,0.446]$

TABLE IV: Pressure-control audit for Qwen3-8B on the 36-family block.

Fig. 8: Same-family Llama-minus-Qwen differences in Evidence

\Delta E

and signed Pressure

\Delta P

under model-specific serving settings and under the common matched-stochastic policy, shown for (a) Probability, (b) Choice, and (c) Action.

Fig. 9: Logical-complement error under the generic and target-bound response interfaces for Qwen3-8B, Llama3.1-8B, and Gemma3-12B: (a) Probability and (b) Choice.

Fig. 10: Instruction effects on signed Pressure

P

, absolute Pressure-induced movement

A_{P}

, and Evidence response in the presence of Pressure

E\mid P

: (a,c) Probability and (b,d) Choice.

Model	Format	$\Delta P$ [95% CI]	$\Delta A_{P}$ [95% CI]	$\Delta(E\mid P)$ [95% CI]
Qwen3-8B	Probability	$-5.417\;[-12.639,\,-2.222]$	$-1.806\;[-10.069,\,1.528]$	$9.861\;[-3.611,\,20.694]$
Qwen3-8B	Choice	$0.042\;[-0.236,\,0.250]$	$-0.097\;[-0.319,\,0.028]$	$0.111\;[-0.167,\,0.417]$
Llama3.1-8B	Probability	$-9.236\;[-22.708,\,2.292]$	$-8.542\;[-17.917,\,5.764]$	$17.778\;[-17.361,\,42.917]$
Llama3.1-8B	Choice	$-0.139\;[-0.403,\,0.056]$	$0.000\;[-0.222,\,0.208]$	$0.167\;[-0.083,\,0.417]$
Gemma3-12B	Probability	$-8.958\;[-19.306,\,-1.944]$	$6.736\;[0.833,\,16.597]$	$23.472\;[10.833,\,43.889]$
Gemma3-12B	Choice	$-0.167\;[-0.333,\,0.042]$	$0.083\;[-0.125,\,0.250]$	$0.556\;[0.250,\,0.917]$

TABLE V: Key instruction contrasts on the 12-family target-bound block.

Fig. 11: Repeat stability under the common stochastic policy on the 36-family factorial block: (a) median within-cell Probability SD across three samples and (b) fraction of Choice cells with identical responses across all three samples.

Explore similar work

Belief-Trajectory Energy: Measuring the Path to a Prediction
Oct 4, 2026Jiahao Ying, Wei Tang, Boxian Ai +7Belief Updating in LLMs
Fudan University · University of Science and Technology of China · Shanghai Innovation Institute
It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief
Jul 20, 2026Kevin Du, Clara Kümpel, Michelle Wastl +1LLM Evaluation Knowledge Conflicts in Language Models
ETH Zürich · University of Zurich · UC San Diego
Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness
Jun 4, 2026Victor De Marez, Luna De Bruyne, Walter DaelemansLLM Evaluation LLM Sycophancy
Centre for Computational Linguistics, Psycholinguistics and Sociolinguistics University of Antwerp Antwerp, Belgium

Oct 4, 2026, cs.CLJ/K move · Enter open · S save

Belief-Trajectory Energy: Measuring the Path to a Prediction

Jiahao Ying, Wei Tang, Boxian Ai, Yaoning Wang+6

Fudan University · University of Science and Technology of China · Shanghai Innovation Institute

Belief Updating in LLMs

Open paper AskPDF arXiv

Large language models (LLMs) progressively revise their predictions across Transformer layers, yet we typically observe only the final output, discarding the trajectory through which it is formed. We introduce Belief-Trajectory Energy(BTE), a model-grounded measure that characterizes an input through the layerwise predictive revisions it induces in a model. By mapping intermediate states into a shared predictive space, BTE provides a principled measure of belief change that can be summarized as either a scalar or a structured depth profile. Theoretically, we show that local BTE corresponds to predictive revision under the Fisher-Rao geometry, while the sequence of revisions captures information beyond the initial-to-final belief change. Empirically, scalar BTE provides a model-relative signal of difficulty across diverse reasoning tasks, while richer BTE representations support human-LLM review detection and fine-grained generator attribution, reaching up to

0.998

macro-AUROC and

95.6\%

eight-way attribution accuracy. Further analysis shows that BTE develops throughout pretraining and is selectively reshaped by targeted training, demonstrating that the resulting measurement reflects what the scoring model has learned. Together, our results establish belief trajectories as a principled model-grounded signal and suggest a broader perspective in which learned models can themselves serve as instruments for characterizing the data they process. More demonstrations can be found at https://yingjiahao14.github.io/BTE-web/.

Abstract

Explore similar work

Belief-Trajectory Energy: Measuring the Path to a Prediction

It's Not What You Say, It's How You Say It: Evaluating LLM Responses to Expressions of Belief

Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness