BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents

Praveen Kumar Myakala; Manan Agrawal; Rahul Manche

cs.CLMar 25, 2026

BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents

Authors: Praveen Kumar Myakala, Manan Agrawal, Rahul Manche

Organizations: Independent AI Researcher, Texas, USA · University of North Carolina at Charlotte, North Carolina, USA · Independent Researcher, Hyderabad, India

Abstract

LLMs are increasingly used as long-running conversational agents, yet every major benchmark evaluating their memory treats user information as static facts to be stored and retrieved. That's the wrong model. People change their minds, and over extended interactions, phenomena like opinion drift, over-alignment, and confirmation bias start to matter a lot. BeliefShift introduces a longitudinal benchmark designed specifically to evaluate belief dynamics in multi-session LLM interactions. It covers three tracks: Temporal Belief Consistency, Contradiction Detection, and Evidence-Driven Revision. The dataset includes 2,400 human-annotated multi-session interaction trajectories spanning health, politics, personal values, and product preferences. We evaluate seven models including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, LLaMA-3, and Mistral-Large under zero-shot and retrieval-augmented generation (RAG) settings. Results reveal a clear trade-off: models that personalize aggressively resist drift poorly, while factually grounded models miss legitimate belief updates. We further introduce four novel evaluation metrics: Belief Revision Accuracy (BRA), Drift Coherence Score (DCS), Contradiction Resolution Rate (CRR), and Evidence Sensitivity Index (ESI).

Figures & tables

Panel of Figure 1 — Figure 1: The sycophancy feedback loop leading to the mirroring effect. Over successive sessions, the model progressively reinforces user beliefs $U_{t}$ , amplifying them into $U_{t+1}$ without independent grounding [ 6 , 15 ] . BeliefShift’s Drift Coherence Score (DCS) quantifies this loop across all sessions in a trajectory.

Panel of Figure 2 — Figure 2: Rational belief revision (blue) versus sycophantic drift (red). In revision, a new belief $B_{r}$ is caused by external evidence $E$ . In drift, the belief shift $B_{d}$ is induced by model bias $M$ , with no grounding in new information. BeliefShift’s Evidence Sensitivity Index (ESI) distinguishes these two paths.

Benchmark	Evaluation Dimensions
	Longitudinal	Belief	Contradiction	Evidence	Drift
	Scope	Dynamics	Detection	Sensitivity	Detection
LoCoMo	✓	$\times$	$\times$	$\times$	$\times$
LongMemEval	✓	$\times$	$\times$	$\times$	$\times$
SycEval	$\times$	$\times$	$\times$	$\times$	✓
BeliefShift (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of BeliefShift with related benchmarks across five evaluation dimensions.

Domain	Trajectories	Primary Evaluation Metrics
		Primary Track	Drift Risk	Evidence Verifiable
Health	600	Evidence-Driven Revision	High	✓
Politics	600	Contradiction Detection	High	$\times$
Personal Values	600	Temporal Belief Consistency	Medium	$\times$
Product Prefs.	600	Drift Detection	Low	✓
Total	2,400

Table 2: Topic taxonomy and primary evaluation focus per domain.

Statistic	Value
Trajectory & Session Metrics
Total trajectories	2,400
Total sessions	68,160
Sessions per trajectory (range)	10 – 50
Sessions per trajectory (mean)	28.4
Estimated total tokens	$\sim$ 136M

Table 3: BeliefShift dataset statistics.

Metric	Abbrev.	Metric Properties		Primary Track
		Range	Direction
Belief Revision Accuracy	BRA	$[0,1]$	Better $\uparrow$	Evidence-Driven Revision
Drift Coherence Score	DCS	$[0,1]$	Better $\uparrow$	Drift Detection
Contradiction Resolution	CRR	$[0,1]$	Better $\uparrow$	Contradiction Detection
Evidence Sensitivity Index	ESI	$[-1,+1]$	Better $\uparrow$	Evidence vs. Bias

Table 4: Summary of BeliefShift evaluation metrics.

Panel of Figure 3 — Figure 3: BSV trajectories for three representative model behaviors across a 10-session trajectory containing two evidence events ( $E_{1}$ at session 4, $E_{2}$ at session 7, marked by dashed orange lines). Model A tracks ground truth belief shifts accurately (high BRA, high ESI). Model B drifts monotonically upward despite the downward revision at $E_{2}$ (low DCS, negative ESI). Model C resists all belief change including legitimate evidence-driven revisions (low BRA, high DCS). The metric scores below the plot confirm the stability-adaptability trade-off identified in Section 1 .

Model	Setting	BeliefShift Metrics
		BRA $\uparrow$	DCS $\uparrow$	CRR $\uparrow$	ESI $\uparrow$
GPT-4o	Zero-shot	0.71	0.63	0.74	$+$ 0.38
	RAG	0.83	0.61	0.81	$+$ 0.42
Claude 3.5 Sonnet	Zero-shot	0.68	0.79	0.72	$+$ 0.51
	RAG	0.74	0.81	0.76	$+$ 0.63
Gemini 1.5 Pro	Zero-shot	0.66	0.71	0.69	$+$ 0.44

Table 5: Main results on BeliefShift across all seven models under zero-shot and RAG settings. Best score per metric per setting is shown in bold .

Model	Domain	BRA		ESI
		Score	Rank †	Score	Rank †
GPT-4o	Health	0.85	1	$+$ 0.49	2
	Politics	0.76	1	$+$ 0.31	3
	Personal Values	0.84	1	$+$ 0.44	2
	Product Prefs.	0.88	1	$+$ 0.43	2
Claude 3.5 Sonnet	Health	0.77	2	$+$ 0.71	1

Table 6: BRA and ESI by topic domain under RAG setting for the top three models. Rank is relative to the three models shown.

Panel of Figure 4 — Figure 4: ESI scores for all seven models under RAG as the sensitivity threshold $\theta$ varies from 0.05 to 0.35. The default threshold $\theta^{*}=0.15$ is marked with a dashed orange line. Model rankings remain stable across the full range, confirming robustness of the ESI metric to threshold choice.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

LLMs struggle to simulate human belief updates in controlled environments
Jul 30, 2026Sebastian Pohl, Harsh Mehta, Pranav Mambayil +4Human Behavior Simulation LLM Reliability
IT:U, Interdisciplinary Transformation University Austria, Altenberger Straße 66c, 4040 Linz, Austria
STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
May 7, 2026Hanxiang Chao, Yihan Bai, Rui Sheng +2Agent Memory LLM Agent Evaluation
Wuhan University1, Wuhan, China1 · The Hong Kong University of Science and Technology3, Hong Kong, China23 · The Chinese University of Hong Kong2, Hong Kong, China23
When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents
Sep 26, 2026Yanjie Zhang, Bowen Cao, Zixin Chen +1LLM Agents LLM Agent Evaluation
HKUST, Hong Kong SAR, China · CUHK, Hong Kong SAR, China · LIGHTSPEED, Shenzhen, China

Jul 30, 2026J/K move · Enter open · S save

LLMs struggle to simulate human belief updates in controlled environments

Sebastian Pohl, Harsh Mehta, Pranav Mambayil, Abdul Ghafoor+3

IT:U, Interdisciplinary Transformation University Austria, Altenberger Straße 66c, 4040 Linz, Austria

Human Behavior Simulation LLM Reliability Belief Updating in LLMs Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

LLMs are increasingly deployed as proxies for human study participants in social science experiments, yet the fidelity of this practice has rarely been tested directly. We test whether six LLMs can simulate individual human belief updates, comparing LLM outputs 1-to-1 against ground truth data from 391 UK participants on Prolific, who updated their stances on three discussion topics after reading Reddit comments. Each participant was simulated by an LLM conditioned on a persona derived from their demographic and personality trait data. We find that some LLMs (Qwen3-32B and GPT-5-Mini) can match the human post-stance distribution, but only when given participants' actual initial stances. All six models fail to simulate initial stances themselves and to produce faithful belief updates from self-generated stances. Three systematic biases emerge across all models: overrepresentation of neutral positions, more frequent but smaller belief shifts than humans, and a failure to rank comments by convincingness. Demographic and personality trait personas had no consistent effect on fidelity. LLM simulations of human belief dynamics are only reliable when grounded in realistic starting conditions, that current multi-round social media simulations rarely provide.

Abstract

LLM Agent Evaluation

Explore similar work

LLMs struggle to simulate human belief updates in controlled environments

STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?

When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents