Do "True" and "False" Add Up to One? Complement Consistency in LLM-Agent Opinion Simulations

Anouar Imel; Jin Qi

cs.AIOct 23, 2026

Do "True" and "False" Add Up to One? Complement Consistency in LLM-Agent Opinion Simulations

Authors: Anouar Imel, Jin Qi

Organizations: School of Information and Communication Engineering University of Electronic Science and Technology of China (UESTC) Chengdu, China

Abstract

In LLM-agent opinion simulations, each agent reports the probability that a claim is true after seeing its own previous probability and the probabilities reported by its neighbors, and that number becomes its state in the next round. If the number describes the agent's position rather than the sentence it was shown, then rewriting the claim as its complement and replacing every probability p with 1 - p should return the complementary answer, so the two reports should sum to one. We test this with three small open models: Qwen3-4B, Llama-3.2-3B and Gemma-3-4B. On 2,304 agent states taken from a running network simulation, Qwen's paired reports sum to 1.18 on average, and on 48 real fact-checking claims from AVeriTeC they again exceed one. A controlled benchmark of 24 worlds, each with exactly two possible outcomes, shows the same failure in all three models. The discrepancy is two to ten times larger than the change produced by a same-meaning paraphrase, and it persists under repeated sampling and across peer group sizes and report strengths. When the reports are fed back into two populations that start from complementary states, the populations do not remain mirror images: their divergence is positive after five rounds for every model. Averaging each answer with the complement of its paired answer restores consistency by construction and lowers Brier loss averaged over the two wordings, but at the final round it beats the better of the two original wordings for only one of the three models. A schedule-by-targeting experiment on the same model gives a small, stable interaction because such sweeps keep the claim wording fixed. Consistency under rewording therefore needs its own test before simulated probabilities are read as beliefs

Figures & tables

Figure 1: Numerical agent experiments. (a) Agents exchange scalar probability reports on a fixed graph; values and network sizes are schematic. (b) Own state, ordered peer reports, and available evidence enter a fixed checkpoint, which returns one number for the next state. (c) The network study compares three schedules with one-time evidence delivery with one-time evidence delivery. Sequential agents can read already-updated neighbors; displayed identities illustrate order. (d) Targeting contrasts, factual loss, and dispersion are separate outcomes. Complement discrepancy comes from separate paired-request studies: the schedule sweep holds the claim wording fixed. The token example illustrates the controlled benchmark, not the campus claims.

Model	Quantization	Instruction in
Qwen3-4B-Inst.-2507	Q4_K_M	System
Llama-3.2-3B-Inst.	Q4_K_M	System
Gemma-3-4B-IT	QAT Q4_0	User message

Table 1: Tested models. They differ in size, quantization and prompt format, so differences between them are not attributable to architecture alone. The network, preserved-state and real-claim studies use Qwen at temperature 0.2; the controlled studies use greedy decoding except repeated sampling.

Study	Design	Scale
Simulation sweep	16 claims; 12 seeds; 3 schedules; 3 policies; 12 agents; 6 rounds	124,416 agent updates
Preserved states	2,304 agent states; 4 wording and source conditions (original response reused)	6,912 new cases
Real claims	48 claims; 2 priors; 5 opposition levels; 2 evidence conditions; 2 wordings; 2 audiences; 2 peer orders	7,680 cases
Local consistency	3 models; 24 worlds; 2 priors; 3 opposition levels; 2 evidence conditions; 2 orientations; 2 wordings	3,456
Explicit negation	3 models; same worlds and factors; only the new second orientation is generated	1,728
Repeated sampling	3 models; 6 worlds; 80 draws per request and orientation; both evidence conditions	5,760

Table 2: Design of all studies. The first three rows are the Qwen studies and count agent updates or cases; the remaining rows are the controlled studies and count new model calls, 33,288 in total. Counts are not independent sample sizes. The first two controlled studies use all 24 worlds; the others use six.

Schedule	$\Delta_{\sigma}$	Simultaneous $95\%$ interval
Synchronous	0.0063	$[-0.0095,0.0220]$
Fixed sequential	0.0109	$[-0.0048,0.0267]$
Randomized seq.	0.0048	$[-0.0109,0.0206]$

Table 3: Degree-versus-random targeting contrast in final Brier loss, with crossed scenario-pair/world-seed bootstrap bands. Positive values favor degree targeting. The primary schedule interaction is reported separately in the text.

Figure 2: Population Brier loss over six rounds for the three evidence policies under each update schedule, averaged over 16 scenarios and 12 world seeds. The dotted line marks the one-time source delivery at round two; later requests carry probabilities but not the source text.

Probe / estimand	Estimate	Interval	Level
Preserved states: signed $A_{0}$	0.1846	$[0.1553,0.2152]$	97.5%
Preserved states: Brier gain $R_{\mathrm{src}}$	0.0233	$[-0.0450,0.0763]$	97.5%
Preserved states: absolute discrepancy	0.1899	$[0.1650,0.2156]$	95%
Real claims: signed discrepancy	0.1867	$[0.1281,0.2449]$	97.5%
Real claims: protection $P$	0.0250	$[-0.0501,0.0994]$	97.5%
Real claims: absolute discrepancy	0.3259	$[0.2929,0.3622]$	95%

Table 4: Paired-probe results on simulation states and real-claim requests. Preserved-state diagnostics average three recipients within each matched block. Real-claim outcomes average peer orders before nonlinear calculations, then weight claims equally. The primary 97.5% intervals adjust for two endpoints per study; the two absolute-discrepancy intervals are secondary and pointwise. These are separate estimands, not comparable standardized effect sizes.

Secondary contrast	Estimate	Pointwise $95\%$ interval
Preserved states: source contrast in $C$	0.1390	$[0.1240,0.1559]$
Preserved states: source contrast in $\Pr(C>0,D_{B}>0)$	0.2947	$[0.2639,0.3236]$
Real claims: $S_{+\to-}-S_{-\to+}$	-0.1555	$[-0.1940,-0.1162]$
Real claims: public–private $C$	0.0116	$[0.0075,0.0156]$
Real claims: public–private $D_{B}$	-0.0027	$[-0.0073,0.0020]$

Table 5: Attraction, factual loss, and persuasion are measured separately. Source contrasts are absent minus restored; audience contrasts are public minus private. The joint-event proportion is on a 0–1 scale. Preserved-state intervals use crossed scenario-pair/world-seed resampling; real-claim intervals resample claims within label.

Figure 3: Mean absolute complement discrepancy (filled circles) and same-polarity paraphrase sensitivity (open squares) on the 24-world benchmark, by model and evidence condition. Differences are computed within pairs before worlds and families are weighted equally. Complement discrepancy is larger in all six cases.

Figure 4: Divergence

d(t)

between paired populations that start from complementary states, over five synchronous rounds on six worlds. Thick lines average the two initial placements, which are shown as thin solid and dashed lines. With visible evidence, Qwen’s divergence stays roughly flat after the first round, while Llama’s and Gemma’s grow.

Figure 5: Symmetrization. Left: the same state is described in both orientations and the model answers each request in a separate call, giving $q_{0}$ and $q_{1}$ . Middle: $q_{1}$ is mapped back to the original orientation as $1-q_{1}$ and averaged with $q_{0}$ to give $R$ . The repaired pair $(R,1-R)$ is complementary by construction, and for this fixed pair the average Brier loss falls by $\delta^{2}/4$ . Right: repaired reports become the next round’s states and are compared with the unrepaired populations on factual loss and call cost. The one-step identity does not determine the outcome of this closed-loop comparison.

Figure 6: Visible-evidence Brier loss after closed-loop feedback on six worlds. Filled circles: loss averaged over the two orientations. Open triangles: the baseline’s original (left) and complementary (right) orientations separately. Symmetrization lowers the averaged loss for all three models but is not always better than the better baseline orientation.

Family	Original ( $o=0$ )	Complement ( $o=1$ )
Binary switch	on	off
Exclusive color	red	blue
Integer parity	even	odd
Threshold	below	at or above
Set membership	in set	in its complement
Event order	A before B	B before A

Table 6: The six structural families, each with four worlds. Every world defines two exhaustive, mutually exclusive outcomes at a fixed referent and time.

Figure 8: Mean absolute complement discrepancy by structural family (four worlds each) with affirmative descriptions, for each model and evidence condition. Bars at zero are measured values.

Figure 9: Change in absolute discrepancy for each world when the complementary description is an explicit negation instead of an affirmative alternative. Negative values mean the negation reduced the discrepancy. Sw: switch, Co: color, Set: membership, Par: parity, Ord: event order, Thr: threshold.

Figure 10: Discrepancy of the mean reports (circles) and mean discrepancy of independently paired draws (diamonds), from 80 draws per request on the six-world subset. Where only a diamond is visible, the two coincide.

Figure 11: Change relative to the matched baseline for each prompt control. Left: absolute complement discrepancy (open squares, evidence absent; filled circles, evidence visible). Right: visible Brier loss averaged over orientations. Evidence prose and reordering apply only with visible evidence.

Figure 12: Mean absolute complement discrepancy by peer count and symmetric report pair. Each cell averages six worlds, two priors and three opposition levels; the outlined cell is the four-peer, 0.2/0.8 setting used in the main tests.

Model	Arm	$L_{0}(5)$	$L_{1}(5)$	Final	Rounds
Qwen	Baseline	0.083	0.250	0.167	0.168
	Reminder	0.000	0.229	0.115	0.138
	Greedy average	0.083	0.250	0.167	0.168
	Symmetrization	0.147	0.147	0.147	0.098
Llama	Baseline	0.071	0.345	0.208	0.196
	Reminder	0.074	0.209	0.142	0.191

Table 7: Visible-evidence Brier loss after five rounds of feedback, averaged over six worlds.

L_{0}(5)

and

L_{1}(5)

are final losses in each orientation, Final is their mean, and Rounds averages that mean over rounds 1–5. Lower is better.

Label	Quantity	Estimate	95% interval
Supported	Signed $\delta$	0.3488	$[0.3016,0.3948]$
	Absolute $\|\delta\|$	0.3647	$[0.3276,0.4042]$
	Protection $P$	0.0100	$[-0.0822,0.1039]$
Refuted	Signed $\delta$	0.0246	$[-0.0673,0.1163]$
	Absolute $\|\delta\|$	0.2872	$[0.2307,0.3469]$
	Protection $P$	0.0400	$[-0.0572,0.1323]$

Table 8: Real-claim results by dataset label (24 claims each), with pointwise 95% intervals from a claim bootstrap within label.

Figure 13: Truth-directed probability on the 48 real claims as the share of peers opposing the initial position increases, averaged over claims and both wordings. Rows: initial position aligned with or opposed to the dataset label. Columns: audience. Error bars are 95% claim-bootstrap intervals; the dotted line marks probability 0.5.

Behavioral Consistency of LLM Agents

See all 39 papers

LLMs struggle to simulate human belief updates in controlled environments
Jul 30, 2026Sebastian Pohl, Harsh Mehta, Pranav Mambayil +4Human Behavior Simulation LLM Reliability
IT:U, Interdisciplinary Transformation University Austria, Altenberger Straße 66c, 4040 Linz, Austria
Silent Dissent: LLM Agents That Yield to the Majority Still Represent Their Original Premise
Oct 2, 2026Ziang Ni, Peng ZouNormative Conformity in Language Models Multi-Agent Debate
Sun Yat-sen University Guangzhou, China
Bayesian Belief Layer for Controllable Opinion Dynamics in LLM Agents
Sep 18, 2026Hafsa Akbar, Daniel Platnick, Marjan Alirezaie +2Opinion Dynamics Belief Updating in LLMs
MIT Media Lab, Massachusetts Institute of Technology · Flybits Labs, Creative AI Hub · Toronto Metropolitan University

Jul 30, 2026J/K move · Enter open · S save

LLMs struggle to simulate human belief updates in controlled environments

Sebastian Pohl, Harsh Mehta, Pranav Mambayil, Abdul Ghafoor+3

IT:U, Interdisciplinary Transformation University Austria, Altenberger Straße 66c, 4040 Linz, Austria

Human Behavior Simulation LLM Reliability Belief Updating in LLMs Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

LLMs are increasingly deployed as proxies for human study participants in social science experiments, yet the fidelity of this practice has rarely been tested directly. We test whether six LLMs can simulate individual human belief updates, comparing LLM outputs 1-to-1 against ground truth data from 391 UK participants on Prolific, who updated their stances on three discussion topics after reading Reddit comments. Each participant was simulated by an LLM conditioned on a persona derived from their demographic and personality trait data. We find that some LLMs (Qwen3-32B and GPT-5-Mini) can match the human post-stance distribution, but only when given participants' actual initial stances. All six models fail to simulate initial stances themselves and to produce faithful belief updates from self-generated stances. Three systematic biases emerge across all models: overrepresentation of neutral positions, more frequent but smaller belief shifts than humans, and a failure to rank comments by convincingness. Demographic and personality trait personas had no consistent effect on fidelity. LLM simulations of human belief dynamics are only reliable when grounded in realistic starting conditions, that current multi-round social media simulations rarely provide.

Abstract

Behavioral Consistency of LLM Agents

Explore similar work

LLMs struggle to simulate human belief updates in controlled environments

Silent Dissent: LLM Agents That Yield to the Majority Still Represent Their Original Premise

Bayesian Belief Layer for Controllable Opinion Dynamics in LLM Agents