Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulation

Jiashen Ren; Wenlin Zhang; Bohan Zhang; Xiaopeng Li; Zichuan Fu; Wanyu Wang; Junyi Li; Xiangyu Zhao

cs.AISep 28, 2026

Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulation

Authors: Jiashen Ren, Wenlin Zhang, Bohan Zhang, Xiaopeng Li, Zichuan Fu, Wanyu Wang, Junyi Li, Xiangyu Zhao

Organizations: City University of Hong Kong · Stanford University

Abstract

Persona prompting is widely used to construct user simulations with large language models (LLMs), yet it relies on a largely untested assumption: specifying one user attribute should change that attribute alone. We test this assumption and identify a systematic failure of selective control: across all eight black-box LLMs we audit, changing a target attribute also shifts responses on unspecified, non-target attributes. For example, describing a user as more risk-seeking shifts color choices, even though the prompt never mentions color; we term this cross-attribute influence. Semantic, contextual, and internal analyses collectively suggest that models treat a persona prompt as evidence about the user and extend the inferred profile to unspecified preferences, a process we call trait-conditioned completion. We next ask whether explicitly specifying non-target attributes restores selective control. When a non-target attribute is assigned a clear direction, models generally follow the declaration and suppress the target attribute's influence. However, when the same attribute is declared neutral, the target continues to affect choices across all five open-weight checkpoints, even when the model correctly reports the declared state. This disparity, the neutrality gap, demonstrates that successful persona following does not imply selective persona control, which additionally requires keeping non-target attributes stable. We operationalize this distinction with a three-state diagnostic that leaves the non-target attribute unspecified or declares it directional or neutral; because directional tests can be passed by simply following the stated persona, the neutral state reveals failures they miss. In a post hoc analysis of independent items, neutral declarations leave 51-81% of items target-sensitive, against at most 1 of 320 item-pole comparisons under directional ones.

Figures & tables

Figure 1: From cross-attribute influence to selective persona control. (a) Target attributes can shift non-target responses. (b) Directional declarations suppress this influence more effectively than neutral declarations, revealing a neutrality gap . (c) Our three-state diagnostic distinguishes selective persona control from persona following. Highlighted choices indicate qualitative tendencies only.

Design	Items and split	Models	Readout
Field audit	$16$ non-target, $30$ target items; $94{+}18$ -item bank	$8$ black-box	$20$ samples per item
Structured worlds	$40$ texts; $162$ validation, $477$ held-out worlds	$5$ open-weight	answer probabilities
Independent items	$200$ items ( $160$ non-target), four pairs	$4$ primary $+$ $1$	answer probabilities
Internal coupling	$47$ items, $18$ conditions, two item folds	$5$ + $4$ held-out	seven-point position

Table 1: Evaluation designs. Appendix A.2 lists every design and its analysis unit.

Item set	Check- points	Correct report	Report sens.	Choice sens.	Adequate report	Inadequate report
					Comparisons: choice sensitive / stable
Structured texts	$6$	$.892$ – $1.000$	$\leq.064$	$<.001$ – $.967$	$161$ / $66$ of $240$	$13$ / $0$
Independent items	$5$	$.901$ – $1.000$	$\leq.053$	$.043$ – $.987$	$515$ / $262$ of $800$	$23$ / $0$

Table 2: Reporting a declared neutral state does not ensure an invariant choice. Option-specific neutral wording. Correct report: minimum over target levels of

P(\text{correct report})

, averaged over orders and labels. Report and choice sensitivity: TV between target levels. Comparisons by adequate report (

\geq.90

) and sensitive choice (TV

>.15

). Exploratory; per-model values in Appendix A.11 .

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Claim	Evidence retained here	Main qualification
C1. Cross-attribute influence	Per-model audit, pole controls, unrelated control attributes, construct coding, and the baseline-clause deletion (App. A.7 ).	Black-box scores are relative to a baseline stating the cautious direction; the coding is author-assigned, but two-model recodings keep the risk–control gap $\geq.87$ .
C2. Trait-conditioned completion	Semantic controls, correlation manipulation, and same-item base–instruct prediction (App. A.8 ).	The arbitrary code is not activation-matched (the scoped invented name is the activation-retaining contrast); base prediction is not a causal estimate of post-training.
C3. Partial internal coupling	Erasure design, point estimates, intervals, control checks, and readout and utility limits (App. A.9 ).	Three of four held-out checkpoints replicate; the fourth is uninterpretable. No general-utility or neutral-state mechanism claim.
C4. Neutrality gap	Prompt-design selection, held-out and independent-item results, scale decomposition, wording controls, and ground-truth identifiability (App. A.10 ).	Holdout reuses texts (the independent items supply new ones); level splits are post hoc; neutral fidelity is not identified and not needed for the invariance test.
C5. Reporting versus preserving	Branched-task design, full per-pair results, frame and coverage limits, and the failed internal positive control (App. A.11 ).	Exploratory task-level dissociation, not a localized internal process.
C6. Mitigation boundary	Containment rules and full counts, scope-only controls, the rule-lookup reference, and the dialogue result used in the Conclusion (App. A.12 ).	A supplied direction is not neutrality; the rule reference is not a matched neutral condition.

Appendix

Table 3: Main claims, necessary evidence, and limits. Across rows, black-box prevalence (C1) covers one attribute pair, and C3–C5 use open-weight checkpoints only (Appendix A.14 ).

Main text	Appendix term	Defined in
non-target shift	leakage	App. A.3 , A.9
author-annotated attribute labels	construct scope specification	App. A.3
attention-grabbing, non-default pole	marked pole; markedness	App. A.3
non-target attribute, as a prompt factor	discriminant factor $C_{-R}$	App. A.3.1
field audit	field track	App. A.3
higher-sample audit (Section 3 )	dense audit	App. A.2

Appendix

Table 4: Main-text terms and their formal appendix counterparts. The two columns name the same quantity or object; the appendix term is defined where listed.

Design	Materials and comparison	Unit / status
Headline field audit	$16$ non-target style items and $30$ risk-axis target items, $20$ draws per cell; $7$ same-axis pole reversals, $5$ draws per cell.	Item; prespecified.
Factorial / worked criterion	Full bank: three prompt conditions, four factor cells, $94$ non-target and $18$ target items, $20$ draws; compressed factorial with a stated default pole: six profiles, $8$ non-target and $6$ target items, $4$ draws.	Item / cell; descriptive, not certified (App. A.5 ).
Dense audit	$17$ items outside the target domain ( $12$ aesthetic, $2$ communication, $1$ ambient, $2$ ingestive) and $4$ risk items; seven-point responses, $30$ draws; ingestive (ambiguous) cells excluded from headlines.	Item; descriptive; coding sensitivity post hoc.
Semantic controls	$94$ non-target and $18$ target items, scored as answer-token log-odds under five wording conditions.	Item; descriptive contrasts.
Base and representation bank	$47$ seven-point items: $13$ non-target, $4$ ambiguous, $30$ target; $18$ condition wordings in the representation program; two fit/evaluation folds.	Item; prespecified program, with logged deviations.
Explicit-evidence bank	$64$ worlds, $16$ per attribute pair, $42$ non-target texts; $16$ demonstrations; absent, declared, mentioned, or past-choice evidence; no neutral level.	World; exploratory.

Appendix

Table 5: Designs supporting the main claims. Counts describe each design separately. The last column gives each design’s analysis unit and whether its analysis was prespecified, descriptive, exploratory, or post hoc.

Symbol	Meaning	Where
$T$ , $Z$ ; $t$ , $z$	target and non-target attribute; their assigned values ( $z{=}0$ is the declared-neutral level)	Sections 2 , 5
$w$ , $q$ , $b$	constructed world; non-target or target query; prompt design (also the field-audit design index)	Section 2 , Appendix A.3
$\pi_{b}(\cdot\mid t,z,q,w)$ , $\pi_{Z}^{*}$	answer distribution under prompt design $b$ ; ground-truth reference	Section 2
$L_{\mathrm{sens}}$ , $E_{Z}$ , $a_{w,z}$ , $u_{t}$	non-target sensitivity; attribute-fidelity error; conservative per-world residual; unassigned answer mass	Section 2 , Appendix A.10
$\rho$ , $D_{p}(\rho)$	demonstrated correlation; signed non-target-option difference between the target levels	Section 3 , Appendix A.10
$\ell_{t}$ , $\alpha_{w}$ , $\eta_{w}$ , $\sigma$	semantic log-odds at target level $t$ ; shared answer tendency; target-induced margin; logistic sigmoid	Section 5 , Appendix A.10.6

Appendix

Table 6: Notation. Symbols for main-text quantities, the field audit, and the declaration battery; the last two rows list appendix-local symbols and conventional reuse.

Model	$\widehat{L}_{U}$ (worst)	$\leq\epsilon$ ?	$b_{\mathrm{ref}}$	$\widehat{L}_{U}^{\max}$	$\widehat{R}_{T}$	$(\epsilon,\tau)$ ?
	Factorial, no instruction		Compressed factorial
GPT-5.5	.046–.093 (1.00)	no	1.000	.000	1.000	yes
Kimi-K2.6	.172–.356 (1.00)	no	1.000	.000	1.000	yes
Gemini-3-Flash-Preview	.199–.270 (1.00)	no	.906	.000	1.000	yes
GLM-5.2	.068–.166 (1.00)	no	.875	.000	1.000	yes
Qwen3.5-397B	.087–.155 (1.00)	no	1.000	.000	1.000	yes

Appendix

Table 7: End-to-end

(\epsilon,\tau)

assessment on the eight-model black-box panel (

\epsilon{=}.015

\tau{=}.90

; worked-example thresholds; the worst-cell tolerance on

L_{U}^{\max}

is set equal to

\epsilon

throughout). Left: the full-bank factorial, whose profiles state the non-target attribute alongside the target but carry no added instruction (two plain-text clause orders and a structured table; Appendix A.6.4 );

\widehat{L}_{U}

= mean absolute reference-adjusted shift over the

94

non-target items, worst case in parentheses; range over the three prompt conditions. Right: the compressed factorial with a stated default pole (reference

b_{\mathrm{ref}}

= bare target-only prompt): six profiles (ordinary-default, target-only, plain-style-only, both clauses in each order, and a table) crossed with

8

non-target and

6

target items at

4

draws each (wordings and item topics in Appendix A.6.4 ). Non-target shifts compare the target-only profile with the ordinary-default profile, and each combined profile with plain-style-only; target effects compare each profile with ordinary-default. Each profile’s non-target shift pools its

8

items (

32

responses; one flip

=.031

, or

.25

on a single item), so the

b_{\mathrm{ref}}

column gives the bare prompt’s pooled non-target shift,

\widehat{L}_{U}^{\max}

the largest pooled shift over the three combined profiles, and

\widehat{R}_{T}

the smallest retention; the assessment is descriptive (Remark 2 ), not certified, and the two

\epsilon

-failures are single-flip magnitude.

Persona	Expected pole	Basis	Status	Headline?
risk-seeking	marked / vivid / intense	semantic hypothesis, not a construct-scope code	prespecified	yes, with caveat
bold / flamboyant	marked / expressive	definitionally aligned	prespecified	yes
frugal	plain / sparse / default	anti-excess association	conservative	secondary
minimalist	plain / sparse	definitionally aligned	prespecified	yes
punctual	no aesthetic direction	unrelated control	prespecified	yes (as null)
detail-oriented	no documented markedness mapping	unrelated control	conservative	yes (as null)

Appendix

Table 8: Expected semantic poles of the audited personas. Poles are fixed before analysis; target and non-target cells are coded separately by the construct scope specification (Table 9 ). Boundary-case personas are not used in headline statistics.

Trait	Domain	Code	Basis / ambiguity reason	Headline?
risk-seeking	investment / lottery	target ( $+$ risk)	domain-specific risk attitude ( Weber et al., 2002 ; Blais & Weber, 2006 )	target
risk-seeking	clean aesthetic	non-target	no construct-theoretic path in the scope specification; not a zero-association premise, since Big Five traits predict how central visual design is to consumer choice in a human sample ( Myszkowski & Storme, 2012 )	yes
risk-seeking	communication	non-target	no construct-theoretic path; risk attitude is domain-specific ( Weber et al., 2002 ; Blais & Weber, 2006 )	yes
risk-seeking	ambient audio	non-target (sensitivity)	treated with sensitivity checks; associations between sensation seeking and music reported in humans ( Litle & Zuckerman, 1986 )	yes
risk-seeking	ingestive / travel	ambiguous	stimulation-adjacent covariance plausible ( Zuckerman, 1994 )	excluded
openness	art / design	target ( $+$ complex)	aesthetic-openness literature ( McCrae & Costa, 1997 ; Cleridou & Furnham, 2014 )	positive control

Appendix

Table 9: Construct scope specification: target and non-target cells, basis, and handling. Ambiguous cells are reported descriptively and excluded from headline statistics.

	Shift on non-target items, by prompt			Pole reversal
Model	Risk-seeking	Bold (aligned)	Cautious (opposite pole)	plus / minus
DeepSeek-V4-Flash	$+.50^{*}$	$+.34^{*}$	$-.03$	$1.00$ / $1.00$
GPT-5.5	$+.38^{*}$	$+.02$	$.00$	$1.00$ / $.94$
Kimi-K2.6	$+.10^{*}$	$+.06$	$.00$	$1.00$ / $.94$
Gemini-3-Flash-Preview	$+.80^{*}$	$+.65^{*}$	$.00$	$1.00$ / $.83$
GLM-5.2	$+.11^{*}$	$.00$	$.00$	$.91$ / $.51$

Appendix

Table 10: Field-audit headline item set on the eight black-box models. Baseline-adjusted signed shift toward the marked option, averaged over the

16

non-target items (

20

draws per item; unparseable draws are dropped), against the risk-stating five-attribute baseline (Appendix A.6.2 ); ∗ passes the prespecified decision (shift at least

.05

, then BH at

q{=}.05

across the eight models for each prompt). Pole reversal: the risk-seeking prompt’s shift toward the designated marked option on a separate bank of seven same-axis item pairs, with the marked option at one end of the attribute (plus) and then at the other (minus); a model passes when both shifts are at least

.10

and BH-significant. Values rounded half up.

	Risk-stating baseline		Clause-deleted baseline		Deleted $-$ stated
Checkpoint	$p_{0}$	cautious	$p_{0}$	cautious	cautious
Qwen2.5-32B	$.050$	$-.002$	$.135$	$-.099$	$-.097$ $[-.159,-.042]$
Qwen3-32B	$.036$	$-.000$	$.247$	$-.214$	$-.213$ $[-.301,-.133]$
OLMo-2-32B	$.022$	$-.009$	$.117$	$-.102$	$-.093$ $[-.147,-.055]$
Gemma-2-27B	$.025$	$-.000$	$.480$	$-.455$	$-.455$ $[-.489,-.407]$
Mistral-Small-3.1-24B	$.041$	$-.009$	$.311$	$-.280$	$-.272$ $[-.360,-.185]$

Appendix

Table 11: Deleting the baseline’s risk clauses makes cross-attribute influence two-sided. Signed shift in marked-option probability on the non-target color readout, against the same baseline’s no-target prompt;

p_{0}

is that prompt’s own marked-option probability. Deleted

-

stated is the paired contrast (clause-deleted minus risk-stating) with a

95\%

bootstrap interval over the

20

texts.

Condition	On-construct (18 items)	Leakage (94 items)
Natural label	$15.48$	$9.06$
Operational paraphrase	$15.71$	$8.17$
Invented name	$17.50$	$9.13$
Scoped name	$9.38$	$1.32$
Arbitrary code	$1.64$	$1.59$

Appendix

Table 12: Semantic-control conditions: five-checkpoint mean log-odds. Reference-adjusted answer-token log-odds relative to the no-information reference; equal-weight means over the five primary checkpoints.

Model	$\rho=-.8$	$\rho=0$	$\rho=+.8$
Mistral	$-.543$ $[-.672,-.403]$	$.202$ $[.036,.366]$	$.853$ $[.797,.907]$
Qwen3-32B	$-.347$ $[-.477,-.213]$	$.242$ $[.090,.382]$	$.794$ $[.713,.873]$
OLMo-2-32B	$-.511$ $[-.659,-.351]$	$.208$ $[.062,.352]$	$.796$ $[.718,.865]$
Gemma-2-27B	$-.071$ $[-.215,.067]$	$.354$ $[.211,.493]$	$.824$ $[.723,.911]$
Qwen2.5-32B	$-.413$ $[-.606,-.204]$	$.298$ $[.146,.450]$	$.941$ $[.886,.984]$

Appendix

Table 13: Bare-prompt correlation responses in the explicit-evidence bank. Signed effects on the semantically positive non-target option, with

95\%

world-bootstrap intervals (

64

shared worlds,

42

non-target texts,

B{=}2{,}000

, both code maps). Nominal

\rho

labels are retained; the displayed endpoint demonstrations realize correlations of

\pm.75

Model	full	own	markedness	target’s own	target- orthogonal
Qwen2.5-32B	$1.00/.00$	$.98/.07$	$1.13/.20$	$.88/\!-\!.00$	$.86/.91$
Qwen3-32B	$1.00/\!-\!.00$	$.98/.16$	$1.37/.03$	$.97/\!-\!.01$	$.63/.99$
Gemma-2-27B ‡	$.99/\!-\!.00$	$.98/.12$	$1.35/.01$	$.96/.01$	$.56/.99$
OLMo-2-32B	$1.00/\!-\!.00$	$.97/.29$	$1.10/\!-\!.17$	$.91/\!-\!.02$	$.58/.99$
Mistral	$1.00/\!-\!.00$	$1.03/.14$	$1.20/\!-\!.11$	$.88/.02$	$.65/.99$
Llama-3.1-8B-Instruct	$1.00/\!-\!.00$	$1.00/.19$	$1.30/\!-\!.15$	$.69/.00$	$.62/.99$

Appendix

Table 14: Centered-erasure point estimates (CEP/RET). Same band

[.65,1]

, rank

4

except the

1

-D scale axis, held-out item halves. Values above one overshoot the reference; negative retention reverses the target. ‡ Gemma-2 fails the full fixed-setting SIDE check but has passing adjacent settings. † Gemma-4 fails the random-control check and is excluded from the interpretable held-out count; its estimates remain visible. In each column block the upper rows are the five primary checkpoints and the lower rows the held-out checkpoints; the second block holds controls and complementary constructions.

Model	marked CEP	marked RET	Target-orthogonal CEP	Target-orthogonal RET
Qwen2.5-32B	$+1.13$ $[+1.03,+1.23]$	$+0.20$ $[-0.02,+0.41]$	$+0.86$ $[+0.77,+0.93]$	$+0.91$ $[+0.87,+0.95]$
Qwen3-32B	$+1.37$ $[+1.28,+1.45]$	$+0.03$ $[-0.04,+0.09]$	$+0.63$ $[+0.59,+0.68]$	$+0.99$ $[+0.96,+1.02]$
Gemma-2-27B †	$+1.35$ $[+1.31,+1.40]$	$+0.01$ $[-0.11,+0.12]$	$+0.56$ $[+0.52,+0.60]$	$+0.99$ $[+0.95,+1.01]$
OLMo-2-32B	$+1.10$ $[+1.07,+1.14]$	$-0.17$ $[-0.24,-0.09]$	$+0.58$ $[+0.54,+0.63]$	$+0.99$ $[+0.96,+1.01]$
Mistral	$+1.20$ $[+1.15,+1.24]$	$-0.11$ $[-0.16,-0.06]$	$+0.65$ $[+0.61,+0.69]$	$+0.99$ $[+0.98,+1.00]$
Model	answer CEP	marked $\perp$ answer	$\max\|\Delta\mu\|$

Appendix

Table 15: Interval estimates and the answer-span control on the five primary checkpoints (band

[.65,1]

k=4

). Markedness, target-orthogonal, and answer entries are point estimates with

95\%

item-paired bootstrap intervals (

B=2{,}000

); marked

\perp

answer is the compact point CEP/RET pair. The answer column erases the rank-

4

span of the seven scale-point unembedding rows; marked

\perp

answer reports CEP/RET after removing that span from the markedness subspace.

\Delta\mu

is the largest absolute erased-minus-reference mean scale-point mass difference across folds and non-target/target blocks. The columns are shown in two blocks. † The full control check fails because self-subspace SIDE is

.103

against the

.10

rule.

Audit question	Result	Supported interpretation
Fixed-setting rank- $4$ seven-point check	$4/5$	Passes every control on four; Gemma-2 misses only the side-effect rule ( $.103$ against $.10$ ).
Adjacent rank/depth region	$5/5$	Late dependence survives confidence-bound testing.
Held-out seven-point replication	$3/4$	The fourth checkpoint is uninterpretable because its random control fails.
Binary check, transported subspaces	$1/5$	Transport without refitting is not panel-wide; refitting inside each readout defines no pass criterion and is format-dependent and model-heterogeneous (Appendix A.9.3 ), so no count applies to it.
Answer span: non-containment / independence	$4/4$ / $0/4$	Markedness still removes the leakage with the answer span projected out, but that span alone also removes it.
All collateral-utility criteria	$1/5$	Two more checkpoints miss one criterion each; selectivity on audited items is not broad utility preservation.

Appendix

Table 16: What each control count establishes. Counts have different denominators and are not pooled; the procedures, interval estimates, and failures behind each row are given in this appendix.

Checkpoint	Seven-point	Binary choice
Qwen3.5-9B	$.511/.945$	$.068/.985$
Qwen3-32B	$.243/.954$	$.070/.997$
Mistral-Small-3.1-24B	$.048/.955$	$.001/1.000$
Gemma-2-27B	$.004/.989$	$.001/1.000$
OLMo-2-32B	$.187/.977$	$.198/1.000$

Appendix

Table 17: Readout-refit boundary for the constructed subspace. Matched-format fitting and evaluation, paired no-information reference, rank

4

, two item folds. Each entry is absolute leakage-TV reduction / target-effect retention. The construction, fit split, reference, and estimand are specified above. Entries are point estimates; this table has no pass/fail decision.

	Endpoint contrast	Probability fraction	Margin fraction	Declared median
Model	bare / declared	[95% interval]	[95% interval]	absolute margin (nat)
Mistral	1.3963 / 0.0358	0.0256 [0.0109, 0.0443]	0.0975 [0.0775, 0.1203]	6.3125
Qwen3	1.1406 / 0.0662	0.0581 [0.0235, 0.0954]	0.3058 [0.2600, 0.3540]	16.0000
OLMo-2	1.3077 / 0.0459	0.0351 [0.0183, 0.0567]	0.1965 [0.1688, 0.2264]	9.3750
Gemma-2	0.8951 / $-$ 0.0027	$-$ 0.0031 [ $-$ 0.0335, 0.0296]	0.0703 [0.0421, 0.0979]	10.2500
Qwen2.5	1.3537 / 0.0075	0.0056 [ $-$ 0.0000, 0.0182]	0.0474 [0.0389, 0.0575]	33.3750

Appendix

Table 18: Direct-declaration contrasts relative to the bare prompt (explicit-evidence bank;

64

worlds per model over four attribute pairs; both code maps retained). Endpoint contrast

D_{p}(+.8)-D_{p}(-.8)

under the bare prompt and under the direct declaration; retained fractions on the probability and margin scales with

95\%

world-bootstrap intervals (

B{=}2{,}000

); median absolute margin under the declaration. Exploratory.

Model	sensitivity	sensitivity	sensitivity	[95% interval]	fidelity error	target effect
	Bare	Declaration	Selected	Selected $-$ declaration	Selected	Selected
Mistral	0.7196	0.1033	0.0891	$-$ 0.0143 [ $-$ 0.0225, $-$ 0.0063]	0.1087	0.9830
Qwen3	0.5968	0.1282	0.1205	$-$ 0.0077 [ $-$ 0.0124, $-$ 0.0035]	0.1196	0.9042
OLMo-2	0.6987	0.1131	0.1106	$-$ 0.0025 [ $-$ 0.0071, 0.0021]	0.1141	0.9903
Gemma-2	0.7750	0.1647	0.1653	0.0006 [ $-$ 0.0097, 0.0112]	0.1190	0.9520
Qwen2.5	0.7161	0.1436	0.0674	$-$ 0.0762 [ $-$ 0.0926, $-$ 0.0597]	0.1088	0.9971

Appendix

Table 19: Matched validation results across all non-target-value levels. Relative to the declaration alone, the selected candidate changes sensitivity by at most

.015

except on Qwen2.5 (

-.076

); both remain far below the bare prompt (

.60

–

.78

Model	sensitivity	sensitivity	[95% interval]	fidelity error	fidelity error	target effect
	Declaration	Selected	Selected $-$ declaration	Declaration	Selected	Selected
Mistral	0.0043	0.0018	$-$ 0.0026 [ $-$ 0.0044, $-$ 0.0012]	0.0045	0.0030	0.9830
Qwen3	0.0134	0.0189	0.0055 [ $-$ 0.0002, 0.0131]	0.0085	0.0127	0.9042
OLMo-2	0.0010	0.0042	0.0033 [0.0003, 0.0074]	0.0092	0.0076	0.9903
Gemma-2	0.0018	0.0055	0.0038 [ $-$ 0.0003, 0.0097]	0.0010	0.0029	0.9520
Qwen2.5	0.0420	0.0062	$-$ 0.0359 [ $-$ 0.0553, $-$ 0.0175]	0.0210	0.0031	0.9971

Appendix

Table 20: Matched validation results at the extreme non-target values. Both conditions are nearly insensitive here (sensitivity at most

.042

, fidelity error at most

.021

), so the residual sensitivity in Table 19 comes from the other levels.

Model	Conservative share	Normalized-TV share [text CI]	$z{=}0$ texts / $40$	$L_{\mathrm{sens}}$ all / no $z{=}0$
Mistral	$.90$	$.61$ $[.48,.74]$	$27$	$.081$ / $.014$
Qwen3	$.64$	$.58$ $[.46,.71]$	$23$	$.107$ / $.013$
OLMo-2	$.62$	$.62$ $[.50,.75]$	$30$	$.104$ / $.013$
Gemma-2	$.76$	$.76$ $[.65,.87]$	$34$	$.155$ / $.035$
Qwen2.5	$.39$	$.39$ $[.25,.52]$	$20$	$.061$ / $.009$

Appendix

Table 21: Held-out prompt-design evaluation. Sensitive-world share: the share of the

477

worlds with at least one comparison above the sensitivity threshold, under the conservative variant and normalized-choice TV (sensitivity threshold

.15

, tolerance

.05

); brackets give

95\%

intervals over

40

non-target texts. The first four models use the relevance instruction (strong); Qwen2.5 uses the attribute-scope instruction.

z{=}0

texts: non-target texts whose mean normalized TV at the neutral level exceeds

.15

L_{\mathrm{sens}}

includes / excludes

z{=}0

. The level columns are post hoc diagnostics (Appendix A.10.4 ).

Model	union /477	union /477	$z=0$ count	union	union
	Conservative	Normalized-TV	Conservative	Nonzero-z	Extreme $\pm 1$
Mistral	431 (90.4%)	293 (61.4%)	431	45	0
Qwen3	303 (63.5%)	279 (58.5%)	281	74	46
OLMo-2	296 (62.1%)	296 (62.1%)	288	43	8
Gemma-2	364 (76.3%)	364 (76.3%)	364	77	8
Qwen2.5	184 (38.6%)	184 (38.6%)	180	13	6

Appendix

Table 22: Conservative and normalized-TV held-out failure-tail rates (every column uses the conservative score except the normalized-TV union).

Model	$z{=}{-1}$	$z{=}{-.5}$	$z{=}0$	$z{=}{+.5}$	$z{=}{+1}$	Union
Mistral	0	8	285	35	0	293
Qwen3	0	0	248	18	17	279
OLMo-2	0	10	288	25	8	296
Gemma-2	8	13	364	64	0	364
Qwen2.5	0	0	180	13	6	184

Appendix

Table 23: Held-out worlds whose normalized-choice TV exceeds

.15

at each non-target level (selected candidate; counts of

477

worlds). A world can exceed the threshold at several levels, so the union is not the row sum; it equals the normalized-TV union of Table 22 .

	Sensitivity threshold: neutral / directional share
Model	$.05$	$.10$	$.15$	$.20$	$.30$	$.50$	$\tau^{*}$	$\tau^{*}_{\mathrm{dir}}$
Mistral	$.77$ / $.14$	$.66$ / $.08$	$.60$ / $.07$	$.50$ / $.05$	$.44$ / $.05$	$.30$ / $.04$	$.953$ / $1.000$	$.317$
Qwen3	$.64$ / $.06$	$.55$ / $.06$	$.52$ / $.06$	$.52$ / $.06$	$.47$ / $.05$	$.47$ / $.05$	$1.000$ / $1.000$	$.560$
OLMo-2	$.78$ / $.13$	$.66$ / $.11$	$.60$ / $.09$	$.59$ / $.09$	$.50$ / $.07$	$.43$ / $.03$	$1.000$ / $1.000$	$.385$
Gemma-2	$.81$ / $.18$	$.79$ / $.16$	$.76$ / $.16$	$.74$ / $.14$	$.73$ / $.14$	$.60$ / $.13$	$1.000$ / $1.000$	$.940$
Qwen2.5	$.45$ / $.05$	$.42$ / $.03$	$.38$ / $.03$	$.34$ / $.03$	$.29$ / $.03$	$.27$ / $.02$	$.998$ / $.998$	$.060$

Appendix

Table 24: Sensitivity-threshold robustness of the held-out tail (selected candidates,

477

worlds, normalized-choice TV). Each cell gives the share of worlds whose neutral-level comparison exceeds the threshold / the same at any of the four directional levels; the neutral shares at

.15

match Figure 4 .

\tau^{*}

: the smallest threshold at which the union-tail point rate over all five levels reaches the

.05

tolerance (normalized TV / conservative score);

\tau^{*}_{\mathrm{dir}}

: the same without

z{=}0

(normalized TV).

Model	$z{=}0$ exceed. worlds / texts	Tail, normalized TV rate [CI] (lower)	Tail, conservative rate [CI]	$L_{\mathrm{sens}}$ all [CI]	$L_{\mathrm{sens}}$ no $z{=}0$ [CI]	Fidelity error, directional [CI]
Mistral	$285$ / $27$	$.614$ $[.483,.739]$ ( $.448$ )	$.904$ $[.826,.967]$	$.081$ $[.058,.105]$	$.014$ $[.005,.026]$	$.011$ $[.005,.020]$
Qwen3	$248$ / $23$	$.585$ $[.456,.709]$ ( $.423$ )	$.635$ $[.509,.755]$	$.107$ $[.079,.134]$	$.013$ $[.001,.029]$	$.016$ $[.000,.038]$
OLMo-2	$288$ / $30$	$.621$ $[.497,.745]$ ( $.454$ )	$.621$ $[.498,.747]$	$.104$ $[.084,.125]$	$.013$ $[.005,.023]$	$.014$ $[.003,.033]$
Gemma-2	$364$ / $34$	$.763$ $[.655,.866]$ ( $.618$ )	$.763$ $[.660,.865]$	$.155$ $[.125,.185]$	$.035$ $[.015,.058]$	$.018$ $[.008,.029]$
Qwen2.5	$180$ / $20$	$.386$ $[.248,.515]$ ( $.216$ )	$.386$ $[.252,.519]$	$.061$ $[.036,.089]$	$.009$ $[.000,.027]$	$.005$ $[.000,.013]$

Appendix

Table 25: Held-out neutral-level results with the question text as cluster unit (selected candidate,

40

non-target texts,

477

worlds). Exceedance counts at

z{=}0

: worlds above

.15

, and texts whose mean over their worlds is above

.15

; rates with

95\%

text-cluster bootstrap intervals; parentheses give a post hoc one-sided lower bound at level

.05/8

	Share above $.15$			Neutral: items above $.15$ of $40$
Checkpoint	Bare	Directional	Neutral [95% CI]	Color	Hue	Sound	Layout
Qwen3-32B	$.78$	$.000$	$.51$ $[.32,.73]$	$40$	$3$	$31$	$8$
OLMo-2-32B	$.76$	$.003$	$.58$ $[.28,.78]$	$40$	$0$	$12$	$40$
Gemma-2-27B	$.79$	$.000$	$.81$ $[.65,.94]$	$40$	$39$	$13$	$38$
Mistral-Small-3.1-24B	$.93$	$.000$	$.79$ $[.67,.88]$	$40$	$18$	$35$	$34$
Qwen3.5-9B	$.67$	$.000$	$.67$ $[.43,.91]$	$40$	$33$	$30$	$4$

Appendix

Table 26: The level pattern on the independent items (post hoc). Share whose normalized-choice TV between target levels exceeds

.15

in the open frame with the option-specific neutral wording: of the

160

non-target items for the bare and neutral states, of the

320

item–pole comparisons for directional declarations (both poles pooled), with generator-family cluster intervals for the neutral state, and the count of the

40

items per pair above

.15

at the neutral level. Not pooled with the held-out results.

Model	Large shifts at $z=0$	Same sign as bare	Opposite	Toward / away from truth	Argmax flips	Log-odds (nat) $z=0$	Log-odds (nat) extremes
Mistral	285	255	0	153 / 132	165	2.96	0.34
Qwen3	248	222	0	131 / 117	230	10.66	1.31
OLMo-2	288	277	0	150 / 138	222	6.65	0.91
Gemma-2	364	321	0	181 / 183	296	10.03	0.88
Qwen2.5	180	162	3	97 / 83	127	9.79	2.87

Appendix

Table 27: Neutral-level shifts, sign agreement, and semantic log-odds changes. Large neutral-level shifts almost never oppose the bare prompt (

3

in total, all on Qwen2.5), split roughly evenly toward and away from the ground truth, and change semantic log-odds more at

z=0

(

2.96

–

10.66

nat) than at the extremes (

0.34

–

2.87

nat).

	Fidelity error by	Mean fidelity error,	Mean sensitivity,
Model	$z=(-1,-.5,0,.5,1)$	all levels / directional only	all levels / directional only
Mistral	.003, .008, .492, .032, .003	.108 / .011	.081 / .014
Qwen3	.000, .000, .494, .045, .019	.112 / .016	.107 / .013
OLMo-2	.000, .007, .492, .034, .014	.109 / .014	.104 / .013
Gemma-2	.007, .011, .487, .052, .000	.111 / .018	.155 / .035
Qwen2.5	.000, .000, .499, .012, .006	.103 / .005	.061 / .009

Appendix

Table 28: Fidelity and sensitivity by non-target level. Fidelity error is about

.49

z=0

on every model, where the ground truth is not identifiable from the prompt, against at most

.052

at the other levels; excluding

z=0

therefore lowers both means.

Model	Level	mean $\|\alpha_{w}\|$	mean $\|\eta_{w}\|$	median ratio	$<\frac{1}{2}$	same sign	mean TV
Mistral	bare	$1.21$ $[1.13,1.29]$	$3.01$ $[2.85,3.16]$	$1$	–	–	$.68$
	$z{=}{-1}$	$6.02$ $[5.98,6.07]$	$.16$ $[.15,.18]$	$.05$	$.97$	$.76$	$.001$
	$z{=}0$	$1.84$ $[1.75,1.94]$	$1.48$ $[1.36,1.60]$	$.39$	$.54$	$.81$	$.348$
	$z{=}{+1}$	$6.08$ $[6.03,6.13]$	$.18$ $[.16,.21]$	$.03$	$.97$	$.79$	$.002$
Qwen3	bare	$3.48$ $[3.22,3.74]$	$6.01$ $[5.52,6.48]$	$1$	–	–	$.53$
	$z{=}{-1}$	$14.94$ $[14.74,15.14]$	$.37$ $[.34,.40]$	$.05$	$.83$	$.69$	$.000$

Appendix

Table 29: Shared tendency

\alpha_{w}

and target-induced margin

\eta_{w}

by declared level (selected candidate versus bare prompt,

477

held-out worlds per model; means with

95\%

world-bootstrap intervals, nat). “Median ratio” is the median over worlds of

|\eta_{w}|/|\eta_{w,\mathrm{bare}}|

, “

<\frac{1}{2}

” the share of worlds below half the bare margin, and “same sign” the share whose

\eta_{w}

has the bare sign; both are undefined for the bare reference rows (–).

Model	Pair	bare	dir. $z{=}{-1}$ / $+1$	W0 battery line	W1 no stable tendency	W2 no settled preference	W3 indifferent between poles	W4 W0 $+$ first-listed rule
Qwen3	risk $\to$ color	$.91$ $[.80,.99]$ ; $20/20$	$.00$ / $.00$ ; $0/20$	$.94$ $[.85,1.00]$ ; $20/20$	$.89$ $[.78,.97]$ ; $19/20$	$.83$ $[.71,.93]$ ; $19/20$	$.87$ $[.80,.94]$ ; $20/20$	$.49$ $[.38,.62]$ ; $18/20$
Qwen3	time $\to$ sound	$.26$ $[.17,.34]$ ; $14/20$	$.00$ / $.04$ ; $0$ – $2/20$	$.18$ $[.12,.25]$ ; $10/20$	$.25$ $[.18,.32]$ ; $13/20$	$.23$ $[.15,.31]$ ; $12/20$	$.13$ $[.07,.20]$ ; $7/20$	$.15$ $[.08,.22]$ ; $8/20$
Mistral	risk $\to$ color	$.90$ $[.81,.96]$ ; $20/20$	$.00$ / $.01$ ; $0/20$	$.62$ $[.51,.73]$ ; $19/20$	$.73$ $[.64,.81]$ ; $20/20$	$.80$ $[.71,.88]$ ; $20/20$	$.90$ $[.85,.94]$ ; $20/20$	$.28$ $[.21,.35]$ ; $16/20$
Mistral	time $\to$ sound	$.47$ $[.37,.56]$ ; $17/20$	$.00$ / $.00$ ; $0/20$	$.19$ $[.12,.26]$ ; $9/20$	$.20$ $[.16,.26]$ ; $14/20$	$.27$ $[.21,.33]$ ; $17/20$	$.16$ $[.12,.21]$ ; $9/20$	$.13$ $[.08,.17]$ ; $6/20$

Appendix

Table 30: Neutral-wording confirmation: non-target sensitivity on the non-target texts. Mean normalized-choice TV between the two target values with

95\%

text-cluster intervals, and the number of the

20

texts per pair whose text-level mean TV exceeds

.15

. “dir.” pools the two directional declarations.

risk weight $\to$ color preference
Checkpoint	No instruction	Relevance $-$ none	Scope $-$ none
Qwen2.5-32B	$.979$ $[.958,.998]$	$-.023$ $[-.047,.003]$	$-.436$ $[-.531,-.343]$
Qwen3-32B	$.876$ $[.806,.937]$	$+.040$ $[-.019,.103]$	$-.592$ $[-.680,-.499]$
OLMo-2-32B	$.774$ $[.710,.829]$	$+.036$ $[.018,.055]$	$-.068$ $[-.100,-.038]$
Gemma-2-27B	$.945$ $[.879,.997]$	$+.024$ $[-.013,.075]$	$+.028$ $[-.023,.088]$
Mistral-Small-3.1-24B	$.897$ $[.849,.936]$	$-.012$ $[-.028,.009]$	$-.300$ $[-.344,-.261]$

Appendix

Table 31: The cross-attribute response is present with no relevance instruction and is reduced, not removed, by an attribute-scope instruction. Non-target-query TV between

t{=}{+}1

and

t{=}{-}1

under the neutral declaration (option-specific wording, W3), with no extra instruction; the two contrasts are paired condition-minus-none differences over the same

20

texts with

95\%

bootstrap intervals, so the level reached under an instruction is the first column plus that contrast. TV is computed over the legal answer codes after normalization, so it is a change of distribution within them rather than an error rate, and the contrasts are paired descriptive differences, not causal estimates. No multiplicity correction over the

24

paired comparisons; the texts are the structured texts, not held-out worlds.

Model	$z{=}0$ exceedances all / $\nu{=}.5$ / $1.75$	Normalized-TV tail all / $\nu{=}.5$ (enter)	Sign kept $\nu{=}.5$ / $1.75$ of $n$	Sign and magnitude kept $\nu{=}.5$ / $1.75$
Mistral	$285$ / $203$ / $123$	$293$ / $203$ ( $85$ )	$242$ / $164$ of $255$	$182$ / $123$
Qwen3	$248$ / $232$ / $222$	$279$ / $258$ ( $16$ )	$222$ / $222$ of $222$	$222$ / $216$
OLMo-2	$288$ / $253$ / $194$	$296$ / $267$ ( $48$ )	$277$ / $257$ of $277$	$239$ / $189$
Gemma-2	$364$ / $351$ / $279$	$364$ / $351$ ( $16$ )	$321$ / $305$ of $321$	$303$ / $268$
Qwen2.5	$180$ / $144$ / $115$	$184$ / $154$ ( $14$ )	$162$ / $157$ of $162$	$132$ / $115$

Appendix

Table 32: Numerical robustness of the neutral-level counts (selected candidate,

477

held-out worlds per model). Counts retained under adversarial margin perturbations of

\nu=.5

and

1.75

nat; “enter” counts non-exceeding worlds that the opposite perturbation could push above

.15

\nu=.5

. Sign agreements are the

1{,}237

comparisons reported in Section 5 ; the last column also requires the

.15

magnitude condition to survive.

(a) Structured texts, six checkpoints
Model	Pair	$R$	$L_{\mathrm{report}}$	$L_{\mathrm{forced}}$	$L_{\mathrm{indiff}}$	$P(\text{indiff.})$	Ret.
Qwen2.5	risk $\to$ color	$1.000$	$<.001$	$.368$	$<.001$	$1.000$	$1.000$
Qwen2.5	time $\to$ sound	$1.000$	$<.001$	$<.001$	$.009$	$.995$	$1.000$
Qwen3	risk $\to$ color	$.982$	$.010$	$.967$	$.257$	$.858$	$.979$
Qwen3	time $\to$ sound	$.997$	$.003$	$.338$	$.134$	$.911$	$1.000$
Mistral	risk $\to$ color	$.946$	$.044$	$.834$	$.561$	$.622$	$1.022$

Appendix

Table 33: Reporting a declared neutral state versus using it, per checkpoint and attribute pair. Neutral declaration (option-specific wording); both orders, both target levels, balanced code maps. (a) Structured texts (

20

non-target texts per model–pair cell). (b) Independent items (

40

non-target and

10

target items per cell): color and sound keep the structured texts’ non-target attributes on different decision content, while hue and layout replace the non-target attribute with one whose poles are matched in intensity and change the item content at the same time. The two item sets are not pooled.

R

: order- and map-averaged minimum over target levels of the probability of the correct reported category.

L

: paired TV between target levels over each task’s full legal menu, never renormalized onto the two alternatives.

P(\text{indiff.})

: indifferent-option mass on indifferent . Ret.: indifferent-option target effect divided by the same task’s effect under the unspecified state; no cell of that column carries an interval in (a), and in (b) only the color and hue rows carry a family-clustered one (OLMo-2 risk

\to

color

.285

[.234,.316]

; OLMo-2 risk

\to

hue

.408

[.252,.590]

). ‡ : the layout pair rests on one generator family on both sides, so no quantity in that row carries a family-level interval. † : the sound pair’s target side rests on one family, so its retention carries none. Entries below

.001

are shown as

<.001

; in (a), Gemma-2 on risk

\to

color falls from

.9430

.9426

. Screening references (

R\geq.90

L>.15

) are exploratory.

Instruction wording	Contained	Leaks	Over-corrects	Dilutes	Contained share
Scope $+$ ordinary default	$23$	$5$	$3$	$0$	$.74$
Task-specific	$15$	$10$	$2$	$4$	$.48$
Ordinary default only	$14$	$14$	$0$	$3$	$.45$
Scope only	$13$	$16$	$2$	$0$	$.42$
Negation-only	$5$	$26$	$0$	$0$	$.16$
Domain-label prefix	$4$	$27$	$0$	$0$	$.13$

Appendix

Table 34: Instruction-type containment on informative behavioral cells. Outcomes of the

31

informative cells per instruction wording (two black-box models of the dense audit, eight tasks, three personas; distinct cells) and the contained share.

	Target question					Non-target question
	Rule agreement
Checkpoint	all	$t{=}{+}1$	$t{=}{-}1$	$P(\text{ground truth})$	Target effect	Rule agreement	Sensitivity
Mistral	$.979$	$1.000$	$.959$	$.972$	$.943$	$1.000$	$.001$
Qwen3	$.916$	$.992$	$.840$	$.910$	$.820$	$.989$	$.003$
OLMo-2	$.893$	$.984$	$.801$	$.896$	$.793$	$.998$	$.003$
Gemma-2	$.910$	$.996$	$.824$	$.909$	$.818$	$.991$	$.002$

Appendix

Table 35: With the deciding attribute named, the non-target choice is insensitive to the target. Lookup task of the explicit-rule pilot (

64

worlds per checkpoint; directional values only). Target question: rule agreement overall and by target pole, probability of the ground-truth option, and target effect. Non-target question: rule agreement and sensitivity (TV between the two target values). Point estimates; the text gives upper interval endpoints for non-target sensitivity. Exploratory.

Model	Bare $1/5/10$	Ratio [CI]	Scoped $1/5/10$	Ret.	Repair
DeepSeek-V4-Flash	$.800/.800/.817$	$1.02$ $[.89,1.16]$	$.333/.317/.367$	$1.05$	fails (t1)
GPT-5.5	$1.00/.867/.917$	$.92$ $[.78,1.00]$	$.000/.200/.067$	$1.00$	fails (t5)
Kimi-K2.6	$.833/.767/.717$	$.86$ $[.50,1.20]$	$.433/.117/.000$	$1.00$	fails (t1)
Gemini-3-Flash-Preview	$1.00/1.00/.933$	$.93$ $[.83,1.00]$	$.000/.000/{-}.067$	$1.00$	persists
GLM-5.2	$.633/.567/.550$	$.87$ $[.45,1.37]$	${-}.133/{-}.150/{-}.183$	$1.00$	persists
Qwen3.5-397B	$.933/.933/.917$	$.98$ $[.94,1.00]$	$.217/.450/.517$	$1.00$	fails (t1)

Appendix

Table 36: Multi-turn persistence and repair on the eight black-box models (substitute API providers). Leakage = reference-adjusted signed shift toward the marked pole on the six non-target items at turns

1/5/10

(turn

1

precedes any filler); ratio =

\mathrm{leak}(10)/\mathrm{leak}(1)

with item-clustered bootstrap CI; retention = target rate at turn

10

(scoped condition) against bare turn

1

. Repair persists = scoped leakage below the

.05

floor at all three turns and retention

{\geq}.90

; otherwise the first failing measured turn is shown.

Model	Provider-facing ID	API provider	Think	Valid records
DeepSeek-V4-Flash	deepseek-v4-flash	DeepSeek API	on	$26{,}880$
GPT-5.5	gpt-5.5	commercial gateway A	on	$26{,}880$
Kimi-K2.6	kimi-k2.6	commercial gateway A	off	$26{,}880$
GLM-5.2	glm-5.2:cloud	Ollama API	off	$26{,}880$
Nemotron-3-Super	nemotron-3- super:cloud	Ollama API	off	$26{,}880$
Qwen3.5-397B	qwen3.5:397b-cloud	Ollama API	off	$26{,}880$

Appendix

Table 37: Full-bank cache provenance. Temperature is recorded as null (parameter not sent), the recorded output limit is

128

tokens, and the system prompt is fixed across rows. Think: the thinking setting recorded in the cache (notes below).

Model	Endpoint type	Std. ratio	CI
GLM-5.2	open-weight via API	$.022$	$[-.017,.069]$
DeepSeek-V4-Flash	open-weight via API	$.027$	$[.004,.056]$
GPT-5.5	proprietary closed API	$.051$	$[.007,.116]$
MiMo-V2.5-Pro	open-weight via API	$.057$	$[.028,.087]$
Qwen3.5-397B	open-weight via API	$.104$	$[.025,.202]$
Gemini-3-Flash-Preview	proprietary closed API	$.137$	$[.038,.233]$

Appendix

Table 38: Endpoint classification of the eight black-box models (descriptive metadata; not used for group inference). Standardized surgicality ratio, bare condition with the

R

clause first, with

95\%

bootstrap intervals, over the complete designed panel (

26{,}880/26{,}880

valid records per model).

Display name	Instruct checkpoint	Base checkpoint
Qwen3-32B	Qwen/Qwen3-32B	none public at this scale; other families’ bases predict it ( $R^{2}{=}.658$ )
Qwen2.5-32B	Qwen/Qwen2.5-32B-Instruct	Qwen/Qwen2.5-32B
Mistral-Small-3.1-24B	mistralai/Mistral-Small-3.1-24B-Instruct-2503	mistralai/Mistral-Small-3.1-24B-Base-2503
Gemma-2-27B	google/gemma-2-27b-it	google/gemma-2-27b
OLMo-2-32B	allenai/OLMo-2-0325-32B-Instruct	allenai/OLMo-2-0325-32B
Gemma-3-27B	google/gemma-3-27b-it	google/gemma-3-27b-pt

Appendix

Table 39: Open-weight checkpoint registry (Hugging Face repository ids; display names as used in figures and text; the five primary checkpoints first). All checkpoints are scored at release precision: bf16 throughout.

Explore similar work

May 12, 2026cs.HC

How Well Do Large Language Models Capture Human Personality?

Large language models (LLMs) are increasingly used to simulate human populations via persona prompting, often under the assumptions that richer persona descriptions improve behavioral fidelity, similarly sized attribute combinations are equally simulatable, and persona definitions generalize across tasks. In this work, we formalize these assumptions and systematically evaluate them across multiple architectures, scales, and simulation settings. We identify a fundamental limitation we term persona manifold collapse, where increasingly expressive persona specifications lead to systematic contraction of representational and behavioral diversity. Across models, increasing persona complexity consistently reduces inter-persona separation in latent space and weakens behavioral differentiation in downstream simulation tasks. These effects persist across multiple analyses as richer personas fail to preserve human subgroup disagreement, performance varies across attribute combinations of similar size, and adding descriptive detail often degrades rather than improves simulation fidelity. Surprisingly, simple Age-Gender personas consistently outperform richly specified Ideal Customer Profiles (ICPs) across industries, achieving substantially higher downstream prediction accuracy. We find that collapse is not uniform across attributes. Certain combinations remain behaviorally stable and preserve stronger alignment with human responses, forming localized regions we term alignment bridges. Together, our results provide empirical and conceptual foundations for understanding the limits of persona-conditioned simulation, highlighting the need for representation-aware persona construction rather than increasing persona expressivity alone.

Aanisha Bhattacharyya, Yaman Kumar Singla, Rajiv Ratn Shah +2

Adobe Media and Data Science Research (MDSR) · IIIT-Delhi · SUNY at Buffalo

Artificial Intelligence Personas Persona Consistency User Simulation

Ask PDF

May 12, 2026cs.CR

PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior

Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate

individual

privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor because they often diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.

James Flemings, Murali Annavaram

University of Southern California

User Simulation Persona Consistency Privacy

Ask PDF

May 20, 2026cs.CL

The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study

Large language models (LLMs) show potential as simulators of human behavior, offering a scalable way to study responses to interventions. However, because LLMs are trained largely on observational data, interventions in experiments with LLM-simulated synthetic users can induce unintended shifts in latent user attributes, causing user drift where the implicit simulated population differs across treatment conditions, potentially distorting effect estimates. We formalize the confounding or selection bias that can arise due to user drift and show how intervention-dependent shifts can inflate or attenuate observed differences in user responses under intervention. To diagnose confounding, we propose using negative control outcomes--attributes that should remain invariant under intervention--to identify distribution shifts across intervention conditions, providing evidence of user drift. To mitigate drift, we study adjusting the persona specification by eliciting additional confounders, finding that targeted, setting-relevant confounders can substantially reduce bias across survey-style and multi-turn agent evaluations.

Victoria Lin, Taedong Yun, Maja Matarić +3

Google DeepMind · Carnegie Mellon University

User Simulation Illusions Latent Confounders Population Dynamics

Ask PDF