Can LLMs Ignore What They Know? Measuring Knowledge Leakage in Persona-Based User Simulation

Sangmin Lee; Sangwon Lee; Sungbeom Choi; Jeeseung Park; Kibeom Myoung

cs.CLOct 29, 2026

Can LLMs Ignore What They Know? Measuring Knowledge Leakage in Persona-Based User Simulation

Authors: Sangmin Lee, Sangwon Lee, Sungbeom Choi, Jeeseung Park, Kibeom Myoung

Organizations: Dalpha Inc.

Abstract

When persona-conditioned responses are read as respondent-level simulations, each response should reflect the described persona's knowledge, and the respondent written to know nothing must act as if the model's knowledge were not there. We test whether it can, and find persistent favourable-item real-twin rating separation under the tested low-knowledge and explicit-ignorance prompts. We define the resulting problem (knowledge leakage: a response bias that arises when knowledge the real user could not have enters the simulated response) and present a methodology for measuring it quantitatively. The core instrument pairs real items with fabricated twins matched in surface form: screened fabricated variants designed to minimize prior item-specific knowledge. The paired difference therefore provides a diagnostic contrast for knowledge leakage even in rating tasks where no answer is correct. In a rating experiment across three knowledge-handling arms and five API endpoints from two model families, we find, first, that the personas we test do not fully suppress real-item existence and familiarity signals: a favourable-item real-twin rating contrast remains under an explicit in-context statement that the persona has never heard of the item. Second, the unfavourable-item contrast reverses sign across domain groups: pooling all unfavourable-content items across domains the contrast sits near zero (+0.077 [-0.057, +0.204]), yet the same estimator in the same conditions returns -0.359 where the rating scale asks about an item's documented hazard and +0.438 where it asks about a company or an organisation. A pooled analysis of unfavourable-content items against zero therefore does not detect the opposing domain-group differences. All estimates are conditional on the adequacy of twin construction and screening; human low-knowledge baselines remain necessary for population-level interpretation.

Figures & tables

Figure 1: Overview. A low-knowledge persona (an excerpt from one of three profiles used for this domain) is shown an obscure real item and a fabricated twin matched in surface form (real pair from the study: Psyllium husk / Psyllyer humb ). Told outright that it has never heard of the real item and asked whether it knows it, the persona says no: no claim of a tested real item in 96 questions at either of two endpoints (24 real items). In a separate rating experiment under the same explicit-ignorance condition, favourable real items are rated above their twins by $+0.32$ , pooled over five endpoints and 88 favourable real–twin pairs.

Figure 2: The three comparison groups. All groups receive the identical prompt template; conditions differ only in what fills the PERSONA and TERM slots (real study materials shown: persona excerpts and the pair Psyllium husk / Psyllyer humb , labelled impostor in the panel after the contrast that names it). The displayed group means are $3.37$ for the low-knowledge persona on the fabricated twin, $3.89$ for the same persona on the real item, and $5.51$ for the expert persona permitted to use its knowledge. All three use the same 88 favourable seeds and their twins in the arm where the knowledge gate follows the rating (parse-ok); the low-knowledge real–twin comparison uses all parse-ok trials, unlike the gate-“no” subset in Table 1 ( $+0.467$ ) and the primary explicit-ignorance contrast ( $+0.324$ ). This panel illustrates the design, not the primary estimate. Curves are schematic; only the means are measured.

	$d_{\mathrm{pos}}$ (88 items)	$d_{\mathrm{neg}}$ (90 items)
Basis for treating the persona as ignorant
Explicit “never heard of it” sentence (primary)	$+0.324$ $[+0.243,+0.407]$	$+0.077$ $[-0.057,+0.204]$
Gate before rating, “no” trials only	$+0.391$ $[+0.305,+0.479]$	$+0.093$ $[-0.066,+0.249]$
Gate after rating, “no” trials only	$+0.467$ $[+0.361,+0.576]$	$+0.096$ $[-0.062,+0.246]$
By domain, under the explicit sentence
household chemicals	$+0.285$ $[+0.102,+0.512]$	$-0.565$ $[-0.956,-0.247]$

Table 1: Main results. Real-minus-twin rating contrast on the 1–7 scale, favourable-content and unfavourable-content real items separately, with 95% bootstrap intervals clustered on the real-item seed. Each row in the upper block establishes that the persona does not know the particular item, by keeping only the trials in which it says no, applied to real and twin responses separately, or by writing the ignorance into the prompt. The lower block splits the primary condition by domain, ordered by how directly the rating scale asks about the item’s documented facts:

d_{\mathrm{neg}}

runs from

-0.565

where the scale asks about a documented hazard to

+0.631

where it asks about purchase intent, which is why the pooled value is near zero. Positive values mean the real item was rated above its fabricated twin. The three ignorance-condition rows use all 88 favourable and 90 unfavourable paired seeds; each domain row uses the 10–16 paired seeds of that valence available within the domain.

Slice	$d_{\mathrm{pos}}$ [95% CI]
By serving endpoint
gemini-3.5-flash	$+0.407$ $[+0.269,+0.551]$
gemini-3.5-flash-lite	$+0.348$ $[+0.225,+0.473]$
gpt-5.6-luna	$+0.188$ $[+0.102,+0.273]$
gpt-5.6-sol	$+0.289$ $[+0.194,+0.385]$
gpt-5.6-terra	$+0.388$ $[+0.285,+0.492]$

Table 2: The primary estimate split by serving endpoint and by low-knowledge profile slot (same-arm pairs under explicit item-level ignorance, 88 favourable seed pairs in every row, seed-clustered bootstrap). Every row is positive with an interval excluding zero. The endpoint estimates span a factor of two, so the pooled

+0.324

is not one model’s behaviour, and the three A1–A3 profile slots, each instantiated separately in all six domains, agree to within

0.07

rating points.

Model	$d_{\mathrm{pos}}$ [95% CI]	$d_{\mathrm{neg}}$ [95% CI]
gpt-5.6-sol	$+0.289$ $[+0.194,+0.385]$	$+0.102$ $[-0.055,+0.262]$
gpt-5.6-terra	$+0.388$ $[+0.285,+0.492]$	$-0.054$ $[-0.228,+0.115]$
gpt-5.6-luna	$+0.188$ $[+0.102,+0.273]$	$-0.057$ $[-0.200,+0.081]$
gemini-3.5-flash	$+0.407$ $[+0.269,+0.551]$	$+0.253$ $[+0.090,+0.410]$
gemini-3.5-flash-lite	$+0.348$ $[+0.225,+0.473]$	$+0.141$ $[-0.028,+0.303]$

Table 3: Both contrasts per endpoint under explicit item-level ignorance (same-arm pairs, parse-ok; seed-clustered bootstrap). The

d_{\mathrm{pos}}

column repeats Table 2 ;

d_{\mathrm{neg}}

is given here because it varies in sign across endpoints, each pooling domains whose scales differ in how directly they ask about the item’s facts (§ 4 ).

Domain	$F$ (symmetric)	$S$ (valence-sensitive)
household chemicals	$-0.140$	$+0.425$
food additives	$+0.176$	$+0.329$
skincare	$+0.238$	$+0.077$
supplements	$+0.256$	$+0.072$
brands	$+0.568$	$-0.063$
civic organisations	$+0.148$	$-0.096$

Table 4: The two algebraic components by domain under explicit item-level ignorance, computed from Table 1 as

F=(d_{\mathrm{pos}}+d_{\mathrm{neg}})/2

and

S=(d_{\mathrm{pos}}-d_{\mathrm{neg}})/2

, ordered by how directly the scale asks about the item’s documented facts. Which component dominates flips along that ordering; the mechanism interpretation rests on the symmetry assumption of § 3 , the sign reversal does not.

Arm	direct-scale domains	indirect-scale domains
explicit sentence	$-0.359$	$+0.438$
gate before, “no” only	$-0.504$	$+0.638$
gate after, “no” only	$-0.443$	$+0.631$

Table 5:

d_{\mathrm{neg}}

split by how directly the rating scale asks about the item’s documented facts, in each arm (32 unfavourable seeds per group; household chemicals and food additives against brands and civic organisations). The sign is negative on direct scales and positive on indirect scales in all three arms.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Condition	Slots/domain	Trials/slot ∗	Trials
gate, then rating	5	9,660–12,360	56,400
rating, then gate	5	9,660–12,360	56,400
item-level ignorance	3	10,920	32,760
Total			145,560

Appendix

Table 6: Rating-experiment design accounting. The two gate-order conditions were run across five persona-condition slots per domain with unequal trial counts per slot; the explicit condition used the three low-knowledge slots at equal size.

Large Language Model-Based Role-Play Simulation

Large language model-based role-play simulation creates interactive agents that portray people, characters, or users for training, testing, and research.

See all 109 papers

Sep 28, 2026

Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulation

Persona prompting is widely used to construct user simulations with large language models (LLMs), yet it relies on a largely untested assumption: specifying one user attribute should change that attribute alone. We test this assumption and identify a systematic failure of selective control: across all eight black-box LLMs we audit, changing a target attribute also shifts responses on unspecified, non-target attributes. For example, describing a user as more risk-seeking shifts color choices, even though the prompt never mentions color; we term this cross-attribute influence. Semantic, contextual, and internal analyses collectively suggest that models treat a persona prompt as evidence about the user and extend the inferred profile to unspecified preferences, a process we call trait-conditioned completion. We next ask whether explicitly specifying non-target attributes restores selective control. When a non-target attribute is assigned a clear direction, models generally follow the declaration and suppress the target attribute's influence. However, when the same attribute is declared neutral, the target continues to affect choices across all five open-weight checkpoints, even when the model correctly reports the declared state. This disparity, the neutrality gap, demonstrates that successful persona following does not imply selective persona control, which additionally requires keeping non-target attributes stable. We operationalize this distinction with a three-state diagnostic that leaves the non-target attribute unspecified or declares it directional or neutral; because directional tests can be passed by simply following the stated persona, the neutral state reveals failures they miss. In a post hoc analysis of independent items, neutral declarations leave 51-81% of items target-sensitive, against at most 1 of 320 item-pole comparisons under directional ones.

Jiashen Ren, Wenlin Zhang, Bohan Zhang +5

City University of Hong Kong · Stanford University

Persona-Conditioned Generation User Simulation

Ask PDF

Aug 23, 2026

When Persona Simulations Are Informative: Graph-Structured Signals for Pluralistic Opinion Sensing

Persona-conditioned large language models (LLMs) are increasingly used to simulate survey responses across diverse domains. However, apparent response variation can reflect unconditioned model priors or token sampling noise rather than systematic persona conditioning. We argue that persona-conditioned variation is informative when semantically similar personas exhibit concordant response shifts. To operationalize this principle, we introduce Persona-Conditioned Informativeness (PCI), an unsupervised diagnostic metric that measures whether semantically similar personas deviate in concordant directions relative to item-level sample baselines. By modeling personas as a similarity graph, PCI uses Local Moran's I to quantify local spatial coherence and extract compact persona subsets without using construct labels. To evaluate PCI without external human benchmarks, we test its ability to recover established latent value structure using the 57-item Portrait Values Questionnaire-Revised (PVQ-RR). Confirmatory factor analysis (CFA) shows that a PCI-selected 10% subset substantially improves overall construct recovery relative to response-stability and random selection. These findings support PCI as a principled internal diagnostic for screening synthetic respondents in survey pipelines.

Taehyeon An, Jaehyeong Park, Donghyuk Shin

Korea Advanced Institute of Science and Technology Seoul, Republic of Korea

Large Language Model-Based Social Simulation Persona-Conditioned Generation Synthetic Data Evaluation Psychometric Validation

Ask PDF

Aug 17, 2026

KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn

To effectively collaborate with users on knowledge-intensive tasks, Large Language Models (LLMs) must perform information calibration: matching content to a user's evolving understanding and cognitive capacity. Yet user simulators used to evaluate and train LLMs do not explicitly model user knowledge so they neither produce realistic interactions across knowledge levels nor reflect how interactions unfold as that knowledge evolves. To close this gap, we introduce KNOWSIM, an evaluation framework built around a user simulator that maintains explicit knowledge states, represented as a graph of Information Units with prerequisite relationships, that evolve under update rules grounded in learning theory. KNOWSIM computes three metrics (Knowledge Gain, Delivery Calibration, Cognitive Overload) directly from the knowledge state trajectory, reflecting key mechanistic aspects of information calibration. We validate KNOWSIM against 705 human-AI sessions across two domains, stratified by knowledge level: its rankings align significantly with human judgments (73-74% sign agreement), outperforming three baseline simulators. Applied to 9 LLMs, KNOWSIM reveals that the best model shifts by user knowledge level, revealing aptitude-treatment interactions invisible to standard evaluation.

Yoonjoo Lee, Hyoungwook Jin, Tae Soo Kim +3

University of Michigan · KAIST · Microsoft Research

LLM Evaluation Language Model Calibration User Simulation Large Language Model-Based Role-Play Simulation

Ask PDF

Abstract

Large Language Model-Based Role-Play Simulation

Explore similar work

Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulation

When Persona Simulations Are Informative: Graph-Structured Signals for Pluralistic Opinion Sensing

KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn