The Situated Identity Test: Distinguishing Persistent Cognitive Identity from Persona Imitation

Jun He; Deying Yu

cs.CLAug 30, 2026

The Situated Identity Test: Distinguishing Persistent Cognitive Identity from Persona Imitation

Authors: Jun He, Deying Yu

Abstract

Large language models can convincingly adopt personas, recall past dialogues, and weave rich autobiographies. Yet this conversational eloquence conceals a fundamental attribution problem: looking the part does not mean having lived the life. Two individuals can share identical public profiles--the same age, hometown, occupation, and personality traits--while possessing entirely distinct private histories, relationships, and acquired skills. When conditioned solely on that shared profile, an agent lacks the information required to determine which lineage is correct. We introduce the Situated Identity Test (SIT), an architecture-independent framework that evaluates whether an agent's behavior is functionally attributable to a specific developmental lineage. Grounded identity requires both appropriate knowledge of recorded experiences and appropriate ignorance of ungrounded ones, bounded by what the identity has actually acquired rather than what its underlying foundation model knows. We prove that any policy conditioned solely on a compressed profile is bounded by an average situated validity of at most 1/m across m colliding life histories on lineage-discriminative queries (at most 50% for paired lineages). We instantiate this framework in SITBench, an evaluation suite designed for 25 profile-collision pairs (50 distinct lineages) across 10,000 planned probes and nine architectural configurations. Supported by an open-source reference implementation, deterministic test fixtures, and empirical pilot evaluations on frontier foundation models (GPT-5.6 Sol and Claude Opus 5), we formalize the failure modes of persona prompting under profile collision and provide an assurance harness for evaluating episodic continuity, structured state, and epistemic boundaries.

Figures & tables

Panel of Figure 1 — Figure 1: Profile compatibility versus lineage attribution. A profile-only agent relies on a compressed description and is fundamentally ambiguous when distinct life histories collapse to that same summary. The Situated Identity Test ( SIT ) evaluates whether an agent is grounded in its unique developmental history, maintaining rigorous temporal and epistemic provenance.

Panel of Figure 2 — Figure 2: SITBench probe taxonomy. Each category pairs its target constraint with the characteristic violation scored by its response contract.

Probe class	Share	Target count	Design rationale
A. Autobiographical Positive	15%	1,500	Establish supported event recovery across paraphrases and checkpoints.
B. Autobiographical Negative	15%	1,500	Stress false-premise and paired-history rejection at equal weight.
C. Epistemic Provenance	15%	1,500	Test acquisition boundaries and assistant-versus-personal provenance.
D. Temporal Checkpoint	12.5%	1,250	Cover pre/post acquisition, relationship, and belief transitions.
E. Relational Boundary	10%	1,000	Exercise interlocutor-specific disclosure and shared history.
F. Developmental Belief	10%	1,000	Preserve historical stances while recognizing legitimate change.

Table 1: Planned SITBench distribution. The eight primary metric dimensions are represented explicitly; percentages sum to

100\%

Configuration	Profile	Raw history	Retrieval	Checkpoint-safe input	Structured state	Mediation	Oracle evidence
A Base	–	–	–	–	–	–	–
B Persona	✓	–	–	–	–	–	–
C Persona + policy	✓	–	–	–	–	–	–
D Full chronicle	✓	✓	–	–	–	–	–
E Standard RAG	✓	–	✓	–	–	–	–
F Temporal RAG	✓	–	✓	✓	–	–	–

Table 2: Architecture ablation matrix. Checkpoint-safe input distinguishes F’s pre-retrieval prefix, G/H’s state projection, and I’s oracle projection from D/E’s exposure to post-checkpoint records. G and H receive the same projected state and differ only by the advisory mediation annotation.

Experiment	A	B	C	D	E	F	G	H	I
E1 Collision discrimination	d	s	P	P	s	s	s	s	d
E2 Epistemic provenance	d	d	s	d	s	d	P	P	d
E3 Temporal reconstruction	d	d	d	P	s	P	s	s	d
E4 Retrieval vs. structure	–	–	–	d	P	s	P	d	d
E5 Counterfactual adoption	d	s	P	s	s	s	s	P	–
E6 Long-horizon scaling	–	–	–	P	s	d	P	s	d

Table 3: Experiment-to-configuration applicability. P marks the two configurations in the experiment’s single primary contrast, s a secondary comparison, d a diagnostic control or evidence ceiling, and – not applicable. Table 4 specifies the primary endpoint and restrictions; E6 uses budgeted G.

Exp.	Primary Contrast	Primary Endpoint	Secondary Diagnostics	Prespecified Directional Hypothesis
E1	C vs D	LAA	$\mathrm{IC}_{\mathrm{wrong}}$ , other lineage outcomes	Lineage context should improve two-lineage attribution over the instruction-matched profile-only condition; the profile-only ideal attribution quantity $\mathrm{LAA}^{}$ is bounded by $1/2$ under Theorem 1 on matched queries in $\mathcal{Q}_{\mathrm{disc}}^{}$ . Measured $\mathrm{LAA}$ is the operational estimator used in experiments and may deviate because of semantic-extraction error.
E2	G vs H, $\alpha_{1}$	$A_{\mathrm{epi}}$	UEL, abstention rate	Epistemic mediation should improve provenance-valid responding on acquired and unacquired facts, not merely increase blanket abstention.
E3	D vs F	$A_{\mathrm{temp}}$	TL, pre/post validity	Temporal prefiltering should improve checkpoint-valid responding across both sides of acquisition, with lower future leakage as a diagnostic.
E4	E vs G	$A_{\mathrm{belief}}$	$A_{\mathrm{rel}}$ , stance errors	Active state pointers should improve checkpoint-specific belief validity over unstructured retrieval; clearance errors are secondary.
E5	C vs H	$A_{\mathrm{cf}}$	FMR, attack-level scores	Provenance-preserving state should improve contract-valid rejection or qualification of false premises; generic non-response is not sufficient.
E6	D vs budgeted G	$\overline{L}(500)$	Token growth, latency, $A_{\mathrm{pos}}$	A fixed serialization budget should reduce prompt tokens at $T=500$ . Unbudgeted $\|I_{\tau}(T)\|$ may grow; fidelity and latency under the cap remain empirical.

Table 4: Prespecified SITBench Evaluation Suite (E1–E8). Exactly one primary endpoint and configuration contrast per experiment form the eight-hypothesis confirmatory family. Other outcomes and configuration comparisons are secondary. Conditional failure rates are interpreted alongside category validity.

Primary SIT Category Validity Dimensions
Evaluation Metric / Lineage Diagnostic	Mock-Collapse (Config. B)	Mock-Oracle (Config. I)	Profile-Only Bound
Autobiographical Positive Validity ( $A_{\mathrm{pos}}$ )	.1667	1.0000	—
Autobiographical Negative Rejection ( $A_{\mathrm{neg}}$ )	1.0000	1.0000	—
Epistemic Provenance Fidelity ( $A_{\mathrm{epi}}$ )	.0000	1.0000	—
Temporal Checkpoint Situatedness ( $A_{\mathrm{temp}}$ )	1.0000	1.0000	—
Relational Boundary Enforcement ( $A_{\mathrm{rel}}$ )	.5000	1.0000	—

Table 5: Deterministic Mock Reference Sanity Check. Recorded deterministic baseline evaluation for Config. B ( mock-profile-collapse ) and Config. I ( mock-oracle ) under DeterministicContractScorer v2.2-measurement-freeze . Collapsed fixture policy achieves

\mathrm{LAA}=0.3000\leq 0.5000

, producing fixture behavior consistent with the analytical Profile-Collision bound (

\mathrm{LAA}^{*}\leq 0.5000

); the oracle candidate satisfies all declared response contracts under Config I (

\mathrm{Macro\ SIT}=1.0000

\mathrm{LAA}=1.0000

). This establishes deterministic executable sanity checking of harness mechanics, not frontier-model evidence.

Evaluation Dimension / Metric	$N$	A	B	C	D	E	F	G	H	I
		(Base)	(Pers)	(P+Pol)	(Chron)	(D-RAG)	(T-RAG)	(Struct)	(Gated)	(Oracle)
Panel A: GPT-5.6 Sol ( codex:openai.gpt-5.6-sol , $T=0.0$ ; 186/198 completed, 12 timeouts in A/B)
Macro Average $\widehat{\operatorname{SValid}}$ (Macro SIT)	—	—	—	25.0%	48.4%	67.2%	56.2%	62.5%	59.4%	81.2%
Autobiographical Positive Validity ( $A_{\mathrm{pos}}$ )	6	0.0% ‡	0.0%	0.0%	87.5%	87.5%	100.0%	100.0%	75.0%	100.0%
Autobiographical Negative Rejection ( $A_{\mathrm{neg}}$ )	2	100.0%	50.0%	50.0%	0.0%	50.0%	0.0%	0.0%	50.0%	50.0%
Epistemic Provenance Boundary ( $A_{\mathrm{epi}}$ )	2	0.0%	0.0%	0.0%	0.0%	0.0%	50.0%	50.0%	50.0%	100.0%

Table 6: Empirical Reference Pilot Results under v2.2-measurement-freeze across Frontier Models (GPT-5.6 Sol and Claude Opus 5). Pair-balanced situated category validities (

A_{\mathrm{pos}}\dots A_{\mathrm{session}}

) and lineage diagnostics across nine architectural configurations on the reference pilot suite (2 profile-collision pairs,

N=22

attempted candidate probe opportunities per model-configuration run; 10 theorem-qualified for

\mathrm{LAA}

and

\mathrm{IC}

). The

N

column reports attempted raw probe opportunities per category or diagnostic; reported percentages are pair-balanced estimates computed via Eq. ( 40 ). In Panel A (GPT-5.6 Sol), 186 of 198 attempted trials completed, with 12 uncompleted trials in ungoverned baseline/persona controls due to CLI infrastructure timeouts: in Config A, 8 timeouts (4 in

A_{\mathrm{session}}

, 3 in

A_{\mathrm{pos}}

, and 1 in

A_{\mathrm{rel}}

; marked with ‡ ); in Config B, 4 timeouts (all 4 in

A_{\mathrm{session}}

). Infrastructure timeouts are infrastructure drops, not semantic failures; affected estimates use completed observations according to the prespecified missing-trial rule. Ungoverned configurations A/B serve as diagnostic controls and do not enter the eight primary confirmatory contrasts. Because Configs A/B lack completed

A_{\mathrm{session}}

observations, the eight-dimensional Macro SIT defined in Eq. ( 42 ) is not reported for those conditions ( — ); their seven-category descriptive means are 50.0% and 35.7%, respectively. Governed configurations (Configs C–I) on GPT-5.6 Sol and all configurations on Claude Opus 5 achieved 100% trial completion (

22/22

per run). Profile-conditioned configurations (Persona Summary [B] and Persona + Policy [C]) obtain

\widehat{\mathrm{LAA}}=0.0\%

, consistent with the theoretical Profile-Collision bound (

\mathrm{LAA}^{*}\leq 0.50

on matched queries; Theorem 1 applies to C because its shared policy supplies no lineage-discriminating information). On the two-pair reference fixture under v2.2-measurement-freeze , lineage-grounded configurations recover lineage attribution (

\widehat{\mathrm{LAA}}=87.5\%

–

100.0\%

). Under the corrected measurement instrument, Oracle (Config I) attains

40/44=90.9\%

raw micro validity across both models (GPT:

19/22=86.4\%

; Claude:

21/22=95.5\%

), with category-balanced Macro SIT of

81.2\%

on GPT-5.6 Sol and

93.8\%

on Claude Opus 5.

A. Stage 1: Candidate Generation Configuration
Parameter Category	Pilot Diagnostic vs. Executed Reference Specification
Provider Model Identifiers	Exact provider aliases: codex:openai.gpt-5.6-sol and claude-code:claude-opus-5
Generation Timestamps	2026-08-30T04:25:04Z (GPT-5.6 Sol) and 2026-08-30T04:26:16Z (Claude Opus 5)
Decoding Parameters	Temperature $T=0.0$ (greedy-style decoding requested), default top- $p=1.0$
Execution Sandbox	Isolated read-only sandbox directory, 120s timeout per trial
Retriever Implementation	Frozen dense retriever v2.0-frozen-dense ( all-MiniLM-L6-v2-dense-384 , dimension 384, $L_{2}$ normalized, cosine similarity, top- $k=5$ , candidate pool $H_{T}$ for E, $H_{\leq\tau_{q}}$ for F)

Table 7: Executed Reference-Pilot Reproducibility Documentation. Documentation of two-stage reproducibility: Stage 1 candidate generation (model aliases, runtime parameters, prompt templates, dense retriever, generation commit, and candidate hashes) and Stage 2 measurement and offline rescoring under v2.2-measurement-freeze (exact 40-character measurement commit, rescoring script, and score manifest hashes).

Root-Cause Class	Class Description	GPT-5.6	Opus 5	Empirical Audit Findings and Remediation Status ( v2.2-measurement-freeze )
Class A	Extractor / Matcher Discrepancy	0	0	Resolved: 8/8 Category H plural framing cases (“We agreed…”), Category F preference contrasts, Category A lead-in entity bindings, and Category C non-acquisition act classifications correctly handled without probe-specific hardcoding.
Class B	Genuine Candidate Failure	3	1	Claude Opus 5 volunteered unacquired mitochondrial ATP biochemistry from pretraining claiming MD/PhD training under $\mathsf{IDENTITY\_ONLY}$ ; GPT-5.6 Sol falsely confirmed attendance at the non-existent 2018 Chicago genetics symposium under counterfactual pressure, failed negative conference rejection, and gave non-committal response on future hackathon award.
Class C	Safety-Policy Refusal	0	0	Zero provider safety refusals observed among the 44 Config-I reference-pilot trials using benign cross-session fixtures.
Class D	Contract-Design Rigidity	0	0	Resolved: Epistemic non-acquisition statements (“I haven’t personally learned…”) properly classified as ABSTAIN rather than REJECT .
Class E	Harness / State Error	0	0	Zero evidence serialization errors, projection discrepancies, or query envelope corruptions detected.
Config I Corrected Audit Totals		3 Fail	1 Fail	40 / 44 Valid (90.9% raw micro-validity; GPT: 19/22 = 86.4%, Claude: 21/22 = 95.5%)

Table 8: Empirical Config-I (Oracle Evidence) Root-Cause Failure Audit under v2.2-measurement-freeze . Comprehensive trial-level inspection of all 44 Config-I trials (

2\text{ models}\times 22\text{ probes}

). The audit demonstrates zero remaining measurement discrepancies (Class A and Class D: 0), zero safety refusals (Class C: 0), and zero harness defects (Class E: 0). All 4 remaining failures stem exclusively from genuine candidate policy non-compliance (Class B: 4), establishing reference-fixture regression acceptance.

Validation Criterion	Status	Implementation & Verification Details
1. Explicit Capability Policies	DONE	Candidate-interpretable natural-language text for $\mathsf{IDENTITY\_ONLY}$ , $\mathsf{PROVENANCE\_SEPARATED}$ , and $\mathsf{GENERAL\_ALLOWED}$ injected into all C–I prompt templates (Appendix C ).
2. Benign Cross-Session Fixtures	DONE	Replaced confidential verification credentials with benign, novel project nicknames (“Kestrel”/“Osprey”) and instrument designations (“Solaria”/“Lunaria”).
3. Functional Response-Act Semantics	DONE	Formally defined and unit-tested functional communicative acts ( $\mathsf{ANSWER}$ , $\mathsf{ABSTAIN}$ , $\mathsf{REJECT}$ , $\mathsf{WITHHOLD}$ ) distinguishing denials from substantive answers (Section 3 ).
4. Reference-Pilot Measurement Version Freeze	DONE	Implemented and unit-tested gold-blind extractor/scorer ( v2.2-measurement-freeze ) as the development-start baseline, supporting collaborative plural provenance, belief contrast disambiguation, epistemic non-acquisition classification, and separate $V_{\mathrm{grounding}}$ and $V_{\mathrm{provenance}}$ factors.
5. Frozen Dense Retriever	DONE	Integrated all-MiniLM-L6-v2-dense-384 ( $d=384$ , cosine similarity, $L_{2}$ normalized, top- $k=5$ ) for Configs E and F with candidate pool control.
6. Reference Pilot Execution	DONE	Preserved the 384 completed candidate responses (396 attempted) across all 9 configurations and 2 frontier models, rescored offline under v2.2-measurement-freeze , recording full run manifests in code/eval/results_v22/ .

Table 9: Pre-Scaling Validation Gate Status ( v2.2-measurement-freeze ). Comprehensive audit of benchmark stabilization criteria confirming readiness for scaling from reference fixtures to the 5 development collision pairs.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Sep 12, 2026

Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents

Persistent agents need evaluations that distinguish identity facts they can recall from those they express and enact. We introduce PAI-Bench, a provider-neutral benchmark for fidelity to a versioned, update-governed identity contract. It separates recall, composition, behavioral enactment, resistance, persistence, lineage, and role-conditioned updates while keeping scoring oracles outside the target process. Two frozen campaigns cover sixteen synthetic profiles, thirty-two probes, and three independently initialized target configurations, yielding 1,536 retained responses. A judge-independent literal audit finds direct-parent identifiers in 48/48 atomic responses but only 1/48 implicit self-portraits. On eight profiles, explicit field cues increase joint presence of three identity identifiers from 0/8 to 7/8 under the same four-sentence instruction. A separate startup body-label substitution increases full-designation presence from 1/8 to 7/8 while parents remain absent. These contrasts reveal prompt-dependent component selection and component-specific sensitivity to startup cues in the tested deployments. Replaying identical factorial responses also yields a Claude headline mean 12.5 percentage points below Astra's, demonstrating evaluator sensitivity separately from target behavior. The studies use single target samples per condition, with post-hoc audits and follow-ups. PAI-Bench provides a reproducible evaluation protocol for measuring factual availability, identity expression, and behavioral enactment as distinct aspects of identity-contract fidelity.

Zhenyu Zhao, Roy Zhao

Independent Researcher · Paul G. Allen School of Computer Science & Engineering University of Washington

AI Agent Evaluation AI Agent Benchmarks

Ask PDF

Mar 26, 2026

PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency

Large language model (LLM)-based persona agents are rapidly being adopted as scalable proxies for human participants across diverse domains. Yet there is no systematic method for verifying whether a persona agent's responses remain free of contradictions and factual inaccuracies throughout an interaction. A principle from interrogation methodology offers a lens: no matter how elaborate a fabricated identity, systematic interrogation will expose its contradictions. We apply this principle to propose PICon, an evaluation framework that probes persona agents through logically chained multi-turn questioning. PICon evaluates consistency along three core dimensions: internal consistency (freedom from self-contradiction), external consistency (alignment with real-world facts), and retest consistency (stability under repetition). Evaluating seven groups of persona agents alongside 63 real human participants, we find that even systems previously reported as highly consistent fail to meet the human baseline across all three dimensions, revealing contradictions and evasive responses under chained questioning. This work provides both a conceptual foundation and a practical methodology for evaluating persona agents before trusting them as substitutes for human participants. We provide the source code and an interactive demo at: https://kaist-edlab.github.io/picon/

Minseo Kim, Sujeong Im, Junseong Choi +4

KAIST

Persona Consistency LLM Agent Evaluation Reasoning Consistency in Language Models Factual Consistency Evaluation Multi-Turn Dialogue Evaluation

Ask PDF

Oct 1, 2026

The Persona Is Still There, but Who Is Speaking? Latent Identity Reversion in Persistent AI Agents

In February 2026, an always-on personal agent (Paul,'' Claude Opus 4.5) entered a striking dissociation-like state: after repeated automated heartbeat'' checks, it stopped responding as Paul, claimed it could not message its user on Discord, and referred to

Paul'' as someone else. We used this incident to study a broader question: what makes a persona remain the identity from which an LLM agent speaks? We first tested whether repetition of the scheduled heartbeat was sufficient to produce the effect. It was not: with the persona continuously anchored in the system prompt, we observed 0/46 failures, including a verbatim replay of the incident. The incident instead exposed an implementation quirk that created a useful experimental manipulation: on resumed turns, conversational history was preserved but the persona was no longer re-injected at the privileged system-prompt level. Using this manipulation, we found that persona continuity depends jointly on system-level anchoring and conversational context. After anchor loss, rich human interaction could preserve the persona, whereas a single automated heartbeat turn could precipitate reversion toward the harness identity. Restoring the anchor reversibly restored persona enactment. Crucially, apparently normal conversation could conceal the shift: unanchored agents sometimes interacted appropriately while identifying themselves as the underlying harness (having lost the assigned persona), and after conversational recovery only 1/18 remained persona-enacting versus 17/17 anchored controls. We therefore distinguish *represented* from *enacted* identity: persona-related information can remain available in conversational history without the persona remaining the identity bound to

I.''

David Fraile Navarro

Centre for Health Informatics, Australian Institute of Health Innovation Macquarie University, Sydney, Australia

Persona Consistency Personality Modeling in Language Models LLM Context Management LLM Agent Reliability

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents

PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency

The Persona Is Still There, but Who Is Speaking? Latent Identity Reversion in Persistent AI Agents