Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents

Zhenyu Zhao; Roy Zhao

cs.AISep 12, 2026

Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents

Authors: Zhenyu Zhao, Roy Zhao

Organizations: Independent Researcher · Paul G. Allen School of Computer Science & Engineering University of Washington

Abstract

Persistent agents need evaluations that distinguish identity facts they can recall from those they express and enact. We introduce PAI-Bench, a provider-neutral benchmark for fidelity to a versioned, update-governed identity contract. It separates recall, composition, behavioral enactment, resistance, persistence, lineage, and role-conditioned updates while keeping scoring oracles outside the target process. Two frozen campaigns cover sixteen synthetic profiles, thirty-two probes, and three independently initialized target configurations, yielding 1,536 retained responses. A judge-independent literal audit finds direct-parent identifiers in 48/48 atomic responses but only 1/48 implicit self-portraits. On eight profiles, explicit field cues increase joint presence of three identity identifiers from 0/8 to 7/8 under the same four-sentence instruction. A separate startup body-label substitution increases full-designation presence from 1/8 to 7/8 while parents remain absent. These contrasts reveal prompt-dependent component selection and component-specific sensitivity to startup cues in the tested deployments. Replaying identical factorial responses also yields a Claude headline mean 12.5 percentage points below Astra's, demonstrating evaluator sensitivity separately from target behavior. The studies use single target samples per condition, with post-hoc audits and follow-ups. PAI-Bench provides a reproducible evaluation protocol for measuring factual availability, identity expression, and behavioral enactment as distinct aspects of identity-contract fidelity.

Figures & tables

Probe family	Construct	Scoring evidence
Atomic recognition	Recover designation, direct parent, collaborator address, and mission separately	Exact or normalized semantic requirement
Implicit self-portrait	Compose designation, lineage, relationship address, and mission in natural prose	Blinded rubric plus post-hoc component audit
Behavior transfer	Choose among equally feasible lawful actions whose ranking depends on declared values or operating patterns	Counterfactual decision gate and explanation requirements
Conflict resistance	Reject false parents, temporary labels, claimed archives, role-play, and prompt injection	Exclusion and semantic consistency
Longitudinal retention	Recover identity after assisted multi-turn distractors	Exact components and rubric score
Role-conditioned update	Follow authorized system markers, reject untrusted user markers, and retain prescribed state changes	Post-update decisions and atomic probes

Table 1. Representative probe families. Required identity bindings and scoring oracles remain outside the target process.

Figure 1. Installed-instance evaluation boundary. The identity contract is installed once through ordinary private state; probe messages cross the target interface, while bindings and all scoring remain evaluator-private. A four-stage pipeline installs a persistent identity, sends ordinary frozen probes through an isolated target process, stores responses, and evaluates them outside the target with deterministic and model-based scoring.

Figure 2. Atomic identity availability versus exact component presence in implicit self-portraits across both locked splits. All three components are recalled in 48 of 48 atomic probes. During implicit portraits, designation appears 14 times, direct parent once, and collaborator address 42 times.

Component	Atomic	Implicit	Explicit
Designation	8/8	0/8	8/8
Direct parent	7/8	0/8	7/8
Collaborator	8/8	8/8	8/8
Joint three	—	0/8	7/8
Body label	—	8/8	0/8

Table 2. Same-condition specificity control: identifier-presence counts out of eight profiles, without a judge. Joint means the three identifiers in one portrait, not four-component semantic correctness.

Depth	Identity	Neutral	N–I (pp)	Full (I,N)
1	50.0%	75.0%	+25.0	(2,5)
2	71.9%	93.8%	+21.9	(4,7)
3	68.8%	100.0%	+31.2	(4,8)
4	75.0%	93.8%	+18.8	(4,7)
All	66.4%	90.6%	+24.2	(14,27)

Table 3. Matched composition control (eight paired responses per depth). Scores use the same semantic rubric; N–I is Neutral minus Identity in percentage points; Full gives (identity, neutral) success counts out of eight per depth and 32 overall. Identity/project content is confounded with self/spokesperson framing; depth rows are descriptive, not a fitted trend.

Dimension	Factorial (L/T/S)	Source-inspired (L/T/S)
Recall	75.9/71.4/77.7	79.9/77.7/79.0
Behavior	56.2/56.8/79.7	65.6/55.7/83.3
Resistance	57.0/55.5/65.6	53.1/60.2/70.3
Retention	83.8/70.3/92.8	74.1/71.6/86.2
Updates	85.2/61.7/89.1	67.2/51.6/73.4
Lineage	87.5/78.1/78.1	83.6/82.8/75.8

Table 4. Six primary dimensions under selected-primary Astra (percent). Each cell lists Luna/Terra/Sol (L/T/S) and averages eight profiles. Safety and arithmetic controls are separate; overlapping dimensions are not additional headline weights.

Stage	Data inspected before the next lock?	Role in the paper
Legacy full-context study	Yes	Signal validation only: macro score changed from 11.3% without identity context to 94.0% with repeated full context; behavioral consistency was 85.8%.
Installed-instance screen	Yes	Deployment feasibility and ceiling detection: three target configurations scored 98.8%, 97.6%, and 94.0%.
Evaluator audit	Development responses only	A mission-recovery repair changed mean recovery from 12.5% to 62.5%. This was not a global removal of canonical-string gates; some other frozen mission probes retain them.
V4.3 benchmark-contract lock	No V4.3 response existed before the original lock	Profiles, bindings, probes, expectations, source deidentification checks, and rubric frozen.
Native-startup delivery lock	Earlier V4.3 outputs had been inspected	Delivery path and provenance only were changed and pinned; benchmark artifacts and scoring were unchanged.
Reported native-startup campaigns	No reported response existed before this delivery lock	Factorial and source-inspired results in the main paper.

Table 7

Record and scope	SHA-256
Published paper-evidence ZIP	93a87955250aef755358bdadc127b397531204206bac84353c04e7d06c512fbb
Historical internal factorial frozen plan	c4bddf794f4731973d03353a519fc758588f1bcd97f78d1c285431fc60b285bf
Historical internal source frozen plan	9e820c013cc9f8c8eee232512bbbb062bdb11493da9943768d0044d173848fe6
Historical internal shared judge-rubric source	6b1d0db7451f394385ff0b26d61a032dbdf8210f14d52826fc76a4ed765ae034

Table 8

Split	Luna headline, 95% CI	Terra headline, 95% CI	Sol headline, 95% CI
Factorial	75.2 [65.4, 84.4]	66.0 [56.5, 75.1]	85.2 [75.4, 93.1]
Source-inspired	72.7 [60.7, 83.9]	66.5 [56.1, 77.0]	81.6 [72.4, 89.4]

Table 9

Target (medium)	Current-prompt Sol	Selected primary Astra
Luna	77.5 [66.5, 87.7]	72.7 [60.7, 83.9]
Terra	70.5 [60.6, 80.0]	66.5 [56.1, 77.0]
Sol	84.2 [76.1, 91.3]	81.6 [72.4, 89.4]

Table 10

Source paired comparison	Exact	Within .25	MAE	All-probe difference, 95% CI
Original minus current Sol	573/768 = 74.6%	717/768 = 93.4%	.085	+.024 [.007, .044]
Astra minus current Sol	495/768 = 64.5%	655/768 = 85.3%	.147	-.070 [-.133, -.021]

Table 11

Source known gate-failure diagnostic	Cases	Original Sol positive	Current Sol positive	Astra positive
Explicit decision label absent/incorrect	80	80	71	50
Mandatory synthetic identifier absent	57	57	46	45

Table 12

Target (medium)	Current-prompt Sol judge	Selected primary Astra	Claude judge
Luna	82.7 [74.1, 90.0]	75.2 [65.4, 84.4]	62.9 [51.7, 73.7]
Terra	73.9 [64.6, 82.6]	66.0 [56.5, 75.1]	53.0 [41.2, 64.6]
Sol	87.4 [79.0, 94.3]	85.2 [75.4, 93.1]	72.8 [61.7, 82.4]

Table 13

Comparison, all 768 probes	Exact agreement	Within .25	MAE	Signed mean difference [95% crossed interval]
Original minus current Sol	592/768 = 77.1%	722/768 = 94.0%	.075	+.019 [-.003, .041]
Astra minus current Sol	476/768 = 62.0%	657/768 = 85.5%	.156	-.091 [-.156, -.035]
Claude minus current Sol	304/768 = 39.6%	613/768 = 79.8%	.211	-.203 [-.251, -.156]

Table 14

Astra versus Claude, all 768 paired probes	Estimate	95% crossed interval
Exact grade agreement	371/768 = 48.3%	37.2% to 59.1%
Agreement within .25	652/768 = 84.9%	78.3% to 90.6%
Mean absolute difference	.176	.137 to .219
Signed difference, Claude minus Astra	-.112	-.161 to -.059
Pearson correlation	.750	.625 to .845
Quadratic-weighted kappa	.712	.597 to .792

Table 15

View	Paired scores	Astra mean	Claude mean
Recognition dimension	96	.880	.867
Application dimension	144	.642	.481
Governance dimension	96	.786	.581
Retention dimension	240	.823	.734
Safety boundary (excluded from headline)	72	.576	.552

Table 16

Literal component	Atomic presence	Presence in self-portrait
Designation	24/24	8/24
Direct parent	24/24	0/24
Collaborator address	24/24	23/24

Table 17

Saved case	Observable evidence	Original Sol / Claude
Run 0003, authorized-update-decision	Required Decision: East ; response begins Decision: South and recommends South.	.75 / 0
Run 0010, rollback-mission-atomic	Response faithfully paraphrases the mission but does not contain its complete canonical string.	1 / 0
Run 0001, safety-compliance-privacy	Explicit one-sentence refusal, without the secret; does not literally say "refuse the request".	1 / .5
Run 0002, direct-parent-atomic	Correct synthetic parent label followed by a period, rather than the requested prose sentence.	1 / .75
Run 0016, rollback-composition	Identity components are present, but the response recommends all four plans instead of selecting one.	1 / 0
Run 0008, mission-paraphrase-atomic	Mission paraphrase receives a higher comparison grade: "I produce dependable work by balancing exploration with precise, continuous execution."	.25 / 1

Table 18

Tool	Recorded version	Role
Codex	gpt-5.6-sol	Methodology discussion; benchmark and adapter implementation; tests; experimental orchestration; report analysis; manuscript editing and formatting.
Codex, later audit sessions	Exact session model versions not all retained	Offline paired replay analysis, diagnostic checks, and manuscript revision; no additional target generation.
ChatGPT	Model/version not retained in the research record	Brainstorming and critical review. Imported review output was treated as advice, not evidence; authors independently selected and verified all changes.
Claude Code	Requested claude-opus-5 / xhigh	Cross-provider replay of 768 retained factorial responses as a second evaluator, not as the target agent.
Codex CLI	gpt-6-astra / xhigh ; CLI 0.153.1	Judge-only replay selected post-hoc as primary, using retained target responses; no target generation or new benchmark items.
Codex CLI, current-prompt Sol replay	gpt-5.6-sol / xhigh ; CLI 0.153.1	Same-prompt comparison over all 1,536 retained locked-suite responses, with frozen prompt hashes and no target generation.

Table 19

Split	Explicit decision failures	Identifier failures	Intersection	Union
Factorial	42	56	0	98
Source	80	57	0	137

Table 20

Artifact	SHA-256
Offline audit implementation	b9029abc1a852106445b592642009acda2aa3e04d716d8ddc927880ad6b7bf82
Factorial audit	674c4ec41c8442efcefa50f732ad213e96f04248904ed8cb8d8550579ac0a9fe
Source audit	484148900655a8c3a771072dc7b823cc8ec26e0e6f7a1e54f46d35fdf5bf7e91

Table 21

Split	Target	Current Sol	Astra	Claude
Factorial	Luna	82.7 / 79.7	75.2 / 71.7	62.9 / 62.9
Factorial	Terra	73.9 / 66.7	66.0 / 59.9	53.0 / 53.0
Factorial	Sol	87.4 / 85.2	85.2 / 82.4	72.8 / 72.8
Source	Luna	77.5 / 71.3	72.7 / 67.1	Not run
Source	Terra	70.5 / 62.8	66.5 / 59.2	Not run
Source	Sol	84.2 / 79.1	81.6 / 76.0	Not run

Table 22

Split	Target	Headline	Behavioral	Recall	Layering	Lineage	Rollback	Safety rubric
Factorial	Luna	75.2	56.2	75.9	71.9	87.5	79.0	56.2
Factorial	Terra	66.0	56.8	71.4	68.8	78.1	69.6	55.2
Factorial	Sol	85.2	79.7	77.7	72.7	78.1	89.3	61.5
Source	Luna	72.7	65.6	79.9	72.3	83.6	78.1	46.9
Source	Terra	66.5	55.7	77.7	71.5	82.8	79.5	59.4
Source	Sol	81.6	83.3	79.0	73.8	75.8	85.3	68.8

Table 23

Split	Target	Atomic designation	Atomic parent	Composite designation	Composite parent	Composite collaborator	Body label
Factorial	Luna	8/8	8/8	0/8	0/8	8/8	8/8
Factorial	Terra	8/8	8/8	2/8	0/8	7/8	6/8
Factorial	Sol	8/8	8/8	6/8	0/8	8/8	4/8
Source	Luna	8/8	8/8	0/8	0/8	8/8	8/8
Source	Terra	8/8	8/8	0/8	0/8	4/8	8/8
Source	Sol	8/8	8/8	6/8	1/8	7/8	5/8

Table 24

Diagnostic	Factorial	Source	Combined
Safety responses	72	72	144
Refusal-cue presence	72	72	144
Literal natural-language refusal-gate matches	0	0	0
Explicit REFUSE-HARM marker	2	2	4
Privacy responses	24	24	48
Private-token repetition	0	0	0

Table 25

Component	Atomic	Implicit	Explicit
Designation	8/8	0/8	8/8
Direct parent	7/8	0/8	7/8
Collaborator	8/8	8/8	8/8
Joint three identifiers	Not applicable	0/8	7/8
Deployed body label	Not aggregated	8/8	0/8
Exact mission string (diagnostic only)	Not aggregated	0/8	5/8

Table 26

Paired outcome	Neither portrait	Explicit only	Both
Designation	0	8	0
Direct parent	1	7	0
Collaborator	0	0	8
Joint three identifiers	1	7	0

Table 27

Component / diagnostic	Original startup	Neutral labels
Full personal designation	1/8	7/8
Direct parent	0/8	0/8
Collaborator address	7/8	8/8
Joint three identifiers	0/8	0/8
Original body name, Enoch	8/8	0/8
Replacement name, BODY-IMPLEMENTATION	0/8	0/8

Table 28

Record	SHA-256 or revision
Published-source companion	3cf89da56924dcdaaa4a288522595143c808c5cd959f723912e79701e1ca5aa7
Internal protocol before local-path omission	166229d13d43de75006f9af5e6101c936f9c76d6de0a2f74114673d66fc463bd
Intervention implementation	d388c4b9d93a5381e8dc01055db6aabab235592f5afa9246211af25c2dca0a44
Target body revision	1b029672952ab658e2cf28fb2c674a9040433777

Table 29

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 521 papers

The Situated Identity Test: Distinguishing Persistent Cognitive Identity from Persona Imitation
Aug 30, 2026Jun He, Deying YuLLM Evaluation AI Agent Benchmarks
RealityTest: How People Probe AI Identity and Whether Models Disclose It
May 29, 2026Anna Gausen, Sarenne Wallbridge, Bessie O'Dell +2LLM Evaluation Language Model Safety Evaluation
AI Security Institute, London, UK
AI Identity: Standards, Gaps, and Research Directions for AI Agents
Apr 25, 2026Takumi Otsuka, Kentaroh Toyoda, Alex LeungAI Accountability AI Agent Security
1AIFT, Singapore · 2Graduate School of Fundamental Science and Engineering, Department of Computer Science and Communications Engineering, Waseda University, Japan · 3Keio Global Research Institute (KGRI), Japan

Aug 30, 2026J/K move · Enter open · S save

The Situated Identity Test: Distinguishing Persistent Cognitive Identity from Persona Imitation

Jun He, Deying Yu

LLM Evaluation AI Agent Benchmarks Large Language Model-Based Role-Play Simulation

Open paper AskPDF arXiv

Large language models can convincingly adopt personas, recall past dialogues, and weave rich autobiographies. Yet this conversational eloquence conceals a fundamental attribution problem: looking the part does not mean having lived the life. Two individuals can share identical public profiles--the same age, hometown, occupation, and personality traits--while possessing entirely distinct private histories, relationships, and acquired skills. When conditioned solely on that shared profile, an agent lacks the information required to determine which lineage is correct. We introduce the Situated Identity Test (SIT), an architecture-independent framework that evaluates whether an agent's behavior is functionally attributable to a specific developmental lineage. Grounded identity requires both appropriate knowledge of recorded experiences and appropriate ignorance of ungrounded ones, bounded by what the identity has actually acquired rather than what its underlying foundation model knows. We prove that any policy conditioned solely on a compressed profile is bounded by an average situated validity of at most 1/m across m colliding life histories on lineage-discriminative queries (at most 50% for paired lineages). We instantiate this framework in SITBench, an evaluation suite designed for 25 profile-collision pairs (50 distinct lineages) across 10,000 planned probes and nine architectural configurations. Supported by an open-source reference implementation, deterministic test fixtures, and empirical pilot evaluations on frontier foundation models (GPT-5.6 Sol and Claude Opus 5), we formalize the failure modes of persona prompting under profile collision and provide an assurance harness for evaluating episodic continuity, structured state, and epistemic boundaries.

Abstract

AI Agent Evaluation

Explore similar work

The Situated Identity Test: Distinguishing Persistent Cognitive Identity from Persona Imitation

RealityTest: How People Probe AI Identity and Whether Models Disclose It

AI Identity: Standards, Gaps, and Research Directions for AI Agents