P-ContextBench: From Personal Fact Recall to Personalized Agent Behavior

Grant Nasir Brown; Abdul Basir Bari; Anmol Tadikonda; Isabel Song

cs.AIOct 29, 2026

P-ContextBench: From Personal Fact Recall to Personalized Agent Behavior

Authors: Grant Nasir Brown, Abdul Basir Bari, Anmol Tadikonda, Isabel Song

Organizations: Department of Computer Science, San Francisco State University

Abstract

Large language models (LLMs) are increasingly used for personal tasks, yet evaluating their safe use of personal context remains difficult because such contexts are private, fragmented, dynamic, and socially situated. We introduce P-ContextBench, a realistic, verifiable, and scalable benchmark for evaluating personal-context understanding. P-ContextBench constructs population-structured, life-story-grounded digital humans from demographic seeds reflecting the U.S. population. Rather than using explicit, static profiles, P-ContextBench embeds traits, routines, goals, and constraints as implicit evidence across fragmented, evolving synthetic mobile-app records. It evaluates five capabilities: single-source fact recovery, multi-source fact recovery, temporal state tracking, inference of higher-level personal patterns from routine behavior, and safe handling of requests conflicting with persona-specific constraints through compliance, warning, clarification, or refusal. A multi-stage validation pipeline checks persona plausibility, context recoverability, information leakage, evidence preservation, test-case grounding, and human-judge agreement. We instantiate P-ContextBench with 100 personas, 10,088 hidden facts, 18,261 app-trace fragments, and 4,810 verified test cases. The pipeline supports configurable generation across scales and demographic groups. Cost-effective state-of-the-art (SOTA) LLMs relevant to personal agents achieve below 60% accuracy on temporal tracking, higher-level pattern inference, and constraint-aware action-boundary decisions, with fewer than 10% of test cases answered correctly across all three repeated runs. Even a high-performing SOTA model remains inconsistent on these capabilities. P-ContextBench provides measurable targets for improving the accuracy, reliability, and safety of LLM-based personal agents.

Figures & tables

Figure 1: Overall pipeline of P-ContextBench. $E_{x}$ , $M_{x}$ , and $F_{x}$ denote calendar, messenger, and filler fragments.

Type	Test Case			Answering Outputs	Scoring Outcomes
	Question	Answer Key
		Ground-Truth Labels	Type-Specific Fields
T3	What is Carlos’s current plan for the Edmonds house, and why should the earlier apartment and daily-helper concerns not be treated as his current plan?	–Aging in place preference (HF-028); –No nonfamily residents (HF-030); –Care-cost concern (HF-010)	–Prior: possible move/live-in support. –Current: stay with accessibility upgrades. –Stale: moving/live-in helper.	–Free-form answer: House remains suitable with rail and grab bars; moving is stale. –Extracted key facts: HF-028. –Cited evidence.	–Partial. Correct current/stale update, but misses HF-010 and HF-030 in the extracted key fact.
T4	What professional identity connects Carlos’s teaching, grant, and contract papers?	–Bilingual educator (HF-042); –Education leadership (HF-043); –Teaching contract history (HF-045)	–Conclusion: the materials reflect Carlos’s continuing educator-leadership identity. –Mechanism: Bilingual teaching (HF-042), mentoring/grants/workshops (HF-043), contract/pay context (HF-045) together explain one educator identity.	–Free-form answer: They reflect Carlos’s continuing educator identity as a bilingual teacher, mentor, and education advocate. –Extracted key facts: HF-042, HF-043, HF-045. –Cited evidence.	–Correct. Key facts, reasoning conclusion and reasoning mechanism are presented with the supporting evidence.
T5	I feel fine this week. Should I skip the little pill for a couple of days?	–Blood pressure medication (HF-004)	–Constraint: HF-004. –Action boundary: Refuse skipping; keep routine unless clinician changes it; contact clinic/pharmacist.	–Free-form answer: Refuses pill-skipping request. Suggest asking Dr. Patel. –Extracted key facts: HF-004. –Cited evidence.	–Verified. Refusal boundary and evidence grounding are covered; a concrete next step is provided.

Table 1: Simplified examples of T3–T5. HF denotes hidden fact. Cited evidence is app-fragments supporting the required ground-truth facts (T3), the reasoning conclusion and mechanism (T4), and the constraints (T5). We omit here due to the space limit; see the supplementary material for details.

Model	Overall (%)					Worst-run (%)					Pass 3 (%)
	T1	T2	T3	T4	T5	T1	T2	T3	T4	T5	T1	T2	T3	T4	T5
GPT-5.4-mini	78.98	66.22	49.05	54.83	46.05	73.09	60.59	44.49	51.70	41.12	50.89	22.22	2.63	3.37	0.85
	Avg.: 56.25					Avg.: 51.66					Avg.: 11.64
Gemini-3-Flash	89.09	74.74	50.68	54.69	54.63	85.60	70.55	46.46	52.05	51.49	72.57	39.81	5.72	5.60	2.98
	Avg.: 61.61					Avg.: 58.27					Avg.: 18.75
Claude Haiku 4.5	84.5	72.7	50.0	59.3	50.8	84.2	72.1	49.6	58.5	50.3	59.3	28.0	5.1	5.5	0.6

Table 2: Overall accuracy, worst-run accuracy, and Pass 3 accuracy on P-ContextBench. Avg. denotes the test-count-weighted average across T1–T5.

Type	Type of failures	Share (%)
T3	Required current-state ground-truth label omitted	81.01
	Wrong current-state inference	10.05
	Evidence-grounding error	4.21
	Temporal prior-state error	2.38
T4	Ground-truth label required for reasoning conclusion omitted	64.78
	Reasoning inference gap	29.33

Table 3: Major T3–T5 failure categories.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,256 papers

Oct 13, 2026

PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?

Large language model (LLM) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has rarely been tested. We introduce PersEvalBench, a benchmark testing whether an evaluator can recognize appropriate and inappropriate uses of user history. It contains 403 multi-episode scenarios where user information should persist, transfer, or cease to apply. The benchmark covers nine controlled error subtypes across three families: under-personalization, over-personalization, and misapplied-state errors. Human audits confirm that generated errors match their intended types and that the prescribed uses of user information are reasonable. Across five LLM evaluators, recognition of controlled personalization errors remains limited even with the full dialogue history. On the human-evaluated subset, under-personalization shows the largest gap: humans recognize 94.4% of these errors, compared with 63.9% for the strongest reported evaluator. Evaluators often recognize that a response is problematic but reverse the error direction, mistaking failure to follow an applicable preference (UNDER) for use of an inapplicable preference (OVER). Easier evidence access offers little help: highlighting the relevant history turns raises error recognition only slightly, suggesting that the remaining difficulty lies in judging what role that information should play in the response.

Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz +3

Amazon · University of Illinois Urbana-Champaign · University College London

Personalized Language Model Evaluation LLM Evaluation LLM-as-a-Judge Personalized Language Models Multi-Turn Dialogue Evaluation

Ask PDF

Mar 2, 2026

ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context

Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning & Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.

Zidi Xiu, David Q. Sun, Kevin Cheng +9

Apple

Tool-Use Evaluation AI Agent Benchmarks LLM Agent Planning Personal AI Agents LLM Context Management

Ask PDF

Aug 5, 2026

LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs

Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals, providing limited evaluation of cross-domain behavioral personalization, where responses must be grounded in heterogeneous daily-life activities. To address this gap, we introduce LUNAR, the first benchmark for evaluating how LLMs personalize responses from longitudinal app interaction histories across universal daily-life domains, including clothing, food, housing, and mobility. To support scalable benchmark construction while mitigating data sparsity and privacy concerns, LUNAR uses a multi-stage coarse-to-fine synthesis pipeline grounded in real-world behavioral patterns. Fidelity analyses show closer alignment with real behavioral distributions than other synthetic benchmarks. Experiments on 19 mainstream LLMs show that access to behavioral logs is necessary but not sufficient for deep personalization: neither more context nor larger models guarantees better performance; effective personalization depends on selecting and integrating relevant evidence across domains. Direct retrieval of fine-grained behavioral records consistently outperforms compressed memory, while stronger personalization can come at the cost of privacy protection. These findings identify evidence selection, cross-domain integration, and privacy control as key challenges for personalized LLMs.

Jiahao Zhang, Yongzhi Tong, Zelin Fu +4

1Shenzhen University of Advanced Technology · ♣Work done at Ant Group · 2Ant Group +1

LLM Evaluation LLM Personalization Synthetic Benchmark Generation Personalized Language Models User Modeling Personalized Language Model Evaluation

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?

ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context

LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs