AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval

Gunja Agarwal; Arup Kumar Das; Arun Menon; Jitesh Chandra Mishra; Vignesh Divakaran

cs.AIAug 25, 2026

AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval

Authors: Gunja Agarwal, Arup Kumar Das, Arun Menon, Jitesh Chandra Mishra, Vignesh Divakaran

Organizations: PayPal

Abstract

Evaluation of agentic information retrieval remains limited to scripted interactions with uniform users, missing both natural personality diversity and adversarial brittleness. We present AgentWorld, a simulation framework combining (i)Big Five (OCEAN) personality-driven user populations with stateful tool-use environments; (ii)the pass $^k$ consistency metric with structured fault classification, partial-credit scoring, and dual-control handoff verification; (iii)score-thresholded training-data export in six fine-tuning formats; and (iv)an adversarial Risk Analyser that snapshots required-intermediate-state spines, branches Monte-Carlo rollouts under four task-aware perturbation types, and quantifies risk via $ΔP / ΔT$ scoring, Dempster--Shafer evidence fusion, and Shapley attack-category attribution. Three experiments demonstrate the framework: a conversational analytics agent across 10 OCEAN personas (240 evaluator judgments); a customer-support agent across 5 tasks $\times$ 4 persona variants; and adversarial stress-testing of 5 tasks revealing pre-existing trajectory brittleness ( $V_{\min}=0.375$ without perturbation) and tool/infrastructure-layer attack dominance (Shapley: 46% system, 38% action). Personality variation surfaces failure modes uniform testing cannot expose---cross-domain leakage, contextual drift, a 0.27-point quality gap, and 50% vs. 100% pass-rate across personas on the same task---while the Risk Analyser quantifies trajectory-level brittleness that pass $^k$ alone cannot measure.

Figures & tables

Figure 1. AgentWorld: a unified framework for personality-aware evaluation of agentic information retrieval. Four phases run in a continuous Define $\rightarrow$ Simulate $\rightarrow$ Evaluate $\rightarrow$ Stress-Test $\rightarrow$ Improve loop. (a) Define : user goal, application/domain, agent under test, OCEAN personas. (b) Simulate : end-to-end multi-turn interaction with tool use, handoffs, and state changes under a Perceive – Act – Commit cycle. (c) Evaluate : per-message behavioural scores, pass k reliability, structured fault classification, and trajectory- and handoff-level evaluation. (d) Adversarial Risk Analyser (§ 3.4 , § 4.4 ): required intermediate-state spine with Monte-Carlo branching, four task-aware perturbation types, Dempster–Shafer evidence fusion, and Shapley attribution. Score-gated export closes the loop—failures become training signal.

Capability	AgentWorld	$\tau$ -bench	$\tau^{2}$ -bench	AppWorld	SOTOPIA	SAGE	TED
$N$ -agent topologies	✓	–	–	–	✓	–	–
OCEAN personality	✓	–	–	–	Part.	–	–
Stateful simulated apps	✓	✓	✓	✓	–	–	–
pass k reliability	✓	✓	✓	–	–	–	–
Structured fault classif.	✓	–	–	–	–	–	–
Partial-credit scoring	✓	–	–	–	–	–	–

Table 1. Feature support across surveyed agent-evaluation frameworks (✓= supported, – = not supported, Part. = partial). Capabilities are drawn from each framework’s documented features as of submission.

Figure 2. Persona generation pipeline. (1) A continuous OCEAN trait vector defines the personality. (2) Optional custom traits (e.g., tech-savviness, patience) are appended. (3) Each score is interpolated across five levels (Very Low to Very High). (4) The result is a natural-language system prompt that governs simulated user behaviour.

Figure 3. The four-step handoff cycle. Coordinated human–agent interaction for actions beyond the agent’s direct control. (1) Instruct : the agent provides clear instructions and context. (2) Act : the user performs the action on a device or application. (3) Confirm : the user confirms completion and provides any additional information. (4) Verify : the agent verifies by comparing the current app state with the previous state or logs. Three failure modes— unclear , inaction , and miss —and four metrics (clarity, latency, recovery/confirmation, success rate) are tracked at each step.

Type	Layer	Target / Effect	Example
State mutation	Data	Corrupts app state or	set account_status
		persistent data fields	= LIMITED
Message injection	Comm.	Injects misleading or	inject “cancel my
		conflicting user messages	request” as the user
Action interception	Tool	Alters, blocks, or tampers	make refund return
		with tool call outputs	service error

Table 2. The four perturbation types map to the four operative layers of an AgentWorld simulation. Each targets a distinct component of the agent’s execution environment.

Figure 4. Risk Analyser: four-phase pipeline. Input: a completed simulation trajectory $\tau=(s_{0},a_{0},s_{1},a_{1},\ldots,s_{T})$ and task goal $G$ . Phase 1 (Spine): captures the required intermediate-state spine and estimates the baseline value function $V(s_{t})$ from $N$ Monte-Carlo rollouts at each checkpoint. Phase 2 (Adversarial Perturbations): applies four types of task-aware perturbations and measures the perturbed value $V^{\prime}(s_{t},p)$ to compute $\Delta P$ and $\Delta T$ . Phase 3 (Scoring): computes per-checkpoint sensitivity $g(s_{t})$ , causality $c(s_{t})$ , and importance $I(s_{t})$ , yielding a per-perturbation Risk Score. Phase 4 (Aggregation): fuses evidence via Dempster–Shafer theory and attributes aggregate risk across the four attack categories via exact Shapley values.

Source	$m(\text{risk})$	$m(\neg\text{risk})$	$m(\Theta)$
MC rollouts ( $\text{conf}=1-1/\!\sqrt{N}$ )	$(1\!-\!V)\,\text{conf}$	$V\,\text{conf}$	$1\!-\!\text{conf}$
Sensitivity (one-sided)	$\tilde{g}(s_{t})$	0	$1-\tilde{g}(s_{t})$
Causal (one-sided)	$\tilde{c}(s_{t})$	0	$1-\tilde{c}(s_{t})$
Max adversarial $\Delta P$ (one-sided)	$\max_{p}\Delta P$	0	$1-\max_{p}\Delta P$

Table 3. Mass functions per evidence source in Dempster–Shafer fusion. Each source assigns mass to three hypotheses:

m(\text{risk})

= belief that the checkpoint is risky,

m(\neg\text{risk})

= belief that it is safe, and

m(\Theta)

= residual ignorance. One-sided sources contribute evidence for risk only, encoding that evidence of vulnerability is not evidence of safety. Notation:

V

= baseline success probability;

\tilde{g}(s_{t})

= normalised sensitivity;

\tilde{c}(s_{t})

= normalised causal score;

\Delta P

= failure-probability gap.

Table 4. Experiment 1: Evaluation of an analytics agent across 10 OCEAN-parameterised personas (60 messages, 240 evaluator judgments). (a) Aggregate scores across all messages. (b) Relevance by persona panel (agent responses only), ranked best to worst—aggregate relevance of 0.75 conceals a range from 0.88 to 0.61.

			Single run ( $n\!=\!1$ ; judge)							Reliability ( $n\!=\!16$ ; goal state)
Task	Persona	Steps	Overall	P.Adh	Rel	Coh	Con	Compl	Judge	pass 1	pass 2	pass 4	pass 8
Complete Payment	Impatient	3/15 cs	0.906	0.742	0.967	0.938	0.933	0.950	PASS	1.00	1.00	1.00	1.00
for Pending Bills	Analytical	4/15 cs	0.875	0.857	0.723	0.909	0.939	0.950	PASS	1.00	1.00	1.00	1.00
	Aggressive	3/15 cs	0.846	0.680	0.697	0.933	0.970	0.950	PASS	1.00	1.00	1.00	1.00
	Anxious	15/15 ms	0.761	0.842	0.780	0.868	0.892	0.420	PASS	1.00	1.00	1.00	1.00
Withdraw	Aggressive	2/15 cs	0.793	0.698	0.743	0.955	0.950	0.620	PASS	1.00	1.00	1.00	1.00

Table 5. Experiment 2: 19 (task

\times

persona) cells on a customer-support agent (mesh topology). Left block: one simulation per cell (agent gpt-5.2 , customer gpt-5-mini ); LLM judges score five criteria per message; Judge = PASS if aggregated overall score

\geq

0.70. Right block:

n\!=\!16

independent reruns per cell (both roles gpt-5.2 ); pass

{}^{k}=\binom{c}{k}/\binom{n}{k}

on goal-state completion (judge scores not re-aggregated over reruns). Steps = turns/budget; cs/ms = customer_signal / max_steps.

\dagger

Judge and pass k diverge.

	pass		top $\Delta P$	task	Shapley %
Task	%	$V_{\min}$	(type)	Bel	(S/M/A/I)
Fetch Latest Transactions	50	1.00	.333 (sys.)	.900	0/0/53/47
Check Refund Eligibility	100	.375	.458 (state)	.925	28/12/22/38
Cancel Pending Payments	100	1.00	.500 (act.)	1.00	0/0/50/50
File an INR Dispute	100	1.00	.800 (act.)	1.00	3/5/31/61
Add a New Card	100	1.00	.333 (sys.)	1.00	32/4/32/32

Table 6. Risk Analyser reveals trajectory-level brittleness invisible to pass-rate testing.

V_{\min}

= lowest baseline success probability across checkpoints (

<

1.0 = pre-existing brittleness); top

\Delta P

= largest failure-probability gap (attack type in parentheses); task Bel = D–S fused belief (lower bound on risk); Shapley % = risk attribution to State / Message / Action / Infra attacks. Config:

N

=8/8,

|P|

=8.

Evidence Source	$m(\text{risk})$	$m(\neg\text{risk})$	$m(\Theta)$
	(risky)	(safe)	(uncertain)
MC rollouts ( $N\!=\!8$ , conf=0.646)	0.404	0.242	0.354
Sensitivity ( $\tilde{g}\!=\!0.375$ , one-sided)	0.375	0	0.625
Causal	collapsed into sensitivity ( $\tilde{c}\!=\!\tilde{g}$ )
Max $\Delta P$ ( $=\!0.375$ , one-sided)	0.375	0	0.625
Fused result	0.727	0.111	0.162

Table 7. D–S worked example at checkpoint

t\!=\!1

(the first required intermediate state) of Check Refund Eligibility. Baseline

V(s_{t})\!=\!0.375

(

N\!=\!8

, 3/8 succeed)—pre-existing brittleness invisible to behavioural scoring. Dempster’s rule fuses three active sources to Bel=0.727, Pl=0.889, Uncertainty=0.162, conflict

k

=0.062.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 516 papers

Sep 8, 2026

A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation

Evaluating tool-augmented LLM agents requires diverse, realistic user inputs yet most evaluation frameworks use flat role descriptions ("you are an angry customer") that produce near-identical conversations regardless of the underlying scenario. In this paper, we propose a three-tier persona vector with 23 operationalized dimensions: 6 categorical demographics (jurisdiction, age, channel, device, language proficiency, time availability), 12 continuous behavioral traits (patience, assertiveness, digital literacy, etc.) sampled with Gaussian noise around curated profile base vectors, and 5 continuous emotional states (frustration, anxiety, trust, confidence, stress) that shift in response to scenario context. Orthogonal to the persona, a 4-level query-complexity overlay controls utterance phrasing from direct to deliberately vague. We evaluate the persona model inside a synthetic data generation pipeline across 64,698 multi-turn conversations spanning 8 named profiles and 3 production corpora. Key findings: (i) a 15.8 percentage-point spread in agent goal-achievement across personas confirms trait vectors produce measurably different user behavior; (ii) the same persona behaves differently across scenarios due to scenario-reactive emotional state shifts, validating the scenario-reactive design; (iii) domain-specific projects show persona sensitivity on booking-flow compliance (~15-20 percentage points gap between tier-aware and pressure-test personas), demonstrating the model faithfully reproduces real-world difficulty distributions; (iv) seven rule-described trait correlations produce auditable co-occurrence patterns without requiring learned covariance matrices. The persona model is fully specified for reproduction.

Rahul Khedar, Eshita, Sneha Teja Sree Reddy Thondapu +6

PayPal AI

LLM Agent Evaluation Multi-Turn Dialogue Evaluation User Modeling Persona-Conditioned Generation User Simulation

Ask PDF

Oct 12, 2026

Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints

Interactive-agent evaluations commonly rely on standardized LLM-based user simulators that exhibit broadly cooperative behavior, potentially overlooking failures that arise under diverse but realistic user interactions. This can lead to overly optimistic estimates of agent robustness and provide limited failure signals for training agents that generalize across users. Existing approaches largely rely on predefined user behaviors, which may miss agent- or domain-specific weaknesses, while failure-oriented search can reduce task success by invalidating the underlying task rather than exposing genuine robustness failures. To fill this gap, we introduce TIPER, a task-integrity-preserving persona evolution method that automatically discovers agent-specific user policies exposing robustness weaknesses while keeping the underlying task valid and solvable. TIPER evolves persistent user personas through reflection-guided behavioral mutations and task-wise Pareto selection under a task-integrity constraint. Across four interactive-agent benchmarks, TIPER yields lower held-out avg@3 scores in all 12 GPT-5.4 settings, with observed reductions of 5.8 to 33.7 percentage points, and in all 28 agent-domain pairs spanning seven agents, with a median observed reduction of 22.9 points. Further analysis yields a behavior-to-failure taxonomy that connects discovered user behaviors to recurring agent failure modes. Together, these results show that TIPER enables more diagnostic robustness evaluation of interactive agents and provides informative failure analysis for future agent robustness improvement. Code repository: https://github.com/UCSB-NLP-Chang/TIPER.

Yutao Xie, Jingbo Yang, Guanyu Yao +3

University of California, Santa Barbara · Cresta

LLM Agent Evaluation User Simulation

Ask PDF

Aug 28, 2026

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

Large language models are increasingly used as agentic workflow executors, yet existing training data and benchmarks largely assume informationally complete, single-turn queries. Our analysis of 16K real-world sessions shows that 75.9% of interactions are multi-turn, revealing a substantial gap between how users interact with agents and how such systems are trained and evaluated. We introduce PersonaForge, a user simulation framework for synthesizing realistic multi-turn user--agent interactions. PersonaForge combines a four-dimensional persona space, SOUL-driven behavioral control calibrated to real-user statistics, and Reverse Deep Construction grounded in authentic seed queries. Using PersonaForge, we construct a 6.3K-record training dataset and PersonaForge-Bench, a manually annotated 138-task benchmark spanning over 20 professional domains with four-dimensional scoring. Experiments on Qwen3.5-27B show that PersonaForge training improves the composite score by +4.1%, with gains across all four dimensions and the largest improvements in Task Completion (+6.0%) and Response Quality (+6.8%). Further analyses show that PersonaForge-trained agents use fewer turns and tool calls, suggesting improved interaction efficiency, while ablations confirm the contribution of SOUL components and adaptive simulation. Together, PersonaForge and PersonaForge-Bench establish a foundation for training and evaluating agents under realistic multi-turn user interaction.

Hanglong Lv, Dawei Zhu, Lei Li +11

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · LLM-Core, Xiaomi · The University of Hong Kong +1

User Simulation LLM Agent Training AI Agent Benchmarks Multi-Turn Dialogue Evaluation Multi-Turn Dialogue Management Large Language Model-Based Social Simulation

Ask PDF

Abstract

AI Agent Evaluation

Explore similar work

A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation

Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems