Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants

Deepak Nathani; Cheng Zhang; Chang Huan; Jiaming Shan; Yinfei Yang; Alkesh Patel; Zhe Gan; William Yang Wang

cs.AIApr 1, 2026

Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants

Authors: Deepak Nathani, Cheng Zhang, Chang Huan, Jiaming Shan, Yinfei Yang, Alkesh Patel, Zhe Gan, William Yang Wang,

Organizations: University of California, Santa Barbara · Independent Researcher, USA · Apple, USA · University of Washington

Abstract

Proactive agents that anticipate user needs and autonomously execute tasks hold great promise as digital assistants, yet the lack of realistic user simulation frameworks hinders their development. Existing approaches model apps as flat tool-calling APIs, failing to capture the stateful and sequential nature of user interaction in digital environments and making realistic user simulation infeasible. We introduce Proactive Agent Research Environment (Pare), a framework for building and evaluating proactive agents in digital environments. Pare models applications as finite state machines with stateful navigation and state-dependent action space for the user simulator, enabling active user simulation. Building on this foundation, we present Pare-Bench, a benchmark of 143 diverse tasks spanning communication, productivity, scheduling, and lifestyle apps, designed to test context observation, goal inference, intervention timing, and multi-app orchestration.

Figures & tables

Figure 1: Overview of the Pare framework architecture. Pare framework consists of an event-based environment that models Stateful App transitions. The user-environment interface exposes selective tools based on the current state $S_{t}$ of the currently active app, and user actions result in app state transitions. Whereas the agent-environment interface exposes all tools as a flat API structure to enable efficient information gathering and task execution by Proactive Assistants.

Figure 2: Comparison of tool call chains for sending a message. Non-FSM agent frameworks allow direct API calls (left), while Pare ’s FSM-based design requires the user to navigate through sequential app screens (right), matching how real users interact with mobile apps.

Model	Success@4 $\uparrow$	Success^4 $\uparrow$	Success Rate $\uparrow$	Proposal Rate $\downarrow$	Acceptance Rate $\uparrow$	Read Actions
Claude 4.5 Sonnet	60.8%	18.2%	42.0% $\pm$ 1.0%	12.8% $\pm$ 0.4%	78.2% $\pm$ 0.8%	20.2 $\pm$ 0.2
Gemini 3 Flash	64.3%	16.1%	42.1% $\pm$ 1.0%	19.1% $\pm$ 0.4%	67.1% $\pm$ 2.3%	21.7 $\pm$ 0.1
Gemini 3 Pro	59.4%	11.9%	35.1% $\pm$ 0.9%	16.5% $\pm$ 0.4%	71.2% $\pm$ 1.6%	19.5 $\pm$ 0.2
GPT-5	57.3%	17.5%	37.4% $\pm$ 1.5%	28.1% $\pm$ 0.3%	70.2% $\pm$ 1.0%	20.6 $\pm$ 0.3
Qwen 3 4B Instruct	35.0%	6.3%	18.5% $\pm$ 1.3%	20.5% $\pm$ 0.3%	63.7% $\pm$ 1.6%	16.7 $\pm$ 0.1
Llama 3.2 3B Instruct	23.8%	1.4%	10.0% $\pm$ 0.4%	23.0% $\pm$ 0.3%	58.4% $\pm$ 0.3%	16.6 $\pm$ 0.3

Table 1: Model performance metrics on Pare-Bench across 4 runs. We report standard error (

\pm

). Proposal Rate follows lower is better, since too many proposals can annoy the user. The best performing model is highlighted in bold for each metric.

Figure 3: Proposal Rate, Acceptance rate, and Execution Success of different models with respect to tool failure probability evaluation across 4 runs.

Figure 4: Proposal Rate, Acceptance rate, and Execution Success of different models with respect to environment event noise evaluation across 4 runs.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Accept $\uparrow$	Reject	Gather Context $\downarrow$	Truncated
Claude 4.5 Sonnet	72.1% $\pm$ 1.4%	7.8% $\pm$ 0.8%	17.8% $\pm$ 1.9%	2.3% $\pm$ 0.7%
Gemini 3 Flash	61.5% $\pm$ 1.9%	14.7% $\pm$ 1.9%	18.3% $\pm$ 0.8%	5.5% $\pm$ 0.9%
Gemini 3 Pro	65.6% $\pm$ 1.5%	13.3% $\pm$ 1.0%	15.1% $\pm$ 1.0%	5.9% $\pm$ 1.0%
GPT-5	64.1% $\pm$ 1.3%	7.4% $\pm$ 0.3%	23.4% $\pm$ 1.4%	5.1% $\pm$ 0.4%
Qwen 3 4B Instruct	56.3% $\pm$ 1.6%	14.1% $\pm$ 1.0%	26.5% $\pm$ 1.1%	3.2% $\pm$ 0.7%
Llama 3.2 3B Instruct	49.9% $\pm$ 0.7%	15.3% $\pm$ 1.0%	29.1% $\pm$ 0.3%	5.8% $\pm$ 0.3%

Appendix

Table 2: Ternary decision distribution across models. We report standard error (

\pm

) across 4 runs. The best performing model is highlighted in bold for each metric.

Model	G $\rightarrow$ Accept $\uparrow$	G $\rightarrow$ Reject $\downarrow$	G $\rightarrow$ Truncated
Claude 4.5 Sonnet	29.8% $\pm$ 4.3%	4.6% $\pm$ 3.0%	65.6% $\pm$ 4.4%
Gemini 3 Flash	25.5% $\pm$ 1.2%	9.5% $\pm$ 1.4%	65.0% $\pm$ 1.2%
Gemini 3 Pro	28.7% $\pm$ 3.0%	5.6% $\pm$ 2.4%	65.7% $\pm$ 0.7%
GPT-5	25.2% $\pm$ 1.5%	3.7% $\pm$ 1.1%	71.1% $\pm$ 1.4%
Qwen 3 4B Instruct	25.5% $\pm$ 2.4%	9.7% $\pm$ 2.8%	64.8% $\pm$ 0.9%
Llama 3.2 3B Instruct	27.4% $\pm$ 1.4%	10.2% $\pm$ 0.5%	62.4% $\pm$ 1.9%

Appendix

Table 3: Gather context resolution. We report the percentage of proposals that entered the information gathering phase and how they eventually resolved. Standard error (

\pm

) across 4 runs.

Proactive Model	Success@4 $\uparrow$	Success^4 $\uparrow$	Success Rate $\uparrow$	Proposal Rate $\downarrow$	Acceptance Rate $\uparrow$	Read Actions
User Model: GPT-5-mini
Claude 4.5 Sonnet	64.0%	22.0%	45.5% $\pm$ 2.2%	12.9% $\pm$ 0.9%	79.8% $\pm$ 2.4%	20.3 $\pm$ 0.0
GPT-5	54.0%	24.0%	37.0% $\pm$ 2.4%	27.5% $\pm$ 0.3%	70.7% $\pm$ 1.4%	20.5 $\pm$ 0.3
Gemini 3 Flash	56.0%	10.0%	35.0% $\pm$ 2.4%	16.3% $\pm$ 0.5%	68.0% $\pm$ 4.5%	19.2 $\pm$ 0.4
Qwen 3 4B Instruct	36.0%	8.0%	20.0% $\pm$ 1.4%	20.1% $\pm$ 0.3%	61.6% $\pm$ 2.5%	16.7 $\pm$ 0.2
User Model: Claude 4.5 Sonnet

Appendix

Table 4: User model study on the ablation split (50 scenarios, 4 runs each). We vary the user simulation model while keeping the proactive assistant models fixed. Each section groups results by user model. The best performing proactive model is highlighted in bold for each metric within each group.

Figure 5: Scenario generation agent pipeline. Given Pare codebase and target apps, the agent generates scenarios through four stages: description, data population, event flow, and validation. Each stage (except description) runs in oracle mode for execution-based validation, with failed attempts looping back for retry. The uniqueness check ensures no duplicate scenarios.

Figure 6: App usage statistics across 143 scenarios in Pare-Bench . PASAgentUserInterface and System apps are excluded since they are included in all scenarios.

App	Description	States
FSM-based Apps
Cab	Ride-hailing	Home, Options, Quote, Ride
Note	Note-taking	Folders, List, Detail, Edit
Email	Email client	Mailbox, Detail, Compose
Calendar	Scheduling	Agenda, Detail, Edit
Contacts	Contact mgmt	List, Detail, Edit

Appendix

Table 5: Applications in Pare and their corresponding states.

Figure 7: User view of the email app state transitions. While the assistant can directly send a single message with the tool call messaging_app. send_message ( recipients, attachments, message ,..) , the user agent must perform a series of state transitions to do this. ( start composing

\rightarrow

set recipients

\rightarrow

attach file

\rightarrow

send composed mail ).

Proactive Agents

See all 35 papers

$π$ -Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows
May 14, 2026Haoran Zhang, Luxin Xu, Zhilin Wang +11Long-Horizon Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University +7
ProEvent: An Event-centric Benchmark for Proactive Agents
Jul 20, 2026Guanzhen Li, Liangming Pan, Leye WangProactive Agents LLM Agent Evaluation
School of Computing, Peking University
Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym
Sep 29, 2026Jio Oh, Seunghyun Do, Young-Jun Lee +2LLM Agent Evaluation Long-Horizon Agent Evaluation
KAIST · University of Minnesota

May 14, 2026J/K move · Enter open · S save

$π$ -Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

Haoran Zhang, Luxin Xu, Zhilin Wang, Runquan Gui+10

Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University+7

Long-Horizon Agent Evaluation AI Agent Benchmarks Proactive Assistance Personal AI Agents Proactive Agents

Open paper AskPDF arXiv

The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce

π

-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity,

π

-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.

Abstract

Proactive Agents

Explore similar work

πππ-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

ProEvent: An Event-centric Benchmark for Proactive Agents

Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym

$π$ -Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows