AutoBuddy: Evaluating Proactive, Always-On Agents on Task Discovery and Execution in a Simulated Workplace
Organizations: Microsoft
Abstract
As LLM agents become more autonomous, they are increasingly expected to run as always-on assistants on a user's behalf, identifying work to be done and completing it without being asked. Yet existing benchmarks usually give the agent its task, whereas in knowledge work the requests sit in the user's workspace and the agent must first recognise which obligations exist. We study proactive task discovery and execution, in which an agent must infer a user's pending tasks from their workspace and complete them in the systems where the work lives. This setting poses two challenges. For the agent, the tasks are buried in a large volume of mail, chat, documents and records. For evaluation, a benchmark needs a simulated organisation whose people, events and documents are mutually consistent, so that tasks arise realistically and their correct outcomes are known. We introduce AutoBuddy-Bench, a simulator that addresses these challenges. It models an organisation's employees, projects, business systems and event history as a temporal graph, renders it into realistic workspace artefacts, derives ground-truth tasks from the events that create obligations, and computes each task's outcome from the world state; execution is judged by the agent's changes to the workspace. To evaluate current LLMs, we build AutoBuddy, a reference agent with interchangeable backbone LLMs: it identifies tasks from the workspace, grounding each in its source records, and executes each in a separate session with write access. Varying only the backbone, we find a systematic shift in where tasks are lost: weaker models fail mainly at discovery, intermediate models identify tasks but often stop short of carrying them out, and the strongest models fail mainly during execution, most often on business-system operations. Discovery precision is only weakly related to execution quality, suggesting the two are distinct capabilities.
Figures & tables
| Discovery | ||||||
| Model | Provider | Precision | Recall | Execution | End to end | |
| claude-opus-5.5 | Anthropic | 0.905 | 0.947 | 0.925 | 0.739 | 0.698 0.002 |
| claude-opus-5 | Anthropic | 0.691 | 0.975 | 0.808 | 0.698 | 0.680 0.025 |
| gpt-5.6-sol | OpenAI | 0.947 | 0.970 | 0.958 | 0.676 | 0.654 0.014 |
| grok-4.7 | xAI | 0.605 | 0.941 | 0.736 | 0.676 | 0.635 0.023 |
| gpt-5.5 | OpenAI | 0.938 | 0.922 | 0.930 | 0.686 | 0.631 0.039 |
| False alarms | ||||||
|---|---|---|---|---|---|---|
| Model | Missed | Closed | Colleague’s | Background | Logged | Reopened |
| claude-opus-5.5 | 6.3 | 12.0 | 0.0 | 0.0 | 52.0 | |
| claude-opus-5 | 3.0 | 28.3 | 16.7 | 8.0 | 30.0 | |
| gpt-5.6-sol | 3.7 | 4.7 | 0.0 | 2.0 | 49.0 | |
| grok-4.7 | 7.0 | 68.0 | 2.0 | 4.0 | 9.7 | |
| gpt-5.5 | 9.3 | 4.7 | 0.0 | 2.7 | 42.3 | |
| Failure (model) | Case | Score |
| Discovery | ||
| Re-opened closed work grok-4.7 | Re-opened a document request answered on an earlier day | — |
| Colleague’s request claude-haiku-4.5 | Opened Emmanuel’s request to restart failed jobs for its own user | — |
| Misread request claude-haiku-4.5 | Read a support -system reconciliation as an HR one; the real task was never found | 0 |
| Execution | ||
| Stopped to ask gpt-6-luna | Merged two invoice requests, asked which rule applies, sent nothing | 0 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Example tools | Example task |
|---|---|---|
| Collaboration | search_emails , list_chat_messages , list_meetings , search_files , send_email , create_document , share_document | write the review of a completed report run and share it with its audience |
| Research | wandb_list_runs , wandb_run_history , kubernetes_list_jobs , kubernetes_pod_logs , kubernetes_restart_job , coding_run_analysis | report which training jobs failed and why; decide which run to promote |
| Business | zendesk_update_ticket , hubspot_update_contact , quickbooks_send_invoice , bamboohr_update_employee | update the support tickets, contacts or invoices a stated policy selects |
| Family | What the employee must do | Tasks |
|---|---|---|
| Business systems (end state of each named record) | 49 | |
| Customer support | reopen, escalate or close helpdesk tickets by a stated policy | 26 |
| Sales | change CRM contacts’ lifecycle stage by a stated rule | 15 |
| Finance | send the invoices a stated rule selects | 5 |
| Human resources | correct employee records against a roster | 3 |
| Collaboration (document or message content) | 54 | |
| Step | Stages (invocation) | Produces |
|---|---|---|
| 1 Company scope | resolve_company (1), plan_population (2) | a grounding profile of the company; an exact population plan |
| 2 Positions | build_history (3), build_capability (4), build_identity (5), build_work (6), build_context (7), build_access (8) | dated employment histories; what each position does; names and accounts; projects with owners and participants; external parties and resources; groups, roles and access grants |
| 3 Events and keys | build_training (9), build_events (10, 12), build_ledger (11), build_graph (13), build_reporting (14), build_collaboration (16), build_cases (18), build_query_program (22) | experiment runs; dated world events; payroll and numeric outcomes; a temporal graph; report runs; collaboration episodes; anchored cases; scenarios with their oracles and keys |
| 4 Workspace | build_topology (15), build_artifact_corpus (17), build_artifacts (19), build_artifact_data (20), build_graph (21), integrate_artifact_corpus (23), project_artifact_corpus_graph (24), build_query_program (25) | mail, chat, calendar and file surfaces; documents with revisions and audiences; every document bound to an exact revision and its report data; the graph and oracles rebuilt over them |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,304 papers