An email assistant should not complete less work simply because a user phrases the same request differently. Yet most benchmarks test each task with only one canonical request, leaving this form of robustness largely unmeasured. We test whether email assistants remain reliable when the requested information, available evidence, and expected outcome stay fixed, but the communication style or English variety changes. We construct validated variants along five communication-style axes and four rule-based dialect conditions, and evaluate them on three benchmarks: a retrieval-augmented generation (RAG) pipeline and two tool-using agents. Indirect requests reduce performance on all three benchmarks, while formal requests reduce performance on both agentic benchmarks. Examining the systems more closely shows that these failures have different causes. Verbose requests mainly hurt a lexical retriever by making the relevant email harder to find. By contrast, indirect and dialect variants remain harmful even when the relevant email is retrieved. In the agentic setting, indirect and formal requests mainly cause the agents to omit required actions, not to take more unsupported actions. These results show that a successful response is not enough to establish robustness: evaluations should vary how requests are expressed and separately measure whether agents complete the requested work.
Figures & tables
Figure 1: Variant construction, and one EnronQA request carried through it. (a) A request is rewritten into ten communication-style poles and four English varieties; a variant enters the evaluation set only after clearing the gates of § 2.3 . (b) The same request under three variants. The gold email is retrieved at rank 1 in all three cases, but the indirect variant produces an incorrect answer. ref , con , and dia denote reference, contrastive, and dialect conditions, respectively.
Figure 2: Accuracy Δ and gold-email recall Δ for verbose and indirect against unperturbed request under three retrievers on EnronQA. Shading marks the gap between the part of the accuracy change not matched by a change in retrieval. Values are averaged over three answerer models; per-model values in Table 10 .
Figure 3: Paired Δ for the indirect and formal poles. Vermillion marks an omission measure and blue an unsupported-action measure; shade marks the model. (a) ToolTalk required-call recall, -17.0 to -6.8 pp over 6 pole × model cells, 6 significant; (b) ToolTalk incorrect-action rate, -1.9 to +4.1 pp, 0 of 6 significant; (a) and (b) share a scale. (c) WorkBench on the headline model, the share of tasks with no tool call beside the share leaving an unwanted change to the sandbox. Per-pole and per-model values in Table 16 and Table 17 . *p<.05 , **p<.01 , ***p<.001 .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: EnronQA breakage by pole and question-complexity bin. Columns are the four sampling bins: simple (the control), 3-hop , multi-question and threaded . Each cell is the percentage of slots in that bin which the unperturbed request answered correctly and the rewrite broke.
Figure 5: All ten poles in the headline condition of each system, with 95% paired bootstrap confidence intervals. Reference poles are the four meaning-preserving rewrites, contrastive poles the six styled ones. Metrics differ across panels and are not commensurable.
Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.
Enterprise email agents must combine information retrieval, structured state changes, temporal reasoning, and multi-step coordination. Recent agent benchmarks include productivity tasks, but few center on typed email workflows in a self-contained environment. We introduce EmailBench, a benchmark of 206 email and productivity scenarios across 16 task categories. The benchmark couples a typed email API specification with provider-neutral naming, a deterministic synthetic Enron-inspired corpus, and a scenario suite whose topic selection was informed by aggregate task-intent telemetry from an interactive prototype. Its hybrid evaluation protocol combines 258 executable static assertions with 211 LLM rubrics. We evaluate eight LM configurations on a fixed single-user corpus. The best-performing configuration passes only 33.5% of scenarios despite 99.7% of its tool calls completing without an observed API failure, with pass rates varying substantially across task categories. This gap shows that valid tool execution is not equivalent to task completion. EmailBench provides a self-contained environment for end-to-end email-agent evaluation, with broader tool coverage, multi-persona testing, and repeated-run evaluation as future work areas.
People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the faithfulness of reports, and it leaves open whether the requester can understand and use the reply. We call this property reply usability and measure it with a reader model that takes the requester's position and tests whether the reply can be interpreted from what the requester has seen, is in the requester's language and answers the request. Its agreement with the majority of five annotators lies within the range of the individual annotators. Across eight models from seven families, 54% of 480 replies fail, and the best model still fails on 30%. Failed replies mostly point to material the requester has not seen or use labels that the agent coined during its work. Stating explicit requirements for the reply reduces the failure rate from 0.52 to 0.43 together with errors and omissions, and a secondary model that rewrites the reply reduces it to 0.36, which human annotators confirm in a blind comparison. Reply usability can thus be scored alongside task success and improved with one appended message or one call to a secondary model.
Yilin Jiang, Fei Tan, Xuanyu Yin +3
East China Normal University, Shanghai, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China +1
AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first. This decision is usually evaluated by showing a model a proposed action, asking whether it should proceed, and scoring agreement with human labels. We introduce DelegationBench to test whether such scores can be trusted. It has 156 scenarios with four possible responses (act, ask for permission, ask for missing information, refuse), and most scenarios come in matched pairs that change a single feature: whether the action was requested, what is at stake, whether it can be undone, or who will see it. Across ten models from five families, agreement scores mislead in three ways. A simple keyword rule, which we wrote after seeing the benchmark, agrees with our annotators more often than eight of the models, yet its decision changes in only 9 of 48 matched pairs. Equivalent ways of asking the same question change how often a model acts by up to 52.5 percentage points. And every model stops to ask the user less often when it must carry out the task with tools than when it judges a proposed action. When rules are stated explicitly, the same models follow them almost perfectly, so the gaps are not explained by a general inability to follow rules. We release the benchmark and evaluation tools and recommend reporting these properties separately rather than as one score.