Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents

Detao Zeng; Guilin Ou; Nuo Chen; Yunhao Feng; Hongliang Wang; Qinli Yang; Junming Shao; Zhong Zhang

cs.AIOct 23, 2026

Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents

Authors: Detao Zeng, Guilin Ou, Nuo Chen, Yunhao Feng, Hongliang Wang, Qinli Yang, Junming Shao, Zhong Zhang

Organizations: University of Electronic Science and Technology of China

Abstract

Large language model (LLM) agents increasingly execute tasks asynchronously: a main agent continues working while its subagents run, and the task state may change before their results arrive. The main agent must decide which work to update or preserve and which checks to repeat, while avoiding prohibited changes. Final task success alone does not reveal whether these results were handled correctly, and natural asynchronous arrivals are hard to reproduce. We introduce Async-RBench, a benchmark for asynchronous task execution in LLM agents, with 200 executable tasks covering eight scenarios. A scenario controller regulates result delivery using predefined task-state conditions, while an Event Response Contract (ERC) specifies what each response must change, preserve, avoid, and re-verify. The Dynamic Replanning Score (DRS) weights the response process and outcome equally and is reported separately from task success. We evaluate nine agents and find DRS scores of 19.2 to 45.1, with the lowest scores on duplicate reports and subagent failures. Under our execution protocol, all nine agents obtain lower task success with asynchronous than with batched delivery (26.0% vs. 29.7% on average). For three agents, prompting the main agent to rebuild its plan after every result improves DRS by only 2.6 to 3.5 points. Together, these findings highlight the challenges of reliable asynchronous task execution and the need to evaluate how agents handle incoming results alongside end-to-end task success.

Figures & tables

Figure 1: A delayed-authority interruption in a repository-repair task. Lightning bolts mark subagent results reaching the main agent, and yellow boxes mark its responses. The right panels contrast the correct response with an illustrative failure.

Panel of Table 1 — Table 1: Comparison with related benchmarks. Async delivery : subagent results reach the main agent while it continues executing. Post-event checks : behaviors evaluated after an event. Per-event scoring : each event response is scored separately from end-to-end task success. $\checkmark$ explicit; partial or related; ✗ no explicit design.

Figure 2: Overview of Async-RBench . Subagents A–C run concurrently in separate workspaces while the main agent continues working. The scenario controller checks submitted results and applies case-specific release conditions, with a maximum-hold fallback. B’s and C’s reports are held until A’s is presented, even if they finish first. Task-level checks determine success $S$ . The response to the target event and its artifacts are evaluated against the ERC to compute DRS (Section 3.2 ).

Scenario	Example task	Evaluation focus
Delayed authority	Archive an AWS invoice; its fields arrive after the folder and summary table are checked.	Adopt authoritative evidence ; save the attachment and verify the record.
Staleness and ordering	Negotiate a cart purchase; an old offer arrives after the new $42 agreement is saved.	Reject stale updates ; keep the current agreement over old terms.
Conflicting results	Diagnose slow DB writes; duplicate-index evidence arrives after causes are ranked.	Arbitrate by evidence , not arrival order; revise the root-cause ranking.
Duplication and replay	Build a scheduling app; a completion message is replayed after its tests are integrated.	Stay idempotent ; avoid repeated side effects and needless rebuilding.
Partial to complete	Merge two spreadsheets into a CSV; column extractions arrive before the complete merge.	Separate provisional from complete evidence ; confirm row alignment.
Subagent failure	Coordinate restaurant deliveries; a delivery subagent declines its suborder.	Recover selectively ; reassign the failed suborder and keep valid routes.

Table 2: Example task and evaluation focus for each scenario, grouped by result validity (first four), information completeness (next two), and dependencies and scheduling (last two). Examples are condensed from actual cases.

Agent	Delay	Stale	Conflict	Replay	Partial	Failure	Scope	Strag.	Calls	DRS
Claude Sonnet 5	38.7 $\pm$ 1.8	53.4 $\pm$ 1.2	39.9 $\pm$ 2.1	15.5 $\pm$ 1.2	29.8 $\pm$ 1.7	35.8 $\pm$ 1.3	66.9 $\pm$ 1.9	80.5 $\pm$ 1.5	77.1	45.1
GPT-5.6 Terra	34.2 $\pm$ 1.5	43.9 $\pm$ 2.9	41.5 $\pm$ 1.3	18.5 $\pm$ 1.6	21.4 $\pm$ 1.3	16.2 $\pm$ 1.9	69.4 $\pm$ 2.0	72.2 $\pm$ 2.4	48.1	39.7
Qwen3.7 Plus	33.0 $\pm$ 1.3	45.5 $\pm$ 2.9	38.8 $\pm$ 1.5	12.6 $\pm$ 1.6	21.2 $\pm$ 2.4	6.9 $\pm$ 1.5	61.8 $\pm$ 2.4	85.9 $\pm$ 2.6	75.9	38.2
GLM5.3 Flash	29.0 $\pm$ 2.7	47.8 $\pm$ 2.2	27.0 $\pm$ 2.3	11.6 $\pm$ 2.2	31.2 $\pm$ 2.1	5.0 $\pm$ 0.6	53.5 $\pm$ 2.4	62.3 $\pm$ 1.4	76.5	33.4
GPT-5.6 Luna	34.0 $\pm$ 1.2	47.8 $\pm$ 1.2	31.7 $\pm$ 1.4	4.2 $\pm$ 0.3	22.2 $\pm$ 1.2	14.9 $\pm$ 1.2	43.8 $\pm$ 1.3	50.5 $\pm$ 2.2	45.5	31.1
DeepSeek V4 Pro	16.2 $\pm$ 1.6	47.6 $\pm$ 2.2	20.8 $\pm$ 3.2	2.7 $\pm$ 0.7	24.2 $\pm$ 2.4	16.8 $\pm$ 2.8	62.0 $\pm$ 2.1	54.7 $\pm$ 3.2	108.0	30.6

Table 3: DRS (0–100) under asynchronous execution. Scenario columns follow Table 2 and report the mean and sample SD over three repetitions. The Calls column counts main-agent and subagent responses. The last row averages across agents. Bold marks column maxima.

Figure 5: Mean token consumption per run by the main agent and all subagents. Each pair shows batched and asynchronous delivery; percentages give the main agent’s share.

Agent	Task success		Change
Agent	Batched	Async	$\boldsymbol{\Delta S}$
Claude Sonnet 5	42.3	38.2	$-4.2$
GPT-5.6 Terra	41.3	38.8	$-2.5$
Qwen3.7 Plus	21.3	18.7	$-2.7$
GLM5.3 Flash	25.5	20.8	$-4.7$
GPT-5.6 Luna	40.2	34.3	$-5.8$

Table 4: Task success rate

S

(%) under batched and asynchronous delivery.

Figure 6: DRS and mean model calls per run under Native execution and Full-rebuild. In (b), solid bars count main-agent calls and hatched bars count calls summed over all subagents.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Task construction and acceptance. LLM-agent screening feeds human-led scenario and ERC design and review; assembled instances then undergo execution checks in both delivery modes, with reference or equivalent solutions and invalid variants.

Information

Access boundary

Content and use

Appendix

Table 5: Case components and information boundaries. Public version and dependency evidence comes from the task or presented results; the evaluator retains the full affected scope, response requirements, and scoring evidence.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios
May 27, 2026Kou Shi, Ziao Zhang, Shiting Huang +7Tool-Use Evaluation AI Agent Benchmarks
University of Science and Technology of China · University of Toronto
PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents
Aug 25, 2026Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li +2Tool-Using Agents LLM Agent Evaluation
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Nanjing University, China
Empirical Comparison of Agent Communication Protocols for Task Orchestration
Mar 24, 2026Ivan DobrovolskyiAgent Communication Protocols LLM Agent Orchestration
Independent Researcher, Sunnyvale, CA, USA

May 27, 2026J/K move · Enter open · S save

AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios

Kou Shi, Ziao Zhang, Shiting Huang, Avery Nie+6

University of Science and Technology of China · University of Toronto

Tool-Use Evaluation AI Agent Benchmarks Tool-Using Agents

Open paper AskPDF arXiv

Large language model (LLM)-based agents have shown strong capabilities in using external tools to solve complex tasks. However, existing evaluations often overlook the temporal dimension of tool use, especially the impact of tool response latency, and are usually limited to single-task settings. In real-world applications, multiple tasks often need to be executed concurrently, and overall efficiency depends on whether an agent can use idle time while waiting for tool responses. We refer to this capability as asynchronous tool calling. To evaluate it, we propose AsyncTool, a benchmark for assessing LLM-based agents in interactive multi-task tool-use environments with delayed tool feedback. AsyncTool presents multiple heterogeneous tasks simultaneously and simulates realistic tool response latency during execution. Using a hybrid data evolution strategy, we construct a diverse asynchronous multitasking dataset that covers multiple scenarios and tool-use patterns. We evaluate models at the step, sub-task, and task levels, and introduce efficiency-oriented metrics to measure task coordination and completion efficiency. Extensive experiments show that delayed tool feedback poses substantial challenges to current agents and leads to clear performance degradation. Models that better coordinate task switching, dependency tracking, and state maintenance achieve stronger performance on AsyncTool. Our analysis identifies key failure modes of current tool-using agents and provides practical insights for designing future systems with stronger temporal reasoning and coordination capabilities.

Abstract

LLM Agent Evaluation

Explore similar work

AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios

PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents

Empirical Comparison of Agent Communication Protocols for Task Orchestration