MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads

Shengxuan Qiu; Haochen Huang; Shuzhang Zhong; Zizhuo Fu; Zebin Yang; Jianming Tong; Meng Li

cs.MAOct 29, 2026

MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads

Authors: Shengxuan Qiu, Haochen Huang, Shuzhang Zhong, Zizhuo Fu, Zebin Yang, Jianming Tong, Meng Li

Organizations: Peking University · Georgia Institute of Technology

Abstract

Large language model (LLM)-based multi-agent systems (MAS) coordinate specialized agents to solve complex tasks, turning each user request into interdependent model and tool operations. Unlike independent requests, these operations become ready at different times and consume evolving context as agents coordinate and exchange results. This creates changing patterns of concurrency, computation, and memory demand that can expose different bottlenecks in LLM serving hardware. Yet benchmarking these effects requires capturing the collaboration rules that generate serving workloads, beyond the requests observed during execution. We present MAServeBench, a benchmark that treats multi-agent collaboration as a configurable dimension of serving workloads for LLM serving and hardware evaluation. Its composable stages capture how agents coordinate and exchange context in documented MAS workflows, enabling native executions to generate different workloads even when task inputs and tools remain unchanged. Hierarchical Semantic Traces preserve realized operations and dependencies for reproducible comparisons of legal release constraints at fixed work. Task-linked measurements connect these workloads to serving execution and GPU/NPU hardware behavior, revealing why serving efficiency does not always translate into timely task completion. The benchmark code is available at https://github.com/ShengxuanQiu/MAServeBench.

Figures & tables

Figure 1: Collaboration reshapes serving demand. (a) Independent request arrivals. (b) A multi-agent task induces dependent requests with artifact exchange and synchronization. Both can use continuous batching. (c) MAServeBench enables controlled, reproducible evaluation of collaboration-induced serving demand and hardware resource behavior.

Figure 2: MAServeBench hierarchical workload abstraction. (a) Example multi-agent debugging workflow. (b) Four reusable collaboration templates within the workflow model. (c) Unified stage specification of participants, execution control, and context construction. (d) Hierarchical Semantic Trace records at task, stage, session, and operation levels.

Field Name	Data Type	Description
metadata	TraceMetadata	Version and identity
tasks	TaskRecord[]	End-to-end tasks
stages	StageInstance[]	Realized stages
sessions	AgentSession[]	Runtime participants
artifacts	ArtifactRecord[]	Delivered artifacts
shared_state	StateRecord[]	Versioned state

Table 1: MAServeBench Semantic Trace manifest schema. Brackets [] denote repeated records.

Field Name	Data Type	Description
operation_id	string	Unique operation ID
operation_type	enum(OpType)	LLM, tool, transform, state
task_id	string	Parent task
stage_instance_id	string	Parent stage instance
session_id	optional string	Parent agent/session
iteration	uint32	Iteration index

Table 2: MAServeBench semantic operation schema.

Workflow	Templates / participants	Workflow	Templates / participants
MetaGPT startup Hong et al. (2024)	$\Gamma$ : Sequential Handoff Product Manager, Architect, Project Manager, Engineer	OWL Workforce CAMEL-AI (2025)	Spawn Planner, Coordinator, Workers
ChatDev Default Qian et al. (2024)	Refinement Loop Programmer, Code Reviewer, Software Test Engineer	TradingAgents Xiao et al. (2024)	Debate Bull and Bear Researchers, Risk Analysts
Magentic-One Fourney et al. (2024)	Spawn Orchestrator, Selected Workers	Virtual Lab Swanson et al. (2025)	Fork–Join, Debate Meeting Leads and Members, Merge Participants
MindSearch Chen et al. (2025)	Spawn Planner, Searcher	Claude Code PR review Anthropic (2026)	Fork–Join Lead, Security, Performance, Test Reviewers
GPT Researcher GPT Researcher Contributors (2026)	Fork–Join, Refinement Loop Editor, Researcher, Reviewer, Reviser, Writer	DeerFlow literature review ByteDance (2026)	Spawn, Fork–Join Lead, Extraction Subagents
Open Deep Research LangChain (2026)	Spawn, Fork–Join Supervisor, Researchers	Agyn issue resolution Benkovich and Valkov (2026)	Spawn, Refinement Loop Manager, Researcher, Engineer, Reviewer

Table 3: Source-grounded collaboration mappings. Twelve workflows: templates

\mathcal{T}

(first line) and participants

\mathcal{P}

(second line). MetaGPT uses

\Gamma

for sequential handoff (

\mathcal{T}=\varnothing

Platform	Application workload	Matched operator
A800	Qwen3.5-4B/9B/27B	Qwen3.5-9B projections
PRO 5000	Qwen3.5-9B	Qwen3.5-9B projections
910C	Qwen3.5-9B; Qwen3-8B	Matrix, normalization, gather; full attention
950DT	Qwen3.5-9B captures	Matrix, normalization, gather; Qwen3-8B attention

Table 4: Application and operator coverage. 910C applications use one logical chip per dual-chip card; operator probes are separate from application runs.

Figure 3: Demand over workflow progress. Native Qwen3-8B/910C, 32/64/128 users; three-block medians/ranges. (a)–(c) Modeled AI (FLOP/byte), with marked

25\times

axis compression above 100. (d)–(f) Allocated KV (GiB).

Figure 4: Native generation versus fixed-work release. (a) Selected changes from Full: four-call controls (three-block medians/ranges) and GPT Researcher (one native pair per binding; writer input denotes the first call). (b) Barrier/DAG root-latency ratios over three paired replay blocks.

Figure 5: Native execution shapes and kernel mix. Qwen3-8B/910C, 64 users, 128-request limit; Spawn has four children. (a) Time-weighted decoding batches, three-block average. (b) Matrix time by tensor-row

M

. (c) Kernel-time mix. (b,c) One capture per workflow/window; Early/Late: plain/hatched. C1/C2 follow the composition order in § 6.1 .

GPU
Feature	A800 (Ampere)	PRO 5000 (Blackwell)
Architecture	GA100, sm_80	GB202, sm_120
BF16 matrix	3rd-gen Tensor Cores	5th-gen Tensor Cores
L2 cache	40 MiB	96 MiB
Memory	80 GB HBM2e	48 GB GDDR7
Bandwidth	2.04 TB/s	1.34 TB/s

Table 5: GPU/NPU architectural context. Specifications follow NVIDIA NVIDIA (2020) ; NVIDIA (2025) ; NVIDIA (2026) and Huawei Huawei (2025) . A800 is the 80 GB SXM4 variant; NPU vector-access granularity is vendor-reported.

Figure 6: GPU projection response to geometry and weight locality. (a) All 28 timing ratios, A800/RTX PRO 5000; higher favors RTX PRO 5000. (b,c) One/eight-row physical DRAM reads normalized by one BF16 weight, from independent counter passes.

Figure 7: NPU compute response. (a) 910C/950DT Memory-pass ratios for 18 BF16 matrix cases,

K=4096

. (b) Twelve 950DT Gather implementations; binary inspection verifies SIMT and identifies Hybrid as SIMD. O/P: ordered/permuted layouts.

Figure 8: NPU attention response to context geometry. (a) Native Qwen3-8B/910C attention durations. (b) Matched 910C/950DT probes at batch 32 with 1k/4k contexts and contiguous/permuted page layouts (C/P).

Figure 9: KV pressure and task tails. Qwen3-8B/910C, 64 users. (a) Native preemptions by victim workflow at 31.99 GiB; three-block medians/ranges. (b,c) Fixed-work prefill factor and root P95. Connected points: paired blocks; black: medians.

Figure 10: Temporal mixing at fixed work. (a) Time-weighted share of target-containing scheduled intervals co-batched with each background class or all three, 2–110 s (bars: medians; markers: blocks). (b) First/second-call scheduled-residence differences (

A-B

; paired block means, median/range). (c) Root-latency changes.

Figure 11: Fixed Loop targets under near-matched background. (a) Root latency. (b) Added client-observed decoding time per output token; bars/dots: mean/block means, two calls and three frozen inputs. (c) All 27 paired targets; the line denotes equal root-latency and scheduled-residence changes.

Figure 12: Legal release under deadlines. Fixed-work (a) arrival-window deadline goodput and (b) attainment. Colors: release rules; line styles: KV budgets. Points: three-block medians/ranges.

Appendix figures & tables61 assets

Supplementary material from the paper’s appendix.

Appendix

Structure	Operations	Model calls	Depth	Width
Spawn	6	5	3	4
Fork–join	5	5	2	4
Loop	3	2	3	1
Debate	13	12	4	4
C1	13	11	7	4
C2	17	15	8	4

Appendix

Table A.1: Suite graph geometry. Each row summarizes three fixed-bank bindings with the same structure. Width counts model operations at a graph level. C1/C2 follow Figure 5 .

Framework	Execution	Workflow / abstraction	Quality $\uparrow$	LLM	Tools	Input (k)	Output (k)
SWE-bench Verified $N=500$ Resolved rate (%)
Agyn	Reference	Original issue-resolution team	60.4%	37	125	1,156	175
	MAServeBench	Spawn + Refinement Loop	62.4%	41	129	1,074	168
SWE-Debate	Reference	Localization, debate, and MCTS patch search	55.6%	52	146	1,066	167
	MAServeBench	Candidate generation + Debate + iterative search	53.4%	48	143	989	174
GAIA (dev) $N=165$ Answer accuracy (%)

Appendix

Table A.2: Paired functional validation of source MAS workflows and their MAServeBench realizations using Qwen3.5-9B . Each pair shares

N

task instances, model settings, and evaluation procedures. Quality measures resolved rate (SWE-bench Verified), answer accuracy (GAIA), task success (WebArena), or RACE report quality (DeepResearch Bench). LLM/tool calls and input/output tokens (in thousands) are averaged per attempted task. Quality scores across datasets are not directly comparable.

Figure A.1: Request lengths in source workflows and MAServeBench realizations. Reported (a) input and (b) output tokens per LLM request for the paired native evaluations in Table A.2 . Circles and squares denote P50 and P90 over requests; lines join the two distribution percentiles. Colors distinguish reference and realization.

Figure A.2: Source-to-MAServeBench serving-workload correspondence. Eight bindings, three blocks, 24 roots per arm. (a) Antichain width and path depth: binding medians and block ranges. (b) Root-local peak client in-flight ECDF and mean per-root time shares, distinct from batch size. (c) Input tokens over ten progress bins: weighted median, P25–P75 band, dotted P90. (d) Tokens by in-flight at dispatch: median, P25–P75 thick and P10–P90 thin bars; percentages are root-weighted request shares. Empty bins remain unconnected; weighting is defined below.

Metric	Reference	MAServeBench	Paired difference	P90 (R/M)
Model calls	30 [28, 30]	30 [28, 30]	0 [0, 0.2]	30/30
Retrieval calls	4 [4, 4]	4 [4, 4]	0 [0, 0]	4/4
LLM required edges	31 [29, 31]	31 [29, 31]	0 [0, 0.2]	31/31
LLM depth (calls)	20 [18, 20]	20 [18, 20]	0 [0, 0.2]	20/20
LLM level width	3 [3, 3]	3 [3, 3]	0 [0, 0]	3/3
LLM antichain width	3 [3, 3]	3 [3, 3]	0 [0, 0]	3/3

Appendix

Table A.3: Paired serving-workload statistics. Each arm reports median [P25, P75] across eight binding-level medians over three blocks. Paired differences first take the median of three within-binding block differences, then summarize the eight bindings. The last column gives arm-wise P90 over binding medians. Retrieval calls use the shared frozen-document tool boundary; graph metrics use LLM calls. Session rows first summarize eligible sessions within each root; changes and ratios are formed per session before aggregation.

Structure	Native	Replay	Median	Maximum
Composed	2	4	0.118%	0.172%
f&j	1	2	0.136%	0.145%
Loop	2	4	0.471%	0.797%
Earlier f&j	1	2	0.577%	0.658%

Appendix

Table B.1: Bounded native-to-replay calibration. Error is absolute task-latency difference divided by native task latency. Each native trace has two replays.

Protocol	Attempts	Eligible	Replays
Earlier thinking on	2	1	2
Continuation thinking off	5	5	10
Total	7	6	12

Appendix

Table B.2: Replay-calibration attempt ledger. The earlier protocol includes one failed native refinement-loop attempt; successful traces retain their paired replays.

Case	Roots	Cache	Pairs	Median	Root range
FJ	1	Off	3	$-0.33\%$	$[-0.58,-0.01]\%$
FJ	1	On	3	$-0.80\%$	$[-0.93,+0.01]\%$
FJ	16	Off	48	$+0.20\%$	$[-1.74,+2.88]\%$
FJ	16	On	48	$-0.38\%$	$[-1.84,+0.74]\%$
FJR	1	Off	3	$-0.64\%$	$[-0.66,-0.17]\%$
FJR	1	On	3	$-0.50\%$	$[-1.13,-0.37]\%$

Appendix

Table B.3: Replay under concurrency and prefix caching. Median error summarizes paired block means; the range retains individual root errors. FJ: fork–join; FJR: fork–join with refinement.

Workflow	Roots	Cache off	Cache on
FJ	1	0.992 [0.987, 1.003]	0.995 [0.986, 0.997]
FJ	3	0.994 [0.993, 1.003]	0.994 [0.989, 0.997]
FJR	1	1.000 [0.991, 1.002]	1.000 [0.993, 1.001]
FJR	3	0.993 [0.991, 0.995]	0.999 [0.990, 1.004]

Appendix

Table B.4: Optional token/length replay wall-time ratios on the same engine and runner. Each cell is a three-block median [minimum, maximum]. FJR consumes the preceding Fork–Join artifact.

Study	Model / platform	Users	Limit/shape
Demand (Fig. 3 )	Qwen3-8B / 910C	32/64/128	128
Execution shapes (Fig. 5 )	Qwen3-8B / 910C	64	128
Attention captures (Fig. 8 a)	Qwen3-8B / 910C	64	64
State pressure (Fig. 9 )	Qwen3-8B / 910C	64	128
Stage/root (Fig. G.2 )	Qwen3.5-9B / RTX PRO 5000	64	64
NPU hybrid captures (Sec. 6.3 )	Qwen3.5-9B / 910C and 950DT	32	32

Appendix

Table C.1: Application and operator protocol map. Application rows give concurrent root users and request limits; operator rows give execution shape. Timing windows and native captures are retained separately.

Quantity	Scope and unit	Statistic
Arithmetic intensity	Qwen3-8B scheduler steps; FLOP/byte	$F/D$ for scheduled work and logical traffic; progress-bin duration weighting; aggregate $\sum F/\sum D$
Allocated KV	Scheduler pool; GiB	Allocated blocks times block capacity; residency integrates occupancy over cohort wall time
Kernel-duration share	Native capture; percent	Duration sum of one mutually exclusive kernel family divided by all classified kernel duration in that capture
Projection time	Matched primitive; microseconds	CUDA events on GPUs; corresponding native Memory-pass ranges on NPUs
Physical DRAM reads	Counter pass; bytes per complete primitive	All constituent kernels summed per marked primitive, then median across complete samples; one BF16 weight normalizes the plotted reads
Stage and root latency	Unprofiled run; seconds	Within-binding means balanced across common task bindings; paired block ratios summarized by their median

Appendix

Table C.2: Measurement contracts. Numerators and denominators refer to the same stated observation scope.

Rate (roots/s)	Release	KV (GiB)	Goodput: med/min/max	Attainment: med/min/max
0.00677316	DAG	16	0.008466/0.005927/0.010160	1.000/1.000/1.000
0.00677316	DAG	40	0.008466/0.005927/0.010160	1.000/1.000/1.000
0.00677316	Barrier	16	0.002540/0.002540/0.005080	0.429/0.300/0.500
0.00677316	Barrier	40	0.003387/0.002540/0.003387	0.400/0.333/0.429
0.01625559	DAG	16	0.018626/0.014393/0.021166	1.000/0.917/1.000
0.01625559	DAG	40	0.018626/0.014393/0.021166	1.000/0.917/1.000

Appendix

Table C.3: Legal-release/KV deadline outcomes: median/minimum/maximum across three paired blocks. Goodput is timely roots per arrival-window second; attainment is the fraction meeting the shared deadline.

Figure C.1: Offline deadline sensitivity over the complete formal grid. Columns use four multiples of the same independent DAG/40 GiB reference P95. Top: task attainment; bottom: arrival-window deadline goodput. Lines/markers show three-block medians and shaded bands span minimum to maximum for each configuration. Exact offered rates and unchanged arrival/drain counting follow Table C.3 . Every task is re-scored from its original arrival/completion timestamps; no serving execution is repeated.

Release	KV (GiB)	Preemptions	Prefill factor	Root P95 (s)	Output TPS
DAG	16	628/754/0	1.981	948.7	813.3
DAG	40	0/0/0	1.000	801.8	876.2
Barrier	16	480/590/0	1.748	1067.3	775.4
Barrier	40	0/0/0	1.000	999.6	800.5

Appendix

Table C.4: Highest-rate state and service outcomes. Preemptions retain blocks 1/2/3; other columns are three-block medians. These are marginal summaries, not paired-effect ratios.

Figure C.2: High-load KV recomputation. (a) Scheduler preemption events. (b) Processed/source prefill. Every configuration retains all three paired blocks, including the 43-root block without preemption.

Figure C.3: Throughput and task tails over offered load. (a) Output TPS over the drain-inclusive period. (b) Root P95 latency. Points show three-block medians; whiskers span their ranges.

Figure C.4: Batch exposure and state over offered load. (a) Decoding batch weighted by pure-decode step duration through final request completion. (b) Peak allocated KV occupancy. Points show three-block medians and ranges.

Figure C.5: Request-limit sensitivity. Qwen3-8B/910C, higher/lower limit 128/64. (a) Output TPS ratio. (b) Drain-inclusive timely-task-rate ratio. Highest tested rates are 1.2, 0.5, and 0.2 roots/s for ordinary, Loop, and mixed work. Filled points show paired-block medians, whiskers ranges, and open points individual blocks. Dashed line: parity.

Figure C.6: Ordinary-serving load response. Qwen3-8B/910C, shared arrivals across limits 64/128. (a) Arrival-window SLO goodput. (b) Minimum class attainment. Bars show three confirmation-block medians and ranges under locked deadlines.

Figure C.7: Loop load response. Same arrival and SLO contract as Figure C.6 ; each root executes the recorded loop work. Both request limits retain the same per-block arrival stream.

Figure C.8: Mixed-workflow load response. Same contract as Figure C.6 , with seven workflow classes and class-specific locked deadlines. Panel (b) reports the least attained class in each confirmation block.

Context	Block	Mean root time	KV integral
Full	1	1.266	0.953
Full	2	1.264	0.954
Full	3	1.257	0.957
Selected	1	1.280	0.958
Selected	2	1.283	0.957
Selected	3	1.281	0.951

Appendix

Table D.1: Serial/parallel ratios at fixed resolved work. Each cell is one paired 12-root block with exact payload and input/output-length checks.

Figure D.1: Optional control across context settings. Qwen3-8B/910C. (a) All 72 paired root observations. (b) Full-context block-1 KV trajectories under parallel/serial control. Each pair retains inputs, output lengths, and required dependencies.

Control	Context	Input tokens	Root (s)	KV (GiB s)
Parallel	Full	1275.4	11.318	11.522
Serial	Full	1286.8	14.431	11.644
Parallel	Selected	906.7	10.753	8.585
Serial	Selected	895.1	13.850	8.216

Appendix

Table D.2: Native demand of control/context cells: three-block medians of per-root input means and completed-root latency means, and per-cohort KV integrals.

Figure D.2: Native control and context delivery. Qwen3-8B/910C, 12 bindings per cell per block. (a) Generated input tokens per root. (b) Completed-root latency. Native artifacts propagate through synthesis and verification; Table D.2 defines the aggregation.

Binding	Calls	Input (k)	Output (k)
Asyncio	28/28	77.038/73.496	19.589/18.122
cgroup v2	30/30	85.928/77.785	24.324/24.785
HTTP cache	25/30	68.495/87.922	16.154/23.131

Appendix

Table D.3: Source-workflow native work underlying Figure 4 (a). Entries are Full/Selected, one native pair per document binding. Token counts are in thousands.

Binding	Calls	Refine- ments	Writing passes	Edges	Depth
Asyncio	28/28	3/3	3/3	29/29	18/18
cgroup v2	30/30	4/4	3/3	31/31	20/20
HTTP cache	25/30	3/4	2/3	26/31	15/20

Appendix

Table D.4: Native structural trace differences. Entries are Full/Selected, one native pair per binding. Refinements count section-draft revisions; writing passes include the initial pass. Edges and depth use the LLM-only graph after collapsing non-LLM paths and removing transitive edges. Depth counts calls.

Figure D.3: Demand relative to serving. Matched user concurrency. Three-block median ratios: (a) aggregate

\sum F/\sum D

; (b) KV residency integral

\int K(t)\,dt

Figure D.4: Natural input/output demand across 210 task/structure executions. Shared bindings retain generated artifacts and output lengths; model-call medians range from two for loop to sixteen for the composed sequence.

Figure D.5: Absolute-time demand on Qwen3-8B/910C for 64 users. Every block is plotted; block zero is emphasized consistently. Curves end at their own run completion.

Figure D.6: Fixed-call demand controls on Qwen3-8B/910C, request limit 32. Solid lines preserve workflow dependencies; dashed lines release independent calls from the same resolved work set. Each condition contains three blocks. Horizontal axes show each cohort’s own progress.

Figure D.7: Fixed-call demand controls on Qwen3-8B/910C, request limit 64. Solid lines preserve workflow dependencies; dashed lines release independent calls from the same resolved work set. Each condition contains three blocks. Horizontal axes show each cohort’s own progress.

Figure D.8: Fixed-call demand controls on Qwen3-8B/910C, request limit 128. Solid lines preserve workflow dependencies; dashed lines release independent calls from the same resolved work set. Each condition contains three blocks. Horizontal axes show each cohort’s own progress.

Family	$N$	$K$	Instances	MiB/weight
Auxiliary	64	4096	24	0.5
Output	4096	4096	32	32
Down	4096	12288	32	96
Attention input	10240	4096	8	80
Recurrent input	12288	4096	24	96
Gate/up	24576	4096	32	192

Appendix

Table E.1: Qwen3.5-9B projection support. Footprints count distinct BF16 weight instances in the model.

$M$	$N$	A800	RTX PRO 5000
512	4096	90.83	121.85
512	24576	447.50	549.53

Appendix

Table E.2: Wider-row GPU controls: CUDA-event time in

\mu

s, with one reused weight.

Rows	910C	950DT	Ratio
1	4.397	2.007	2.19
8	3.445	2.867	1.20
32	5.480	5.017	1.09

Appendix

Table E.3: Matched normalization: native Memory-pass durations in

\mu

Workflow	Modal $M$	Share	Modal $M$	Share	$M>128$
	Early		Late		Late
Serving	64	100.0	24	92.5	0.0
Spawn	64	100.0	128	84.0	15.6
Fork–Join	128	100.0	128	66.9	32.7
Loop	64	100.0	64	100.0	0.0
Debate	128	100.0	128	89.9	9.8

Appendix

Table E.4: Native matrix rows, Qwen3-8B/910C. Shares are percentages of each capture’s matrix kernel time. Early/Late are capture windows; the last column gives Late time at

M>128

Figure E.1: Batch evolution and final drain, Qwen3.5-9B/RTX PRO 5000, 64 users and a 64-request limit. (b) decoding time with at most eight requests, split at the first scheduling of the last new model call. Final drain accounts for most low-batch exposure.

Family	Rows	Weights	A800	RTX PRO	Ratio
Auxiliary	1	1	6.696	2.964	2.259
Auxiliary	1	8	6.528	2.960	2.205
Output	1	1	23.312	13.089	1.781
Output	1	8	25.136	37.601	0.668
Down	1	1	67.032	37.980	1.765
Down	1	8	66.832	109.479	0.610

Appendix

Table E.5: All 28 GPU projection event-time cases for Figure 6 . Times are microseconds; ratio is A800/RTX PRO 5000. Weight count is the rotating address set size.

Bytes	Layout	Template	Path	Time	Binary SHA
64	ordered	SIMD	SIMD	24.608	377f67f79470
64	ordered	Hybrid	SIMD	24.660	377f67f79470
64	ordered	SIMT	SIMT	3.566	01a1d6e8b3e6
64	permuted	SIMD	SIMD	24.475	377f67f79470
64	permuted	Hybrid	SIMD	24.507	377f67f79470
64	permuted	SIMT	SIMT	3.724	01a1d6e8b3e6

Appendix

Table E.6: All 12 gather path checks for Figure 7 . Within each layout, source/index hashes and the numerical contract match; binary hashes identify the actual compiled path. Times are native Memory-pass microseconds.

Workflow	Arm	Quantile	Full	Half	Ratio
Mixed	MAS	0.25	551.8	288.0	1.92
Mixed	MAS	0.75	743.4	383.3	1.94
Mixed	independent	0.25	691.1	362.1	1.91
Mixed	independent	0.75	742.4	376.3	1.97
f&j	MAS	0.25	415.7	207.8	2.00
f&j	MAS	0.75	512.0	263.0	1.95

Appendix

Table F.1: Application-derived context controls, batch 64. Operator times are in

\mu

s. Quantiles select representative context samples within each arm.

Figure F.1: Full-attention context and native execution distributions, Qwen3-8B/910C, 64 users with a 64-request limit. The four context controls include every pure-decoding step, weighted by step duration. Panel (b) contains four MAS early/late captures; all eight arms/phases contain 53,172 native attention calls.

Family	Rows	Weights	One weight	A800 reads	RTX PRO reads
Auxiliary	1	1	0.500	0.000000	0.000000
Auxiliary	1	8	0.500	0.000000	0.000000
Output	1	1	32.000	25.621948	0.000000
Output	1	8	32.000	32.013550	31.997192
Down	1	1	96.000	96.031555	60.832153
Down	1	8	96.000	96.031921	96.000488

Appendix

Table F.2: All physical reads underlying Figure 6 . Values are MiB per complete sampled primitive call; one-weight size is the plotting denominator. Event timing is retained in a separate pass in Table E.5 .

Block	Preemptions	Victims	Processed prefill	Source input	Factor	Loop factor
0	470	186	1052422	571973	1.840	1.000
1	471	189	1025513	570441	1.798	1.000
2	431	180	1004811	571822	1.757	1.000

Appendix

Table F.3: State-event and repeated-work ledger for Figure 9 . Mixed and loop-only controls use 64 users, request limit 128, and the same 31.99 GiB allocated pool. Victims count distinct mixed-run request IDs.

Block	Budget	Actual	Preemptions	Factor	P95
1	16	15.996	887	2.523	1344.587
2	16	15.996	899	2.557	1347.172
3	16	15.996	900	2.478	1323.505
1	32	31.992	460	1.843	1141.455
2	32	31.992	511	1.933	1157.747
3	32	31.992	425	1.791	1130.362

Appendix

Table F.4: KV-budget run ledger. Actual allocation is GiB; factor is processed/source prefill; P95 is seconds. Every row contains 64 completed roots.

Figure F.2: Root outcomes across KV budgets. Same fixed-work comparison as Figure 9 . (a) All 192 root latencies per budget across three blocks. (b) Each block’s root P95.

Figure F.3: State and batch trajectories across KV budgets. Same Qwen3-8B/910C comparison as Figure 9 . All nine runs are shown over absolute time; every trajectory ends at its own completion.

Block	Calls	Input	Output	A share	B share
1	236	180,249	517,259	96.278%	31.474%
2	236	179,890	516,780	96.275%	31.442%
3	236	180,229	515,145	96.280%	31.461%

Appendix

Table G.1: Balanced temporal-mixing audit. A and B have identical per-block totals. The common background is 19 Loop + 18 Spawn + 18 Fork–Join roots; each arm also retains nine targets. Shares are time-weighted within 2–110 s.

Binding	Block	Call 1 (s)	Call 2 (s)	Root change
GSM8K	1	-24.027	+31.264	+2.169%
GSM8K	2	-21.461	+32.747	+3.439%
GSM8K	3	-23.953	+28.384	+1.351%
PubMedQA	1	-17.206	+23.613	+2.136%
PubMedQA	2	-14.452	+24.905	+3.525%
PubMedQA	3	-16.170	+19.943	+1.253%

Appendix

Table G.2: Balanced temporal mixing, all three paired blocks. Top: three targets per binding/block; call entries are mean paired scheduled-residence differences (A minus B, seconds). Root entries are ratios of means, not means of individual ratios. Bottom: overall mean path differences over nine targets. Pre/Wait are milliseconds; residence and root differences are seconds.

Block	Background	Calls	Input tokens	Output tokens	Target mean	Target P95
1	Loop only	110	76,503	290,861	256.203	295.974
1	Perturbed	116	79,911	304,773	268.939	309.799
2	Loop only	110	76,144	290,382	257.026	296.631
2	Perturbed	116	79,552	304,294	268.380	308.671
3	Loop only	110	76,483	288,747	257.521	296.758
3	Perturbed	116	79,891	302,659	267.643	307.455

Appendix

Table G.3: Near-matched background work and fixed-target outcomes. Each window contains nine targets and 55 background roots. Calls and tokens refer only to the background; mean/P95 are target-root latency in seconds.

Call	Input	Block 1	Block 2	Block 3
1	1	2.654	2.405	2.011
1	2	3.068	2.717	2.704
1	3	3.075	2.711	2.651
2	1	1.868	1.590	1.503
2	2	2.094	1.872	1.552
2	3	2.293	2.376	1.945

Appendix

Table G.4: Paired increases in effective decoding time per output token (ms). Each cell averages three repeated targets of one frozen input within one block.

Figure G.1: Near-matched fixed-target completion paths. (a) Target-root latency distributions. (b) Paired changes in block-mean completion-path terms. Each point represents one block; scheduled residence includes pauses after first scheduling.

Figure G.2: Stage-to-task changes across cohorts. The original seven-structure Separate/Mixed characterization, Qwen3.5-9B/RTX PRO 5000. (a) Selected stage times and paired percentage changes. (b) Root latency. (c) Mixed-minus-separate path changes; positive means longer latency. Separate runs use 64 users per class; mixed runs use 64 total (ten Spawn and nine per remaining class).

Figure G.3: Complete stage response to mixed composition. Qwen3.5-9B/RTX PRO 5000. Separate runs use 64 users per class; mixed runs use 64 total. All 26 stage groups: bars are medians of three binding-balanced block means; labels are medians of paired block ratios. Short arrows use a minimum display length. C1/C2 follow Figure 5 ; (b) is the research–review–writing sequence. Figure G.2 (a) selects six of these groups.

Workflow	Root	Pre-submit	Residence
Loop	+129.48	+118.49	+11.21
Sequence	-635.39	-586.36	-49.05
Spawn–f&j–loop	-178.56	-167.44	-10.93
f&j–debate–loop	-264.74	-231.78	-33.03

Appendix

Table G.5: Mean root-latency changes (mixed minus separate), seconds. Columns follow observed completion paths. Backend and boundary terms complete the exact ledger.

Figure G.4: GPU mixed attribution, Qwen3.5-9B/RTX PRO 5000. (a) all 78 stage/block pairs linking co-batch exposure to stage latency. (b) balanced conditional native matrix-work distributions for early loop-containing batches, matched on the same physical GPU and scheduler conditions.

Class	Separate runs	Roots/ run	Separate total	Mixed/ block	Mixed total	Audit total
Spawn	3	64	192	10	30	222
Fork–Join	3	64	192	9	27	219
Loop	3	64	192	9	27	219
Debate	3	64	192	9	27	219
C1	3	64	192	9	27	219
C2	3	64	192	9	27	219

Appendix

Table G.6: Root-count ledger across the three blocks. Each separate window contains 64 roots of one class. Each of three mixed windows contains 10 Spawn roots and 9 of each other class. All recorded roots complete. The ordinary control is an additional window in the full audit.

Workflow	Stage	S0	M0	S1	M1	S2	M2
Spawn	S1	275.73	311.09	275.35	316.60	277.97	319.44
Fork–Join	S1	272.41	252.68	270.12	256.39	269.67	274.15
Loop	S1	154.64	276.15	153.14	292.11	154.54	282.51
Debate	S1	593.07	428.92	591.09	419.20	590.47	437.42
C1	S1	302.02	341.09	302.86	312.72	305.40	298.56
C1	S2	412.10	211.89	411.93	221.74	409.25	235.69

Appendix

Table G.7: All 26 stage groups underlying Figure G.2 . S/M are separate/mixed binding-balanced stage means in seconds; columns retain every paired block.

Figure G.5: Unmatched background demand on fixed targets. (a) All 27 target-root latencies per arm. (b) Background input/output work by block. The heterogeneous arm supplies substantially more work.

Block	Background	Target mean (s)	Background tokens
1	Same class	254.946	367,364
1	Heterogeneous	451.322	1,808,641
2	Same class	256.105	366,526
2	Heterogeneous	411.475	1,798,945
3	Same class	257.084	365,230
3	Heterogeneous	491.703	1,838,309

Appendix

Table G.8: Unmatched-demand fixed-target run ledger. Root time is the mean over nine tagged targets; background work sums source input and output tokens across 55 roots.

Block	A/B	$\Delta$ Root (s)	$\Delta$ Pre (ms)	$\Delta$ Wait (ms)	$\Delta$ Res (s)
1	0.9975	-0.657	-0.005	-6.73	-0.651
2	1.0035	+0.924	-0.002	-5.14	+0.930
3	0.9920	-2.124	+0.042	+9.33	-2.134

Appendix

Table G.9: Fixed-population temporal mixing. A/B is interleaved/clustered mean target latency; deltas are A minus B. Pre: ready to submission; Wait: submission to first scheduling; Res: first scheduling to completion, including pauses.

LLM Agent Serving

Large language model agent serving runs tool-using, multi-step agents efficiently by coordinating inference, memory, and workflow execution.

See all 37 papers

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
Oct 3, 2026Qizhi Chu, Zekai Yu, Sijie Wen +5Multi-Agent LLM Systems Multi-Agent Collaboration
Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University · Tsinghua University
ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration
Aug 9, 2026Guo Chen, Ziwen Li, Reed Li +4Multi-Agent LLM Systems AI Agent Evaluation
Southwest University Chongqing, China · Tencent Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences Beijing, China
Competence-Aware Orchestration for LLM-Based Multi-Agent Systems
Oct 9, 2026Songmiao Wang, Yuchen Wang, Chenglin Yu +2LLM Agent Orchestration Multi-Agent LLM Systems
The Hong Kong Polytechnic University

Oct 3, 2026J/K move · Enter open · S save

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability

Qizhi Chu, Zekai Yu, Sijie Wen, Yang Liu+4

Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University · Tsinghua University

Multi-Agent LLM Systems Multi-Agent Collaboration AI Agent Benchmarks Multi-Agent Communication

Open paper AskPDF arXiv

Large language models (LLMs) have progressively evolved into the core of autonomous agents. Building on this progress, LLM-based multi-agent systems (MAS) coordinate multiple agents into a synergistic team to accomplish complex tasks that exceed the capabilities of individual agents. The effectiveness of such systems depends not only on the agents themselves, but also on how collaboration mechanisms are designed and organized. Note that real-world collaboration is typically partially observable, where each agent can only access partial information about the environment due to physical or privacy-related constraints. However, many existing multi-agent benchmarks assume global observability, and leave limited support for systematically evaluating collaboration mechanisms. To bridge this gap, we introduce MASBench, a multi-agent collaboration benchmark designed under partially observable constraints. It is organized into three progressive task categories: Reasoning, Scheduling, and Game. Through this structure, we progressively evaluate three representative collaboration mechanisms: Protocol, Memory, and Routing. MASBench further provides deterministic evaluation metrics, including performance score, communication cost, and cost effectiveness, to characterize both collaboration outcomes and communication overhead. Experiments across diverse LLM backbones and mechanism configurations offer empirical guidance for effective MAS design. Code is available at: https://github.com/BUPT-GAMMA/MASBench

Abstract

LLM Agent Serving

Explore similar work

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability

ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration

Competence-Aware Orchestration for LLM-Based Multi-Agent Systems