In a multi-agent system (MAS), removing an expensive agent call can save tokens without making the answer arrive sooner: latency depends on the longest chain of dependent calls. We present Latency-Aware Multi-agent System (LAMaS), a modular platform for learning low-latency orchestration. A framework can send a query to LAMaS and receive an answer and execution trace, or keep its own graph builder and use the platform's latency-optimized orchestration layer. During training, the layer gives operators on the critical path the greatest latency penalty while targeting a floor on expected accuracy. During inference, a lightweight controller uses partial results to decide whether to skip work that has yet to start. Across four benchmarks, LAMaS reduces end-to-end latency by over 50% relative to the evaluated learning-based MAS baselines while maintaining competitive or better accuracy. GDesigner and AgentDropout retain their graph builders and train the controller on their own traces; both achieve lower latency across the same benchmarks.
Figures & tables
Figure 1: LAMaS connects agent frameworks to model servers and returns an answer and execution trace for each query. We integrate the full pipeline into GDesigner and AgentDropout, retaining each framework’s graph builder and retraining the execution controller on its traces.
Figure 2: Two execution graphs that produce the same answer at similar cost but differ in latency. Nodes denote agent invocations and arrows denote data flow; red edges mark the critical execution path, the longest chain of dependencies that must complete before an answer is available. Graph B has a shorter critical path and therefore lower wall-clock latency than Graph A.
Figure 3: Training and inference in the LAMaS orchestration layer. (1) Material: A library of reusable operators and tools. (2) Training: Phase 1 performs latency-aware graph optimization via Lagrangian-constrained learning and critical-path credit assignment, where S0 denotes the target accuracy floor enforced during optimization. Phase 2 trains a lightweight execution controller on the orchestrator’s traces to predict the control advantage. (3) Inference: The trained orchestrator first constructs the execution graph; the execution controller then performs online execution control, monitoring operator completions and adaptively eliminating redundant future agent interactions.
Dataset
Method
Acc (%) ↑
Cost ( ×10−4 ) ↓
Latency (s) ↓
CP Tokens ↓
GSM8K
Generate
87.46±0.20
2.87±0.05
5.43±0.18
280±8
CoT
88.97±0.43
4.53±0.04
6.92±0.21
331±15
SC
89.86±0.19
18.61±0.09
10.64±0.27
591±22
GDesigner
93.30±0.93
17.86±0.28
32.31±1.07
1624±29
AgentDropout
93.11±0.27
53.81±6.78
81.24±12.74
4098±538
MaAS
93.36±0.72
31.52±0.78
48.11±3.49
2501±53
Table 1: Comparison with non-MAS and MAS baselines across four benchmarks. Each cell reports mean ± standard deviation across 3 independent runs. For MAS methods, the best mean in each column per dataset is shown in bold . Cost is reported in units of ×10−4 USD. Latency is the average end-to-end wall-clock time per query (in seconds). CP Tokens denotes the average number of output tokens on the critical execution path.
Dataset
Method
Acc (%) ↑
Cost ( ×10−4 ) ↓
Latency (s) ↓
GSM8K
LAMaS
93.65
7.71
11.73
w/o controller
93.27
9.43
13.84
w/o CP Credit
92.73
8.71
12.93
w/o latency obj. ( λt=0 )
93.59
9.71
15.83
naive lat. only
91.85
9.86
15.52
HumanEval
LAMaS
95.42
12.92
21.00
Table 2: Ablation study of inference-time execution control, critical-path (CP) credit assignment, and the latency objective across four benchmarks. Cost is in units of ×10−4 USD; latency is end-to-end wall-clock time per query (in seconds).
Dataset
Method
Acc (%) ↑
Cost ( ×10−4 ) ↓
Latency (s) ↓
CP Tokens ↓
GSM8K
GDesigner
93.30
17.86
32.31
1624
GDesigner+LAMaS
93.27
12.08
20.55
1138
AgentDropout
93.11
53.81
81.24
4098
AgentDropout+LAMaS
93.18
39.67
58.14
2989
HumanEval
GDesigner
85.50
19.86
49.68
2983
GDesigner+LAMaS
85.75
12.93
30.85
1571
Table 3: Transferability of the full LAMaS pipeline (training-time graph optimization, critical-path credit assignment, and inference-time execution control) to other learning-based MAS architectures. The controller is retrained on each architecture’s own traces (Phase 2 of the LAMaS training procedure). Latency is the average end-to-end wall-clock time per query (in seconds). CP Tokens denotes the average critical-path output tokens.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Dataset
#Train
#Test
Metric
Code Generation
HumanEval
33
131
pass@1
Math Reasoning
GSM8K
264
1055
Accuracy
MATH
119
486
Accuracy
Knowledge Reasoning
MMLU-Pro
125
500
Accuracy
Appendix
Table 4: Dataset statistics.
Setting
Pearson r
Spearman ρ
MAS methods × 4 benchmarks
0.995
0.985
Appendix
Table 5: Correlation between CP Tokens and measured wall-clock latency across MAS-method/benchmark cells in Table 1 .
Benchmark
Decision Acc (%) ↑
Precision ↑
Recall ↑
F1 ↑
HumanEval
73.5
0.72
0.66
0.69
MMLU-Pro
70.8
0.69
0.63
0.66
Appendix
Table 6: Decision-level agreement between the learned execution controller and the oracle stopping rule on HumanEval and MMLU-Pro. The positive class is the “stop” decision.
Benchmark
Controller
Acc (%) ↑
Latency (s) ↓
HumanEval
No controller
95.17
24.47
Learned
95.42
21.00
Oracle
95.67
18.84
MMLU-Pro
No controller
66.00
17.94
Learned
66.27
14.81
Oracle
66.73
13.07
Appendix
Table 7: End-to-end accuracy and latency under different controllers on HumanEval and MMLU-Pro. The “Learned” rows correspond to the LAMaS configuration in Table 2 , and “No controller” rows correspond to the w/o controller ablation.
Figure 4: Average operator selection probabilities on HumanEval after inserting an unseen operator, Debate , at inference time. The unseen operator receives non-trivial probability mass comparable to several seen operators, indicating that the learned policy can still reasonably select operators not exposed during training.
Dataset
λt
Acc (%) ↑
Latency (s) ↓
HumanEval
0
95.67
35.62
0.001
95.93
23.50
0.005 ⧫
95.42
21.00
0.01
94.91
13.91
0.05
93.89
11.29
MMLU-Pro
0
66.33
22.46
Appendix
Table 8: Sensitivity to the latency penalty weight λt . Increasing λt consistently reduces latency, with mild accuracy variation across the sweep.
Dataset
τ
Acc (%) ↑
Latency (s) ↓
HumanEval
0.1
93.89
13.57
0.2
94.91
15.35
0.3 ⧫
95.42
21.00
0.4
95.67
22.80
MMLU-Pro
0.1
65.13
13.42
0.2
65.93
14.23
Appendix
Table 9: Sensitivity to the activation threshold τ . A higher threshold activates more operators per step, increasing accuracy and latency. A lower threshold produces leaner graphs with lower latency but reduced accuracy.
Dataset
L
Acc (%) ↑
Latency (s) ↓
HumanEval
2
94.15
12.85
4 ⧫
95.42
21.00
6
95.67
23.50
8
95.93
25.54
MMLU-Pro
2
64.93
12.81
4 ⧫
66.27
14.81
Appendix
Table 10: Sensitivity to the number of supernet layers L . More layers provide greater reasoning depth and slightly higher accuracy, but increase latency due to deeper critical paths.
Dataset
S0/Sref
Acc (%) ↑
Latency (s) ↓
HumanEval
0.95
94.66
17.85
1.00 ⧫
95.42
21.00
1.02
95.67
22.00
1.05
96.18
23.54
MMLU-Pro
0.95
65.40
12.60
1.00 ⧫
66.27
14.81
Appendix
Table 11: Sensitivity to the accuracy floor S0 . We vary S0 as a fraction of the architecture’s reference accuracy Sref ( S0=Sref is the default, marked by ⧫ ).
Dataset
Backbone
Method
Acc (%) ↑
Latency (s) ↓
HumanEval
qwen3.5-flash
MaAS
94.91
100.87
LAMaS
96.95
38.93
qwen3.5-9b
MaAS
95.17
105.18
LAMaS
95.93
42.18
MMLU-Pro
qwen3.5-flash
MaAS
82.40
174.60
LAMaS
82.93
98.04
Appendix
Table 12: Generalizability across Qwen backbones and serving setups: closed-source qwen3.5-flash served through OpenRouter and open-weight qwen3.5-9b served locally with vLLM. Latency is end-to-end wall-clock time per query (s). Best mean among MAS methods per column per dataset is in bold .
Dataset
Method
Acc (%) ↑
Cost ( ×10−4 ) ↓
Latency (s) ↓
GSM8K
Tool-augmented CoT
89.79
11.38
18.27
LAMaS
93.65
7.71
11.73
HumanEval
Test-time repair
86.51
6.35
15.43
LAMaS
95.42
12.92
21.00
MMLU-Pro
Self-Refine
57.13
5.51
15.19
LAMaS
66.27
10.93
14.81
Appendix
Table 13: Comparison with single-agent agentic baselines. LAMaS rows are taken from Table 1 . Cost is in units of ×10−4 USD; latency is end-to-end wall-clock time per query (in seconds).
Dataset
Method
Acc (%) ↑
Cost ( ×10−4 ) ↓
Latency (s) ↓
GSM8K
GDesigner
93.30
17.86
32.31
GDesigner + consensus-2 stop
92.61
13.97
26.92
GDesigner + half-agents
91.37
11.24
21.37
GDesigner+LAMaS
93.27
12.08
20.55
MMLU-Pro
GDesigner
64.27
25.03
46.66
GDesigner + consensus-2 stop
62.93
19.02
39.48
Appendix
Table 14: GDesigner under two naive inference-time latency-cutting rules. The GDesigner and GDesigner+LAMaS rows are taken from Tables 1 and 3 . Cost is in units of ×10−4 USD; latency is end-to-end wall-clock time per query (in seconds).
Multi-agent systems (MAS) coordinate multiple LLM-powered agents through structured workflows, gaining reasoning power but incurring high inference latency from multi-step execution and repeated model invocations. Existing orchestration methods primarily optimize task performance and inference cost, leaving latency largely unaddressed. In MAS, end-to-end latency is governed by the critical execution path, so reducing total cost alone does not reliably reduce latency. Moreover, optimizing latency while preserving accuracy remains non-trivial: naive latency optimization can misassign operator-level credit and degrade task accuracy. To address this gap, we propose Latency-Aware Multi-agent System (LAMaS), a latency-aware orchestration framework for learning-based multi-agent systems. LAMaS addresses this challenge at two levels: at training time, it learns latency-aware execution graphs through constrained optimization with critical-path-aware credit assignment; at inference time, since a graph committed at training time cannot exploit runtime evidence, it complements graph construction with a lightweight controller that adaptively eliminates redundant future agent interactions as execution unfolds. Experiments on four benchmarks show that LAMaS achieves the best latency among evaluated learning-based MAS baselines, reducing end-to-end latency by over 50% while maintaining competitive or better accuracy. LAMaS is also modular and transfers to other MAS with minimal changes, consistently yielding latency reductions.
Multi-agent LLM systems spend additional computation on reasoning and collaboration, but shared serving queues can prevent that computation from producing answers before a deadline. This creates a coupled decision: the reasoning workflow determines the load, while changing load determines which reasoning choices remain affordable. We introduce INFRAMIND, a hierarchical policy that adapts workflow planning and execution to serving conditions. At arrival, a planner selects the collaboration graph using task information, load and the time budget. Before each agent call, an executor jointly selects the model and reasoning strategy using refreshed telemetry and remaining time. A shared latency penalty connects the two policies. Our analysis characterizes why observing load matters when congestion reverses the preferred computation. Across five benchmarks on a shared five-model deployment, INFRAMIND improves low-load accuracy by up to 7.6 percentage points over the strongest evaluated baseline. On 500 MATH questions with matched EDF scheduling and per-request deadlines of 10-300 s, it increases correct, on-time answers from 20.4% to 33.8%. With 15 models and Azure timings replayed on the same benchmark and deadline tiers, this fraction rises from 11.6% to 54.8% over MasRouter. These results support adapting reasoning decisions to the serving capacity available throughout a workflow.
LLM-based multi-agent systems (MAS) increasingly use latent collaboration to avoid the information loss and repeated encoding-decoding overhead of natural-language communication. However, directly forwarding all sender latents makes the receiver-side context scale with both the number of agents and the reasoning length, increasing computation, memory usage, and collaboration latency. A natural solution is latent compression. But we find that cross-agent redundancy remains unresolved in existing latent compression approaches, which typically compress each sender independently and then concatenate the results. We propose LatCom, a cross-agent latent compression framework for efficient multi-agent latent collaboration. LatCom maps multiple sender latents into a fixed number of receiver-readable and task-relevant slots. Rather than reconstructing all sender hidden states, it optimizes the compressed latents for receiver-side task utility. LatCom trains the compressor in two stages: single-sender readability learning first establishes a latent interface interpretable by the frozen receiver, and multi-sender fusion learning then trains the compressor to fuse complementary evidence and remove redundancy across agents. Experiments on multiple benchmarks with Qwen3-4B show that LatCom achieves an average 2.46x inference speed-up over LatentMAS and reduces output token usage by 70.3% while maintaining comparable average accuracy.
Shinan Zhang, Tao Zhang, Qihui Zhu +7
University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center