Large language models (LLMs) are increasingly deployed as the execution core of autonomous agents rather than as standalone text generators. Agentic workloads induce a temporal shift from single-turn inference to multi-turn LLM-tool loops, and a spatial shift from chat-scale, GPU-only execution to repository-scale, GPU-CPU co-located execution. Consequently, coordinating heterogeneous resource demands of agentic execution has emerged as a critical system challenge. We design and implement MARS, an efficient and adaptive co-scheduling system that globally coordinates heterogeneous agentic workloads under coupled GPU-CPU resource pressure. By establishing holistic visibility across GPU inference and CPU tool execution via a unified information stream, an external control plane in MARS decouples admission from execution to prevent heterogeneous resource oversubscription. An internal agent-centric scheduler further minimizes the end-to-end critical path by prioritizing latency-sensitive continuations and adaptively retaining KV cache state only when warm resumption yields a latency benefit. Our evaluations show that MARS reduces end-to-end latency by up to 5.94x while maintaining nearly maximal system throughput. We further integrate MARS as the serving backend for the OpenHands coding agent framework, demonstrating its real-world effectiveness by accelerating end-to-end task completion time by up to 1.87x. Our source code is publicly available at https://github.com/Afterglow231/MARS_preview .
Figures & tables
Figure 1. Agentic workloads induce both temporal and spatial shifts. Together, these shifts create dynamically misaligned GPU and CPU demand across concurrent requests.
Figure 2. Token throughput vs. Dynamic SLO-Aware Goodput. While baselines sustain high token throughput, they experience complete collapse as load increases while MARS successfully maintains high request completion rates.
Figure 3. KV cache eviction dynamics and multi-turn responsiveness. (A) The rate of KV block removals over the normalized execution progress of the workload. (B) The average Time to First Token (TTFT) per round across different latency percentiles.
Figure 4. MARS architecture overview.
Subsystem
Event
Signal
Systemic Role
GPU Plane
gpu_submit
Projected KV
Triggers capacity check
gpu_1st_token
Launch Delay
Estimates prefill bounds
gpu_end
Freed Blocks
Releases capacity
CPU Plane
tool_num
Active Tools
Signals host congestion
tool_start/end
Exec Duration
Isolates CPU starvation
Control Plane
window_update
Dual-Pressure
Modulates Wadm
Table 1. Mapping Unified Info Stream to control actions. By recording phase boundaries, MARS translates opaque hardware states into actionable scheduling telemetry.
Figure 5. Internal Agent-Centric Scheduler.
Figure 6. Input-length distributions and ideal execution time of the workloads.
Figure 7. Mean, P90, and P95 end-to-end latencies for Qwen3-Coder-30B on the H100 testbed under four input-length regimes ( ILR-1 to ILR-4 ). Red text denotes the speedup of MARS over the fastest baseline at each specific load point.
Figure 8. Mean, P90, and P95 end-to-end latencies for GPT-OSS-120B on the H100 testbed across baselines S-ILR1-3 . Red text denotes the speedup of MARS over the fastest baseline.
Figure 9. Overall Goodput Analysis for Qwen3-Coder-30B on the H100 testbed across baselines ILR-1 to ILR-4 .
Figure 10. Mean and P95 end-to-end latencies for Qwen3-Coder-30B on the H200 testbed across baselines ILR-1 to ILR-4 . Red text denotes the speedup of MARS over the fastest baseline.
Figure 11. Mean, P90, and P95 end-to-end latencies for Qwen3-Coder-30B on H200 in the realistic OpenHands deployment across baselines ILR-1 to ILR-4 . Red text denotes the speedup of MARS over the fastest baseline. Red crosses ( × ) indicate configurations where baseline systems were truncated by framework timeouts.
Figure 12. Task completion rates across workload regimes. Error bars indicate the variation in completion rates across all request rates within each regime.
Figure 13. Mean end-to-end latency for Qwen3-Coder-30B on the H100 testbed across baselines ILR-1 to ILR-4 . MARS is compared against three ablated variants to isolate the performance contribution of each core mechanism.
Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.
Mixture-of-Agents (MoA) systems improve reasoning accuracy by routing each query to multiple expert LLMs and aggregating their outputs. Efficiently executing this workload on limited GPU resources has bottlenecks. Skill-based routing creates skewed expert demand, and combining instruction-tuned LLMs with long-reasoning models results in extreme variability in generation lengths. Consequently, traditional scheduling strategies suffer from significant GPU idling and throughput collapse due to load imbalances. We present MOSAIC, a scheduling framework to accelerate MoA workloads. First, we formulate an Integer Linear Program (ILP) based scheduler that jointly optimizes expert placement and per-worker prompt assignment from offline-profiled costs, replicating reasoning experts across workers while pinning lightweight ones. Second, MOSAIC uses confidence-aware adaptive aggregation, leveraging inter-expert agreement to bypass the heavy final aggregator LLM for consensus queries. In our 4-GPU system, MOSAIC achieves up to 2.5x expert-stage, 4.23x aggregator-stage and 1.7~2.3x end-to-end speedups over the baseline scheduler, while matching accuracy within 0.1pp.
Saptarshi Mitra, Yifan Zhang, Rachid Karami +5
University of California, Irvine, USA · University of Southern California, Los Angeles, USA · Intel, USA
Large language model (LLM)-based multi-agent systems (MAS) have become a promising paradigm for complex information-seeking and reasoning tasks by enabling collaborative problem solving among specialized agents. However, existing MAS frameworks tightly couple task reasoning with coordination operations, including task selection, role assignment, message routing, and context management. As interactions grow, using powerful LLMs for these bounded control decisions introduces substantial token overhead and latency, limiting the scalability of agentic Web services. In this paper, we investigate whether coordination can be decoupled from expensive reasoning without compromising collaborative performance. We propose S1-MAS, a token-efficient multi-agent framework based on System One-guided computational division of labor. S1-MAS assigns bounded coordination decisions to lightweight System One models while reserving open-ended reasoning for capable LLM workers. Specifically, a lightweight controller selects inspection conditions, chooses subsequent tasks, and determines termination, while a compact reader retrieves condition-relevant evidence from authorized sources to support these decisions. Through a decision-evidence loop, selected tasks dynamically determine worker roles and source access, enabling adaptive collaboration without task-specific training. Extensive experiments on seven diverse benchmarks demonstrate that S1-MAS achieves superior accuracy while substantially reducing the inference cost. Across individual comparisons with AgentVerse, DyLAN, and SelfOrg on seven benchmarks, S1-MAS reduces GPT-4o token consumption by 44.9%-97.2% and measured end-to-end latency by 37.8%-93.0%. These results highlight its potential for scalable and cost-effective agentic Web applications.
Zihan Zhou, Xinzhe Hu, Hanxu Yang +2
University of Electronic Science and Technology of China · Southwestern University of Finance and Economics
Agentic workflows carry out complex tasks by orchestrating multiple large language models (LLMs) and tools. Serving them at a target throughput with low latency is hard because they are written in arbitrary agentic frameworks and their execution times are unpredictable: execution branches, fans out, or recurs in data-dependent ways. Since their LLMs often outnumber the available GPUs, they also oversubscribe GPUs. We describe Scepsy, a serving system that schedules arbitrary multi-LLM agentic workflows onto a GPU cluster. Scepsy exploits the insight that, while the end-to-end latency of an agentic workflow is unpredictable, each LLM's fraction of execution time is comparatively stable across requests. Scepsy profiles each LLM under different parallelism degrees and combines the profiles with these fractions into an Aggregate LLM Pipeline, a lightweight throughput and latency predictor for allocations. To minimize latency at a target throughput, Scepsy uses the Aggregate LLM Pipeline to search over fractional GPU shares, tensor parallelism degrees, and replica counts. A hierarchical heuristic then places the chosen allocation onto the cluster, minimizing fragmentation and respecting network topology. On realistic agentic workflows, Scepsy achieves up to 2.5x higher throughput before saturation and 1.0-3.3x lower latency than systems that optimize LLMs independently or rely on user-specified allocations.
Otto White, Marcel Wagenländer, Britannio Jarrett +7