cs.OSApr 14, 2026

MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems

Authors: Yifei Wang, Hancheng Ye, Yechen Xu, Cong Guo, Chiyue Wei, Qinsi Wang, Dongting Li, Tingjun Chen, 

Organizations: Duke University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) are increasingly deployed as the execution core of autonomous agents rather than as standalone text generators. Agentic workloads induce a temporal shift from single-turn inference to multi-turn LLM-tool loops, and a spatial shift from chat-scale, GPU-only execution to repository-scale, GPU-CPU co-located execution. Consequently, coordinating heterogeneous resource demands of agentic execution has emerged as a critical system challenge. We design and implement MARS, an efficient and adaptive co-scheduling system that globally coordinates heterogeneous agentic workloads under coupled GPU-CPU resource pressure. By establishing holistic visibility across GPU inference and CPU tool execution via a unified information stream, an external control plane in MARS decouples admission from execution to prevent heterogeneous resource oversubscription. An internal agent-centric scheduler further minimizes the end-to-end critical path by prioritizing latency-sensitive continuations and adaptively retaining KV cache state only when warm resumption yields a latency benefit. Our evaluations show that MARS reduces end-to-end latency by up to 5.94x while maintaining nearly maximal system throughput. We further integrate MARS as the serving backend for the OpenHands coding agent framework, demonstrating its real-world effectiveness by accelerating end-to-end task completion time by up to 1.87x. Our source code is publicly available at https://github.com/Afterglow231/MARS_preview .

Figures & tables

Figure 1. Agentic workloads induce both temporal and spatial shifts. Together, these shifts create dynamically misaligned GPU and CPU demand across concurrent requests.
Figure 2. Token throughput vs. Dynamic SLO-Aware Goodput. While baselines sustain high token throughput, they experience complete collapse as load increases while MARS successfully maintains high request completion rates.
Figure 3. KV cache eviction dynamics and multi-turn responsiveness. (A) The rate of KV block removals over the normalized execution progress of the workload. (B) The average Time to First Token (TTFT) per round across different latency percentiles.
Figure 4. MARS architecture overview.
SubsystemEventSignalSystemic Role
GPU Planegpu_submitProjected KVTriggers capacity check
 gpu_1st_tokenLaunch DelayEstimates prefill bounds
 gpu_endFreed BlocksReleases capacity
CPU Planetool_numActive ToolsSignals host congestion
 tool_start/endExec DurationIsolates CPU starvation
Control Planewindow_updateDual-PressureModulates WadmW_{\mathrm{adm}}
Table 1. Mapping Unified Info Stream to control actions. By recording phase boundaries, MARS translates opaque hardware states into actionable scheduling telemetry.
Figure 5. Internal Agent-Centric Scheduler.
Figure 6. Input-length distributions and ideal execution time of the workloads.
Figure 7. Mean, P90, and P95 end-to-end latencies for Qwen3-Coder-30B on the H100 testbed under four input-length regimes ( ILR-1 to ILR-4 ). Red text denotes the speedup of MARS over the fastest baseline at each specific load point.
Figure 8. Mean, P90, and P95 end-to-end latencies for GPT-OSS-120B on the H100 testbed across baselines S-ILR1-3 . Red text denotes the speedup of MARS over the fastest baseline.
Figure 9. Overall Goodput Analysis for Qwen3-Coder-30B on the H100 testbed across baselines ILR-1 to ILR-4 .
Figure 10. Mean and P95 end-to-end latencies for Qwen3-Coder-30B on the H200 testbed across baselines ILR-1 to ILR-4 . Red text denotes the speedup of MARS over the fastest baseline.
Figure 11. Mean, P90, and P95 end-to-end latencies for Qwen3-Coder-30B on H200 in the realistic OpenHands deployment across baselines ILR-1 to ILR-4 . Red text denotes the speedup of MARS over the fastest baseline. Red crosses ( ×\times ) indicate configurations where baseline systems were truncated by framework timeouts.
Figure 12. Task completion rates across workload regimes. Error bars indicate the variation in completion rates across all request rates within each regime.
Figure 13. Mean end-to-end latency for Qwen3-Coder-30B on the H100 testbed across baselines ILR-1 to ILR-4 . MARS is compared against three ablated variants to isolate the performance contribution of each core mechanism.
More in this topic

LLM Inference Scheduling

Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.

See all 160 papers

Explore similar work

CardsList

  1. MOSAIC: Efficient Mixture-of-Agent Scheduling via Adaptive Aggregation and Inference Concurrency

    Jun 2, 2026Saptarshi Mitra, Yifan Zhang, Rachid Karami +5LLM Inference EfficiencyMulti-Agent System Optimization

    University of California, Irvine, USA · University of Southern California, Los Angeles, USA · Intel, USA

  2. Token-Efficient Multi-Agent Collaboration via System One-Guided Computational Division of Labor

    Oct 6, 2026Zihan Zhou, Xinzhe Hu, Hanxu Yang +2Multi-Agent LLM SystemsMulti-Agent System Optimization

    University of Electronic Science and Technology of China · Southwestern University of Finance and Economics

  3. Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines

    Apr 16, 2026Otto White, Marcel Wagenländer, Britannio Jarrett +7LLM InferenceLLM Inference Scheduling

    Imperial College London · Independent Researcher