GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading
Authors: Qi Qi, Yifan Sui, Hanfei Yu, Hong Zhang, Hao Wang
Organizations: University of Massachusetts Amherst Amherst, Massachusetts, USA · Hong Kong University of Science and Technology Hong Kong, China · Stevens Institute of Technology Hoboken, New Jersey, USA · University of Waterloo Waterloo, Ontario, Canada
Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAG-aware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the end-to-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1x and 10.0x over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2x strong scaling on four workers.
Figures & tables
Figure 1. Per-node execution timelines of AMBER Alert request #317 under different serving approaches.
Figure 2. DAG of AMBER Alert ( Romero et al., 2021 ; Poms et al., 2018 ) , a typical DAG-based ML inference workload for image-based emergency alerts. Each vertex denotes an ML function instance.
Figure 3. GPU memory allocation and utilization for AMBER Alert request #317.
Method
DAG- Aware
ML- Aware
Proactive Pre-load
DAG-Guided Pre-load
PISeL ( Rahimi Jafari et al., 2024 )
✗
✓
✗
✗
InstaInfer ( Sui et al., 2024 )
✗
✓
✓
✗
iRoute ( Li et al., 2026 )
✓
✗
✗
✗
Kraken ( Bhasi et al., 2021 )
✓
✗
✗
✗
Orion ( Mahgoub et al., 2022 )
✓
✗
✗
✗
SMIless ( Lu et al., 2024 )
✓
✓
✗
✗
Table 1. Comparison of ML and DAG serving approaches. GearUp uniquely combines ML-aware pre-loading with DAG dependencies to proactively determine when downstream ML instances should be loaded.
Figure 4. Overview of GearUp . Deployment-time profiling (0) precedes admission (1–3), per-node policy generation for root and downstream nodes (4–6), and execution with online feedback (7–9).
Figure 5. EST/LST schedule and slack windows for FINRA-ML. Bars show per-node invocation durations.
Figure 6. Probabilistic keep-alive optimization for AMBER Alert with w=0.95 .
Figure 7. DAG of FINRA ( Financial Industry Regulatory Authority, [n. d.] ; Li et al., 2026 ) , a typical DAG-based ML inference workload for market surveillance. Each vertex denotes an ML function instance.
Abbr.
Function
Model / Dataset
AMBER Alert
OD
Object Detection
YOLOv5 / COCO
HAP
Human Attribute Prediction
ResNet50 / ImageNet
FR
Face Recognition
FaceNet / ImageNet
IR
Image Recognition
ResNet50 / ImageNet
TRS
Transcription
T5 / SQuAD
Table 2. DAG nodes, functions, models, and datasets for the AMBER Alert and FINRA-ML workloads.
Figure 8. End-to-end latency and speedup distributions for AMBER Alert and FINRA-ML. Dotted lines mark the cold-execution references (latency and 1× speedup).
Figure 9. GPU and host memory usage for (a) AMBER Alert and (b) FINRA-ML, excluding inter-request idle intervals.
Figure 10. Traffic-adaptive keep-alive behavior for FINRA-ML: request and invocation counts (top), and container counts and instance-reuse hit rate (bottom).
Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may restrict their applicability in on-premises scenarios. We present Strait, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfer and accounts for kernel execution interference through an adaptive prediction model. By drawing on these predictions, it performs priority-aware scheduling to deliver differentiated handling. Evaluation results under intense workloads suggest that Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points while incurring acceptable costs on low-priority tasks. Compared to software-defined preemption approaches, Strait also exhibits more equitable performance.
Haidong Zhao, Nikolaos Georgantas
Inria & Sorbonne University · Paris, France · Inria
Multi-agent LLM systems spend additional computation on reasoning and collaboration, but shared serving queues can prevent that computation from producing answers before a deadline. This creates a coupled decision: the reasoning workflow determines the load, while changing load determines which reasoning choices remain affordable. We introduce INFRAMIND, a hierarchical policy that adapts workflow planning and execution to serving conditions. At arrival, a planner selects the collaboration graph using task information, load and the time budget. Before each agent call, an executor jointly selects the model and reasoning strategy using refreshed telemetry and remaining time. A shared latency penalty connects the two policies. Our analysis characterizes why observing load matters when congestion reverses the preferred computation. Across five benchmarks on a shared five-model deployment, INFRAMIND improves low-load accuracy by up to 7.6 percentage points over the strongest evaluated baseline. On 500 MATH questions with matched EDF scheduling and per-request deadlines of 10-300 s, it increases correct, on-time answers from 20.4% to 33.8%. With 15 models and Azure timings replayed on the same benchmark and deadline tiers, this fraction rises from 11.6% to 54.8% over MasRouter. These results support adapting reasoning decisions to the serving capacity available throughout a workflow.
LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs). DeltaServe integrates with existing inference engines through a compact hook interface that requires only multi-LoRA batching support. It exploits the shared execution structure of inference prefill and LoRA fine-tuning forward passes, and uses an SLO-aware scheduler to admit and execute fine-tuning only when sufficient inference headroom is available. The scheduler is driven by a CUDA-graph-aware latency model calibrated offline and refined online. We integrate DeltaServe with vLLM, SGLang, and S-LoRA. On a production trace from Company X, DeltaServe on vLLM delivers 2.9x higher fine-tuning throughput than LLMStation at 100% inference SLO compliance, versus 85% for LLMStation. It also achieves 39% higher fine-tuning throughput than a baseline running vLLM+torchtune, using no additional hardware and maintaining full SLO compliance.