cs.DCOct 28, 2026

GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

Authors: Qi Qi, Yifan Sui, Hanfei Yu, Hong Zhang, Hao Wang

Organizations: University of Massachusetts Amherst Amherst, Massachusetts, USA · Hong Kong University of Science and Technology Hong Kong, China · Stevens Institute of Technology Hoboken, New Jersey, USA · University of Waterloo Waterloo, Ontario, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAG-aware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the end-to-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1x and 10.0x over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2x strong scaling on four workers.

Figures & tables

Figure 1. Per-node execution timelines of AMBER Alert request #317 under different serving approaches.
Figure 2. DAG of AMBER Alert ( Romero et al., 2021 ; Poms et al., 2018 ) , a typical DAG-based ML inference workload for image-based emergency alerts. Each vertex denotes an ML function instance.
Figure 3. GPU memory allocation and utilization for AMBER Alert request #317.
MethodDAG- AwareML- AwareProactive Pre-loadDAG-Guided Pre-load
PISeL ( Rahimi Jafari et al., 2024 )✗✓✗✗
InstaInfer ( Sui et al., 2024 )✗✓✓✗
iRoute ( Li et al., 2026 )✓✗✗✗
Kraken ( Bhasi et al., 2021 )✓✗✗✗
Orion ( Mahgoub et al., 2022 )✓✗✗✗
SMIless ( Lu et al., 2024 )✓✓✗✗
Table 1. Comparison of ML and DAG serving approaches. GearUp uniquely combines ML-aware pre-loading with DAG dependencies to proactively determine when downstream ML instances should be loaded.
Figure 4. Overview of GearUp . Deployment-time profiling (0) precedes admission (1–3), per-node policy generation for root and downstream nodes (4–6), and execution with online feedback (7–9).
Figure 5. EST/LST schedule and slack windows for FINRA-ML. Bars show per-node invocation durations.
Figure 6. Probabilistic keep-alive optimization for AMBER Alert with w=0.95w=0.95 .
Figure 7. DAG of FINRA ( Financial Industry Regulatory Authority, [n. d.] ; Li et al., 2026 ) , a typical DAG-based ML inference workload for market surveillance. Each vertex denotes an ML function instance.
Abbr.FunctionModel / Dataset
AMBER Alert
ODObject DetectionYOLOv5 / COCO
HAPHuman Attribute PredictionResNet50 / ImageNet
FRFace RecognitionFaceNet / ImageNet
IRImage RecognitionResNet50 / ImageNet
TRSTranscriptionT5 / SQuAD
Table 2. DAG nodes, functions, models, and datasets for the AMBER Alert and FINRA-ML workloads.
Figure 8. End-to-end latency and speedup distributions for AMBER Alert and FINRA-ML. Dotted lines mark the cold-execution references (latency and 1×1\times speedup).
Figure 9. GPU and host memory usage for (a) AMBER Alert and (b) FINRA-ML, excluding inter-request idle intervals.
Figure 10. Traffic-adaptive keep-alive behavior for FINRA-ML: request and invocation counts (top), and container counts and instance-reuse hit rate (bottom).
Figure 11. AMBER Alert ablation results: (a) end-to-end latency CDF and (b) speedup CDF.
Figure 12. GPU and host memory usage in the AMBER Alert ablation study, excluding inter-request idle intervals.
Figure 13. Strong and weak scaling with one, two, and four workers. Dotted lines indicate ideal scaling.
More in this topic

LLM Inference Serving

Large language model inference serving studies how to run model requests efficiently while meeting latency, throughput, cost, and energy goals.

See all 74 papers

Explore similar work

CardsList

  1. Strait: Perceiving Priority and Interference in ML Inference Serving

    Apr 30, 2026Haidong Zhao, Nikolaos Georgantas

    Inria & Sorbonne University · Paris, France · Inria

  2. INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration

    Jun 9, 2026Ahasan Kabir, Mengxin Zheng, Qian LouMulti-Agent System OptimizationMulti-Agent Orchestration

    University of Central Florida

  3. DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

    Jul 30, 2026Jiaxuan Chen, Jianshu She, Ye Yuan +5LLM ServingLLM Fine-Tuning

    McGill University · Canada · MBZUAI +3