AEGIS: Runtime-Guided GPU Collocation for Multi-Tenant Deep Learning Training

Ehsan Yousefzadeh-Asl-Miandoab; Büşra Karatay Demiray; Florina M. Ciorba; Pamela Delgado; Pınar Tözün

cs.DCAug 26, 2025

AEGIS: Runtime-Guided GPU Collocation for Multi-Tenant Deep Learning Training

Authors: Ehsan Yousefzadeh-Asl-Miandoab, Büşra Karatay Demiray, Florina M. Ciorba, Pamela Delgado, Pınar Tözün

Organizations: IT University of Copenhagen Denmark · University of Basel Switzerland

Abstract

Deep learning training commonly runs on shared multi-tenant GPU servers, where exclusive allocation provides isolation but can leave resources underutilized and increase queueing time. Collocation can improve efficiency, but interference-agnostic placement may cause severe slowdowns, while inaccurate memory information can lead to out-of-memory (OOM) failures. We present AEGIS, a server-scale runtime scheduling system for controlled collocation of deep learning training workloads on shared multi-GPU servers. AEGIS integrates memory feasibility, post-placement observation, runtime-pressure filtering, placement, and OOM-aware recovery in a single scheduling loop. After placement, AEGIS observes workload activity before permitting further collocation, then uses low-overhead telemetry to determine whether a GPU can safely accept additional work. OOM failures trigger retries under progressively safer memory conditions, eventually falling back to exclusive execution. This online approach avoids costly offline pairwise compatibility profiling. We evaluate AEGIS using vision, Transformer, recommendation, and LLM-style workloads across three production-derived traces. AEGIS reduces geometric-mean makespan by 16% relative to Lucid, 21% relative to Horus, and 27% relative to exclusive allocation. Sensitivity studies show that activity-anchored observation and runtime-pressure filtering balance conservative isolation against interference-agnostic collocation, improving makespan while limiting sharing-induced per-task slowdown.

Figures & tables

Figure 1 . Interference-agnostic MPS-enabled collocation on an A100 40 GB GPU. We start with ResNet50 (BS128) and VGG16 (BS64), then add Xception (BS64), followed by ResNet50 (BS32), all on ImageNet ( Deng et al., 2009 ) , where BS denotes batch size. All jobs complete in every configuration. Throughput gain is the sum of the jobs’ solo runtimes divided by the collocated makespan.

Figure 2 . Overview of AEGIS . Numbered markers show the control flow: submit a job ( 1 ), select the next job ( 2 ), read its memory input ( 3 ), read the current per-GPU state ( 4 ), place the job ( 5 ), start a mandatory post-placement observation hold ( 6 ), detect activity and update the activity-anchored monitoring window ( 7 ), release the hold and refresh runtime-pressure eligibility ( 8 ), and recover from an OOM ( 9 ).

Group	Model	Dataset	BS	GPUs	Mem. (GB)
$\ast$	ResNet 18/34/50	ImageNet/CIFAR-100	32–128	1	0.6–21.2
$\ast$	MobileNet	ImageNet/CIFAR-100	32–128	1	2.4–3.3
$\ast$	EfficientNet	ImageNet/CIFAR-100	32–128	1	2.6–4.5
$\ast$	VGG16	ImageNet	32–128	1	6.8–21.2
$\ast$	Xception	ImageNet	32–128	1	6.7–20.7
$\ast$	Inception	ImageNet	32–128	1	5.5–17.0

Table 1 . Summary of workloads used in the evaluation. Mem. reports the observed/profiled peak GPU memory range in GB across evaluated specifications (BS: batch size).

Metric (h)	Trace	Exclusive	Horus	Lucid	AEGIS Est.-Free
Makespan	Philly	11.84	10.84	9.94	8.39
	Saturn	13.17	12.01	11.22	9.24
	Venus	12.35	11.67	11.17	9.66
Mean JCT	Philly	5.23	4.52	3.93	3.31
	Saturn	4.78	4.00	3.56	2.79
	Venus	4.55	4.16	3.90	2.83

Table 2 . Absolute end-to-end performance across traces. Values are hours; p95 values are within-trace job percentiles. Lower is better for all metrics; bold marks the best value in each row.

Panel of Figure 3 — Figure 3 . End-to-end trace performance. Values are normalized to Exclusive on the corresponding trace; lower is better. Bars show per-trace values and GeoMean. In the JCT panel, diamonds show p95 JCT normalized to Exclusive p95 JCT on the same trace. Results include failed-attempt runtime and recovery wait.

Figure 4 . Per-job normalized JCT distributions across traces. Each job’s JCT is divided by its isolated solo runtime; curves farther left are better. The x-axes are logarithmic.

Panel of Figure 5 — Figure 5 . Queueing and execution-time trade-off. Values are normalized to Exclusive on the corresponding trace; lower is better. Bars show means. Diamonds show p95 values normalized to the corresponding Exclusive p95 for the same metric and trace.

		JCT		Queue		Exec.
Mem. input	Mkspn	Mean	P95	Mean	P95	Mean	P95	F/R
PeakMem	0.712	0.594	0.607	0.442	0.520	1.785	2.110	0/0
Est-free	0.730	0.613	0.598	0.494	0.546	1.550	1.971	12/12
AnalyticalMem	0.733	0.624	0.638	0.520	0.584	1.453	1.964	3/3
GPUMemNet	0.737	0.629	0.636	0.510	0.573	1.575	2.057	1/1
FakeTensor	0.794	0.648	0.679	0.543	0.641	1.483	1.833	0/0

Table 3 . Effect of memory input. Values are GeoMeans across traces, normalized to Exclusive; lower is better for the time metrics; F/R reports failed/recovered attempts. Bold marks the best value in each metric column; the italic row label marks the default AEGIS configuration. Queue includes recovery wait; Exec. includes failed-attempt runtime.

		JCT		Queue		Exec.
Policy	Mkspn	Mean	P95	Mean	P95	Mean	P95	F/R
RR	0.776	0.635	0.636	0.501	0.573	1.686	1.993	28/28
MU	0.755	0.587	0.619	0.449	0.556	1.683	2.160	13/13
MFM	0.730	0.613	0.598	0.494	0.546	1.550	1.971	12/12

Table 4 . Effect of placement policies. Values are GeoMeans across traces, normalized to Exclusive; lower is better for the time metrics; F/R reports failed/recovered attempts. Bold marks the best value in each metric column; the italic row label marks the default AEGIS policy. Queue includes recovery wait; Exec. includes failed-attempt runtime.

		JCT		Queue		Exec.
Config.	Mkspn	Mean	P95	Mean	P95	Mean	P95	F/R
MFM	0.730	0.613	0.598	0.494	0.546	1.550	1.971	12/12
MFM-NF	0.732	0.622	0.646	0.470	0.570	1.821	2.265	18/18
MU	0.755	0.587	0.619	0.449	0.556	1.683	2.160	13/13
MU-NF	0.770	0.616	0.626	0.480	0.580	1.740	2.250	19/19

Table 5 . Runtime-pressure eligibility ablation. Values are GeoMeans across traces, normalized to Exclusive; lower is better for the time metrics; F/R reports failed/recovered attempts. Bold marks the best value in each metric column; the italic row label marks the default. Queue includes recovery wait; Exec. includes failed-attempt runtime. NF: saturation-based pressure filtering disabled; the post-placement observation hold remains enabled.

		JCT		Queue		Exec.
Config.	Mkspn	Mean	P95	Mean	P95	Mean	P95	F/R
AEGIS (Mem-only)	1.007	0.998	1.076	0.116	0.155	1.731	1.955	0/0
AEGIS (U70%)	0.884	0.892	0.858	0.585	0.754	1.172	1.115	0/0
Horus	0.897	0.890	0.844	0.647	0.760	1.112	1.044	0/0
Lucid	0.950	0.876	0.832	0.510	0.631	1.195	0.963	0/0

Table 6 . Limited-telemetry evaluation on the heterogeneous RTX/GTX server. Values are GeoMeans across traces, normalized to Exclusive; lower is better for the time metrics; F/R reports failed/recovered attempts. Bold marks the best value in each metric column; the italic row label marks the selected AEGIS fallback. Queue includes recovery wait; Exec. denotes execution time. U70%: 70% utilization threshold.

Figure 6 . Stability of activity-anchored monitoring windows. Points show weighted mean absolute error of the AEGIS pressure indicators relative to a 200 s activity-anchored reference.

Figure 7 . Pressure-threshold trade-off on calibrated one-GPU workload sequences. The top panel shows throughput gain and slowdown; the bottom panel shows collocation deferrals. The selected operating point lies near the knee of the trade-off.

Explore similar work

Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
Aug 12, 2026Josef Liyanjun ChenCpu-Gpu Hybrid Designs Cohort
Independent Researcher
COMPASS-ABS: Reducing Fragmentation in Shared GPU Clusters for Deep Learning Training Workloads
Sep 16, 2026Yukai Zhou, Hongfan WuInference Workloads Fragmentation
SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
May 1, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuAgentic Inference Large Language Model Tool Use

Aug 12, 2026, cs.DCJ/K move · Enter open · S save

Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

Josef Liyanjun Chen

Independent Researcher

Cpu-Gpu Hybrid Designs Cohort Path Relative Position

Open paper AskPDF arXiv

LLM-agent services repeatedly execute small deterministic transitions between model and tool calls: route an outcome, update state, and emit the next effect. We ask when this control path exposes enough concurrent work for GPU execution, and what changes when a GPU-computed route decision remains on device. We formalize the ready-cohort boundary using fixed-partition share F, exact offline share P*, local upper bound U, and online achieved share A. Under zero service time, unlimited capacity, and equal relative launch deadlines, a specialized dynamic program computes P* exactly. In a stationary Poisson replay of one pinned 851-session public trace panel, the primary condition at 100,000 target active sessions, K=256, and a 50 ms launch deadline gives F=30.19%, P*=43.00%, and U=45.85%. Exact packing recovers 81.83% of the opportunity lost at fixed window boundaries. The outcome-derived route key is a conditioning proxy, not proof of executable identity. A separate mechanism study keeps a GPU-computed binary decision on device instead of returning four bytes to the host and redispatching. Across four named GPU placements, the device-resident path is faster in all 36 configurations; within-placement row-median ratios range from 1.19x to 2.39x. Across both admissible mechanisms, all 14,557,440 tested batched invocations match a separately implemented host oracle. A fixed nested device graph that removes no host decision is slower in all 60 configurations across five placements. Together, the studies establish two measurable gates for GPU agent control: deadline-feasible cohort supply and observation placement. A joined finite online runtime is required to measure A, CPU displacement, and service-level benefit.

Abstract

Explore similar work

Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

COMPASS-ABS: Reducing Fragmentation in Shared GPU Clusters for Deep Learning Training Workloads

SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters