AutoCompress: Automated Hardware-Guided Model Compression

Sheldon Liang; Ali Taghibakhshi; Saurav Muralidharan; Sharath Turuvekere Sreenivas; Ashwath Aithal; Pavlo Molchanov; Nima Tajbakhsh

cs.LGNov 5, 2026

AutoCompress: Automated Hardware-Guided Model Compression

Authors: Sheldon Liang, Ali Taghibakhshi, Saurav Muralidharan, Sharath Turuvekere Sreenivas, Ashwath Aithal, Pavlo Molchanov, Nima Tajbakhsh

Abstract

LLM pruning must preserve quality while meeting hardware- and workload-specific constraints on latency, throughput, and memory. Existing pruning workflows often target parameter counts, requiring practitioners to manually select model sizes that meet latency, throughput, and GPU memory limits. However, parameter count alone does not predict these metrics, which depend on the model architecture, workload, hardware, and serving engine. We introduce AutoCompress, a hardware-guided compression framework that directly incorporates such constraints into differentiable architecture search. AutoCompress jointly optimizes global and per-layer structure to produce heterogeneous architectures, guided by serving-engine cost models validated against end-to-end latency, throughput, and memory measurements. We evaluate AutoCompress on long-context local coding, batched datacenter serving, and latency-sensitive in-game interaction. Compared to the state-of-the-art Minitron pruning framework, AutoCompress improves average accuracy across eight benchmarks by 6.57, 2.35, and 0.57 percentage points on RTX 5070, RTX 5080, and H100, respectively. AutoCompress also compresses Nemotron 3.5 Super from 120B to 74B parameters for single-B200 BF16 serving at 128K context, retaining 95% of its mean benchmark score.

Figures & tables

Figure 1: Parameter count is a poor proxy for serving cost. (a) Two model architectures with the same parameter count (3.61B total, 1.26B active); the narrow, deep one has 35% higher latency on an RTX 5070. (b) Two model architectures that meet the same throughput on an H100; one has 2.2

\times

the parameters of the other.

Figure 2: AutoCompress overview. Given a base model and deployment constraints, AutoCompress automatically selects, prunes, and distills an architecture for on-device gaming, local coding, or datacenter serving. The compressed models meet their latency, memory, or throughput targets, measured on real GPUs.

Figure 3: Joint search over global and per-layer structure. (a) Router outputs define soft architecture masks, updated through gradients from distillation and budget losses. (b) As search progresses, the distributions sharpen toward discrete choices, yielding a heterogeneous architecture.

Figure 4: Measuring per-layer latency and memory (1) A model with multiple different layer configurations is instantiated. (2) Inference engine serves the model under the deployment setting while Nsight Systems traces runtime. (3) The trace is split at layer boundaries, for prefill and for every decode step. (4) Each layer’s runtime becomes one row of the latency table; One profiling run obtains many layer measurements along with total memory usage, whose residual over

W+C+P+R

(Eq. 4 ) sets the tolerance

\epsilon

	On-device game NPC	Local coding agent	Datacenter inference
Hardware	1 $\times$ RTX 5070 (12 GB)	1 $\times$ RTX 5080 (16 GB)	8 $\times$ H100, TP1 $\times$ DP8
Parent	Nemotron Nano 3 Elastic 12B (12.4B / 2.1B active)	Nemotron 3.5 Lightning (31.6B / 3.6B active)	Nemotron 3.5 Lightning (31.6B / 3.6B active)
Workload	batch size 1; 2,048 input + 256 output tokens	batch size 1; 262,144 cached + 16,384 input + 1,024 output tokens	batch size 128 per GPU; 8,192 input + 65,536 output tokens
Latency / throughput target	latency $\leq$ 750 ms	latency $\leq$ 11.5 s	throughput $\geq$ 7,330 tok/s per GPU (+50% over the parent)
Memory target	$\leq$ 4.0 GiB	$\leq$ 13.96 GiB	–
Parent cost	1,147 ms, 8.76 GiB	12.44 s, 23.67 GiB	4,922 tok/s per GPU

Table 1: Deployment settings.

Dimension	Lightning 3.5: RTX 5080 / H100	Nano 3 12B: RTX 5070
Hidden size $d$	2688, 2432, 2176, 1920, 1664, 1408	1920, 1792, 1664, 1536, 1408, 1280
Mamba $H$	64, 56, 48, 40, 32 (global)	64 or remove (per layer)
Routed experts $E_{\ell}$	128, 112, 96, 80, 64	128, 96, 64, 32
Routed FFN width $F_{\ell}$	1856, 1536, 1280, 1024	960, 704, 448, 192, 0
Shared FFN width $S_{\ell}$	3712, 2816, 1920	3712, 2688, 1856, 1024, 0
Depth	Fixed	Mamba and MoE layers may be removed

Table 2: Candidate values for learned architecture search. Hidden size is global; expert counts and FFN widths vary by layer.

Figure 5: Budget adherence. All selected architectures satisfy their deployment budgets when measured on real hardware.

Contract	Method	Search tokens $\downarrow$	KD tokens	KD loss $\downarrow$	AIME25	LiveCodeBench	MMLU-Pro	MATH-500	GSM8K	IFEval	IFBench	GPQA	Avg 8
RTX 5070	AutoCompress	4.18B	30B	0.1909	0.42	1.11	26.46	36.40	47.54	55.13	24.49	21.78	26.67
	Minitron++	8.49B	30B	0.2366	0.00	0.48	19.76	29.90	34.29	43.76	17.43	15.21	20.10
RTX 5080	AutoCompress	2.85B	10B	0.1637	52.71	49.44	67.45	93.60	92.87	83.64	55.61	55.87	68.90
	Minitron++	8.57B	10B	0.1710	48.75	43.81	67.89	92.70	92.93	79.57	49.83	56.94	66.55
8 $\times$ H100	AutoCompress	3.00B	10B	0.1809	55.42	48.02	64.80	92.80	92.49	82.53	53.74	53.09	67.86
	Minitron++	9.98B	10B	0.1820	53.54	47.78	64.43	92.85	92.72	81.38	52.04	53.60	67.29

Table 3: Accuracy and architecture-selection tokens under matched deployment budgets. Search tokens count architecture selection; KD tokens count recovery training. KD loss is averaged over the final 25 iterations on matched batches. Benchmark scores are final-checkpoint pass@1 (%), with prompt-level strict accuracy for IFEval and IFBench. Evaluation follows § 4.6 ; Avg 8 averages the eight benchmarks.

Figure 6: Compression at scale. (a) The Super 3.5 parent’s estimated BF16 serving footprint of approximately 280 GB exceeds the B200’s 180 GB capacity; the AutoCompress model supports a maximum batch size of 16 at 131,072-token context, with measured serving memory of 178 GB. (b) Mean benchmark score versus serving memory; the arrow connects the parent to the AutoCompress model, and the red dashed line marks the 180 GB memory limit. The mean includes 9 benchmarks and the breakdown is available in Appendix D .

Figure 7: Router initialization and warmup. Distillation loss of architectures selected by two RTX 5080 searches. Zero router-output initialization without router warmup yields 10.9% lower loss than random initialization with warmup.

Figure 8: MoE routed expert masking. Comparison of top-k score masking and expert output masking in expert token share, router KD gradient and architecture quality.

Figure 9: RTX 5080 router diagnostics. Top: maximum choice probability on each architecture axis. Bottom: ratio of distillation-gradient norm to budget-gradient norm; a value above one indicates a larger distillation-gradient norm.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Architectures. Both parent models have 52 total layers, 23 Mamba layers, 23 MoE layers and 6 attention layers. Attention layers are all kept full.

Figure 11: Cost-model accuracy : predicted vs. measured end-to-end latency (RTX 5070, RTX 5080) and throughput per GPU (H100) on held-out architectures, including the searched ones.

Contract	Runs	Table entries	GPU-h
Local coding (RTX 5080)	60	468	3.5
Datacenter throughput (H100)	180	408	24.8
On-device NPC (RTX 5070)	96	660	4.5

Appendix

Table 4: Measurement cost of each contract’s latency cost model (Table 1 ).

Figure 12: A conventional memory formula misses about a third of the memory. (a, b) Memory predicted by a conventional formula (weights at deployed precision, per-sequence KV cache and Mamba state, an MoE activation buffer) and by our memory cost model (§ 3.2 ), against measured vLLM memory: a 3.6B-parameter, 39-layer model on the RTX 5070 and a 13.6B-parameter, 52-layer model on the RTX 5080. (c) What the formula misses, as shares of measured memory.

Figure 13: Final MoE architecture selected by AutoCompress . (a) MoE routed expert counts and (b) MoE intermediate sizes across all 40 MoE layers, ordered by backbone depth. Dashed lines show the parent’s uniform dimensions: 512 routed experts and intermediate size 2688.

Figure 14: Benchmark scores after compression and distillation. Nemotron 3.5 Super and AutoCompress , with Nemotron 3 Super as a reference. The unweighted means across these nine metrics are 57.58, 54.74, and 45.44, respectively.

\dagger

GDPval-AA v2 is an index; the other scores are percentages.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 294 papers

LLM Compression with Jointly Optimizing Architectural and Quantization choices
Jun 2, 2026Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi +1Mixed-Precision Quantization LLM Quantization
UiT The Arctic University of Norway · University of Oslo, Norway
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
Aug 13, 2026Palaash Goel, Ayan Sengupta, Akshay Nambi +1LLM Pruning Structured Pruning
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi · Microsoft Research (MSR) India
IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
Sep 30, 2026Pietro Moriello, Pietro Buzzega, Angelo Porrello +1LLM Compression LLM Inference Acceleration
AImageLab, University of Modena and Reggio Emilia, Italy

Jun 2, 2026J/K move · Enter open · S save

LLM Compression with Jointly Optimizing Architectural and Quantization choices

Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi, Phuong Hoai Ha

UiT The Arctic University of Norway · University of Oslo, Norway

Mixed-Precision Quantization LLM Quantization On-Device Language Model Inference LLM Compression Neural Architecture Search

Open paper AskPDF arXiv

Deploying large language models (LLMs) is challenging due to their significant memory and computational requirements. While some methods address this by developing small or tiny language models from scratch, these approaches demand extensive GPU training. Compressing pre-trained LLMs for edge devices offers a compelling alternative. Beyond pruning and quantization, Neural Architecture Search (NAS) enables effective compression, yet prior NAS approaches often limit the search space and decouple architecture from quantization. We introduce a differentiable NAS framework that explores the entire space and jointly optimizes architectural configurations alongside mixed-precision quantization for linear layers of LLMs. Experiments demonstrate superior accuracy-latency trade-offs: our models achieve up to 1.4x faster inference than sequential NAS-then-quantization baselines at comparable accuracy, or up to 6% higher average accuracy across seven reasoning tasks at equivalent latency.

Abstract

LLM Compression

Explore similar work

LLM Compression with Jointly Optimizing Architectural and Quantization choices

Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization

IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs