Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs

Jihao Xin; Tian Lyu; Qilong Pan; Kesen Wang; Marco Canini

cs.DCMar 5, 2026

Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs

Authors: Jihao Xin, Tian Lyu, Qilong Pan, Kesen Wang, Marco Canini

Organizations: KAUST · HUMAIN

Abstract

Post-training compression reduces LLM parameter counts but often produces irregular tensor dimensions that degrade GPU performance -- a phenomenon we call dimensional misalignment. We present a full-stack analysis tracing root causes at three levels: framework, library, and hardware. The key insight is that model inference becomes slower because the resulting dimensions are unfriendly with the GPU execution stack. For example, compressing Llama-3-8B with activation-aware singular value decomposition (ASVD) has 15% fewer parameters yet runs no faster than the uncompressed baseline, because 95% of its dimensions are misaligned. We propose GAC (GPU-Aligned Compression), a new compression paradigm that wraps any dimension-reducing compressor and re-selects hardware-aligned dimensions via multi-choice knapsack optimization under the same parameter budget. We evaluate GAC on Llama-3-8B with ASVD and LLM-Pruner, achieving 100% alignment and recovering up to 1.5 $\times$ speedup while preserving model quality.

Figures & tables

Panel of Figure 1 — Figure 1. GEMM: $Y{=}X{\cdot}W$ with $X{\in}\mathbb{R}^{M\times K}$ , $W{\in}\mathbb{R}^{K\times N}$ .

Figure 2. Llama-3-8B at

\rho=20\%

. Shape:

\circ{=}

Q Head,

\square{=}

K Head,

\triangle{=}

V Head; color: green =8-aligned, red =misaligned.

Figure 3. Llama-3-8B after PaLU compression at $\rho=20\%$ .

Method	Score	Works
Magnitude	$\\|W_{i}\\|_{F}$	SVD-LLM ( Wang et al., 2025 )
Activation	$\\|X_{i}\\|_{F}$	ASVD ( Yuan et al., 2025 )
Gradient	$\left\|\frac{\partial\mathcal{L}}{\partial h_{i}}\cdot h_{i}\right\|$	Taylor Pruning ( Molchanov et al., 2019 )
Fisher	$\mathrm{tr}(\mathbf{F}_{i})$	PaLU ( Chang et al., 2025 )

Table 1. Importance scores for budget allocation.

Panel of Figure 4 — Figure 4. PyTorch SDPA execution stack.

Figure 5. PyTorch SDPA latency across dimensions.

Region	Template	$B_{r}\times B_{c}$	Latency	vs. $t{=}64$
$d{=}64$	64	128 $\times$ 128	0.74 ms	1.0 $\times$
$d\in(64,96]$	96	128 $\times$ 64	1.12 ms	1.5 $\times$
$d\in(96,128]$	128	128 $\times$ 64	1.47 ms	2.0 $\times$
$d\in(128,160]$	160	128 $\times$ 32	2.00 ms	2.7 $\times$
$d\in(160,256]$	192–256	128 $\times$ 32	2.3–2.9 ms	3–4 $\times$

Table 2. FA2 template tiers and performance (

B{=}4

S{=}2048

H{=}32

Figure 6. GEMV latency (

M{=}1

, stride-1 sweep near 4096).

Tier	Condition	Kernel	MMA Instr.
1	$d\bmod 8=0$	cuBLAS-native sm80	mma.m16n8k16
2	$d\bmod 2=0$	CUTLASS sm80 align2	mma.m16n8k16
3	odd	CUTLASS sm75 align1	mma.m16n8k8

Table 3. cuBLAS GEMM kernel tiers (Nsight Compute).

Figure 7. GEMM latency with dimension sweep.

Figure 8. Hardware-level alignment penalties (sweep near 4096): (a,b) Tensor Core throughput, (c) L2 Cache bandwidth.

Level	Mechanism	Constraint	Penalty
Framework	SDPA backend	$d$ % $8{=}0$	${\sim}$ 90%
Framework	FA2 template	$d$ % $32{=}0$	${\sim}$ 30%
Library	cuBLAS GEMM	$K$ / $N$ % $8{=}0$	${\sim}$ 30%
Library	cuBLAS GEMV	$K$ % $8{=}0$	${\sim}$ 12%
Hardware	TC MMA	$K$ % $16{=}0$ , $N$ % $8{=}0$	${\sim}$ 70%
Hardware	L2 sectors	$K$ % $16{=}0$	${\sim}$ 50%

Table 4. Full-stack alignment constraints summary.

Panel of Figure 9 — Figure 9. GAC Pipeline.

Method	Align	PPL	PiQA	HSwag	Latency (ms)
Baseline	100%	6.14	0.80	0.50	99.6
ASVD	5%	34.7	0.58	0.28	100.5 (+1%)
ASVD (GAC)	100%	31.3	0.57	0.26	67.1 ( $-$ 33%)
LLM-Pruner	83%	9.88	0.80	0.49	137.7 (+38%)
LLM-Pruner (GAC)	100%	9.87	0.78	0.47	88.0 ( $-$ 12%)

Table 5. Preliminary results on Llama-3-8B (

\rho{=}15\%

). Measured with batch

{=}1

, sequence length

{=}1024

Figure 10. Llama-3-8B latency across sequence lengths.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 11 — Figure 11. Compressed dimension distributions across compression ratios.

Figure 12. PyTorch SDPA latency across dimensions on H100.

Figure 13. Hardware-level alignment penalties on H100: (a,b) Tensor Core throughput, (c) L2 Cache bandwidth.

Figure 14. Llama-3-8B latency across sequence lengths on H100.

Model Compression

Model compression reduces the storage, computation, or latency of machine-learning models while preserving task performance.

See all 122 papers

Layer-wise Curriculum Learning for Efficient LLM Compression
Sep 16, 2026Donggeon Lee, Dooyeon Na, Seungmin Oh +1LLM Compression Curriculum Learning
Ajou University, South Korea
ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
Mar 18, 2026Ruibo Fan, Xiangrui Yu, Xinglin Pan +5Model Compression LLM Compression
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Harbin Institute of Technology, Shenzhen, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong, Hong Kong SAR
More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
Aug 25, 2026Srikanta Datta Tumkur, Mehar Simhadri, Anshu Bansal +5LLM Serving LLM Inference Efficiency
MIT and Vizuara

Sep 16, 2026J/K move · Enter open · S save

Layer-wise Curriculum Learning for Efficient LLM Compression

Donggeon Lee, Dooyeon Na, Seungmin Oh, Jongbin Ryu

Ajou University, South Korea

LLM Compression Curriculum Learning Knowledge Distillation Memory-Efficient Training

Open paper AskPDF arXiv

In this paper, we introduce layer-wise curriculum learning for efficient LLM compression. The proposed method facilitates the knowledge transfer from the teacher model to the student model, utilizing a curriculum learning approach that begins with easier optimization tasks and progressively tackles harder ones. In order to adopt the layer-wise learning in LLM compression, we partition the whole model into multiple segments consisting of layers, thereby enabling more computationally efficient knowledge transfer for LLMs. Based on our theoretical analysis of cumulative error phenomenon, layer-wise curriculum learning accelerates convergence while stabilizing the knowledge transfer process. In addition, we present a feature caching method with a multi-threading strategy to efficiently address feature misalignment across layers, maximizing GPU utilization. Consequently, our method exhibits advanced model compression performance, as well as high computational efficiency in terms of minimized memory usage and short training hours. Experiments on multiple datasets show that the proposed method achieves state-of-the-art performance while reducing GPU memory usage and training hours by more than 50% on BERT and GPT-2. Moreover, it outperforms the other pruning methods on LLaMA-family and Qwen models under the same training hours, with a lower GPU memory footprint.

Abstract

Model Compression

Explore similar work

Layer-wise Curriculum Learning for Efficient LLM Compression

ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression

More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving