LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

Zishan Shao; Lixun Zhang; Kangning Cui; Wenhao Wu; Jinhee Kim; Yixiao Wang; Ting Jiang; Hancheng Ye

cs.CLAug 26, 2026

LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

Authors: Zishan Shao, Lixun Zhang, Kangning Cui, Wenhao Wu, Jinhee Kim, Yixiao Wang, Ting Jiang, Hancheng Ye,

Organizations: Duke University · Wake Forest University

Abstract

SVD-based low-rank compression has become a fast-growing direction for reducing the memory and computational cost of large language models (LLMs). However, meaningful comparison across existing studies remains difficult as prior evaluations use varied benchmarks, inconsistent ratios, and diverse setups, often failing to isolate low-rank effects from auxiliary techniques. As a result, it remains unclear whether reported gains reflect method-level improvements or differences in evaluation protocol. This lack of comparability highlights the need for a unified, reproducible evaluation platform. To address this problem, we present LowRankArena, a standardized evaluation platform for SVD-based LLM compression. LowRankArena unifies task versions, uniform-precision compression budgets, comparison regimes, and inference measurements, and provides a reproducible pipeline with over 3 TiB released compressed checkpoints. Using LowRankArena, our aligned audit of five representative SVD methods reveals that prior findings are highly conditional under standardized protocols: clear leaders and performance tiers shift across backbones and keep ratios, multiple-choice accuracy can hide large perplexity degradation, and nominal low-rank savings yield workload-dependent and often limited end-to-end speedups. Our code is available at: https://github.com/Zishan-Shao/lowrankarena.git.

Figures & tables

Figure 1: Overview of LowRankArena . Left: The current evaluation of SVD-based LLM compression suffers from inconsistent settings and high reproduction costs. Middle: LowRankArena addresses this by providing a standardized, fully reproducible evaluation platform with over 3 TiB of released checkpoints. Right: By auditing representative publicly reproducible methods under this unified protocol, we characterize the reproducible SVD-compression frontier, highlighting unstable rankings, no consistent advantage over structured pruning, and a mismatch between theoretical and practical end-to-end speedups. Checkpoints: https://huggingface.co/Duke-CEI-SVD/LowRankArena .

Figure 2: Recurring reviewer concerns in recent SVD evaluations. Bars show the share of public OpenReview reviewer/Area Chair notes (

n=76

) mentioning each concern category, highlighting systematic evaluation fragmentation and motivating LowRankArena.

Figure 3: Evaluation scope of LowRankArena. Our platform standardizes SVD assessment across five tracks: (1) Main leaderboard for reasoning/PPL; (2) Literature alignment for prior claims; (3) Broader evaluations for architecture and baseline scaling; (4) inference for end-to-end speed; and (5) Feasibility for compression costs. All tracks utilize a unified setup with matched budgets.

Panel of Figure 4 — Figure 4: Method Performance Ranking Shifts Across Model Families. Method rankings shift substantially when moving from legacy backbones (Llama-1) to modern model families (Llama-3.1 and Qwen3). These shifts show that the apparent superiority of a given SVD-based method is architecture-dependent and does not consistently transfer across model families or keep ratios.

			PPL $\downarrow$		General MCQ $\uparrow$							MCQ $\uparrow$	Math $\uparrow$		Ret. $\uparrow$
Model	Budget	Method	WikiT.	C4	BoolQ	ARC-E	ARC-C	WinoG.	PIQA	HellaS.	OBQA	Avg.	MathQA	MMLU-M	Q. Ret.
Llama-1-7B	Dense FP		5.67	7.20	0.737	0.726	0.445	0.693	0.786	0.749	0.408	0.649	0.261	0.278	1.000
	80%	ASVD	8.60	11.04	0.743	0.628	0.388	0.669	0.750	0.696	0.392	0.609	0.241	0.211	0.787
		SVD-LLM v1	7.88	16.42	0.633	0.481	0.317	0.584	0.681	0.477	0.332	0.501	0.224	0.291	0.767
		DoBi-SVD	9.23	19.01	0.620	0.500	0.305	0.635	0.665	0.570	0.385	0.526	0.229	0.272	0.732
		Basis Sharing	7.74	15.51	0.645	0.614	0.367	0.645	0.711	0.633	0.400	0.574	0.239	0.205	0.747

Table 1: Main standardized leaderboard. SVD-style compression methods are evaluated across keep ratios on Llama and Qwen anchor models under matched task versions, evaluation scripts, and budget definitions. The table highlights that method rankings are not intrinsic: the strongest recipe changes across backbone families, compression budgets, and metric groups.

Figure 5: The capability cliff in zero-shot low-rank compression. SVD vs. structured pruning on Llama-3.1-8B under matched ratios (no added post-compression recovery, uniform-choice macro floor 35.7%). Left: SVD remains competitive with pruning in reasoning accuracy (MCQ Avg). Right: Several SVD methods show severe language-modeling degradation in C4 PPL at aggressive keep ratios, while pruning degrades more gradually.

Figure 6: Inference speedup across diverse request profiles. End-to-end gains are evaluated across the representative workloads defined in Sec. A.2 . While SVD reduces nominal computation, empirical throughput gains remain marginal and inconsistent, especially during decode-heavy generation.

Method	WikiText-2	C4	Pile-Val
MoDeGPT	$0.7699\pm 0.0039$	0.7782	0.7801
Basis Sharing	$0.5369\pm 0.0037$	0.4980	0.5150
SVD-LLM	$0.5212\pm 0.0025$	0.5033	0.4919
ASVD	$0.3278\pm 0.0145$	0.3033	0.3320

Table 2: Calibration sensitivity on Llama-3.1-8B at an 80% keep ratio (

R_{Q}

, higher is better). WikiText-2 uses 3 resamples.

Profile	TTFT speedup	E2E speedup
Prefill-heavy	3.37 $\times$ ( $3417\!\to\!1013$ ms)	1.06 $\times$ ( $4925\!\to\!4659$ ms)
Balanced	3.80 $\times$ ( $1668\!\to\!439$ ms)	0.80 $\times$ ( $5905\!\to\!7358$ ms)
Decode-heavy	0.91 $\times$ ( $187\!\to\!205$ ms)	0.68 $\times$ ( $15136\!\to\!22359$ ms)

Table 3: Fixed-setting speedup audit. Llama-3.1-8B, SVD-LLM v1, 0.6 keep ratio on RTX A5000.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Configuration
Benchmark suite	LowRankArena speed/serve_e2e .
Inference backend	vllm serve ; backend version recorded as vLLM 0.18.1 in the normalized outputs.
Client driver	vllm bench serve with --backend openai and --endpoint /v1/completions .
Tensor parallelism	tensor_parallel_size=1 ; one GPU per server.
Precision	Fixed within each table. The A100 launch script uses float16 . BF16 reruns set bfloat16 for every method in the same table.
KV cache and scheduler	vLLM defaults for the chosen dtype and engine settings; no method-specific scheduler options.

Appendix

Table 4: Inference configuration. Primary LowRankArena online inference settings are taken from benchmark/speed/serve_e2e.yaml and the corresponding inference-summary launch scripts.

Profile	Input tokens	Output tokens	Purpose
Prefill-heavy	4096	32	Measures prompt processing with a long input and short output.
Balanced	2048	128	Measures a mixed prompt-processing and generation setting.
Decode-heavy	512	512	Measures autoregressive generation with a long output.

Appendix

Table 5: Inference profiles. Each method in a inference table is evaluated on the same prefill-heavy, balanced, and decode-heavy profiles with the same request count, request arrival process, and engine settings.

Paper	Core idea	Speed evidence	How we classify it
SAES-SVD	Jointly optimizes local reconstruction and accumulated cross-layer error compensation (CEALC + ACES).	Single A6000 speedup plot on LLaMA-3-8B; workload shape under-specified.	Real hardware speedup exists, but protocol is too incomplete for strong tokens/s comparison.
SVD-LLM	Truncation-aware whitening + sequential low-rank weight update.	A100 + EPYC throughput sweeps with explicit batch/seq settings; repo eval uses use_cache=True .	One of the cleanest single-device autoregressive benchmarks; still not a serving study.
ASVD	Activation-aware transformation handles outliers; iterative calibration; extends to KV-cache compression.	No real throughput benchmark recovered.	Method/quality paper, not a speed paper.
Dobi-SVD	Differentiable rank/truncation selection + optimal weight update + optional remapping.	A100 throughput plus TITAN Xp 12GB case study.	Real speed paper, but TITAN Xp result is offload-confounded; remapping alters ratio semantics.
Basis Sharing	Shared basis vectors across layers with layer-specific coefficients.	Single A100, batch $=512$ , seq $=32$ , one-token generation.	Decode-side microbenchmark at one operating point.
DipSVD	Dual-level importance protection (local singular-vector protection + global layer allocation).	Real A100-class throughput figure with batch/seq sweeps.	Real throughput paper, but protocol is less explicit than SVD-LLM.

Appendix

Table 6: Summary of representative SVD-style LLM compression papers (Part I).

Paper	Core idea	Speed evidence	How we classify it
Swift-SVD	Closed-form activation-aware compression with one eigendecomposition; dynamic rank allocation.	Single 5090 inference benchmark + end-to-end compression-time benchmark.	Strong efficiency paper, but still a microbenchmark rather than serving evaluation.
OBD-LLM	Hessian-aware bi-directional whitening using input and output information.	Decomposition-runtime latency only.	Not an inference-throughput paper.
D-Rank	Layer-wise dynamic rank allocation under a fixed global budget.	Real throughput figure exists, but protocol is under-specified in accessible text.	Useful method paper; weaker system reproducibility.
DF-SVD	Fast rank selection / recovery dynamics tied to singular-spectrum structure and Hessian isotropy.	Reports both compression-runtime and inference throughput.	Stronger-than-average efficiency reporting; still not serving-style evaluation.
CPSVD	Preserves hard columns and factorizes only easy columns; supports within-layer non-uniform compression.	Single RTX 3090, batch $=4$ , seq $=1024$ .	Real hardware at one operating point only.
ERC-SVD (ResSVD)	Residual-compensated SVD with selective late-layer compression.	A100 throughput figure, seq $=32$ , varying batch size.	Real throughput paper; protocol not fully explicit.

Appendix

Table 7: Summary of representative SVD-style LLM compression papers (Part II).

	Prefill-heavy ( $4096\rightarrow 32$ )				Balanced ( $2048\rightarrow 128$ )				Decode-heavy ( $512\rightarrow 512$ )
Method	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$
Dense BF16	1033.71	3507.69	11521.19	90.03	73.00	1730.49	10896.70	681.38	51.19	6101.17	769.37	770.88
ASVD	906.88	3602.72	11154.51	87.17	93.72	2981.86	6328.40	395.72	72.32	11053.07	423.96	424.79
SVD-LLM v1	976.19	3852.98	10439.98	81.58	94.42	3031.04	6232.10	389.70	73.24	11251.25	416.68	417.49
SVD-LLM v2	967.90	3831.87	10490.16	81.97	101.52	3101.10	6058.19	378.82	75.84	11557.80	405.04	405.83
DoBi-SVD	1011.50	3943.80	10222.12	79.88	94.96	2712.51	6957.39	435.05	69.85	10040.88	467.48	468.39

Appendix

Table 8: A100 end-to-end serving summary for the Llama-3.1-8B keep-ratio-0.8 bf16 benchmark. Lower TTFT/E2E is better; higher prompt and generation throughput is better.

	Prefill-heavy ( $4096\rightarrow 32$ )				Balanced ( $2048\rightarrow 128$ )				Decode-heavy ( $512\rightarrow 512$ )
Method	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$
Dense BF16	932.72	3793.12	10583.97	82.71	433.55	4179.39	4592.98	287.20	85.81	11919.05	393.47	394.24
ASVD	1046.45	4248.33	9467.95	73.99	406.70	4612.59	4188.58	261.91	75.98	14164.50	335.53	336.19
SVD-LLM v1	1027.94	4111.04	9788.40	76.49	392.13	3883.31	4940.85	308.95	82.92	11251.31	417.29	418.11
SVD-LLM v2	1093.06	4436.42	9060.56	70.80	434.85	4859.52	3966.78	248.04	76.79	14803.03	320.04	320.66
DoBi-SVD	1001.48	4006.10	10045.33	78.50	384.81	3781.46	5074.26	317.30	77.27	10897.85	430.63	431.48

Appendix

Table 9: L40S end-to-end serving summary for the Llama-3.1-8B keep-ratio-0.8 bf16 benchmark. Lower TTFT/E2E is better; higher prompt and generation throughput is better.

	Prefill-heavy ( $4096\rightarrow 32$ )				Balanced ( $2048\rightarrow 128$ )				Decode-heavy ( $512\rightarrow 512$ )
Method	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$	TTFT $\downarrow$	E2E $\downarrow$	In tok/s $\uparrow$	Out tok/s $\uparrow$
Dense BF16	2426.34	8668.29	4722.81	36.90	1509.29	7298.74	2717.86	169.87	623.80	14447.66	328.66	328.66
ASVD	2544.26	10977.00	3739.81	29.22	1934.54	9760.96	2025.96	126.62	957.00	20533.82	230.95	230.95
SVD-LLM v1	2231.26	9636.49	4254.03	33.23	1677.82	8998.35	2180.49	136.28	797.32	20901.97	224.94	224.94
SVD-LLM v2	2639.96	11344.07	3625.66	28.33	2000.00	9878.82	1996.81	124.80	980.79	21198.33	222.23	222.23
DoBi-SVD	2123.16	9163.42	4470.79	34.93	1593.67	8790.49	2239.36	139.96	725.63	20962.44	224.96	224.96

Appendix

Table 10: RTX A5000 end-to-end serving. Lower TTFT/E2E is better; higher prompt and generation throughput is better.

Method	Source-model scope	7B/8B	Peak VRAM	70B one-H200 evidence
ASVD	Hugging Face AutoModel ; Llama-style scripts	Ready	74.2 GiB	Loads Llama-3.1-70B under the H200 memory cap, but stops on a legacy lm_eval.base import inside the sensitivity path before artifact export.
SVD-LLM v1	LLaMA/OPT components	Ready	139.1 GiB	Enters the low-resource whitening path, then OOMs during whitening/eigendecomposition near the H200 memory ceiling; no artifact is produced.
Basis Sharing	LLaMA/OPT/Mistral configs	Ready	15.8 GiB	Enters the local low-resource profiling path, then fails in the same CUDA eigensolver path as SVD-LLM v2 before artifact export.
MoDeGPT	Llama/OPT/Qwen adapters	Ready	123.6 GiB	Loads Llama-3.1-70B, then fails when moving an Accelerate-offloaded model to GPU before compression; no artifact is produced.
DoBi-SVD	AutoModel updater for decoder LMs	Ready	139.1 GiB	The outer wrapper exits successfully, but the internal trainer reaches the H200 memory ceiling and records trainer OOM before training/export, so no artifact is counted.

Appendix

Table 11: Released-code artifact-generation readiness for 70B on one H200. Rows audit whether the available implementation can directly produce a standardized LowRankArena artifact for Llama-3.1-70B at keep ratio 0.6, using one H200-141GB accelerator and no multi-GPU sharding or method-specific repair.

			Perplexity $\downarrow$		General MCQ $\uparrow$							MCQ Avg. $\uparrow$	Math $\uparrow$
Model	Budget	Method	WikiT.	C4	BoolQ	ARC-E	ARC-C	WinoG.	PIQA	HellaS.	OBQA	MCQ Avg.	MathQA	MMLU-M
Llama-2-7B	Dense FP		5.47	7.11	0.794	0.738	0.450	0.694	0.787	0.762	0.440	0.666	0.283	0.288
	80%	ASVD	10.82	15.04	0.685	0.562	0.333	0.639	0.709	0.620	0.382	0.561	0.248	0.304
		SVD-LLM v1	8.39	20.20	0.430	0.490	0.295	0.622	0.671	0.550	0.356	0.488	0.234	0.251
		SVD-LLM v2	9.77	24.76	0.485	0.421	0.282	0.606	0.631	0.505	0.328	0.465	0.227	0.297
		DoBi-SVD	10.06	25.27	0.463	0.470	0.283	0.618	0.645	0.477	0.336	0.470	0.228	0.259

Appendix

Table 12: Llama-2-7B leaderboard. All methods are evaluated from normalized summaries under the primary full-precision comparison scope with matched task versions, evaluation scripts, and budget definition. The table reports the full 80/70/60/50/40 keep-ratio grid with perplexity, MCQ Avg., and base-math metrics.

			Perplexity $\downarrow$		General MCQ $\uparrow$							MCQ Avg. $\uparrow$	Math $\uparrow$
Model	Budget	Method	WikiT.	C4	BoolQ	ARC-E	ARC-C	WinoG.	PIQA	HellaS.	OBQA	MCQ Avg.	MathQA	MMLU-M
Qwen3-8B-Base	Dense FP		7.00	11.78	0.830	0.800	0.570	0.727	0.793	0.787	0.420	0.704	0.542	0.729
	80%	ASVD	11.88	20.54	0.792	0.758	0.483	0.651	0.745	0.641	0.414	0.641	0.420	0.460
		SVD-LLM v1	11.08	33.90	0.688	0.649	0.424	0.669	0.712	0.616	0.408	0.595	0.330	0.355
		SVD-LLM v2	11.59	35.63	0.669	0.618	0.411	0.654	0.715	0.609	0.398	0.582	0.321	0.352
		DoBi-SVD	51.21	236.65	0.460	0.335	0.284	0.519	0.550	0.388	0.262	0.400	0.206	0.264

Appendix

Table 13: Qwen3-8B-Base leaderboard. All methods are evaluated from normalized summaries under the primary full-precision comparison scope with matched task versions, evaluation scripts, and budget definition. The table reports the full 80/70/60/50/40 keep-ratio grid with perplexity, MCQ Avg., and base-math metrics.

Model	Budget	Method	IFEval Strict $\uparrow$	GSM8K EM $\uparrow$	MMLU-Pro $\uparrow$	Avg $\uparrow$
Llama-3.1-8B-Instruct	FP	Dense FP	0.756	0.767	0.374	0.632
	80%	ASVD	0.113	0.000	0.112	0.075
		SVD-LLM v1	0.128	0.027	0.122	0.092
		SVD-LLM v2	0.109	0.024	0.109	0.081
		DoBi-SVD	0.111	0.000	0.105	0.072
		Basis Sharing	0.107	0.047	0.137	0.097

Appendix

Table 14: Llama-3.1-8B-Instruct auxiliary evaluation. Methods are evaluated across keep ratios with matched task versions and evaluation scripts. Avg. is the macro mean over the displayed instruction-style metrics.

Model	Budget	Method	IFEval Strict $\uparrow$	GSM8K EM $\uparrow$	MMLU-Pro $\uparrow$	Avg $\uparrow$
Qwen3-8B	FP	Dense FP	0.344	0.883	0.477	0.568
	80%	ASVD	0.220	0.119	0.217	0.185
		SVD-LLM v1	0.115	0.192	0.246	0.184
		SVD-LLM v2	0.124	0.249	0.228	0.200
		DoBi-SVD	0.129	0.000	0.115	0.081
		Basis Sharing	0.104	0.318	0.258	0.227

Appendix

Table 15: Qwen3-8B instruction-style auxiliary evaluation. Methods are evaluated across keep ratios with matched task versions and evaluation scripts. Avg. is the macro mean over the displayed instruction-style metrics.

Calibration sequences	BoolQ range	Other-6 range
128	0.2691	0.0030
256	0.1505	0.0020

Appendix

Table 16: Calibration-draw sensitivity for MoDeGPT on Llama-3.1-8B at an 80% keep ratio. Each range is the maximum minus the minimum score across three WikiText-2 sampling keys. “Other-6” denotes the average over ARC-Easy, ARC-Challenge, WinoGrande, PIQA, HellaSwag, and OpenBookQA.

Figure 7: LowRankArena repository organization. Benchmark policy is specified declaratively in benchmark/, checkpoint metadata is centralized in checkpoints/, reusable execution logic lives in src/, and all runners emit normalized JSON results for downstream analysis.

LLM Compression

Large language model compression reduces the memory, compute, and context costs of LLMs while aiming to preserve their capabilities.

See all 285 papers

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
Apr 2, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +6LLM Compression Low-Rank Compression
Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland +2
Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
Oct 23, 2026Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler +8LLM Compression Low-Rank Compression
Technical University of Munich, Germany · BMW Group, Germany · University of Glasgow, United Kingdom +1
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Jul 3, 2026Zhuowen Liu, Longkun Hao, Shiyu Feng +3Model Compression LLM Compression
Zhuowen Liu1 · Longkun Hao2 · Shiyu Feng3 +3

Apr 2, 2026J/K move · Enter open · S save

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

Ruoling Qi, Yirui Liu, Xuaner Wu, Xiangyu Wang+5

Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland+2

LLM Compression Low-Rank Compression Neural Network Compression Low-Rank Approximation Singular Value Decomposition

Open paper AskPDF arXiv

The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.

Abstract

LLM Compression

Explore similar work

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression