WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models

Haiyu Wang; Yutong Wang; Jack Jiang; Sai Qian Zhang

cs.CVApr 2, 2026

WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models

Authors: Haiyu Wang, Yutong Wang, Jack Jiang, Sai Qian Zhang

Organizations: Tandon School of Engineering, New York University · Courant Institute of Mathematical Sciences, New York University

Abstract

Singular Value Decomposition (SVD) has become an important technique for reducing the computational burden of Vision Language Models (VLMs), which play a central role in tasks such as image captioning and visual question answering. Although multiple prior works have proposed efficient SVD variants to enable low-rank operations, we find that in practice it remains difficult to achieve substantial latency reduction during model execution. To address this limitation, we introduce a new computational pattern and apply SVD at a finer granularity, enabling real and measurable improvements in execution latency. Furthermore, recognizing that weight elements differ in their relative importance, we adaptively allocate relative importance to each element during SVD process to better preserve accuracy, then extend this framework with quantization applied to both weights and activations, resulting in a highly efficient VLM. Collectively, we introduce~Weighted SVD (WSVD), which outperforms other approaches by achieving over $1.8\times$ decoding speedup while preserving accuracy. We open source our code at: \href{https://github.com/SAI-Lab-NYU/WSVD}{`https://github.com/SAI-Lab-NYU/WSVD`

Figures & tables

Figure 1: (a) Architecture of vision-language model. (b) Overview of WSVD framework.

Figure 2: (a) Latency evaluation of VLM including self-attention (SA) and feed-forward (FFN) modules. (b) Conventional SVD: the left side illustrates SVD of $W_{k}$ , and the right side shows the reconstruction of $K_{h}$ from the shared latent. (c) Per-head SVD: the left side illustrates per-head SVD of $W_{Kh}$ , and right side shows per-head reconstruction of $K_{h}$ from per-head latent.

Figure 3: (a) Naive reconstruction requires materializing and writing back full $K_{h}$ to VRAM (global GPU memory), leading to excessive memory usage and I/O. (b) Our fused kernel consumes $C_{Kh}$ and $C_{Vh}$ tiles on-chip with flash decoding, reducing both peak memory footprint and I/O traffic. All the step numbers are shown in circle.

Figure 4: WSVD decoding pipeline. Each token is down-projected to low-rank latents, and

K

and

V

latents are appended to the cache, while

Q

latent is up-projected and consumed together with cached

C_{Kh},C_{Vh}

in the fused kernel.

Acc.	Method	ScienceQA-IMG $\uparrow$					SEED-Bench $\uparrow$					Avg. $\uparrow$
Acc.	Method	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	$\rho_{1}:50\%$	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	$\rho_{1}:50\%$	Avg. $\uparrow$
LLaVA-v1.5 7B	ASVD	49.93%	50.12%	47.10%	36.69%	19.19%	54.27%	53.53%	48.35%	37.17%	24.17%	42.05%
	SVD-LLM	65.44%	63.71%	61.92%	57.41%	55.53%	57.89%	57.50%	55.33%	54.64%	55.31%	58.47%
	QSVD-noQ	67.72%	68.22%	67.08%	65.05%	62.37%	59.84%	59.07%	59.78%	59.00%	58.23%	62.64%
	WSVD-noQ	68.17%	67.72%	67.28%	65.89%	65.49%	60.10%	60.17%	59.89%	60.18%	60.46%	63.54%
	FP16	Accuracy: 68.01%					Accuracy: 60.18%					64.10%

Table 1: Accuracy evaluation of different methods under FP16 (detailed results in Appendix A.3 ).

Method	ScienceQA-IMG $\uparrow$				SEED-Bench $\uparrow$				Avg. $\uparrow$
Method	v1.5 7B	v1.5 13B	Next 7B	Next 13B	v1.5 7B	v1.5 13B	Next 7B	Next 13B	Avg. $\uparrow$
DuQuant	57.36%	67.22%	66.34%	70.20%	54.11%	61.43%	63.64%	66.15%	63.31%
QVLM	55.24%	66.46%	60.60%	65.28%	50.13%	59.22%	50.38%	65.39%	59.09%
QASVD	41.92%	65.34%	49.37%	64.85%	41.26%	59.30%	49.63%	66.54%	54.78%
QSVD	65.61%	70.12%	66.10%	70.43%	58.49%	62.95%	65.63%	69.21%	66.07%
WSVD	64.25%	72.14%	66.94%	73.08%	60.23%	62.01%	67.49%	70.67%	67.10%

Table 2: Accuracy evaluation of different methods under low-precision on LLaVA-v1.5 7B, LLaVA-v1.5 13B, LLaVA-Next 7B and LLaVA-Next 13B.

Acc.	Method	$\rho_{1}{=}90\%$	$\rho_{1}{=}70\%$	$\rho_{1}{=}50\%$
v1.5 7B	FP16	68.01%
	WSVD-noFT	67.82%	66.82%	65.09%
	WSVD-noQ	68.17%	67.28%	65.49%

Table 3: Results of weighted finetuning ablation under different

\rho_{1}

settings.

Method	ScienceQA-IMG $\uparrow$				Avg. $\uparrow$
Method	v1.5 7B	v1.5 13B	Next 7B	Next 13B	Avg. $\uparrow$
W/o QAT	63.91%	71.99%	66.59%	72.68%	68.79%
WSVD	64.25%	72.14%	66.94%	73.08%	69.10%

Table 4: Results of local QAT ablation.

Figure 5: Latency evaluation and normalized latency on: (a) RTX 4090 and (b) RTX 5090.

Table 5: Latency (ms) on RTX 4090 (left) and RTX 5090 (right) for different sequence lengths.

Table 6: Latency (ms) on RTX 4090 (left) and RTX 5090 (right) for different batch sizes.

Table 12

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{L}_{\text{WSVD}}(A_{h},B_{h})=\bigl\|F_{h}^{1/2}\odot(W_{h}-A_{h}B_{h})\bigr\|_{F}^{2}

Appendix

Algorithm 1 Weighted SVD Fine-tuning (WSVD) pseudo code

Acc.	Method	ScienceQA-IMG $\uparrow$					SEED-Bench $\uparrow$					Avg. $\uparrow$
Acc.	Method	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	$\rho_{1}:50\%$	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	$\rho_{1}:50\%$	Avg. $\uparrow$
LLaVA-v1.5 7B	ASVD	49.93%	50.12%	47.10%	36.69%	19.19%	54.27%	53.53%	48.35%	37.17%	24.17%	42.05%
	SVD-LLM	65.44%	63.71%	61.92%	57.41%	55.53%	57.89%	57.50%	55.33%	54.64%	55.31%	58.47%
	QSVD-noQ	67.72%	68.22%	67.08%	65.05%	62.37%	59.84%	59.07%	59.78%	59.00%	58.23%	62.64%
	WSVD-noQ	68.17%	67.72%	67.28%	65.89%	65.49%	60.10%	60.17%	59.89%	60.18%	60.46%	63.54%
	FP16	Accuracy: 68.01%					Accuracy: 60.18%					64.10%

Appendix

Table 9: Accuracy evaluation of different methods under FP16.

Acc.	Method	ScienceQA-IMG $\uparrow$
Acc.	Method	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	$\rho_{1}:50\%$
LLaVA-v1.5 7B	DobiSVD	67.19%	60.94%	59.38%	56.64%	54.69%
	ASVD	49.93%	50.12%	47.10%	36.69%	19.19%
	SVD-LLM	65.44%	63.71%	61.92%	57.41%	55.53%
	QSVD-noQ	67.72%	68.22%	67.08%	65.05%	62.37%
	WSVD-noQ	68.17%	67.72%	67.28%	65.89%	65.49%

Appendix

Table 10: Accuracy evaluation of different methods under FP16.

Acc.	Method	ScienceQA-IMG $\uparrow$				SEED-Bench $\uparrow$				Avg. $\uparrow$
Acc.	Method	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	Avg. $\uparrow$
Qwen-VL 7B	ASVD	63.86%	63.11%	60.54%	59.49%	61.67%	61.00%	59.94%	58.69%	61.04%
	SVD-LLM	65.29%	65.29%	64.55%	65.29%	62.99%	62.80%	62.69%	62.51%	63.93%
	QSVD-noQ	66.78%	66.24%	66.68%	65.20%	63.25%	63.00%	62.06%	61.26%	64.31%
	WSVD-noQ	68.77%	68.12%	67.23%	65.49%	63.32%	63.53%	63.60%	63.68%	65.47%
	FP16	Accuracy: 68.32%				Accuracy: 63.52%				65.92%

Appendix

Table 11: Accuracy evaluation of different methods under FP16 on Qwen-VL 7B and Molmo-7B-O.

Acc.	Method	OCRBench $\uparrow$					HRBench-4K $\uparrow$					Avg. $\uparrow$
Acc.	Method	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	$\rho_{1}:50\%$	$\rho_{1}:90\%$	$\rho_{1}:80\%$	$\rho_{1}:70\%$	$\rho_{1}:60\%$	$\rho_{1}:50\%$	Avg. $\uparrow$
LLaVA-Next 13B	ASVD	53.10%	52.50%	51.30%	50.50%	47.70%	44.00%	44.13%	43.88%	43.00%	42.00%	47.21%
	SVD-LLM	52.10%	51.90%	51.00%	49.90%	48.20%	43.00%	44.25%	42.62%	43.75%	43.25%	47.00%
	QSVD-noQ	52.80%	52.40%	52.40%	51.40%	48.90%	44.25%	44.88%	43.88%	43.37%	42.88%	47.72%
	WSVD-noQ	53.30%	53.30%	53.50%	52.20%	48.70%	46.13%	44.88%	44.50%	44.88%	44.50%	48.59%
	FP16	Accuracy: 53.30%					Accuracy: 45.63%					49.47%

Appendix

Table 12: Accuracy evaluation of different methods under FP16 on OCRBench and HRBench-4K.

Table 18

QSVD		WSVD
Step	Time	Step	Time
Input calibration	1 min	Input calibration	6 min
SVD on all layers	2 min 15 s	SVD on all layers	1 min
Gradient collection & rank allocation	10 min 12 s	Gradient collection & rank allocation	13 min 40 s
SVD results fusion	30 s	Local FT	9 min
$\beta$ tuning & quantization	82 min	Local QAT & quantization	8 min

Appendix

Table 15: Calibration time breakdown for QSVD and WSVD.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

LSVD: Loss-Aware Low-Rank Approximation for Efficient Low-Precision Vision-Language Models
May 30, 2026Haiyu Wang, Yutong Wang, Leshu Li +2Vision-Language Models Efficient VLM Inference
Tandon School of Engineering, New York University · Courant Institute of Mathematical Sciences, New York University · Brookhaven National Laboratory
SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models
Sep 21, 2026Kewei Zhang, Zheng Chen, Haotong Qin +1VLM Quantization Efficient VLM Inference
Shanghai Jiao Tong University · The Hong Kong Polytechnic University
Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
Apr 2, 2026Ruoling Qi, Yirui Liu, Xuaner Wu +6LLM Compression Low-Rank Compression
Institute of Artificial Intelligence (TeleAI), China Telecom · Shanghai Jiao Tong University · University of Maryland +2

May 30, 2026J/K move · Enter open · S save

LSVD: Loss-Aware Low-Rank Approximation for Efficient Low-Precision Vision-Language Models

Haiyu Wang, Yutong Wang, Leshu Li, Yihui Ren+1

Tandon School of Engineering, New York University · Courant Institute of Mathematical Sciences, New York University · Brookhaven National Laboratory

Vision-Language Models Efficient VLM Inference Low-Rank Matrix Decomposition Low-Rank Compression Singular Value Decomposition

Open paper AskPDF arXiv

Vision-language models (VLMs) deliver strong multimodal reasoning capabilities, but their large computational cost and high parameter counts make deployment challenging on resource-constrained devices. Low-rank decomposition has emerged as a promising compression technique, yet existing methods often optimize local matrix reconstruction error, rely on uniform or heuristic rank allocation, and focus mainly on attention projections while leaving feed-forward networks underexplored. In this paper, we propose~LASER (Loss-Aware Singular-value dEcomposition and Rank allocation), a low-rank compression framework for efficient low-precision VLM inference. LASER derives a curvature-weighted SVD objective from a second-order approximation of the model loss and uses Kronecker-factored Fisher information to guide decomposition toward downstream performance rather than reconstruction alone. We further introduce a loss-aware cross-layer rank allocation strategy based on calibration gradients, enabling more effective parameter budgeting across layers. Finally, we extend low-rank compression to FFN layers through a hybrid scheme that combines SVD with quantization. The evaluation results show that LASER achieves more than

2.3\times

decoding speedup over previous work while preserving strong accuracy under low-precision inference.

Abstract

Efficient VLM Inference

Explore similar work

LSVD: Loss-Aware Low-Rank Approximation for Efficient Low-Precision Vision-Language Models

SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression