Fast Differentiable SVD on GPU via Polar Decomposition

Uliana Parkina; Askar Tsyganov; Sergei Kudriashov; Sergey Samsonov; Maxim Rakhuba

math.NASep 26, 2026

Fast Differentiable SVD on GPU via Polar Decomposition

Authors: Uliana Parkina, Askar Tsyganov, Sergei Kudriashov, Sergey Samsonov, Maxim Rakhuba

Organizations: HSE University

Abstract

We present a fully GPU-oriented SVD pipeline based on polar decomposition, motivated by iterative methods that rely solely on matrix multiplications, such as the Newton-Schulz iteration. We show that this approach enables up to a $2\times$ speedup compared to standard implementations. Furthermore, we derive a numerically stable backward pass for the polar decomposition and leverage it to obtain a fully differentiable SVD. Our methods are released as open-source implementations in both PyTorch and JAX: https://github.com/fallnlove/cans_svd.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of different parts of the SVD computation by their share of the total runtime for a random matrix $M\in\mathbb{R}^{n\times n}$ with $\kappa_{2}(M)=10$ on an NVIDIA B200 GPU. We report the shares of polar-factor computation (in blue), symmetric EVD (EIGH) (in pink), and other routines (in green) for different polar-decomposition-based SVD algorithms.

Panel of Figure 2 — Figure 2: Comparison of SVD algorithms on random matrices. The total wall-clock time is averaged over $100$ trials and reported with the standard deviation. Note: The algorithms have different accuracy levels, see Tables 1 and 2 for a more comprehensive analysis. The experiments are conducted on an NVIDIA B200 GPU.

Method	$10$	$10^{2}$	$10^{3}$	$10^{4}$	$10^{6}$
CANS SVD (fp32)	$4.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$4.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$4.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-6}$	$4.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-6}$
CANS SVD (tf32)	$9.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$8.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$8.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$8.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$1.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-3}$
QDWH SVD	$3.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$6.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CUDA POLAR	$4.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$7.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{1}$	$7.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{1}$	$7.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{1}$
CUDA QR	$1.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-5}$	$2.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-5}$	$1.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-5}$	$1.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-5}$	$1.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-5}$
CUDA JACOBI	$9.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.4}\cdot 10^{-4}$	$8.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.4}\cdot 10^{-4}$	$6.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.8}\cdot 10^{-4}$	$6.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.6}\cdot 10^{-4}$	$5.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.7}\cdot 10^{-4}$

Table 1: Comparison of the accuracy of SVD algorithms on a random square matrix

M\in\mathbb{R}^{4096\times 4096}

for different condition numbers. The reported metric is the relative reconstruction error

\|M-U\Sigma V^{\top}\|_{F}/\|M\|_{F}

, with standard deviation. Results are averaged over

100

sampled matrices. Red indicates cases with significantly large errors.

Method	$10$	$10^{2}$	$10^{3}$	$10^{4}$	$10^{6}$
CANS SVD (fp32)	$3.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.9}\cdot 10^{-6}$	$3.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$2.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CANS SVD (tf32)	$6.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$6.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$2.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$6.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$6.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$
QDWH SVD	$3.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CUDA POLAR	$4.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$4.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$8.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$8.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$7.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CUDA QR	$9.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-6}$	$1.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-5}$	$1.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-5}$	$1.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-5}$	$9.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.4}\cdot 10^{-6}$
CUDA JACOBI	$2.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-3}$	$2.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-3}$	$2.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-3}$	$2.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-3}$	$2.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-3}$

Table 2: Comparison of the accuracy of SVD algorithms on a random square matrix

M\in\mathbb{R}^{4096\times 4096}

for different condition numbers. The reported metric is the relative orthogonality error

\max\{\|U^{\top}U-I\|_{F}/\|I\|_{F},\|V^{\top}V-I\|_{F}/\|I\|_{F}\}

, with standard deviation. Results are averaged over

100

sampled matrices.

Method \ Spec.gap	$1$	$10^{-1}$	$10^{-2}$	$10^{-3}$	$10^{-4}$	$10^{-5}$	$10^{-6}$
Analytical Townsend (2016)	$66.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.6}$	$67.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.0}$	$100.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 41.0}$	$690.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 580.0}$	$5500.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4600.0}$	NaN	NaN
Taylor Wang et al. (2021)	$49.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.3}$	$49.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.3}$	$49.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.3}$	$49.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.3}$	$49.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.3}$	$49.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.3}$	$49.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.3}$
Tikhonov (ours)	$66.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.6}$	$67.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.0}$	$100.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 41.0}$	$690.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 580.0}$	$5500.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4600.0}$	$66.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.6}$	$66.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.6}$

Table 3: Comparison of the gradient norm of SVD backward pass algorithms on a random square matrix

M\in\mathbb{R}^{4096\times 4096}

for different spectral gaps. The reported metric is the Frobenius norm of the gradient

\|\overline{M}\|_{F}

. Results are averaged over

100

sampled matrices.

Ratio	Method	PPL	Val Loss	sec. / iter.
100%	LLaMA-2-7B	${\color[rgb]{0.5,0.5,0.5}{5.53}}$	${\color[rgb]{0.5,0.5,0.5}{6.25}}$	${\color[rgb]{0.5,0.5,0.5}{-}}$
	Dobi-SVD	6.95	7.17	75.09
80%	Our ( $\varepsilon=0.1$ )	6.90	7.19	55.62
	Our ( $\varepsilon=1.0$ )	6.87	7.11	55.62
	Our ( $\varepsilon=10$ )	6.93	7.17	55.62
	Dobi-SVD	7.47	7.83	75.09

Table 4: Metric values of various compression methods. Experiments were conducted using the LLaMA-2-7B model on the WikiText-2 dataset.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Comparison of computation time for matrix multiplications in tf32 precision (left plot) and QR decompositions (right plot) across different GPUs on random square matrices of various sizes. The total wall-clock time is averaged over $100$ trials.

Panel of Figure 4 — Figure 4: Comparison illustrating how many matrix multiplications can be executed within the runtime of a single QR decomposition for square matrices of different sizes across different GPUs. In the left plot, matrix multiplications are performed in tf32 precision. In the right plot, matrix multiplications are performed in fp32 precision. The total wall-clock time is averaged over $100$ trials.

Figure 5: Comparison of SVD algorithms on a random low rank matrix

M\in\mathbb{R}^{n\times n}

with

\operatorname{rank}(M)=\lfloor n/10\rfloor

. Total wall-clock time was averaged over

100

trials and reported with the standard deviation. Note: we do not report CUDA POLAR for this setting, since it has substantially large reconstruction error. And algorithms have different accuracy, see Tables 5 and 6 for a more comprehensive analysis. The experiments are conducted on an NVIDIA B200 GPU.

Method	$1$	$64$	$1024$	$4095$	$4096$
CANS SVD (fp32)	$1.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$2.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-6}$	$4.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$8.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.4}\cdot 10^{-6}$	$4.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CANS SVD (tf32)	$5.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-4}$	$7.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-4}$	$1.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-3}$	$9.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$9.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$
QDWH SVD	$1.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-6}$	$1.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$2.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CUDA POLAR	$8.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 22.7}\cdot 10^{0}$	$7.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{1}$	$7.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{1}$	$7.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{1}$	$4.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CUDA QR	$7.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.5}\cdot 10^{-7}$	$1.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-6}$	$6.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.4}\cdot 10^{-6}$	$1.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-5}$	$1.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-5}$
CUDA JACOBI	$3.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.6}\cdot 10^{-5}$	$9.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-5}$	$5.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.4}\cdot 10^{-4}$	$1.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-4}$	$9.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.4}\cdot 10^{-4}$

Appendix

Table 5: Comparison of SVD algorithms accuracy on a random square matrix

M\in\mathbb{R}^{4096\times 4096}

for different ranks. Reported relative reconstruction error

\|M-U\Sigma V^{\top}\|_{F}/\|M\|_{F}

, with standard deviation. Metrics were averaged over

100

sampled matrices. Red indicates cases with significantly large error.

Method	$1$	$64$	$1024$	$4095$	$4096$
CANS SVD (fp32)	$1.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$1.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$1.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.2}\cdot 10^{-6}$	$3.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CANS SVD (tf32)	$1.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-6}$	$1.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$2.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$6.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$
QDWH SVD	$1.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$1.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$1.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$3.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CUDA POLAR	$7.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.4}\cdot 10^{-6}$	$3.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$6.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$1.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-5}$	$4.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
CUDA QR	$7.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-6}$	$7.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-6}$	$6.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-6}$	$9.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.3}\cdot 10^{-6}$	$9.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-6}$
CUDA JACOBI	$2.5_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.2}\cdot 10^{-6}$	$3.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-4}$	$1.7_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-3}$	$2.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-3}$	$2.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-3}$

Appendix

Table 6: Comparison of SVD algorithms accuracy on a random square matrix

M\in\mathbb{R}^{4096\times 4096}

for different ranks. Reported relative orthogonality

\max\{\|U^{\top}U-I\|_{F}/\|I\|_{F},\|V^{\top}V-I\|_{F}/\|I\|_{F}\}

, with standard deviation. Metrics were averaged over

100

sampled matrices.

Method	$10$	$10^{2}$	$10^{3}$	$10^{4}$	$10^{6}$
CANS SVD (fp32)	$4.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$4.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$4.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-6}$	$4.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-6}$
Extended	$5.8_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.6_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-6}$	$5.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.1}\cdot 10^{-6}$	$1.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-5}$	$9.0_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
SciPy	$5.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.2_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$5.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$
Gram	$4.4_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-6}$	$2.3_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-5}$	$1.1_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$3.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$	$6.9_{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm.0}\cdot 10^{-4}$

Appendix

Table 7: Comparison of the accuracy of SVD algorithms on a random square matrix

M\in\mathbb{R}^{4096\times 4096}

for different condition numbers. The reported metric is the relative reconstruction error

\|M-U\Sigma V^{\top}\|_{F}/\|M\|_{F}

, with standard deviation. Results are averaged over

100

sampled matrices.

Figure 6: Comparison of SVD algorithms on a square random matrices

M\in\mathbb{R}^{n\times n}

with

\kappa_{2}(M)=10

. Total wall-clock time was averaged over

100

trials and reported with the standard deviation. The experiments are conducted on an NVIDIA B200 GPU.

Figure 7: Comparison of different methods for computing polar decomposition on a square random matrices

M\in\mathbb{R}^{n\times n}

with

\kappa_{2}(M)=10^{6}

. The reported metric is relative orthogonality error

\|W^{\top}W-I\|_{F}/\|I\|_{F}

Figure 8: Pareto front for different CUDA JACOBI tolerance settings on square random matrices

M\in\mathbb{R}^{4096\times 4096}

with

\kappa_{2}(M)=10^{2}

$\kappa_{2}(M)\downarrow\ \delta\rightarrow$	$0.9$	$0.99$	$0.999$	$0.9999$
$10^{2}$	$3.30\cdot 10^{-6}$	$4.26\cdot 10^{-6}$	$7.25\cdot 10^{-6}$	$1.84\cdot 10^{-5}$
$10^{4}$	$3.77\cdot 10^{-6}$	$4.41\cdot 10^{-6}$	$1.05\cdot 10^{-5}$	$2.51\cdot 10^{-5}$
$10^{6}$	$4.00\cdot 10^{-6}$	$5.38\cdot 10^{-6}$	$1.69\cdot 10^{-5}$	$4.88\cdot 10^{-5}$

Appendix

Table 8: Ablation on different

\delta

for preprocessing on a random matrix

M\in\mathbb{R}^{4096\times 4096}

for different condition numbers. The reported metric is the relative reconstruction error of CANS SVD (fp32).

$\kappa_{2}(M)\downarrow\ \delta\rightarrow$	$0.9$	$0.99$	$0.999$	$0.9999$
$10$	$7.20\cdot 10^{-4}$	$9.73\cdot 10^{-4}$	$2.98\cdot 10^{-3}$	$9.15\cdot 10^{-3}$
$10^{2}$	$6.57\cdot 10^{-4}$	$8.24\cdot 10^{-4}$	$1.84\cdot 10^{-3}$	$4.33\cdot 10^{-3}$
$10^{3}$	$7.04\cdot 10^{-4}$	$8.52\cdot 10^{-4}$	$1.91\cdot 10^{-3}$	$3.63\cdot 10^{-3}$

Appendix

Table 9: Ablation on different

\delta

for preprocessing on a random matrix

M\in\mathbb{R}^{4096\times 4096}

for different condition numbers. The reported metric is the relative reconstruction error of CANS SVD (tf32).

Panel of Figure 9 — Figure 9: Performance comparison of different precisions (tf32 and bf16) for matrix multiplication. The left plot shows the runtime of the matrix multiplications themselves. The right plot shows how the SVD runtime changes when the multiplication precision is varied. The total wall-clock time is averaged over $100$ trials. Experiments are conducted on an NVIDIA B200 GPU.

Panel of Figure 10 — Figure 10: Comparison of different methods for solving the symmetric Lyapunov equation under different scenarios. The total wall-clock time was averaged over $100$ trials and reported with the standard deviation. The experiments were conducted on an NVIDIA B200 GPU.

Explore similar work

Singular value soft-thresholding via the polar decomposition
Jul 24, 2026Stephen Becker
University of Colorado Boulder, USA
Quasi-SVD: Learning a Lie-constrained matrix factorisation for real-time imaging
Jul 28, 2026Christopher HahneLie Group Methods Computational Imaging
University of Bern 3008 Bern, Switzerland
Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations
Apr 7, 2026Chris Choy3D Representation Learning Representation Learning
NVIDIA

Abstract

Explore similar work

Singular value soft-thresholding via the polar decomposition

Quasi-SVD: Learning a Lie-constrained matrix factorisation for real-time imaging

Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations