Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations

Chris Choy

cs.LGApr 7, 2026

Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations

Authors: Chris Choy

Organizations: NVIDIA

Abstract

Recent work has shown that removing orthogonalization during training and applying it only at inference improves rotation estimation in deep learning, with empirical evidence favoring 9D representations with SVD projection. However, the theoretical understanding of why SVD orthogonalization specifically harms training, and why it should be preferred over Gram-Schmidt at inference, remains incomplete. We provide a detailed gradient analysis of SVD orthogonalization specialized to $3 \times 3$ matrices and $SO(3)$ projection. Our central result derives the exact spectrum of the SVD backward pass Jacobian: it has rank $3$ (matching the dimension of $SO(3)$ ) with nonzero singular values $2/(s_i + s_j)$ and condition number $κ= (s_1 + s_2)/(s_2 + s_3)$ , creating quantifiable gradient distortion that is most severe when the predicted matrix is far from $SO(3)$ (e.g., early in training when $s_3 \approx 0$ ). We further show that even stabilized SVD gradients introduce gradient direction error, whereas removing SVD from the training loop avoids this tradeoff entirely. We also prove that the 6D Gram-Schmidt Jacobian has an asymmetric spectrum: its parameters receive unequal gradient signal, explaining why 9D parameterization is preferable. Together, these results provide the theoretical foundation for training with direct 9D regression and applying SVD projection only at inference.

Figures & tables

Figure 1: Gradient of $\mathcal{L}$ w.r.t. $M_{11}$ (or $t^{\prime}_{11}$ ) vs. the error $M_{11}-R^{*}_{11}$ , with $\sigma=0.5$ Gaussian noise ( $10{,}000$ samples). Left: Direct 9D gradients lie on the diagonal (each element depends only on its own error). Center: SVD-Train gradients scatter across all quadrants; color encodes the singular value gap $\delta=s_{2}+s_{3}$ (small $\delta$ in red = most erratic). Right: GS-Train (6D) also produces ambiguous gradients from cross-column coupling.

Figure 2: SVD vs. Gram-Schmidt projection error at inference, restricted to the small-

\sigma

regime where the first-order theory of Levinson et al. (2020) is accurate (and where trained networks operate). Left: Expected squared Frobenius error

\mathbb{E}[\left\lVert R_{\mathrm{proj}}-R^{*}\right\rVert_{F}^{2}]

with

M=R^{*}+\sigma N

. Empirical results (solid,

5{,}000

samples per

\sigma

) closely match the first-order predictions (dashed):

3\sigma^{2}+O(\sigma^{3})

for SVD and

6\sigma^{2}+O(\sigma^{3})

for GS. Right: The ratio of GS to SVD error stays near the theoretical value of 2 across the entire range.

Figure 3: Coordinate dependence test. For random $M$ and $R_{2}\in\mathrm{SO}(3)$ , we compare $g(M)$ with $g(MR_{2})R_{2}^{\top}$ : for a coordinate-independent projector, these should be identical. Left: SVD produces zero inconsistency (spike at $0°$ ), confirming $\mathrm{SVDO}^{+}(MR_{2})R_{2}^{\top}=\mathrm{SVDO}^{+}(M)$ . GS spreads over $10$ – $60°$ , showing its result depends on the choice of coordinate frame. Right: SVD projection error is identical regardless of coordinates (points on the diagonal), while GS error changes unpredictably under coordinate rotation.

Figure 4: Per-column RMS projection error at inference (

\sigma=0.5

50{,}000

samples). Left: Gram-Schmidt produces increasing error from column 1 (privileged, normalized first) to column 3 (derived via cross product), reflecting its sequential, greedy forward-pass structure. Right: SVD distributes error equally across all three columns, a consequence of its coordinate-independence. SVD also achieves lower absolute error per column.

Representation	Dim	Cont. $g$	Double	Jacobian	Training	Gradient	Null	Inference
			cover	rank	$\kappa$	blow-up	space	proj.
Euler angles	3	No	No ∗	3 †	$\to\infty$ †	Gimbal lock	0 †	None
Exp. coordinates	3	No	Yes	3 †	$\to\infty$ †	At $\left\lVert\omega\right\rVert=2\pi$	0 †	None
Axis-angle	4	No	Yes	3	$\to\infty$ †	At $\theta=\pi$	1	Normalize
Quaternion	4	No	Yes	3	$=1$	Never ‡	1	Normalize
$\mathbb{R}^{6}$ + GS ( Zhou et al., 2019 )	6	Yes	No	$\leq 5$	$\geq\frac{\left\lVert\mathbf{t}_{1}^{\prime}\right\rVert}{\left\lVert\mathbf{r}_{2}^{\prime\prime}\right\rVert}$	$O(1/\left\lVert\mathbf{r}_{2}^{\prime\prime}\right\rVert)$	$\geq 1$	GS

Table 1: Comparison of rotation representations for gradient-based optimization. “Continuous

g

” refers to whether the mapping from

\mathrm{SO}(3)

to the representation space is continuous ( Zhou et al., 2019 ) . “Training

\kappa

” is the condition number of the representation Jacobian during training. “Inference proj.” indicates the projection method applied at inference to obtain a valid rotation. Representations above the mid-rule suffer from topological obstructions (discontinuity and/or double cover); those below are continuous but differ in gradient properties.

Explore similar work

Jun 1, 2026

Riemannian Gradient Descent for Low-Rank Architectures

We explore Riemannian optimization techniques for rank-factored matrix parameters, targeting contemporary deep learning applications. We examine ten points in the algorithm design space: two geometries for rank-

r

matrices, three geometries for rank-

r

partial isometries, and block-matrix variants of these five, where factors are shared across block-rows and block-columns. We apply our methods to the multihead attention parameters in small language models. After tuning learning rates, our methods do not conclusively outperform an AdamW baseline. Our implementations are available online.

Nicholas Knight

NVIDIA

Riemannian Optimization Low-Rank Approximation Matrix Optimization Riemannian Geometry Multi-Head Attention

Ask PDF

Sep 26, 2026

Fast Differentiable SVD on GPU via Polar Decomposition

We present a fully GPU-oriented SVD pipeline based on polar decomposition, motivated by iterative methods that rely solely on matrix multiplications, such as the Newton-Schulz iteration. We show that this approach enables up to a

2\times

speedup compared to standard implementations. Furthermore, we derive a numerically stable backward pass for the polar decomposition and leverage it to obtain a fully differentiable SVD. Our methods are released as open-source implementations in both PyTorch and JAX: https://github.com/fallnlove/cans_svd.

Uliana Parkina, Askar Tsyganov, Sergei Kudriashov +2

HSE University

Singular Value Decomposition Matrix Factorization GPU Acceleration

Ask PDF

Aug 18, 2026

Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale

Differentially private training adds isotropic Gaussian noise to clipped gradients, corrupting every singular direction equally. In vision models, where spatial correlation concentrates gradient energy into a low-rank subspace, most of this noise falls in directions that carry little signal. Spectral gradient orthogonalization via polar decomposition is introduced as a post-processing step that recovers directional signal from the noisy gradient's low-rank structure at zero additional privacy cost. A phase transition governs the utility of this approach: orthogonalization improves accuracy only when the per-direction spectral signal-to-noise ratio (SNR) suffices for singular vector recovery; in low-SNR regimes, the directional bias of the gradient is replaced by a nearly random orthogonal update, and the transformation is harmful. The recovery threshold is determined by the spectral gap of the gradient and is surpassed at large batch sizes. Empirically, the benefit scales with model capacity: spectral orthogonalization achieves a +20.9% improvement over DP-SGD on WRN-28-10 (B = 4096) and +14.9% on ResNet-18, while reducing inter-run variance by a factor of two to three. In the fine-tuning regime, spectral orthogonalization matches the stability of DP-Adam while maintaining a first-order memory footprint. Combining spectral with temporal denoising yields 50.3% on CIFAR-10 (epsilon = 4), the highest accuracy in any tested configuration. These gains are specific to moderate-to-high-SNR regimes such as large-batch training of higher-capacity models. Small-batch or low-SNR settings are better served by DP-SGD or temporal denoising.

Sabari Shanmugam, Nick Barnes, Kerry Taylor

School of Computing, Australian National University, Canberra, Australia

Privacy-Preserving ML Differential Privacy Differentially Private Stochastic Gradient Descent

Ask PDF