Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates

Ahmad Bdeir; Niels Landwehr

cs.CVSep 26, 2026

Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates

Authors: Ahmad Bdeir, Niels Landwehr

Organizations: Data Science Group University of Hildesheim Hildesheim, Germany

Abstract

Hyperbolic space can embed intrinsic hierarchies in data with low distortion due to the exponential growth of volume with distance from the origin. However, current Lorentz transformer blocks are formulated in ambient coordinates, where numerical errors increase at large radii due to instability in the Lorentzian inner product, leading many models to limit the radius to avoid this issue. This prevents us from utilizing the regions of hyperbolic space that motivate the geometry. To address this, we revisit the components of the transformer block in polar coordinates, where hyperbolic operations such as distance calculation and attention centroids can be computed without the numerical cancellation in their ambient-coordinate formulations. Specifically, we propose a polar fully connected layer that separately maps an embedding's direction and radius, allowing the radius of a feature to be learned rather than determined by the norm of a linear map. We additionally introduce horospherical shifts as relative positional encodings whose query-key distances grow logarithmically with the token gap. Finally, we reformulate the residual connection as average radius Lorentz boosts. Combining these components, we develop a fully hyperbolic transformer that substantially improves performance over Euclidean and hyperbolic baselines on standard vision tasks. We further evaluate our model on ImageNet and demonstrate its ability to generalize to other datasets using pre-trained weights, similar to Euclidean counterparts.

Figures & tables

Model	Size	CIFAR-100	Tiny-ImageNet
Euclidean ViT	Tiny	$74.64\pm 0.41$	$60.74\pm 0.36$
HexFormer	Tiny	$75.65\pm 0.43$	$60.87\pm 0.64$
HexFormer-Hybrid	Tiny	$75.85\pm 0.34$	$62.93\pm 0.45$
Polar-ViT w/o horoshift	Tiny	$\underline{78.04\pm 0.53}$	$\underline{65.12\pm 0.47}$
Polar-ViT	Tiny	$\mathbf{80.15\pm 0.26}$	$\mathbf{67.23\pm 0.32}$
HVT ( Fein-Ashley et al., 2024 )	Base	42.77	40.12

Table 1: Top-1 test accuracy on CIFAR-100 and Tiny-ImageNet, results averaged over 10 seeds. Baseline rows are taken from Alyoussef et al. (2026) (Euclidean ViT, HexFormer) and He et al. (2025c) (HVT, LViT), which reports single values.

Table 2

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Operation	Weights	Condition for correctness
Attention aggregation	Softmax row with leak $\delta$	$\delta\theta^{2}\gtrsim 4\varepsilon$
Manifold weight decay, centroid residual	$(1-w,w)$	$w\gtrsim\varepsilon e^{a}$
Batch-norm mean, token pooling	Uniform	$\theta\gtrsim 2\sqrt{\varepsilon}$ or $r\leq r_{\varepsilon}$
Gyro-norm centering $x\oplus(-\bar{x})$	—	$r\leq r_{\varepsilon}$

Appendix

Table 4: Centroid computations in a Lorentz Transformer and the condition under which ambient arithmetic computes them correctly. Here,

a=r/\sqrt{k}

is the rescaled operating radius,

\theta

is the angular spread of the aggregated points and

\delta

is the weight that leaks from the dominant point.

Change	Component	Accuracy
Split-head Lorentz ViT, class-token readout	—	$73.99\pm 1.03$
+ polar change of current ops and gyrolayer norm	-	$76.27\pm 0.81$
+ boost residual	Section 4.4	$77.06\pm 0.43$
+ polar boost residual	Eq. ( 12 )	$77.20\pm 0.13$
+ polar fully connected layer	Eq. ( 8 )	$78.04\pm 0.53$
+ horospherical positional encoding (Polar-ViT)	Eq. ( 10 )	$\mathbf{80.15\pm 0.26}$

Appendix

Table 5: Component ablation on CIFAR-100, ViT-Tiny, test accuracy over ten seeds. Each row adds one change to the row above.

Radius rule	Accuracy
Anchored gain, Eq. ( 8 )	$\mathbf{78.45}$
Anchored gain, $\sigma=0.1$	78.02
Anchored gain, $\sigma=0.2$	77.36
Free power of the pre-activation norm	75.22
Affine in the input radius, Klein input	75.06
Affine in the input radius, unit-direction input	74.42

Appendix

Table 6: Rules for the output radius of the polar fully connected layer on CIFAR-100 with ViT-Tiny, one seed.

\sigma

is the standard deviation of a random initial deviation from the anchor.

Panel of Figure 1 — Figure 1: Iterations needed by a single fully connected layer to reach a target point, as a function of the distance of the target from the origin. We report the mean over $3$ seeds. Runs that do not converge are shown at the limit of $10^{4}$ iterations.

$r$	Ambient	Polar	Ambient	Polar	Ambient	Polar	Resolution limit
	Squared distance		Centroid		Gyro-norm centering
4	$3\cdot 10^{-4}$	$4\cdot 10^{-7}$	$2\cdot 10^{-4}$	$9\cdot 10^{-7}$	$1.2\cdot 10^{-4}$	$5.6\cdot 10^{-6}$	$5\cdot 10^{-6}$
8	$7\cdot 10^{-1}$	$1\cdot 10^{-5}$	$9$	$5\cdot 10^{-5}$	$7.0\cdot 10^{-1}$	$4.0\cdot 10^{-4}$	$2\cdot 10^{-4}$
12	$2\cdot 10^{3}$	$7\cdot 10^{-4}$	$9$	$3\cdot 10^{-3}$	NaN	$2.0\cdot 10^{-2}$	$1\cdot 10^{-2}$

Appendix

Table 7: Single-precision error against double precision on clusters of

64

points with geodesic spread

1

at radius

r

(

k=1

). We report the relative error for the squared distance and the geodesic error for the weighted centroid and gyro-norm centering. The last column is the bound of Proposition 2 for

\Delta\theta=2^{-24}

Model	fp32
Euclidean ViT	21 ms
Lorentz ViT, ambient arithmetic	52 ms
Polar-ViT, polar arithmetic	58 ms

Appendix

Table 8: Forward and backward time per training step of ViT-Tiny on CIFAR-100 with batch size

128

on an RTX A5000, compiled. The times include only the model.

	Model	Params	ISIC 2016	ISIC 2018	Kvasir-SEG	SANET	BUSI
published	Euclidean U-Net	$486.6$ K	$0.92$	$\mathbf{0.89}$	$\mathbf{0.86}$	$0.78$	$\mathbf{0.82}$
published	Poincaré U-Net	$\approx 487$ K	$0.91$	$0.87$	$0.83$	$0.76$	$0.80$
ours	Lorentz U-Net, ambient	$216.0$ K	$0.92$	$0.81$	$0.79$	$0.73$	$0.71$
ours	Lorentz U-Net, polar	$215.5$ K	$\mathbf{0.93}$	$\mathbf{0.89}$	$\mathbf{0.86}$	$\mathbf{0.81}$	$0.78$

Appendix

Table 9: Medical image segmentation on the benchmark of Mishra et al. (2026) . Test Dice (mean over foreground and background). The published rows are taken from Mishra et al. (2026) , and all values are rounded to two decimals as there. The best value of each column is in bold, and ties are both in bold.

Vision Transformer

Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.

See all 352 papers

Intrinsic Lorentz Neural Network
Feb 27, 2026Xianglong Shi, Ziheng Chen, Yunhan Jiang +1Hyperbolic Representation Learning Image Classification
University of Science and Technology of China · University of Trento · Peking University
Geometry-Aware Hyperbolic Residual-Quantized Variational Autoencoders
Sep 22, 2026Alessio Colombo, Melika AyoughiResidual VQ Hierarchical Representation Learning
Universiteit van Amsterdam, Amsterdam, Netherlands
Group-Equivariant Poincaré Convolutional Networks
Jul 1, 2026Aiden Durrant, Rahul Baburajan, Georgios LeontidisEquivariant Neural Networks Hyperbolic Representation Learning
School of Computing Sciences, University of East Anglia, NR4 7TJ, Norwich, UK · Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037, Tromsø, Norway

Feb 27, 2026J/K move · Enter open · S save

Intrinsic Lorentz Neural Network

Xianglong Shi, Ziheng Chen, Yunhan Jiang, Nicu Sebe

University of Science and Technology of China · University of Trento · Peking University

Hyperbolic Representation Learning Image Classification

Open paper AskPDF arXiv

Real-world data frequently exhibit latent hierarchical structures, which can be naturally represented by hyperbolic geometry. Although recent hyperbolic neural networks have demonstrated promising results, many existing architectures remain partially intrinsic, mixing Euclidean operations with hyperbolic ones or relying on extrinsic parameterizations. To address it, we propose the Intrinsic Lorentz Neural Network (ILNN), a fully intrinsic hyperbolic architecture that conducts all computations within the Lorentz model. At its core, the network introduces a novel point-to-hyperplane fully connected layer (FC), replacing traditional Euclidean affine logits with closed-form hyperbolic distances from features to learned Lorentz hyperplanes, thereby ensuring that the resulting geometric decision functions respect the inherent curvature. Around this fundamental layer, we design intrinsic modules: GyroLBN, a Lorentz batch normalization that couples gyro-centering with gyro-scaling, consistently outperforming both LBN and GyroBN while reducing training time. We additionally proposed a gyro-additive bias for the FC output, a Lorentz patch-concatenation operator that aligns the expected log-radius across feature blocks via a digamma-based scale, and a Lorentz dropout layer. Extensive experiments conducted on CIFAR-10/100 and two genomic benchmarks (TEB and GUE) illustrate that ILNN achieves state-of-the-art performance and computational cost among hyperbolic models and consistently surpasses strong Euclidean baselines. The code is available at magenta{this url}.

Abstract

Vision Transformer

Explore similar work

Intrinsic Lorentz Neural Network

Geometry-Aware Hyperbolic Residual-Quantized Variational Autoencoders

Group-Equivariant Poincaré Convolutional Networks