Hyperbolic space can embed intrinsic hierarchies in data with low distortion due to the exponential growth of volume with distance from the origin. However, current Lorentz transformer blocks are formulated in ambient coordinates, where numerical errors increase at large radii due to instability in the Lorentzian inner product, leading many models to limit the radius to avoid this issue. This prevents us from utilizing the regions of hyperbolic space that motivate the geometry. To address this, we revisit the components of the transformer block in polar coordinates, where hyperbolic operations such as distance calculation and attention centroids can be computed without the numerical cancellation in their ambient-coordinate formulations. Specifically, we propose a polar fully connected layer that separately maps an embedding's direction and radius, allowing the radius of a feature to be learned rather than determined by the norm of a linear map. We additionally introduce horospherical shifts as relative positional encodings whose query-key distances grow logarithmically with the token gap. Finally, we reformulate the residual connection as average radius Lorentz boosts. Combining these components, we develop a fully hyperbolic transformer that substantially improves performance over Euclidean and hyperbolic baselines on standard vision tasks. We further evaluate our model on ImageNet and demonstrate its ability to generalize to other datasets using pre-trained weights, similar to Euclidean counterparts.
Figures & tables
Model
Size
CIFAR-100
Tiny-ImageNet
Euclidean ViT
Tiny
74.64±0.41
60.74±0.36
HexFormer
Tiny
75.65±0.43
60.87±0.64
HexFormer-Hybrid
Tiny
75.85±0.34
62.93±0.45
Polar-ViT w/o horoshift
Tiny
78.04±0.53
65.12±0.47
Polar-ViT
Tiny
80.15±0.26
67.23±0.32
HVT ( Fein-Ashley et al., 2024 )
Base
42.77
40.12
Table 1: Top-1 test accuracy on CIFAR-100 and Tiny-ImageNet, results averaged over 10 seeds. Baseline rows are taken from Alyoussef et al. (2026) (Euclidean ViT, HexFormer) and He et al. (2025c) (HVT, LViT), which reports single values.
Table 2
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Operation
Weights
Condition for correctness
Attention aggregation
Softmax row with leak δ
δθ2≳4ε
Manifold weight decay, centroid residual
(1−w,w)
w≳εea
Batch-norm mean, token pooling
Uniform
θ≳2ε or r≤rε
Gyro-norm centering x⊕(−xˉ)
—
r≤rε
Appendix
Table 4: Centroid computations in a Lorentz Transformer and the condition under which ambient arithmetic computes them correctly. Here, a=r/k is the rescaled operating radius, θ is the angular spread of the aggregated points and δ is the weight that leaks from the dominant point.
Change
Component
Accuracy
Split-head Lorentz ViT, class-token readout
—
73.99±1.03
+ polar change of current ops and gyrolayer norm
-
76.27±0.81
+ boost residual
Section 4.4
77.06±0.43
+ polar boost residual
Eq. ( 12 )
77.20±0.13
+ polar fully connected layer
Eq. ( 8 )
78.04±0.53
+ horospherical positional encoding (Polar-ViT)
Eq. ( 10 )
80.15±0.26
Appendix
Table 5: Component ablation on CIFAR-100, ViT-Tiny, test accuracy over ten seeds. Each row adds one change to the row above.
Radius rule
Accuracy
Anchored gain, Eq. ( 8 )
78.45
Anchored gain, σ=0.1
78.02
Anchored gain, σ=0.2
77.36
Free power of the pre-activation norm
75.22
Affine in the input radius, Klein input
75.06
Affine in the input radius, unit-direction input
74.42
Appendix
Table 6: Rules for the output radius of the polar fully connected layer on CIFAR-100 with ViT-Tiny, one seed. σ is the standard deviation of a random initial deviation from the anchor.
Figure 1: Iterations needed by a single fully connected layer to reach a target point, as a function of the distance of the target from the origin. We report the mean over 3 seeds. Runs that do not converge are shown at the limit of 104 iterations.
Squared distance
Centroid
Gyro-norm centering
r
Ambient
Polar
Ambient
Polar
Ambient
Polar
Resolution limit
4
3⋅10−4
4⋅10−7
2⋅10−4
9⋅10−7
1.2⋅10−4
5.6⋅10−6
5⋅10−6
8
7⋅10−1
1⋅10−5
9
5⋅10−5
7.0⋅10−1
4.0⋅10−4
2⋅10−4
12
2⋅103
7⋅10−4
9
3⋅10−3
NaN
2.0⋅10−2
1⋅10−2
Appendix
Table 7: Single-precision error against double precision on clusters of 64 points with geodesic spread 1 at radius r ( k=1 ). We report the relative error for the squared distance and the geodesic error for the weighted centroid and gyro-norm centering. The last column is the bound of Proposition 2 for Δθ=2−24 .
Model
fp32
Euclidean ViT
21 ms
Lorentz ViT, ambient arithmetic
52 ms
Polar-ViT, polar arithmetic
58 ms
Appendix
Table 8: Forward and backward time per training step of ViT-Tiny on CIFAR-100 with batch size 128 on an RTX A5000, compiled. The times include only the model.
Model
Params
ISIC 2016
ISIC 2018
Kvasir-SEG
SANET
BUSI
published
Euclidean U-Net
486.6 K
0.92
0.89
0.86
0.78
0.82
Poincaré U-Net
≈487 K
0.91
0.87
0.83
0.76
0.80
ours
Lorentz U-Net, ambient
216.0 K
0.92
0.81
0.79
0.73
0.71
Lorentz U-Net, polar
215.5 K
0.93
0.89
0.86
0.81
0.78
Appendix
Table 9: Medical image segmentation on the benchmark of Mishra et al. (2026) . Test Dice (mean over foreground and background). The published rows are taken from Mishra et al. (2026) , and all values are rounded to two decimals as there. The best value of each column is in bold, and ties are both in bold.
Real-world data frequently exhibit latent hierarchical structures, which can be naturally represented by hyperbolic geometry. Although recent hyperbolic neural networks have demonstrated promising results, many existing architectures remain partially intrinsic, mixing Euclidean operations with hyperbolic ones or relying on extrinsic parameterizations. To address it, we propose the Intrinsic Lorentz Neural Network (ILNN), a fully intrinsic hyperbolic architecture that conducts all computations within the Lorentz model. At its core, the network introduces a novel point-to-hyperplane fully connected layer (FC), replacing traditional Euclidean affine logits with closed-form hyperbolic distances from features to learned Lorentz hyperplanes, thereby ensuring that the resulting geometric decision functions respect the inherent curvature. Around this fundamental layer, we design intrinsic modules: GyroLBN, a Lorentz batch normalization that couples gyro-centering with gyro-scaling, consistently outperforming both LBN and GyroBN while reducing training time. We additionally proposed a gyro-additive bias for the FC output, a Lorentz patch-concatenation operator that aligns the expected log-radius across feature blocks via a digamma-based scale, and a Lorentz dropout layer. Extensive experiments conducted on CIFAR-10/100 and two genomic benchmarks (TEB and GUE) illustrate that ILNN achieves state-of-the-art performance and computational cost among hyperbolic models and consistently surpasses strong Euclidean baselines. The code is available at magenta{this url}.
Xianglong Shi, Ziheng Chen, Yunhan Jiang +1
University of Science and Technology of China · University of Trento · Peking University
Residual Vector Quantization turns continuous representations into discrete, multi-level token sequences. Yet most methods operate in Euclidean space, despite the coarse-to-fine structure of the resulting codes and the latent hierarchies present in many data domains. Hyperbolic geometry offers a natural alternative for hierarchical representations, but naive hyperbolic extensions introduce geometric inconsistencies: non-associative hyperbolic addition prevents consistent residual aggregation, while standard straight-through gradient estimation ignores the geometry of the latent space. We propose a geometry-aware hyperbolic residual quantization that addresses these issues in both the forward and backward passes. In the forward pass, Hyperbolic Residual Aggregation restores the telescoping behavior of residual quantization on the Poincare ball. In the backward pass, a discounted Hyperbolic Straight-Through Estimator routes the reconstruction gradient through the quantizer as a single geometric block, avoiding unstable recursive gradient transport across residual stages. Evaluations on hierarchical prediction, recommendation, image tokenization, and neural audio coding tasks show that our method improves the stability and structural organization of hyperbolic residual codes over naive hyperbolic baselines. At the same time, we observe a clear structure-compression trade-off: Euclidean residual quantization remains preferable for pure compression, while geometry-aware hyperbolic quantization is most useful for hierarchically organized discrete latent spaces.
Alessio Colombo, Melika Ayoughi
Universiteit van Amsterdam, Amsterdam, Netherlands
While recent advancements like the Poincaré ResNet have demonstrated the potential of learning visual representations directly in hyperbolic space, their optimisation remains hampered by the computationally intensive nature of Riemannian gradients and the strict boundaries of the manifold. Furthermore, standard hyperbolic networks treat spatial transformations of the same object as distinct hierarchical concepts, leading to redundant parameter usage and vanishing signals. We propose Equivariant Poincaré ResNets, combining hyperbolic geometry with discrete symmetry groups (C4 and D4). We identify critical roadblocks in applying Euclidean equivariance to hyperbolic space and propose geometrically safe tensor reshaping, left-regular permutations for hyperbolic group convolutions, and joint-orientation Poincaré Midpoint Batch normalisation. Empirically, embedding equivariance drastically reduces the optimisation space, accelerating convergence while accelerating convergence while respecting the boundary constraints of the Poincaré ball and preserving spatial-group equivariance.
School of Computing Sciences, University of East Anglia, NR4 7TJ, Norwich, UK · Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037, Tromsø, Norway