SO(3)-RoPE for Spherical Transformers

Christian Libner; Chase van de Geijn; Alexander S. Ecker; Maurice Weiler

cs.LGOct 5, 2026

SO(3)-RoPE for Spherical Transformers

Authors: Christian Libner, Chase van de Geijn, Alexander S. Ecker, Maurice Weiler

Organizations: Institute of Computer Science & CIDAS, University of Göttingen · Max Planck Institute for Dynamics and Self-Organization, Göttingen · MIT CSAIL

Abstract

Spherical data arise in many scientific applications. Often spherical transformers disregard the geometry of the underlying spherical domain, causing distortions and coordinate singularities near the poles. We introduce SO(3)-RoPE, a relative positional embedding that incorporates spherical geometry into transformer attention through unitary SO(3) representations. Our formulation is SO(3)-equivariant and compatible with FlashAttention, retaining efficiency of vanilla transformers. On shallow water dynamics prediction over a rotating sphere, our SO3ViT outperforms an S2Transformer baseline with lower errors and reduced runtime.

Figures & tables

Figure 1: SO3ViT. We patchify the sphere and process those patches with a transformer. In the attention we embed the queries and keys with our $\mathrm{SO}(3)$ RoPE.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Depth	Width	Params	$\ell_{1}$	$\ell_{2}$	Memory	Inference Time
S2Transformer	4	128	0.53 M	0.031	0.040	8.01 GB	141.4 ms
S2Transformer	4	256	2.12 M	0.031	0.040	15.96 GB	198.8 ms
S2Transformer	4	512	8.43 M	0.027	0.035	31.92 GB	282.1 ms
S2Transformer	8	256	4.22 M	0.031	0.039	21.99 GB	471.0 ms
SO3ViT	4	128	0.98 M	0.045	0.058	0.74 GB	6.1 ms
SO3ViT	4	256	3.54 M	0.034	0.043	1.28 GB	9.3 ms

Appendix

Table 1: Peak GPU memory during training and inference time. For every S2Transformer configuration, some SO3ViT configuration performs better in

\ell_{1}

error,

\ell_{2}

error, memory, and inference speed.

Explore similar work

URoPE: Universal Relative Position Embedding across Geometric Spaces
Apr 20, 2026Yichen Xie, Depu Meng, Chensheng Peng +4Visual Geometry Grounded Transformer Rotary Position Embedding
Applied Intuition · University of California, Berkeley
HySPE: Positional Encoding via Symplectic Dual Shears
Oct 7, 2026Zhongping Ji
RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
May 11, 2026Byeongchan Kim, Arijit Sehanobish, Avinava Dubey +2Positional Encoding Transformer Attention
1Seoul National University · 2Independent · 3Google Research +2

Apr 20, 2026, cs.CVJ/K move · Enter open · S save

URoPE: Universal Relative Position Embedding across Geometric Spaces

Yichen Xie, Depu Meng, Chensheng Peng, Yihan Hu+3

Applied Intuition · University of California, Berkeley

Visual Geometry Grounded Transformer Rotary Position Embedding Relative Position Transformer Architectures 3D Object Detection

Open paper AskPDF arXiv

Relative position embedding has become a standard mechanism for encoding positional information in Transformers. However, existing formulations are typically limited to a fixed geometric space, namely 1D sequences or regular 2D/3D grids, which restricts their applicability to many computer vision tasks that require geometric reasoning across camera views or between 2D and 3D spaces. To address this limitation, we propose URoPE, a universal extension of Rotary Position Embedding (RoPE) to cross-view or cross-dimensional geometric spaces. For each key/value image patch, URoPE samples 3D points along the corresponding camera ray at predefined depth anchors and projects them into the query image plane. Standard 2D RoPE can then be applied using the projected pixel coordinates. URoPE is a parameter-free and intrinsics-aware relative position embedding that is invariant to the choice of global coordinate systems, while remaining fully compatible with existing RoPE-optimized attention kernels. We evaluate URoPE as a plug-in positional encoding for transformer architectures across a diverse set of tasks, including novel view synthesis, 3D object detection, object tracking, and depth estimation, covering 2D-2D, 2D-3D, and temporal scenarios. Experiments show that URoPE consistently improves the performance of transformer-based models across all tasks, demonstrating its effectiveness and generality for geometric reasoning. Our code is available on our project website: https://urope-pe.github.io/.

Abstract

Explore similar work

URoPE: Universal Relative Position Embedding across Geometric Spaces

HySPE: Positional Encoding via Symplectic Dual Shears

RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings