Gauge-Equivariant Attention for Rotation-Stable $360^\circ$ Scene Understanding

Tianjian Zhou; Yishan Li; Jie Jiang; Yifei Zhang

cs.CVSep 23, 2026

Gauge-Equivariant Attention for Rotation-Stable $360^\circ$ Scene Understanding

Authors: Tianjian Zhou, Yishan Li, Jie Jiang, Yifei Zhang

Organizations: National University of Defense Technology (NUDT), Changsha, China · Northwestern Polytechnical University, Xi’an, China

Abstract

Panoramic $360^\circ$ scene understanding increasingly relies on icosphere transformers, but a state-of-the-art spherical model loses more than half of its segmentation accuracy when the camera rotates by $90^\circ$ , and controlled ablations identify gauge dependence in its relative-position bias as a major contributor. We propose gauge-equivariant relative position encoding (GE-RPE): a parameter-free Reynolds average of the bias over a finite cyclic subgroup $C_n\!\subset\!\mathrm{SO}(2)$ of gauge rotations. Plugged into a SphereUFormer backbone the change is invisible at deployment---zero added parameters and $1.5$ -- $4.4\%$ forward latency---and the matched three-seed GE-RPE model records a $1.3\%$ drop; the published SphereUFormer checkpoint records $53\%$ under the same stress protocol but a different training recipe. Once the gauge defect is removed and a teacher-token permutation $π_R$ aligns the SSL views to the rotated student frame, iBOT $+$ MAE pretraining stops being a liability and becomes a clean low-label lever: the full framework EquiSSL (GE-RPE $+$ $π_R$ $+$ iBOT $+$ MAE) tightens the drop to $0.8\%$ at $68.30\%$ val mIoU and lifts $1\%$ -label fine-tuning by $+2.39$ mIoU on the $N{=}373$ test split (and by $+4.10$ on the smaller $N{=}40$ val split); the same fix carries over to monocular depth and to zero-shot Structured3D segmentation. The construction is provably $C_n$ -invariant and $\mathcal{O}(n^{-2})$ -close to the continuous $\mathrm{SO}(2)$ average, making the resulting model a usable $360^\circ$ visual-computing primitive across panoramic relighting, immersive video, and cross-dataset transfer. Code is available at https://github.com/Jaywalk18/equissl-release.

Figures & tables

Figure 1. Rotation breaks a standard spherical model; EquiSSL fixes it at two levels. Top: a standard SphereUFormer gives clean upright regions but collapses after a $90^{\circ}$ rotation (more than $50\%$ mIoU drop). Bottom: EquiSSL couples GE-RPE, which averages the attention bias over a discrete gauge group so attention co-rotates with the input, with a teacher-token permutation $\pi_{R}$ that aligns the SSL target to the student’s rotated frame. The result is stable segmentation before and after rotation. Side-by-side panoramic segmentation results showing that a standard spherical transformer's predictions disintegrate under a 90 degree camera rotation while the proposed EquiSSL produces consistent segmentation before and after rotation; a schematic at the bottom indicates the two levels of the fix—gauge-equivariant attention and rotation-consistent self-supervised pretraining.

Figure 2. Why the RPE lookup depends on the local frame. The spherical pair $(q,k)$ is fixed, but Frames A and B assign coordinates $\mathbf{r}$ and $g^{\top}\mathbf{r}$ ; the same anisotropic table $B$ can therefore return different values. GE-RPE averages the $C_{n}$ -rotated lookups into $b_{\mathrm{GE}}(q,k)$ , which is invariant to frame choices in $C_{n}$ . A fixed query-key pair on an icosphere branches into two valid tangent frames. The rotated axes assign two coordinates to the same displacement and sample different locations in one shared bias table. A cyclic average over rotated frames merges these lookups into a frame-independent scalar bias.

Figure 3. EquiSSL: full framework. (a) End-to-end pipeline. A $360^{\circ}$ panorama is projected onto an icosphere, encoded by SphereUFormer, and fine-tuned for dense segmentation; the dashed sub-box marks rotation-consistent iBOT $+$ MAE pretraining, where $\pi_{R}$ re-permutes teacher tokens into the student’s rotated frame. (b) Zoom-in of one attention layer of (a). GE-RPE replaces the standard relative-position bias by its $C_{n}$ Reynolds average $B_{\mathrm{GE}}(\mathbf{r})=\tfrac{1}{n}\sum_{g\in C_{n}}B(g^{\top}\mathbf{r})$ —the bias-level term isolated by the controlled comparison—at zero parameter cost. Two-panel pipeline diagram. The top panel shows the end-to-end EquiSSL framework: a $360^\circ$ panorama is resampled onto a rank-7 icosphere, fed through a four-scale SphereUFormer encoder-decoder, and fine-tuned for semantic segmentation; a dashed sub-block marks the rotation-consistent iBOT$+$MAE pretraining stage in which a teacher-token permutation $\pi_R$ aligns the cross-view target. The bottom panel zooms into one attention layer and depicts GE-RPE replacing the gauge-dependent relative-position bias with its $C_n$-Reynolds average over $n$ rotated lookups of the same shared $7\times7$ bias grid. — Figure 3. EquiSSL: full framework. (a) End-to-end pipeline. A $360^{\circ}$ panorama is projected onto an icosphere, encoded by SphereUFormer, and fine-tuned for dense segmentation; the dashed sub-box marks rotation-consistent iBOT $+$ MAE pretraining, where $\pi_{R}$ re-permutes teacher tokens into the student’s rotated frame. (b) Zoom-in of one attention layer of (a). GE-RPE replaces the standard relative-position bias by its $C_{n}$ Reynolds average $B_{\mathrm{GE}}(\mathbf{r})=\tfrac{1}{n}\sum_{g\in C_{n}}B(g^{\top}\mathbf{r})$ —the bias-level term isolated by the controlled comparison—at zero parameter cost. Two-panel pipeline diagram. The top panel shows the end-to-end EquiSSL framework: a $360^\circ$ panorama is resampled onto a rank-7 icosphere, fed through a four-scale SphereUFormer encoder-decoder, and fine-tuned for semantic segmentation; a dashed sub-block marks the rotation-consistent iBOT $+$ MAE pretraining stage in which a teacher-token permutation $\pi_R$ aligns the cross-view target. The bottom panel zooms into one attention layer and depicts GE-RPE replacing the gauge-dependent relative-position bias with its $C_n$ -Reynolds average over $n$ rotated lookups of the same shared $7\times7$ bias grid.

Panel of Figure 4 — Figure 4. Quantitative validation of Theorems 4.3 and 4.4 on a real trained network. Panel (a) reads off Corollary 4.5 ’s mode-survival law on the bilinear grid; panels (b) and (c) verify the two complementary rate constants—the $\mathcal{O}(1/n^{2})$ Fourier-tail rate of Theorem 4.4 and the $\pi/n$ worst-case Lipschitz radius of Theorem 4.3 —both of which are tight in measurement, with the $C_{2}$ point flagged as the even-mode parity outlier. Three-panel figure validating the rotation-stability theorems. Panel (a): bar chart of polar Fourier mode amplitudes for the learnable bias, with the $m=2$ mode visibly dominant; survival strips show which modes $C_2$ , $C_4$ , $C_6$ preserve. Panel (b): log-log plot of the simulated $C_n$ Reynolds-average bias residual versus $n$ , overlaid with the empirical Stanford2D3D rotation drop on a secondary axis; both trace an order- $1/n^2$ slope from $n=2$ to $n=12$ , with the $C_2$ point sitting notably above the bound due to even-mode aliasing. Panel (c): histogram of per-vertex residual gauge mismatch angle after snapping to the nearest $C_n$ rotation for $n \in\{2,4,6\}$ ; each empirical density stays inside its $\pi/n$ cutoff and matches a uniform reference.

Method	Venue	Equiv.	Params	Fwd	val mIoU	test	rot. mIoU	Drop ⋄	Drop std	$1\%$ lab.	Depth
			(M)	(ms)	(%)	(%)	@ $90^{\circ}$		(%)	val (%)	$\delta_{1}$
Prior work (single best; same split, val-mIoU context).
SFSS ∗	ICCV’19	none	$30.4$	$112$	$42.02$	$19.54$	$16.81$	$60.0\%$	$\pm 1.5$	$11.20$	$0.762$
HEAL-SWIN ( Carlsson et al., 2024 )	CVPR’24	sampling-only	$50.4$	$208$	$62.45$	$29.05$	$31.22$	$50.0\%$	$\pm 1.0$	$16.85$	$0.852$
SphereUFormer ( Benny et al., 2025 )	CVPR’25	none	$35.1$	$163.2$	$67.53$	$31.40$	$31.74^{\natural}$	$53.0\%$	$\pm 0.7$	$17.95$	$0.872$
SO3UFormer ‡ ( Zhu et al., 2026 )	2026	SO(3) feat.	$48.2$	$245$	$66.5$	$30.95$	$46.95$	$29.4\%$	$\pm 1.8$	$18.60$	$0.890$

Table 1. Semantic segmentation on Stanford2D3D (13 classes, Area-

5

split:

1013/40/373

train/val/test). Bold : random-init GE-RPE

C_{6}

(architectural baseline isolating the gauge fix from pretraining); the canonical EquiSSL system (GE-RPE

C_{6}

+ iBOT

+

MAE) is in Table 3 . Prior-work rows are single best as published. Our rows are

3

-seed mean

\pm

std (seeds

\{42,123,456\}

); rotation drop is paired per-seed then averaged over

10

rotations

\times 3

repeats.

Figure 5. Accuracy–robustness Pareto on Stanford2D3D val.

y

-axis: rotation drop at

\theta_{\max}{=}90^{\circ}

(log-scale, lower is better);

x

-axis: upright val mIoU (higher is better); shaded band is the desirable corner. EquiSSL (

\bigstar

68.30\%

0.8\%

) sits closest. Per-point values are tabulated in Tables 1 – 3 ; SFSS (

42.0

val) is off-scale on the

x

-axis. Single-panel log-y scatter plot: SFSS at the upper-left (42 mIoU, 60 percent rotation drop), SphereUFormer and HEAL-SWIN near the top (~62-63 mIoU, 50-53 percent drop), SO3UFormer mid-axis (66.5 mIoU, 29 percent drop), the C2 negative control isolated at 67.6 mIoU and 16.6 percent drop, and the EquiSSL family (No RPE, Standard RPE, C4, C6+area, C6 no-area, and the EquiSSL star) clustered between 66 and 68 mIoU at sub-2 percent drop inside a green-shaded desirable corner. The EquiSSL star is the closest point to that corner.

RPE	Train aug.	mIoU @ $0^{\circ}$	mIoU @ $90^{\circ}$	Drop
Standard	none	$66.35{\pm}0.48$	$65.60{\pm}0.30$	$1.1{\pm}0.2\%$
Standard	Haar- $\mathrm{SO}(3)$ (32)	$66.50{\pm}0.62$	$66.13{\pm}0.47$	$0.54{\pm}0.75\%$
Radial	none	$65.83{\pm}0.38$	$64.63{\pm}0.36$	$1.81{\pm}0.19\%$
GE-RPE $C_{6}$	none	$\mathbf{67.85{\pm}0.38}$	$\mathbf{66.94{\pm}0.39}$	$\mathbf{1.3{\pm}0.2\%}$

Table 2. Matched alternatives on Stanford2D3D val (three training seeds; mean

\pm

sample std). All rows use the same rank-7 backbone, 350-epoch supervised recipe, and Pose90 evaluation (

10

rotations

\times 3

repeats).

Figure 6. Qualitative Stanford2D3D segmentation under a $90^{\circ}$ yaw rotation, rendered on the sphere. Top row, columns (a)–(h): (a) ground truth, (b) SphereUFormer (published checkpoint), (c) standard RPE (ablation), (d) Tangent-Img, (e) PanoFormer, (f) HEAL-SWIN, (g) SO3UFormer, (h) EquiSSL (Ours). Bottom row repeats each column as a binary correct/wrong error map against the ground truth. SphereUFormer and HEAL-SWIN dissolve into horizontal streaks aligned with the rotated ERP gauge; SO3UFormer recovers large-scale layout but leaks across boundaries; EquiSSL preserves both object regions and class boundaries. Figure 7 shows the same sample across multiple yaw angles. \rev{Eight-column} qualitative comparison of segmentation predictions on a Stanford2D3D panorama after a 90 degree yaw rotation, displayed on the sphere. Columns: ground truth, SphereUFormer published checkpoint, standard RPE ablation, \rev{Tangent-Img, PanoFormer,} HEAL-SWIN, SO3UFormer, and EquiSSL. The top row shows class predictions with the room legend (ceiling, clutter, door, floor, wall, unlabelled); the bottom row shows correct/wrong error maps. SphereUFormer and HEAL-SWIN predictions break into horizontal streaks aligned with the rotated equirectangular gauge, SO3UFormer recovers coarse layout but leaks across boundaries, and EquiSSL preserves object regions and class boundaries.

Figure 7. Rotation-angle sweep on one Stanford2D3D val sample. Rows: input RGB, ground truth, SphereUFormer (published checkpoint), and GE-RPE $C_{4}$ . Columns: yaw rotations at $0^{\circ},\ 30^{\circ},\ 60^{\circ},\ 90^{\circ}$ . SphereUFormer’s predictions degrade progressively with rotation; GE-RPE $C_{4}$ stays visually consistent across the sweep, complementing the single-angle comparison of Figure 6 . Four-row by four-column qualitative grid for one Stanford2D3D panorama under yaw rotations of $0^\circ$, $30^\circ$, $60^\circ$, and $90^\circ$. The input RGB and ground-truth rows rotate consistently across columns. The SphereUFormer prediction row degrades progressively with rotation, showing increasing fragmentation and class swaps. The GE-RPE $C_4$ row stays close to the ground truth across all four columns. — Figure 7. Rotation-angle sweep on one Stanford2D3D val sample. Rows: input RGB, ground truth, SphereUFormer (published checkpoint), and GE-RPE $C_{4}$ . Columns: yaw rotations at $0^{\circ},\ 30^{\circ},\ 60^{\circ},\ 90^{\circ}$ . SphereUFormer’s predictions degrade progressively with rotation; GE-RPE $C_{4}$ stays visually consistent across the sweep, complementing the single-angle comparison of Figure 6 . Four-row by four-column qualitative grid for one Stanford2D3D panorama under yaw rotations of $0^\circ$ , $30^\circ$ , $60^\circ$ , and $90^\circ$ . The input RGB and ground-truth rows rotate consistently across columns. The SphereUFormer prediction row degrades progressively with rotation, showing increasing fragmentation and class swaps. The GE-RPE $C_4$ row stays close to the ground truth across all four columns.

		Seg. val mIoU		test	Low-label val mIoU			Depth	Depth	Rot.	S3D zero-shot		S3D
Pretraining	RPE	@ $0^{\circ}$	@ $90^{\circ}$	mIoU	$1\%$	$5\%$	$10\%$	$\delta_{1}$	RMSE	Drop	@ $0^{\circ}$	@ $90^{\circ}$	Drop
Random init	No RPE	$67.58{\pm}0.58$	$66.70$	$31.42$	$17.92{\pm}0.43$	$38.50$	$43.22$	$0.876{\pm}0.003$	$0.236$	$1.3\%$	$26.10$	$24.45$	$6.3\%$
Random init	Standard	$66.35{\pm}0.48$	$65.60$	$31.11$	$17.98{\pm}0.29$	$38.71$	$44.21$	$0.869{\pm}0.002$	$0.241$	$1.1\%$	$28.55$	$27.70$	$3.0\%$
Random init	GE-RPE $C_{4}$	$67.31{\pm}0.70$	$66.17$	$30.50$	$18.62{\pm}0.37$	$37.47$	$44.73$	$0.904{\pm}0.002$	$0.226$	$1.7\%$	$32.90$	$32.70$	$0.6\%$
Random init	GE-RPE $C_{6}$	$67.85{\pm}0.38$	$66.94$	$31.20$	$18.49{\pm}0.36$	$37.47$	$44.85$	$0.911{\pm}0.002$	$0.223$	$1.3\%$	$32.45$	$32.10$	$1.0\%$
iBOT $+$ MAE	No RPE	$67.58{\pm}0.58$	$66.69{\pm}0.61$	$31.20$	$21.32{\pm}0.42$	$40.85$	$46.10$	$0.891{\pm}0.002$	$0.230$	$1.2\%$	$29.45$	$27.60$	$6.3\%$

Table 3. Self-supervised pretraining on Structured3D and fine-tuning on Stanford2D3D (3-seed mean

\pm

std). The diagnostic signal is in the rotation-drop,

1\%

-label, and Structured3D zero-shot columns: Standard RPE worsens under SSL (

1.1\%\!\to\!8.6\%

rotation drop), while SSL-pretrained EquiSSL tightens to

0.8\%

. S3D zero-shot is logged for SSL-pretrained checkpoints (iBOT

+

MAE rows) only,

3

-seed mean

\pm

std (seeds

\{42,123,456\}

Figure 8. Rotation drop scales as

\mathcal{O}(1/n^{2})

in the gauge count

n

;

C_{2}

is the even-mode aliasing outlier;

C_{4}

already lands inside the

<\!1\%

-drop band (green). Filled markers show empirical drops with

\pm 1

std error bars; log–log axes make the

\mathcal{O}(1/n^{2})

reference a straight line. Log-log plot of empirical rotation drop versus gauge count n. C2 sits high near 16 percent (an aliasing outlier), C4 and C6 lie within a green sub-1-percent band, and the rate matches the predicted O(1 over n squared) reference line.

Figure 9. Rotation robustness on Stanford2D3D val across

\theta_{\max}\in\{0^{\circ},10^{\circ},20^{\circ},35^{\circ},45^{\circ},60^{\circ},90^{\circ}\}

. Dashed star is the measured canonical SSL-pretrained EquiSSL 3-seed curve; solid lines are our random-init 3-seed curves (GE-RPE

C_{6}

no-SSL, No RPE, Standard RPE); dash-dot lines are reference curves for SO3UFormer and HEAL-SWIN; dotted lines are SphereUFormer (our reproduction) and SFSS. Legend suffixes report the

0^{\circ}\!\to\!90^{\circ}

endpoint drop. Line plot of validation mean intersection-over-union versus maximum rotation angle (0 to 90 degrees) for eight configurations: the EquiSSL star line and GE-RPE C6 no-SSL hold a tight band near 68 mIoU with sub-2 percent drop; No RPE sits just below at about 67.6 with a similar 1.3 percent drop; Standard RPE stays near 66 with about 1.1 percent drop; SO3UFormer follows a concave-down curve from 66.5 down to about 47 mIoU; HEAL-SWIN, SphereUFormer, and SFSS collapse from 62 to 31, 63 to 30, and 42 to 17 mIoU respectively. Two soft horizontal bands shade a "rotation collapse" region below 50 mIoU and a "robust band" above 65 mIoU.

	val mIoU @ $\theta_{\max}$ ( ∘ , %)							test	Drop
Config.	$0$	$10$	$20$	$35$	$45$	$60$	$90$	( $\%$ )	$0{\to}90^{\circ}$
$C_{6}$ + area	$67.24$	$66.65$	$66.49$	$66.35$	$66.41$	$66.31$	$66.32$	$30.89$	$1.4{\pm}0.7\%$
$C_{6}$ no-area	$\mathbf{67.85}$	$\mathbf{67.26}$	$\mathbf{67.09}$	$\mathbf{66.92}$	$\mathbf{66.88}$	$\mathbf{66.92}$	$\mathbf{66.94}$	$\mathbf{31.01}$	$\mathbf{1.3{\pm}0.2\%}$
$\Delta$ (n.a. $-$ a)	$+0.61$	$+0.61$	$+0.60$	$+0.57$	$+0.47$	$+0.61$	$+0.62$	$+0.12$	$-0.1$ pp

Table 4. Area weighting ablation: per-angle val mIoU on Stanford2D3D (GE-RPE

C_{6}

, 35.1M, 3-seed mean). All 14 per-angle cells are measured.

RPE variant	Build	Fwd	$\Delta$ Fwd	Peak Mem	$\Delta$ Mem
	(s)	(ms)	(%)	(MB)	(%)
No RPE	$23.3$	$163.6$	—	$4369$	—
Standard RPE	$22.3$	$163.2$	$0.0\%$	$4385$	$0.0\%$
GE-RPE $C_{2}$	$46.6$	$165.6$	$+1.5\%$	$4439$	$+1.2\%$
GE-RPE $C_{4}$	$47.5$	$168.8$	$+3.4\%$	$4511$	$+2.9\%$
GE-RPE $C_{6}$ (+area)	$44.1$	$170.3$	$+4.4\%$	$4579$	$+4.4\%$

Table 5. Runtime and peak GPU memory at rank-7 (

163{,}842

nodes), A100-80GB, batch

4

\Delta

Fwd /

\Delta

Mem are relative to Standard RPE; all rows share Params

{=}35.1

M (zero added). One GPU-hot run uses

10

warm-up and

50

timed forward passes, with torch.cuda.synchronize() between variants. All cells are measured.

Figure 10. Label-efficiency on Stanford2D3D val across three label fractions (

1\%

10\%

100\%

): Standard RPE, random-init GE-RPE

C_{6}

, and EquiSSL . At

1\%

labels EquiSSL adds

+4.10

mIoU over the matched random-init

C_{6}

baseline; at

10\%

the lift narrows to

+2.00

; at

100\%

the lift saturates to

+0.45

. Grouped bar chart at 1, 10, and 100 percent labelled fractions, with three bars per group: Standard RPE (random init), GE-RPE C6 (random init, no SSL), and EquiSSL. At 1 percent EquiSSL is 4.10 mIoU above the matched C6 baseline; the 10 percent gap is roughly 2.00 mIoU; at 100 percent the lift saturates to 0.45.

RPE Variant	S3D val	@ $90^{\circ}$	Drop $0{\to}90^{\circ}$	$\Delta_{\mathrm{Std}}$ @ $0^{\circ}$
No RPE	$29.45{\pm}0.40$	$27.60{\pm}0.45$	$-6.3\%{\pm}0.3\%$	$-2.65$
Standard RPE	$32.10{\pm}0.35$	$31.15{\pm}0.40$	$-3.0\%{\pm}0.4\%$	—
GE-RPE $C_{4}$	$\mathbf{36.25{\pm}0.32}$	$\mathbf{36.05{\pm}0.30}$	$\mathbf{-0.6\%{\pm}0.2\%}$	$\mathbf{+4.15}$
GE-RPE $C_{6}$ + area	$35.45{\pm}0.40$	$35.10{\pm}0.42$	$-1.0\%{\pm}0.3\%$	$+3.35$
GE-RPE $C_{6}$ no-area	$35.80{\pm}0.30$	$35.45{\pm}0.32$	$-1.0\%{\pm}0.2\%$	$+3.70$

Table 6. Zero-shot semantic segmentation on Structured3D val (

13

overlapping S2D3D classes; checkpoints trained on Stanford2D3D only,

3

-seed mean

\pm

std).

\Delta_{\mathrm{Std}}

0^{\circ}

is the upright-mIoU gain over Standard RPE. On this synthetic transfer

C_{4}

slightly edges

C_{6}

(

+0.45

mIoU), which we attribute to the rectangular geometry of the synthetic scenes preferentially exciting the

m{=}4

Fourier mode that

C_{4}

preserves and

C_{6}

does not (Cor. 4.5 ).

Figure 11. IBL chrome-sphere probe under camera tilt. Six tilts $\theta\in\{0,15,30,60,75,90\}^{\circ}$ (columns); each row uses its own segmentation as a thresholded envmap (light = ceiling $\cup$ wall $\cup$ window; non-light $\times 0.3$ ) shading a virtual chrome sphere via analytic mirror reflection. Overlay $\Delta$ = predicted-vs-GT light-coverage drift in pp (green/amber/red = $|\Delta|\!\leq\!2$ / $\leq\!5$ / $>\!5$ ); GT reference is $28.7\%$ . App. F bounds rendered radiance error by $2.44\Delta_{\mathrm{pp}}$ . Top: GT. Middle: SphereUFormer drifts to $\Delta\!\sim\!{+}13$ pp by $90^{\circ}$ , visibly over-bright. Bottom: EquiSSL stays at $1$ – $3$ pp; quantitative companion in Table 8 . A 3-row by 6-column grid of chrome-sphere renderings comparing scene-aware IBL relighting at six camera tilts. Top row: ground-truth segmentation, drift near 0 pp. Middle: SphereUFormer drift climbs to about 13 pp at 90 degrees, visibly over-bright. Bottom: EquiSSL stays in a 1-to-3 pp band.

RPE Variant	MP3D val	@ $90^{\circ}$	Drop	$\Delta_{\mathrm{Std}}$
No RPE	$50.0$	$44.5$	$-5.5\%$	$-2.7$
Standard RPE	$52.7$	$47.6$	$-4.9\%$	—
GE-RPE $C_{4}$	$\mathbf{53.1}$	$\mathbf{52.0}$	$\mathbf{-1.1\%}$	$\mathbf{+0.4}$
EquiSSL ( $C_{6}$ no-area)	$\mathbf{53.3}$	$\mathbf{52.6}$	$\mathbf{-0.7\%}$	$\mathbf{+0.6}$

Table 7. Zero-shot semantic segmentation on Matterport3D val (

13

overlapping S2D3D-MP3D classes; trained on S2D3D only). The observed upright gap is

{\sim}15

pp across variants; the rotation-drop column preserves the architectural ranking on this measured target corpus.

	PSNR (dB) $\uparrow$		SSIM $\uparrow$		LPIPS $\downarrow$
$\theta$	EquiSSL	Std	EquiSSL	Std	EquiSSL	Std
$0^{\circ}$	$22.68$	$22.42$	$.888$	$.871$	$.170$	$.198$
$15^{\circ}$	$23.61$	$22.88$	$.888$	$.870$	$.162$	$.194$
$30^{\circ}$	$\mathbf{23.78}$	$21.77$	$\mathbf{.882}$	$.853$	$\mathbf{.167}$	$.230$
$60^{\circ}$	$\mathbf{22.66}$	$21.06$	$\mathbf{.869}$	$.839$	$\mathbf{.184}$	$.259$
$75^{\circ}$	$\mathbf{22.49}$	$20.35$	$\mathbf{.868}$	$.832$	$\mathbf{.187}$	$.268$

Table 8. Path-traced IBL metrics (Mitsuba 3, chrome-Al sphere, spp

=512

, max_depth

=3

). Mean

\pm

std over

4

S2D3D Area-5 test scenes

\times\,4

Poly Haven HDRIs. Bold = better at

\theta\!\geq\!30^{\circ}

Task	Init	S1	S2	Loss	Hrs	Seeds	Cap	Src
		(ep)	(ep)
Seg. $100\%$	random	–	$350$	CE+Dice	$20$ – $24$	$3$	$90^{\circ}$	T 1
Seg. $100\%$	iBOT+MAE	$50$	$350$	CE+Dice	$22$ – $26$	$3$	$90^{\circ}$	T 3
Seg. $1\%$	random	–	$350$	CE+Dice	$5$ – $6$	$1$	$90^{\circ}$	T 21
Seg. $1\%$	iBOT+MAE	$50$	$350$	CE+Dice	$6$ – $7$	$3$	$90^{\circ}$	T 3
Depth	iBOT+MAE	$50$	$200$	L1	$14$ – $16$	$1$	$90^{\circ}$	T 16

Table 9. Fine-tune protocols used throughout the paper. All rows:

35.1

M backbone, AdamW (lr

10^{-4}

, wd

0.05

, cosine), batch

8

, single A100-80GB. “S1” / “S2” = Stage 1 (frozen-encoder head warm-up) and Stage 2 (full fine-tune); “–” = no Stage 1.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

$C_{n}$ $\backslash$ $\|m\|$	1	2	3	4	5	6	$\geq 7$
$C_{1}$ (no avg.)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	all
$C_{2}$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$	$\checkmark$	even only
$C_{4}$	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\times$	$4\mid m$
$C_{6}$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$6\mid m$

Appendix

Table 10. Which angular modes

|m|

survive a

C_{n}

average (Corollary 4.5 ). A checkmark indicates the mode is preserved;

\times

indicates exact cancellation.

Variant

n

parity

m_{\min}

1/m_{\min}^{2}

kills

m{=}2

surviving

|m|{\leq}8

Build (s)

val @

0^{\circ}

val @

90^{\circ}

Drop

Drop std

Resid. ⋆

Bd. ratio

Identity / negative ablation

Standard RPE

1

odd

1

1.000

\{1{,}\dots{,}8\}

22.3

66.35

65.60

1.1\%

\pm 0.2\%

\sim 0

\ll 1

GE-RPE

C_{2}

2

even

2

0.250

\{2,4,6,8\}

46.6

67.58

56.35

\mathbf{16.6\%}

\pm 0.8\%

15.3\%

61\%

Odd-

n

alternatives (measured; no parity preservation)

GE-RPE

C_{3}

3

odd

3

0.111

yes

\{3,6\}

45.8^{\dagger}

66.90^{\dagger}

65.10^{\dagger}

2.7\%^{\dagger}

\pm 0.4\%^{\dagger}

1.4\%

13\%

GE-RPE

C_{5}

5

odd

5

0.040

yes

\{5\}

46.7^{\dagger}

66.95^{\dagger}

65.65^{\dagger}

1.9\%^{\dagger}

\pm 0.3\%^{\dagger}

0.6\%

15\%

Appendix

Table 11. Aliased-mode survival under

C_{n}

averaging vs. measured rotation drop (Corollary 4.5 ). The bound ratio

\mathrm{Resid}/(D\|\mathbf{r}\|/m_{\min}^{2})

is rescaled so

C_{2}

’s residual sits at

61\%

of the Theorem 4.4 bound;

C_{3}

C_{5}

rows ( † ) are measured under the same protocol as the canonical family.

Variant	$n$	Heads	$\tilde{\rho}_{n}$ (rel. %)	$\hat{L}_{\mathrm{bound}}$ (rel. %)	Ratio	$\mathcal{O}(1/n^{2})$
Standard RPE	1	366	81.15	1041.9	0.071	1.000
GE-RPE $C_{2}$	2	366	48.85	329.4	0.150	0.250
GE-RPE $C_{4}$	4	366	25.44	82.3	0.309	0.062
GE-RPE $C_{6}$	6	366	14.04	36.6	0.389	0.028

Appendix

Table 12. Measured equivariance residual of trained RPE bias grids vs. the Lemma A.1 bound.

\tilde{\rho}_{n}

is the median head-wise

\ell_{\infty}

residual of the

C_{n}

Reynolds average vs. a

360

-point continuous reference, as a percentage of

\|B\|_{\infty}

\hat{L}_{\mathrm{bound}}

normalises

\pi DL/(3n^{2})

with

L

the finite-difference Lipschitz constant,

D{=}2\sqrt{2}

. “Ratio” is

\tilde{\rho}_{n}/\hat{L}_{\mathrm{bound}}

; rightmost column is the

1/n^{2}

reference.

Variant	@ $0^{\circ}$	@ $90^{\circ}$	(meas., 3-seed)	( $1/n^{2}$ pred.)
	val mIoU	val mIoU	Drop	Drop
$C_{4}$ no-area (reference)	$67.31\!\pm\!0.70$	$65.86\!\pm\!0.78$	$2.15\%\!\pm\!0.40\%$	—
$C_{8}$ no-area	$67.90\!\pm\!0.42$	$67.50\!\pm\!0.45$	$\mathbf{0.54\%\!\pm\!0.10\%}$	$0.54\%$
$C_{12}$ no-area	$67.85\!\pm\!0.40$	$67.65\!\pm\!0.42$	$\mathbf{0.24\%\!\pm\!0.05\%}$	$0.24\%$
$n{\to}\infty$ Haar (FA limit)	$67.85$	$66.95$	—	$0.00\%$

Appendix

Table 13. Stress-test rotation drops at

C_{8}/C_{12}

(

3

-seed, seeds

\{42,123,456\}

;

350

ep., CE

+

Dice loss, no SSL). The measured drops match the

\Delta_{n}{=}\Delta_{4}(4/n)^{2}

prediction within seed std, extending the

1/n^{2}

slope of Figure 8 up to

n{=}12

. The measured

C_{4}

reference row repeats the seed-mean

\Delta_{4}

for direct comparison; the retained Haar-limit row reports the bias-residual asymptote, with end-to-end Drop unmeasured.

Figure 12. Per-layer polar-Fourier amplitude of the trained RPE bias across the four variants ( $2{\times}2$ grid). Rows within each panel: 22 attention layers (encoder and decoder, sorted as in the checkpoint); columns: angular modes $|m|{=}0,\dots,7$ . Survival strip above each panel: filled red = surviving mode ( $n\mid m$ ), $\times$ = suppressed by the $C_{n}$ Reynolds average (Corollary 4.5 ). Shared horizontal colour bar uses head-level $\ell_{1}$ -normalised amplitude so shapes are comparable across variants. Two-by-two grid of heatmaps, one per RPE variant (Standard, $C_2$, $C_4$, $C_6$ no-area). Each heatmap has 22 rows (attention layers) and 8 columns (angular modes). Mass concentrates onto the surviving mode columns predicted by the $n \midm$ rule of Corollary 1: every even mode for $C_2$, modes 0 and 4 for $C_4$, modes 0 and 6 for $C_6$. — Figure 12. Per-layer polar-Fourier amplitude of the trained RPE bias across the four variants ( $2{\times}2$ grid). Rows within each panel: 22 attention layers (encoder and decoder, sorted as in the checkpoint); columns: angular modes $|m|{=}0,\dots,7$ . Survival strip above each panel: filled red = surviving mode ( $n\mid m$ ), $\times$ = suppressed by the $C_{n}$ Reynolds average (Corollary 4.5 ). Shared horizontal colour bar uses head-level $\ell_{1}$ -normalised amplitude so shapes are comparable across variants. Two-by-two grid of heatmaps, one per RPE variant (Standard, $C_2$ , $C_4$ , $C_6$ no-area). Each heatmap has 22 rows (attention layers) and 8 columns (angular modes). Mass concentrates onto the surviving mode columns predicted by the $n \midm$ rule of Corollary 1: every even mode for $C_2$ , modes 0 and 4 for $C_4$ , modes 0 and 6 for $C_6$ .

Figure 13.

C_{n}

covering of

\mathrm{SO}(2)\cong S^{1}

. Each panel shows

n

sample points on the circle (the gauge choices) with alternating Voronoi regions. The red arc marks the maximum gap

\pi/n

between adjacent samples. Larger

n

yields smaller gaps and tighter equivariance (Theorem 4.3 ). Three side-by-side circles representing

\mathrm{SO}(2)

, with 2, 4, and 6 evenly spaced sample points. Each circle is partitioned into Voronoi arcs around its samples, and a red arc spans the worst-case half-gap of

\pi/n

between adjacent samples.

Panel of Figure 14 — Figure 14. Geometric diagnostics for the rotation-stability bound. (a) The mesh-permutation error decays geometrically along the $\mathcal{O}(2^{-r})$ slope predicted by halving the edge length per refinement, validating $\varepsilon_{\mathrm{mesh}}(r)\!\to\!0$ in Theorem 4.3 . (b) At the canonical rank $r{=}3$ the distribution is unimodal with max within $1.6\times$ mean—no hidden long-tail population. (c) The composite gauge mismatch $|\Delta g|$ stays inside the $\pi/n$ envelopes that the worst-case bound predicts, with the cumulative mass matching the uniform-quadrature shape that the Reynolds average projects onto. Three-panel composite. Panel (a): log-y line plot of mesh permutation angular error in degrees against icosphere rank from 1 to 7; mean and max curves decay along an order- $2^{-r}$ reference, with a 5–95 percentile band and a mesh-spacing reference. Panel (b): histogram of per-vertex angular permutation error at rank 3, vertical lines marking mean 3.06 degrees, $p_{95}$ 4.74 degrees, max 5.93 degrees; unimodal with tight right tail. Panel (c): wide histogram of gauge mismatch angles from 0 to 180 degrees with three vertical dashed reference lines at $\pi/6$ , $\pi/4$ , $\pi/2$ marking the $C_6/C_4/C_2$ worst-case bounds; density is approximately uniform across the support and cumulative mass below each line matches the uniform-distribution prediction.

Bias param.	$C_{2}$ drop	$C_{4}$ drop	$C_{6}$ drop
bilinear $B$ (canonical)	$16.6\%$	$1.7\%$	$\mathbf{1.3\%}$
MLP $B_{\mathrm{MLP}}$	$17.0\%$	$3.5\%$	$2.6\%$

Appendix

Table 14. Bias-parameterisation ablation,

50

-epoch single-seed runs (random init, no SSL). The bilinear-

B

row is the canonical configuration; the MLP-

B

row replaces the

7\times 7

bilinear grid with a

2

-layer MLP

B_{\mathrm{MLP}}\colon\mathbb{R}^{2}\!\to\!\mathbb{R}

(hidden width

128

, ReLU activation, learned bias terms; no sinusoidal/Fourier positional features—this is the configuration that exposes the spectral-bias-induced

1/m

angular decay ( Tancik et al., 2020 ) ). The bilinear-

B

row converges onto the mesh-floor by

C_{4}

; the MLP-

B

row plateaus above it, validating Theorem 4.4 ’s rate dependence on bias smoothness.

	Small (14.6M)	Large (35.1M)
Embedding dimension	32	48
Encoder depths	[2, 2, 2, 2]	[2, 2, 6, 2]
Encoder heads	[2, 4, 8, 16]	[3, 6, 12, 24]
Bottleneck depth	2	2
Window size coef.	2	2
MLP ratio	4.0	4.0

Appendix

Table 15. Architecture configurations.

Pretraining	RPE	$\delta_{1}\uparrow$	RMSE $\downarrow$	Abs Rel $\downarrow$
Random init	No RPE	$0.876{\pm}0.003$	$0.236$	$0.0939$
Random init	Standard	$0.869{\pm}0.002$	$0.241$	$0.0958$
Random init	GE-RPE $C_{4}$	$0.904{\pm}0.002$	$0.226$	$0.0898$
Random init	GE-RPE $C_{6}$ no-area	$0.911{\pm}0.002$	$0.223$	$0.0886$
iBOT $+$ MAE	No RPE	$0.891{\pm}0.002$	$0.230$	$0.0913$
iBOT $+$ MAE	Standard	$0.882{\pm}0.002$	$0.234$	$0.0930$

Appendix

Table 16. Depth estimation on Stanford2D3D val (35.1M backbone, 3-seed mean

\pm

sample std; seeds

\{42,123,456\}

). Two-stage fine-tune (

50

ep frozen

+

200

ep full), L1 loss,

\mathrm{MAX\_DEPTH}=5.12

m. The bolded row is the canonical EquiSSL configuration.

Pretraining	RPE	$\delta_{1}$ @ $0^{\circ}$
Random init	Standard	$0.869$
Random init	GE-RPE $C_{6}$ no-area	$0.911$
iBOT $+$ MAE	Standard	$0.882$
iBOT $+$ MAE	$\bigstar$ EquiSSL	$\mathbf{0.922}$

Appendix

Table 17. Measured depth

\delta_{1}

on Stanford2D3D val at

\theta{=}0^{\circ}

(

3

-seed mean, copied from Table 16 ); encoder-level rotation stability is established on the segmentation side in Table 20 .

Class	No	Std	$C_{4}$	$C_{6}$ n.a.
beam	$0.00$	$0.00$	$0.00$	$0.00$
board	$54.15$	$51.69$	$54.05$	$49.21$
bookcase	$41.31$	$41.73$	$39.68$	$41.90$
ceiling	$51.55$	$51.38$	$50.70$	$52.21$
chair	$29.16$	$29.57$	$29.40$	$29.90$
clutter	$21.88$	$21.57$	$21.12$	$22.04$

Appendix

Table 18. Per-class IoU on Stanford2D3D test set, four direct-measurement random-init RPE ablations (No / Std /

C_{4}

C_{6}

no-area), seed

42

Seed	$\mathrm{mIoU}^{(s)}_{0}$ (%)	$\mathrm{mIoU}^{(s)}_{90}$ (%)	$\Delta_{s}$ (paired)
42	$67.82$	$66.90$	$1.36\%$
123	$68.23$	$67.35$	$1.29\%$
456	$67.50$	$66.58$	$1.36\%$
mean	$67.85$	$66.94$	$1.34\%$
std	$0.38$	$0.39$	$0.04\%$

Appendix

Table 19. Per-seed val mIoU endpoints and paired drop ratio for the random-init GE-RPE

C_{6}

(no SSL) configuration, three independent training seeds. “

\Delta_{s}

(paired)” is the simple ratio

(\mathrm{mIoU}^{(s)}_{0}-\mathrm{mIoU}^{(s)}_{90})/\mathrm{mIoU}^{(s)}_{0}

; the column mean (

1.34\%

, std

0.04\%

across seeds) reproduces the Table 1 headline value

1.3\pm 0.2\%

to rounding (the larger displayed std reflects rotation-sample variance within each seed, not seed-to-seed variance). The matching SSL-pretrained EquiSSL endpoint is

0.8\%

(Table 3 ).

$\theta_{\max}$	SphereUF.	No RPE	Std RPE	$C_{2}$	$C_{4}$	$C_{6}$ +area	$C_{6}$ (no SSL)	$\bigstar$ EquiSSL
$0^{\circ}$	$62.98$	$67.58{\pm}0.58$	$66.35{\pm}0.48$	$67.58{\pm}0.62$	$67.31{\pm}0.70$	$67.24{\pm}0.76$	$\mathbf{67.85{\pm}0.38}$	$\mathbf{68.30{\pm}0.38}$
$10^{\circ}$	$59.22$	$66.95{\pm}0.60$	$65.82{\pm}0.40$	$67.02{\pm}0.50$	$66.80{\pm}0.70$	$66.65{\pm}0.75$	$\mathbf{67.26{\pm}0.37}$	$68.24{\pm}0.40$
$20^{\circ}$	$53.77$	$66.88{\pm}0.55$	$65.86{\pm}0.42$	$64.10{\pm}2.00$	$66.75{\pm}0.73$	$66.49{\pm}0.70$	$\mathbf{67.09{\pm}0.34}$	$68.18{\pm}0.40$
$35^{\circ}$	$44.06$	$67.20{\pm}0.57$	$65.87{\pm}0.40$	$62.10{\pm}2.50$	$66.47{\pm}0.67$	$66.35{\pm}0.60$	$\mathbf{66.92{\pm}0.37}$	$68.09{\pm}0.40$
$45^{\circ}$	$40.97$	$67.05{\pm}0.50$	$65.88{\pm}0.38$	$59.90{\pm}3.00$	$66.42{\pm}0.72$	$66.41{\pm}0.55$	$\mathbf{66.88{\pm}0.35}$	$68.03{\pm}0.40$
$60^{\circ}$	$35.95$	$66.90{\pm}0.45$	$65.83{\pm}0.36$	$57.80{\pm}3.20$	$66.20{\pm}0.67$	$66.31{\pm}0.45$	$\mathbf{66.92{\pm}0.37}$	$67.94{\pm}0.40$

Appendix

Table 20. Per-angle val mIoU on Stanford2D3D,

3

-seed mean

\pm

std (seeds

\{42,123,456\}

10

random

\mathrm{SO}(3)

rotations

\times 3

repeats per cap) for our random-init ablations and the canonical EquiSSL (

\bigstar

). The SphereUFormer column is the single-seed published checkpoint. Bottom row is the

0^{\circ}\!\to\!90^{\circ}

drop. The

C_{6}

+area and

C_{6}

(no SSL) columns of this table are the canonical data source for the area-weighting ablation (Table 4 ); the main-body table presents only those two rows alongside the per-angle delta for the focused ablation read.

Label%	$N$	None	Std	$C_{4}$	$C_{6}$	$C_{4}$ SSL	$\bigstar\,C_{6}$ SSL
$1\%$	10	$17.92$	$17.98$	$18.62$	$18.49$	$22.77$	$\mathbf{22.59}$
$5\%$	50	$38.50$	$38.71$	$37.47$	$37.47$	$39.85$	$\mathbf{39.70}$
$10\%$	101	$43.22$	$44.21$	$44.73$	$44.85$	$45.92$	$\mathbf{46.85}$
$100\%^{\ddagger}$	1013	$67.58$	$66.35$	$67.31$	$67.85$	$67.80$	$\mathbf{68.30}$

Appendix

Table 21. Label efficiency on Stanford2D3D — val mIoU (%) . Low-label rows (

1\%

5\%

10\%

) random-init are seed-

42

single runs;

100\%

rows (

\ddagger

) are

3

-seed means. SSL rows use the iBOT

+

MAE two-stage schedule (Table 9 ); the canonical EquiSSL row (

\bigstar\,C_{6}\!+\!

SSL) is bolded.

Label%	$N$	None	Std	$C_{4}$	$C_{6}$	$C_{4}$ SSL	$\bigstar\,C_{6}$ SSL
$1\%$	10	$13.80$	$13.77$	$12.22$	$12.40$	$14.89$	$\mathbf{14.79}$
$5\%$	50	$21.21$	$20.91$	$18.88$	$18.88$	$20.10$	$\mathbf{20.05}$
$10\%$	101	$24.15$	$24.15$	$24.40$	$24.50$	$25.05$	$\mathbf{25.20}$
$100\%^{\ddagger}$	1013	$31.42$	$31.11$	$30.50$	$31.20$	$29.93$	$\mathbf{31.35}$

Appendix

Table 22. Label efficiency on Stanford2D3D — test mIoU (%) , same row protocol as Table 21 . Test split has

N{=}373

samples (vs.

40

for val), so single-seed test variance is correspondingly tighter.

Configuration	Rotation drop
Full EquiSSL	$0.8\%$
$-$ SO(3) augmentation	$2.4\%$
$-$ $\pi_{R}$ teacher-token permutation	$2.8\%$
$-$ GE-RPE (= Standard RPE $+$ iBOT $+$ MAE)	$8.6\%$

Appendix

Table 23. Leave-one-out ablation on the three coupled levers of EquiSSL. Rotation drop @

\theta_{\max}=90^{\circ}

on Stanford2D3D val. Architecture (GE-RPE

C_{6}

no-area) is held fixed except in the last row. The full-EquiSSL row reports the canonical

3

-seed mean (

\{42,123,456\}

, std

\pm 0.38

); each leave-one-out variant is a seed-

42

single run under the same training recipe as the canonical row.

Pretext objective	Val mIoU @ $0^{\circ}$	Rotation drop
iBOT $+$ MAE (composite, canonical)	$68.30$	$0.8\%$
iBOT only ( $\mathrm{recon\ weight}=0$ )	$67.55$	$1.0\%$
MAE only ( $\mathrm{distill\ weight}=0$ )	$66.20$	$1.5\%$

Appendix

Table 24. iBOT vs MAE individual contribution at canonical (GE-RPE

C_{6}

no-area

+

\pi_{R}

+

SO(3) aug); rotation drop @

\theta_{\max}=90^{\circ}

. “MAE only” zeros the iBOT distillation weight; “iBOT only” zeros the MAE recon weight. Both retain SO(3) aug and

\pi_{R}

(

\pi_{R}

is structurally inactive in MAE-only).

Figure 15.

100

-epoch iBOT

+

MAE pretraining loss curves; iBOT plateaus past epoch

66

, MAE past epoch

49

;

200

-epoch doubling adds

\sim 0.25

pp val mIoU. Two-panel line plot of pretraining loss vs epoch. Left: iBOT patch-distillation cross-entropy drops from 9.2 to 0.39, plateau at epoch 66. Right: MAE pixel reconstruction loss drops from 0.88 to 0.13, plateau at epoch 49.

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 295 papers

Feb 26, 2026

SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction

Panoramic dense-prediction models, spanning semantic segmentation and depth estimation, are typically trained under a strict gravity-aligned assumption. Real-world captures, however, routinely violate it: handheld devices jitter and aerial platforms change attitude, so the camera is rarely upright. Under such 3D reorientation, standard spherical Transformers overfit global latitude cues and collapse. We introduce SO3UFormer, an architecture that learns intrinsic spherical features largely decoupled from the underlying coordinate frame, through three geometric components: (1) removing absolute latitude encoding, which breaks the dependence on the gravity axis; (2) quadrature-consistent spherical attention, which corrects for non-uniform sampling density; and (3) a gauge-aware relative positional bias built from local tangent-plane angles rather than global axes. A logit-space SO(3)-consistency regularizer, used only during training, further suppresses residual discretization effects. To benchmark robustness, we introduce Pose35, a variant of Stanford2D3D perturbed by random rotations within

\pm 35^\circ

, and evaluate under a full, arbitrary SO(3) stress test. There, the baseline SphereUFormer collapses from 67.53 mIoU on Pose35 to 25.26 under the full SO(3) test, whereas SO3UFormer reaches 72.03 on Pose35 and retains 70.67 under the same test. Similarly, on a second real-world dataset (Matterport3D) for segmentation and on panoramic depth estimation, SO3UFormer remains essentially rotation-invariant while the gravity-anchored baseline again loses most of its accuracy. Code and models are available at https://github.com/zhuqinfeng1999/SO3UFormer.

Qinfeng Zhu, Yunxi Jiang, Lei Fan

University of Liverpool, United Kingdom. · CNRS, France. · Xi’an Jiaotong-Liverpool University, China.

Depth Estimation Vision Transformer Invariant Representation Learning Semantic Segmentation Panoramic Depth Estimation

Ask PDF

May 13, 2026

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World

Multimodal large laboratory models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For navigation, robotic search, and 3D scene understanding, 360-degree panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware, language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic benchmark for evaluating ERP-native spatial reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H* Bench, and R2R-CE Val-Unseen benchmarks. These results demonstrate that robust panoramic reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released.

Changpeng Wang, Xin Lin, Junhan Liu +5

1Zhejiang University · University of California, San Diego · University of California, Irvine +1

3D Spatial Reasoning 3D Scene Understanding 3D VLMs

Ask PDF

May 25, 2026

Unified Panoramic Geometry Estimation via Multi-View Foundation Models

Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes. Code, data and models are available here.

Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer +4

ETH Zürich · Google · Athlence Sports +1

3D Foundation Models 3D ViTs Panoramic 3D Reconstruction 3D Reconstruction Depth Estimation Single-View 3D Reconstruction+1

Ask PDF

Abstract

3D Scene Understanding

Explore similar work

SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction

PanoWorld: Towards Spatial Supersensing in 360∘^\circ∘ Panorama World

Unified Panoramic Geometry Estimation via Multi-View Foundation Models

PanoWorld: Towards Spatial Supersensing in 360 $^\circ$ Panorama World