cs.CVSep 23, 2026

Gauge-Equivariant Attention for Rotation-Stable 360∘360^\circ Scene Understanding

Authors: Tianjian Zhou, Yishan Li, Jie Jiang, Yifei Zhang

Organizations: National University of Defense Technology (NUDT), Changsha, China · Northwestern Polytechnical University, Xi’an, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Panoramic 360∘360^\circ scene understanding increasingly relies on icosphere transformers, but a state-of-the-art spherical model loses more than half of its segmentation accuracy when the camera rotates by 90∘90^\circ, and controlled ablations identify gauge dependence in its relative-position bias as a major contributor. We propose gauge-equivariant relative position encoding (GE-RPE): a parameter-free Reynolds average of the bias over a finite cyclic subgroup Cn ⁣⊂ ⁣SO(2)C_n\!\subset\!\mathrm{SO}(2) of gauge rotations. Plugged into a SphereUFormer backbone the change is invisible at deployment---zero added parameters and 1.51.5--4.4%4.4\% forward latency---and the matched three-seed GE-RPE model records a 1.3%1.3\% drop; the published SphereUFormer checkpoint records 53%53\% under the same stress protocol but a different training recipe. Once the gauge defect is removed and a teacher-token permutation πRπ_R aligns the SSL views to the rotated student frame, iBOT++MAE pretraining stops being a liability and becomes a clean low-label lever: the full framework EquiSSL (GE-RPE ++ πRπ_R ++ iBOT++MAE) tightens the drop to 0.8%0.8\% at 68.30%68.30\% val mIoU and lifts 1%1\%-label fine-tuning by +2.39+2.39 mIoU on the N=373N{=}373 test split (and by +4.10+4.10 on the smaller N=40N{=}40 val split); the same fix carries over to monocular depth and to zero-shot Structured3D segmentation. The construction is provably CnC_n-invariant and O(n−2)\mathcal{O}(n^{-2})-close to the continuous SO(2)\mathrm{SO}(2) average, making the resulting model a usable 360∘360^\circ visual-computing primitive across panoramic relighting, immersive video, and cross-dataset transfer. Code is available at https://github.com/Jaywalk18/equissl-release.

Figures & tables

Figure 1. Rotation breaks a standard spherical model; EquiSSL fixes it at two levels. Top: a standard SphereUFormer gives clean upright regions but collapses after a 90∘90^{\circ} rotation (more than 50%50\% mIoU drop). Bottom: EquiSSL couples GE-RPE, which averages the attention bias over a discrete gauge group so attention co-rotates with the input, with a teacher-token permutation πR\pi_{R} that aligns the SSL target to the student’s rotated frame. The result is stable segmentation before and after rotation. Side-by-side panoramic segmentation results showing that a standard spherical transformer's predictions disintegrate under a 90 degree camera rotation while the proposed EquiSSL produces consistent segmentation before and after rotation; a schematic at the bottom indicates the two levels of the fix—gauge-equivariant attention and rotation-consistent self-supervised pretraining.
Figure 2. Why the RPE lookup depends on the local frame. The spherical pair (q,k)(q,k) is fixed, but Frames A and B assign coordinates r\mathbf{r} and g⊤rg^{\top}\mathbf{r} ; the same anisotropic table BB can therefore return different values. GE-RPE averages the CnC_{n} -rotated lookups into bGE(q,k)b_{\mathrm{GE}}(q,k) , which is invariant to frame choices in CnC_{n} . A fixed query-key pair on an icosphere branches into two valid tangent frames. The rotated axes assign two coordinates to the same displacement and sample different locations in one shared bias table. A cyclic average over rotated frames merges these lookups into a frame-independent scalar bias.
Figure 3. EquiSSL: full framework. (a) End-to-end pipeline. A 360∘360^{\circ} panorama is projected onto an icosphere, encoded by SphereUFormer, and fine-tuned for dense segmentation; the dashed sub-box marks rotation-consistent iBOT ++ MAE pretraining, where πR\pi_{R} re-permutes teacher tokens into the student’s rotated frame. (b) Zoom-in of one attention layer of (a). GE-RPE replaces the standard relative-position bias by its CnC_{n} Reynolds average BGE(r)=1n∑g∈CnB(g⊤r)B_{\mathrm{GE}}(\mathbf{r})=\tfrac{1}{n}\sum_{g\in C_{n}}B(g^{\top}\mathbf{r}) —the bias-level term isolated by the controlled comparison—at zero parameter cost. Two-panel pipeline diagram. The top panel shows the end-to-end EquiSSL framework: a 360∘360^\circ panorama is resampled onto a rank-7 icosphere, fed through a four-scale SphereUFormer encoder-decoder, and fine-tuned for semantic segmentation; a dashed sub-block marks the rotation-consistent iBOT++MAE pretraining stage in which a teacher-token permutation πR\pi_R aligns the cross-view target. The bottom panel zooms into one attention layer and depicts GE-RPE replacing the gauge-dependent relative-position bias with its CnC_n-Reynolds average over nn rotated lookups of the same shared 7×77\times7 bias grid.
Figure 4. Quantitative validation of Theorems 4.3 and 4.4 on a real trained network. Panel (a) reads off Corollary 4.5 ’s mode-survival law on the bilinear grid; panels (b) and (c) verify the two complementary rate constants—the O(1/n2)\mathcal{O}(1/n^{2}) Fourier-tail rate of Theorem 4.4 and the π/n\pi/n worst-case Lipschitz radius of Theorem 4.3 —both of which are tight in measurement, with the C2C_{2} point flagged as the even-mode parity outlier. Three-panel figure validating the rotation-stability theorems. Panel (a): bar chart of polar Fourier mode amplitudes for the learnable bias, with the m=2m=2 mode visibly dominant; survival strips show which modes C2C_2, C4C_4, C6C_6 preserve. Panel (b): log-log plot of the simulated CnC_n Reynolds-average bias residual versus nn, overlaid with the empirical Stanford2D3D rotation drop on a secondary axis; both trace an order-1/n21/n^2 slope from n=2n=2 to n=12n=12, with the C2C_2 point sitting notably above the bound due to even-mode aliasing. Panel (c): histogram of per-vertex residual gauge mismatch angle after snapping to the nearest CnC_n rotation for n∈{2,4,6}n \in\{2,4,6\}; each empirical density stays inside its π/n\pi/n cutoff and matches a uniform reference.
MethodVenueEquiv.ParamsFwdval mIoUtestrot. mIoUDrop ⋄Drop std1%1\% lab.Depth
   (M)(ms)(%)(%)@ 90∘90^{\circ} (%)val (%)δ1\delta_{1}
Prior work (single best; same split, val-mIoU context).
SFSS ∗ICCV’19none30.430.411211242.0242.0219.5419.5416.8116.8160.0%60.0\%±1.5\pm 1.511.2011.200.7620.762
HEAL-SWIN ( Carlsson et al., 2024 )CVPR’24sampling-only50.450.420820862.4562.4529.0529.0531.2231.2250.0%50.0\%±1.0\pm 1.016.8516.850.8520.852
SphereUFormer ( Benny et al., 2025 )CVPR’25none35.135.1163.2163.267.5367.5331.4031.4031.74♮31.74^{\natural}53.0%53.0\%±0.7\pm 0.717.9517.950.8720.872
SO3UFormer ‡ ( Zhu et al., 2026 )2026SO(3) feat.48.248.224524566.566.530.9530.9546.9546.9529.4%29.4\%±1.8\pm 1.818.6018.600.8900.890
Table 1. Semantic segmentation on Stanford2D3D (13 classes, Area- 55 split: 1013/40/3731013/40/373 train/val/test). Bold : random-init GE-RPE C6C_{6} (architectural baseline isolating the gauge fix from pretraining); the canonical EquiSSL system (GE-RPE C6C_{6} + iBOT ++ MAE) is in Table 3 . Prior-work rows are single best as published. Our rows are 33 -seed mean ±\pm std (seeds {42,123,456}\{42,123,456\} ); rotation drop is paired per-seed then averaged over 1010 rotations ×3\times 3 repeats.
Figure 5. Accuracy–robustness Pareto on Stanford2D3D val. yy -axis: rotation drop at θmax⁡=90∘\theta_{\max}{=}90^{\circ} (log-scale, lower is better); xx -axis: upright val mIoU (higher is better); shaded band is the desirable corner. EquiSSL ( ★\bigstar , 68.30%68.30\% / 0.8%0.8\% ) sits closest. Per-point values are tabulated in Tables 1 – 3 ; SFSS ( 42.042.0 val) is off-scale on the xx -axis. Single-panel log-y scatter plot: SFSS at the upper-left (42 mIoU, 60 percent rotation drop), SphereUFormer and HEAL-SWIN near the top (~62-63 mIoU, 50-53 percent drop), SO3UFormer mid-axis (66.5 mIoU, 29 percent drop), the C2 negative control isolated at 67.6 mIoU and 16.6 percent drop, and the EquiSSL family (No RPE, Standard RPE, C4, C6+area, C6 no-area, and the EquiSSL star) clustered between 66 and 68 mIoU at sub-2 percent drop inside a green-shaded desirable corner. The EquiSSL star is the closest point to that corner.
RPETrain aug.mIoU @ 0∘0^{\circ}mIoU @ 90∘90^{\circ}Drop
Standardnone66.35±0.4866.35{\pm}0.4865.60±0.3065.60{\pm}0.301.1±0.2%1.1{\pm}0.2\%
StandardHaar- SO(3)\mathrm{SO}(3) (32)66.50±0.6266.50{\pm}0.6266.13±0.4766.13{\pm}0.470.54±0.75%0.54{\pm}0.75\%
Radialnone65.83±0.3865.83{\pm}0.3864.63±0.3664.63{\pm}0.361.81±0.19%1.81{\pm}0.19\%
GE-RPE C6C_{6}none67.85±0.38\mathbf{67.85{\pm}0.38}66.94±0.39\mathbf{66.94{\pm}0.39}1.3±0.2%\mathbf{1.3{\pm}0.2\%}
Table 2. Matched alternatives on Stanford2D3D val (three training seeds; mean ±\pm sample std). All rows use the same rank-7 backbone, 350-epoch supervised recipe, and Pose90 evaluation ( 1010 rotations ×3\times 3 repeats).
Figure 6. Qualitative Stanford2D3D segmentation under a 90∘90^{\circ} yaw rotation, rendered on the sphere. Top row, columns (a)–(h): (a) ground truth, (b) SphereUFormer (published checkpoint), (c) standard RPE (ablation), (d) Tangent-Img, (e) PanoFormer, (f) HEAL-SWIN, (g) SO3UFormer, (h) EquiSSL (Ours). Bottom row repeats each column as a binary correct/wrong error map against the ground truth. SphereUFormer and HEAL-SWIN dissolve into horizontal streaks aligned with the rotated ERP gauge; SO3UFormer recovers large-scale layout but leaks across boundaries; EquiSSL preserves both object regions and class boundaries. Figure 7 shows the same sample across multiple yaw angles. \rev{Eight-column} qualitative comparison of segmentation predictions on a Stanford2D3D panorama after a 90 degree yaw rotation, displayed on the sphere. Columns: ground truth, SphereUFormer published checkpoint, standard RPE ablation, \rev{Tangent-Img, PanoFormer,} HEAL-SWIN, SO3UFormer, and EquiSSL. The top row shows class predictions with the room legend (ceiling, clutter, door, floor, wall, unlabelled); the bottom row shows correct/wrong error maps. SphereUFormer and HEAL-SWIN predictions break into horizontal streaks aligned with the rotated equirectangular gauge, SO3UFormer recovers coarse layout but leaks across boundaries, and EquiSSL preserves object regions and class boundaries.
Figure 7. Rotation-angle sweep on one Stanford2D3D val sample. Rows: input RGB, ground truth, SphereUFormer (published checkpoint), and GE-RPE C4C_{4} . Columns: yaw rotations at 0∘, 30∘, 60∘, 90∘0^{\circ},\ 30^{\circ},\ 60^{\circ},\ 90^{\circ} . SphereUFormer’s predictions degrade progressively with rotation; GE-RPE C4C_{4} stays visually consistent across the sweep, complementing the single-angle comparison of Figure 6 . Four-row by four-column qualitative grid for one Stanford2D3D panorama under yaw rotations of 0∘0^\circ, 30∘30^\circ, 60∘60^\circ, and 90∘90^\circ. The input RGB and ground-truth rows rotate consistently across columns. The SphereUFormer prediction row degrades progressively with rotation, showing increasing fragmentation and class swaps. The GE-RPE C4C_4 row stays close to the ground truth across all four columns.
  Seg. val mIoUtestLow-label val mIoUDepthDepthRot.S3D zero-shotS3D
PretrainingRPE@ 0∘0^{\circ}@ 90∘90^{\circ}mIoU1%1\%5%5\%10%10\%δ1\delta_{1}RMSEDrop@ 0∘0^{\circ}@ 90∘90^{\circ}Drop
Random initNo RPE67.58±0.5867.58{\pm}0.5866.7066.7031.4231.4217.92±0.4317.92{\pm}0.4338.5038.5043.2243.220.876±0.0030.876{\pm}0.0030.2360.2361.3%1.3\%26.1026.1024.4524.456.3%6.3\%
Random initStandard66.35±0.4866.35{\pm}0.4865.6065.6031.1131.1117.98±0.2917.98{\pm}0.2938.7138.7144.2144.210.869±0.0020.869{\pm}0.0020.2410.2411.1%1.1\%28.5528.5527.7027.703.0%3.0\%
Random initGE-RPE C4C_{4}67.31±0.7067.31{\pm}0.7066.1766.1730.5030.5018.62±0.3718.62{\pm}0.3737.4737.4744.7344.730.904±0.0020.904{\pm}0.0020.2260.2261.7%1.7\%32.9032.9032.7032.700.6%0.6\%
Random initGE-RPE C6C_{6}67.85±0.3867.85{\pm}0.3866.9466.9431.2031.2018.49±0.3618.49{\pm}0.3637.4737.4744.8544.850.911±0.0020.911{\pm}0.0020.2230.2231.3%1.3\%32.4532.4532.1032.101.0%1.0\%
iBOT ++ MAENo RPE67.58±0.5867.58{\pm}0.5866.69±0.6166.69{\pm}0.6131.2031.2021.32±0.4221.32{\pm}0.4240.8540.8546.1046.100.891±0.0020.891{\pm}0.0020.2300.2301.2%1.2\%29.4529.4527.6027.606.3%6.3\%
Table 3. Self-supervised pretraining on Structured3D and fine-tuning on Stanford2D3D (3-seed mean ±\pm std). The diagnostic signal is in the rotation-drop, 1%1\% -label, and Structured3D zero-shot columns: Standard RPE worsens under SSL ( 1.1% ⁣→ ⁣8.6%1.1\%\!\to\!8.6\% rotation drop), while SSL-pretrained EquiSSL tightens to 0.8%0.8\% . S3D zero-shot is logged for SSL-pretrained checkpoints (iBOT ++ MAE rows) only, 33 -seed mean ±\pm std (seeds {42,123,456}\{42,123,456\} ).
Figure 8. Rotation drop scales as O(1/n2)\mathcal{O}(1/n^{2}) in the gauge count nn ; C2C_{2} is the even-mode aliasing outlier; C4C_{4} already lands inside the < ⁣1%<\!1\% -drop band (green). Filled markers show empirical drops with ±1\pm 1 std error bars; log–log axes make the O(1/n2)\mathcal{O}(1/n^{2}) reference a straight line. Log-log plot of empirical rotation drop versus gauge count n. C2 sits high near 16 percent (an aliasing outlier), C4 and C6 lie within a green sub-1-percent band, and the rate matches the predicted O(1 over n squared) reference line.
Figure 9. Rotation robustness on Stanford2D3D val across θmax⁡∈{0∘,10∘,20∘,35∘,45∘,60∘,90∘}\theta_{\max}\in\{0^{\circ},10^{\circ},20^{\circ},35^{\circ},45^{\circ},60^{\circ},90^{\circ}\} . Dashed star is the measured canonical SSL-pretrained EquiSSL 3-seed curve; solid lines are our random-init 3-seed curves (GE-RPE C6C_{6} no-SSL, No RPE, Standard RPE); dash-dot lines are reference curves for SO3UFormer and HEAL-SWIN; dotted lines are SphereUFormer (our reproduction) and SFSS. Legend suffixes report the 0∘ ⁣→ ⁣90∘0^{\circ}\!\to\!90^{\circ} endpoint drop. Line plot of validation mean intersection-over-union versus maximum rotation angle (0 to 90 degrees) for eight configurations: the EquiSSL star line and GE-RPE C6 no-SSL hold a tight band near 68 mIoU with sub-2 percent drop; No RPE sits just below at about 67.6 with a similar 1.3 percent drop; Standard RPE stays near 66 with about 1.1 percent drop; SO3UFormer follows a concave-down curve from 66.5 down to about 47 mIoU; HEAL-SWIN, SphereUFormer, and SFSS collapse from 62 to 31, 63 to 30, and 42 to 17 mIoU respectively. Two soft horizontal bands shade a "rotation collapse" region below 50 mIoU and a "robust band" above 65 mIoU.
 val mIoU @ θmax⁡\theta_{\max} ( ∘ , %)testDrop
Config.00101020203535454560609090( %\% )0→90∘0{\to}90^{\circ}
C6C_{6} + area67.2467.2466.6566.6566.4966.4966.3566.3566.4166.4166.3166.3166.3266.3230.8930.891.4±0.7%1.4{\pm}0.7\%
C6C_{6} no-area67.85\mathbf{67.85}67.26\mathbf{67.26}67.09\mathbf{67.09}66.92\mathbf{66.92}66.88\mathbf{66.88}66.92\mathbf{66.92}66.94\mathbf{66.94}31.01\mathbf{31.01}1.3±0.2%\mathbf{1.3{\pm}0.2\%}
Δ\Delta (n.a. −- a)+0.61+0.61+0.61+0.61+0.60+0.60+0.57+0.57+0.47+0.47+0.61+0.61+0.62+0.62+0.12+0.12−0.1-0.1 pp
Table 4. Area weighting ablation: per-angle val mIoU on Stanford2D3D (GE-RPE C6C_{6} , 35.1M, 3-seed mean). All 14 per-angle cells are measured.
RPE variantBuildFwdΔ\Delta FwdPeak MemΔ\Delta Mem
 (s)(ms)(%)(MB)(%)
No RPE23.323.3163.6163.6—43694369—
Standard RPE22.322.3163.2163.20.0%0.0\%438543850.0%0.0\%
GE-RPE C2C_{2}46.646.6165.6165.6+1.5%+1.5\%44394439+1.2%+1.2\%
GE-RPE C4C_{4}47.547.5168.8168.8+3.4%+3.4\%45114511+2.9%+2.9\%
GE-RPE C6C_{6} (+area)44.144.1170.3170.3+4.4%+4.4\%45794579+4.4%+4.4\%
Table 5. Runtime and peak GPU memory at rank-7 ( 163,842163{,}842 nodes), A100-80GB, batch 44 . Δ\Delta Fwd / Δ\Delta Mem are relative to Standard RPE; all rows share Params =35.1{=}35.1 M (zero added). One GPU-hot run uses 1010 warm-up and 5050 timed forward passes, with torch.cuda.synchronize() between variants. All cells are measured.
Figure 10. Label-efficiency on Stanford2D3D val across three label fractions ( 1%1\% , 10%10\% , 100%100\% ): Standard RPE, random-init GE-RPE C6C_{6} , and EquiSSL . At 1%1\% labels EquiSSL adds +4.10+4.10 mIoU over the matched random-init C6C_{6} baseline; at 10%10\% the lift narrows to +2.00+2.00 ; at 100%100\% the lift saturates to +0.45+0.45 . Grouped bar chart at 1, 10, and 100 percent labelled fractions, with three bars per group: Standard RPE (random init), GE-RPE C6 (random init, no SSL), and EquiSSL. At 1 percent EquiSSL is 4.10 mIoU above the matched C6 baseline; the 10 percent gap is roughly 2.00 mIoU; at 100 percent the lift saturates to 0.45.
RPE VariantS3D val@ 90∘90^{\circ}Drop 0→90∘0{\to}90^{\circ}ΔStd\Delta_{\mathrm{Std}} @ 0∘0^{\circ}
No RPE29.45±0.4029.45{\pm}0.4027.60±0.4527.60{\pm}0.45−6.3%±0.3%-6.3\%{\pm}0.3\%−2.65-2.65
Standard RPE32.10±0.3532.10{\pm}0.3531.15±0.4031.15{\pm}0.40−3.0%±0.4%-3.0\%{\pm}0.4\%—
GE-RPE C4C_{4}36.25±0.32\mathbf{36.25{\pm}0.32}36.05±0.30\mathbf{36.05{\pm}0.30}−0.6%±0.2%\mathbf{-0.6\%{\pm}0.2\%}+4.15\mathbf{+4.15}
GE-RPE C6C_{6} + area35.45±0.4035.45{\pm}0.4035.10±0.4235.10{\pm}0.42−1.0%±0.3%-1.0\%{\pm}0.3\%+3.35+3.35
GE-RPE C6C_{6} no-area35.80±0.3035.80{\pm}0.3035.45±0.3235.45{\pm}0.32−1.0%±0.2%-1.0\%{\pm}0.2\%+3.70+3.70
Table 6. Zero-shot semantic segmentation on Structured3D val ( 1313 overlapping S2D3D classes; checkpoints trained on Stanford2D3D only, 33 -seed mean ±\pm std). ΔStd\Delta_{\mathrm{Std}} @ 0∘0^{\circ} is the upright-mIoU gain over Standard RPE. On this synthetic transfer C4C_{4} slightly edges C6C_{6} ( +0.45+0.45 mIoU), which we attribute to the rectangular geometry of the synthetic scenes preferentially exciting the m=4m{=}4 Fourier mode that C4C_{4} preserves and C6C_{6} does not (Cor. 4.5 ).
Figure 11. IBL chrome-sphere probe under camera tilt. Six tilts θ∈{0,15,30,60,75,90}∘\theta\in\{0,15,30,60,75,90\}^{\circ} (columns); each row uses its own segmentation as a thresholded envmap (light = ceiling ∪\cup wall ∪\cup window; non-light ×0.3\times 0.3 ) shading a virtual chrome sphere via analytic mirror reflection. Overlay Δ\Delta = predicted-vs-GT light-coverage drift in pp (green/amber/red = ∣Δ∣ ⁣≤ ⁣2|\Delta|\!\leq\!2 / ≤ ⁣5\leq\!5 / > ⁣5>\!5 ); GT reference is 28.7%28.7\% . App. F bounds rendered radiance error by 2.44Δpp2.44\Delta_{\mathrm{pp}} . Top: GT. Middle: SphereUFormer drifts to Δ ⁣∼ ⁣+13\Delta\!\sim\!{+}13 pp by 90∘90^{\circ} , visibly over-bright. Bottom: EquiSSL stays at 11 – 33 pp; quantitative companion in Table 8 . A 3-row by 6-column grid of chrome-sphere renderings comparing scene-aware IBL relighting at six camera tilts. Top row: ground-truth segmentation, drift near 0 pp. Middle: SphereUFormer drift climbs to about 13 pp at 90 degrees, visibly over-bright. Bottom: EquiSSL stays in a 1-to-3 pp band.
RPE VariantMP3D val@ 90∘90^{\circ}DropΔStd\Delta_{\mathrm{Std}}
No RPE50.050.044.544.5−5.5%-5.5\%−2.7-2.7
Standard RPE52.752.747.647.6−4.9%-4.9\%—
GE-RPE C4C_{4}53.1\mathbf{53.1}52.0\mathbf{52.0}−1.1%\mathbf{-1.1\%}+0.4\mathbf{+0.4}
EquiSSL ( C6C_{6} no-area)53.3\mathbf{53.3}52.6\mathbf{52.6}−0.7%\mathbf{-0.7\%}+0.6\mathbf{+0.6}
Table 7. Zero-shot semantic segmentation on Matterport3D val ( 1313 overlapping S2D3D-MP3D classes; trained on S2D3D only). The observed upright gap is ∼15{\sim}15 pp across variants; the rotation-drop column preserves the architectural ranking on this measured target corpus.
 PSNR (dB) ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
θ\thetaEquiSSLStdEquiSSLStdEquiSSLStd
0∘0^{\circ}22.6822.6822.4222.42.888.888.871.871.170.170.198.198
15∘15^{\circ}23.6123.6122.8822.88.888.888.870.870.162.162.194.194
30∘30^{\circ}23.78\mathbf{23.78}21.7721.77.882\mathbf{.882}.853.853.167\mathbf{.167}.230.230
60∘60^{\circ}22.66\mathbf{22.66}21.0621.06.869\mathbf{.869}.839.839.184\mathbf{.184}.259.259
75∘75^{\circ}22.49\mathbf{22.49}20.3520.35.868\mathbf{.868}.832.832.187\mathbf{.187}.268.268
Table 8. Path-traced IBL metrics (Mitsuba 3, chrome-Al sphere, spp =512=512 , max_depth =3=3 ). Mean ±\pm std over 44 S2D3D Area-5 test scenes × 4\times\,4 Poly Haven HDRIs. Bold = better at θ ⁣≥ ⁣30∘\theta\!\geq\!30^{\circ} .
TaskInitS1S2LossHrsSeedsCapSrc
  (ep)(ep)     
Seg. 100%100\%random–350350CE+Dice2020 – 24243390∘90^{\circ}T 1
Seg. 100%100\%iBOT+MAE5050350350CE+Dice2222 – 26263390∘90^{\circ}T 3
Seg. 1%1\%random–350350CE+Dice55 – 661190∘90^{\circ}T 21
Seg. 1%1\%iBOT+MAE5050350350CE+Dice66 – 773390∘90^{\circ}T 3
DepthiBOT+MAE5050200200L11414 – 16161190∘90^{\circ}T 16
Table 9. Fine-tune protocols used throughout the paper. All rows: 35.135.1 M backbone, AdamW (lr 10−410^{-4} , wd 0.050.05 , cosine), batch 88 , single A100-80GB. “S1” / “S2” = Stage 1 (frozen-encoder head warm-up) and Stage 2 (full fine-tune); “–” = no Stage 1.
Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

CnC_{n} \\backslash ∣m∣|m|123456≥7\geq 7
C1C_{1} (no avg.)✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmarkall
C2C_{2}×\times✓\checkmark×\times✓\checkmark×\times✓\checkmarkeven only
C4C_{4}×\times×\times×\times✓\checkmark×\times×\times4∣m4\mid m
C6C_{6}×\times×\times×\times×\times×\times✓\checkmark6∣m6\mid m
Appendix
Table 10. Which angular modes ∣m∣|m| survive a CnC_{n} average (Corollary 4.5 ). A checkmark indicates the mode is preserved; ×\times indicates exact cancellation.
Variantnnparitymmin⁡m_{\min}1/mmin⁡21/m_{\min}^{2}kills m=2m{=}2 ?surviving ∣m∣≤8|m|{\leq}8Build (s)val @ 0∘0^{\circ}val @ 90∘90^{\circ}DropDrop stdResid. ⋆Bd. ratio
Identity / negative ablation
Standard RPE11odd111.0001.000no{1,…,8}\{1{,}\dots{,}8\}22.322.366.3566.3565.6065.601.1%1.1\%±0.2%\pm 0.2\%∼0\sim 0≪1\ll 1
GE-RPE C2C_{2}22even220.2500.250no{2,4,6,8}\{2,4,6,8\}46.646.667.5867.5856.3556.3516.6%\mathbf{16.6\%}±0.8%\pm 0.8\%15.3%15.3\%61%61\%
Odd- nn alternatives (measured; no parity preservation)
GE-RPE C3C_{3}33odd330.1110.111yes{3,6}\{3,6\}45.8†45.8^{\dagger}66.90†66.90^{\dagger}65.10†65.10^{\dagger}2.7%†2.7\%^{\dagger}±0.4%†\pm 0.4\%^{\dagger}1.4%1.4\%13%13\%
GE-RPE C5C_{5}55odd550.0400.040yes{5}\{5\}46.7†46.7^{\dagger}66.95†66.95^{\dagger}65.65†65.65^{\dagger}1.9%†1.9\%^{\dagger}±0.3%†\pm 0.3\%^{\dagger}0.6%0.6\%15%15\%
Appendix
Table 11. Aliased-mode survival under CnC_{n} averaging vs. measured rotation drop (Corollary 4.5 ). The bound ratio Resid/(D∥r∥/mmin⁡2)\mathrm{Resid}/(D\|\mathbf{r}\|/m_{\min}^{2}) is rescaled so C2C_{2} ’s residual sits at 61%61\% of the Theorem 4.4 bound; C3C_{3} / C5C_{5} rows ( † ) are measured under the same protocol as the canonical family.
VariantnnHeadsρ~n\tilde{\rho}_{n} (rel. %)L^bound\hat{L}_{\mathrm{bound}} (rel. %)RatioO(1/n2)\mathcal{O}(1/n^{2})
Standard RPE136681.151041.90.0711.000
GE-RPE C2C_{2}236648.85329.40.1500.250
GE-RPE C4C_{4}436625.4482.30.3090.062
GE-RPE C6C_{6}636614.0436.60.3890.028
Appendix
Table 12. Measured equivariance residual of trained RPE bias grids vs. the Lemma A.1 bound. ρ~n\tilde{\rho}_{n} is the median head-wise ℓ∞\ell_{\infty} residual of the CnC_{n} Reynolds average vs. a 360360 -point continuous reference, as a percentage of ∥B∥∞\|B\|_{\infty} . L^bound\hat{L}_{\mathrm{bound}} normalises πDL/(3n2)\pi DL/(3n^{2}) with LL the finite-difference Lipschitz constant, D=22D{=}2\sqrt{2} . “Ratio” is ρ~n/L^bound\tilde{\rho}_{n}/\hat{L}_{\mathrm{bound}} ; rightmost column is the 1/n21/n^{2} reference.
 val mIoUval mIoUDropDrop
Variant@ 0∘0^{\circ}@ 90∘90^{\circ}(meas., 3-seed)( 1/n21/n^{2} pred.)
C4C_{4} no-area (reference)67.31 ⁣± ⁣0.7067.31\!\pm\!0.7065.86 ⁣± ⁣0.7865.86\!\pm\!0.782.15% ⁣± ⁣0.40%2.15\%\!\pm\!0.40\%—
C8C_{8} no-area67.90 ⁣± ⁣0.4267.90\!\pm\!0.4267.50 ⁣± ⁣0.4567.50\!\pm\!0.450.54% ⁣± ⁣0.10%\mathbf{0.54\%\!\pm\!0.10\%}0.54%0.54\%
C12C_{12} no-area67.85 ⁣± ⁣0.4067.85\!\pm\!0.4067.65 ⁣± ⁣0.4267.65\!\pm\!0.420.24% ⁣± ⁣0.05%\mathbf{0.24\%\!\pm\!0.05\%}0.24%0.24\%
n→∞n{\to}\infty Haar (FA limit)67.8567.8566.9566.95—0.00%0.00\%
Appendix
Table 13. Stress-test rotation drops at C8/C12C_{8}/C_{12} ( 33 -seed, seeds {42,123,456}\{42,123,456\} ; 350350 ep., CE ++ Dice loss, no SSL). The measured drops match the Δn=Δ4(4/n)2\Delta_{n}{=}\Delta_{4}(4/n)^{2} prediction within seed std, extending the 1/n21/n^{2} slope of Figure 8 up to n=12n{=}12 . The measured C4C_{4} reference row repeats the seed-mean Δ4\Delta_{4} for direct comparison; the retained Haar-limit row reports the bias-residual asymptote, with end-to-end Drop unmeasured.
Figure 12. Per-layer polar-Fourier amplitude of the trained RPE bias across the four variants ( 2×22{\times}2 grid). Rows within each panel: 22 attention layers (encoder and decoder, sorted as in the checkpoint); columns: angular modes ∣m∣=0,…,7|m|{=}0,\dots,7 . Survival strip above each panel: filled red = surviving mode ( n∣mn\mid m ), ×\times = suppressed by the CnC_{n} Reynolds average (Corollary 4.5 ). Shared horizontal colour bar uses head-level ℓ1\ell_{1} -normalised amplitude so shapes are comparable across variants. Two-by-two grid of heatmaps, one per RPE variant (Standard, C2C_2, C4C_4, C6C_6 no-area). Each heatmap has 22 rows (attention layers) and 8 columns (angular modes). Mass concentrates onto the surviving mode columns predicted by the n\midmn \midm rule of Corollary 1: every even mode for C2C_2, modes 0 and 4 for C4C_4, modes 0 and 6 for C6C_6.
Figure 13. CnC_{n} covering of SO(2)≅S1\mathrm{SO}(2)\cong S^{1} . Each panel shows nn sample points on the circle (the gauge choices) with alternating Voronoi regions. The red arc marks the maximum gap π/n\pi/n between adjacent samples. Larger nn yields smaller gaps and tighter equivariance (Theorem 4.3 ). Three side-by-side circles representing SO(2)\mathrm{SO}(2), with 2, 4, and 6 evenly spaced sample points. Each circle is partitioned into Voronoi arcs around its samples, and a red arc spans the worst-case half-gap of π/n\pi/n between adjacent samples.
Figure 14. Geometric diagnostics for the rotation-stability bound. (a) The mesh-permutation error decays geometrically along the O(2−r)\mathcal{O}(2^{-r}) slope predicted by halving the edge length per refinement, validating εmesh(r) ⁣→ ⁣0\varepsilon_{\mathrm{mesh}}(r)\!\to\!0 in Theorem 4.3 . (b) At the canonical rank r=3r{=}3 the distribution is unimodal with max within 1.6×1.6\times mean—no hidden long-tail population. (c) The composite gauge mismatch ∣Δg∣|\Delta g| stays inside the π/n\pi/n envelopes that the worst-case bound predicts, with the cumulative mass matching the uniform-quadrature shape that the Reynolds average projects onto. Three-panel composite. Panel (a): log-y line plot of mesh permutation angular error in degrees against icosphere rank from 1 to 7; mean and max curves decay along an order-2−r2^{-r} reference, with a 5–95 percentile band and a mesh-spacing reference. Panel (b): histogram of per-vertex angular permutation error at rank 3, vertical lines marking mean 3.06 degrees, p95p_{95} 4.74 degrees, max 5.93 degrees; unimodal with tight right tail. Panel (c): wide histogram of gauge mismatch angles from 0 to 180 degrees with three vertical dashed reference lines at π/6\pi/6, π/4\pi/4, π/2\pi/2 marking the C6/C4/C2C_6/C_4/C_2 worst-case bounds; density is approximately uniform across the support and cumulative mass below each line matches the uniform-distribution prediction.
Bias param.C2C_{2} dropC4C_{4} dropC6C_{6} drop
bilinear BB (canonical)16.6%16.6\%1.7%1.7\%1.3%\mathbf{1.3\%}
MLP BMLPB_{\mathrm{MLP}}17.0%17.0\%3.5%3.5\%2.6%2.6\%
Appendix
Table 14. Bias-parameterisation ablation, 5050 -epoch single-seed runs (random init, no SSL). The bilinear- BB row is the canonical configuration; the MLP- BB row replaces the 7×77\times 7 bilinear grid with a 22 -layer MLP BMLP ⁣:R2 ⁣→ ⁣RB_{\mathrm{MLP}}\colon\mathbb{R}^{2}\!\to\!\mathbb{R} (hidden width 128128 , ReLU activation, learned bias terms; no sinusoidal/Fourier positional features—this is the configuration that exposes the spectral-bias-induced 1/m1/m angular decay ( Tancik et al., 2020 ) ). The bilinear- BB row converges onto the mesh-floor by C4C_{4} ; the MLP- BB row plateaus above it, validating Theorem 4.4 ’s rate dependence on bias smoothness.
 Small (14.6M)Large (35.1M)
Embedding dimension3248
Encoder depths[2, 2, 2, 2][2, 2, 6, 2]
Encoder heads[2, 4, 8, 16][3, 6, 12, 24]
Bottleneck depth22
Window size coef.22
MLP ratio4.04.0
Appendix
Table 15. Architecture configurations.
PretrainingRPEδ1↑\delta_{1}\uparrowRMSE ↓\downarrowAbs Rel ↓\downarrow
Random initNo RPE0.876±0.0030.876{\pm}0.0030.2360.2360.09390.0939
Random initStandard0.869±0.0020.869{\pm}0.0020.2410.2410.09580.0958
Random initGE-RPE C4C_{4}0.904±0.0020.904{\pm}0.0020.2260.2260.08980.0898
Random initGE-RPE C6C_{6} no-area0.911±0.0020.911{\pm}0.0020.2230.2230.08860.0886
iBOT ++ MAENo RPE0.891±0.0020.891{\pm}0.0020.2300.2300.09130.0913
iBOT ++ MAEStandard0.882±0.0020.882{\pm}0.0020.2340.2340.09300.0930
Appendix
Table 16. Depth estimation on Stanford2D3D val (35.1M backbone, 3-seed mean ±\pm sample std; seeds {42,123,456}\{42,123,456\} ). Two-stage fine-tune ( 5050 ep frozen ++ 200200 ep full), L1 loss, MAX_DEPTH=5.12\mathrm{MAX\_DEPTH}=5.12 m. The bolded row is the canonical EquiSSL configuration.
PretrainingRPEδ1\delta_{1} @ 0∘0^{\circ}
Random initStandard0.8690.869
Random initGE-RPE C6C_{6} no-area0.9110.911
iBOT ++ MAEStandard0.8820.882
iBOT ++ MAE★\bigstar EquiSSL0.922\mathbf{0.922}
Appendix
Table 17. Measured depth δ1\delta_{1} on Stanford2D3D val at θ=0∘\theta{=}0^{\circ} ( 33 -seed mean, copied from Table 16 ); encoder-level rotation stability is established on the segmentation side in Table 20 .
ClassNoStdC4C_{4}C6C_{6} n.a.
beam0.000.000.000.000.000.000.000.00
board54.1554.1551.6951.6954.0554.0549.2149.21
bookcase41.3141.3141.7341.7339.6839.6841.9041.90
ceiling51.5551.5551.3851.3850.7050.7052.2152.21
chair29.1629.1629.5729.5729.4029.4029.9029.90
clutter21.8821.8821.5721.5721.1221.1222.0422.04
Appendix
Table 18. Per-class IoU on Stanford2D3D test set, four direct-measurement random-init RPE ablations (No / Std / C4C_{4} / C6C_{6} no-area), seed 4242 .
SeedmIoU0(s)\mathrm{mIoU}^{(s)}_{0} (%)mIoU90(s)\mathrm{mIoU}^{(s)}_{90} (%)Δs\Delta_{s} (paired)
4267.8267.8266.9066.901.36%1.36\%
12368.2368.2367.3567.351.29%1.29\%
45667.5067.5066.5866.581.36%1.36\%
mean67.8567.8566.9466.941.34%1.34\%
std0.380.380.390.390.04%0.04\%
Appendix
Table 19. Per-seed val mIoU endpoints and paired drop ratio for the random-init GE-RPE C6C_{6} (no SSL) configuration, three independent training seeds. “ Δs\Delta_{s} (paired)” is the simple ratio (mIoU0(s)−mIoU90(s))/mIoU0(s)(\mathrm{mIoU}^{(s)}_{0}-\mathrm{mIoU}^{(s)}_{90})/\mathrm{mIoU}^{(s)}_{0} ; the column mean ( 1.34%1.34\% , std 0.04%0.04\% across seeds) reproduces the Table 1 headline value 1.3±0.2%1.3\pm 0.2\% to rounding (the larger displayed std reflects rotation-sample variance within each seed, not seed-to-seed variance). The matching SSL-pretrained EquiSSL endpoint is 0.8%0.8\% (Table 3 ).
θmax⁡\theta_{\max}SphereUF.No RPEStd RPEC2C_{2}C4C_{4}C6C_{6} +areaC6C_{6} (no SSL)★\bigstar EquiSSL
0∘0^{\circ}62.9862.9867.58±0.5867.58{\pm}0.5866.35±0.4866.35{\pm}0.4867.58±0.6267.58{\pm}0.6267.31±0.7067.31{\pm}0.7067.24±0.7667.24{\pm}0.7667.85±0.38\mathbf{67.85{\pm}0.38}68.30±0.38\mathbf{68.30{\pm}0.38}
10∘10^{\circ}59.2259.2266.95±0.6066.95{\pm}0.6065.82±0.4065.82{\pm}0.4067.02±0.5067.02{\pm}0.5066.80±0.7066.80{\pm}0.7066.65±0.7566.65{\pm}0.7567.26±0.37\mathbf{67.26{\pm}0.37}68.24±0.4068.24{\pm}0.40
20∘20^{\circ}53.7753.7766.88±0.5566.88{\pm}0.5565.86±0.4265.86{\pm}0.4264.10±2.0064.10{\pm}2.0066.75±0.7366.75{\pm}0.7366.49±0.7066.49{\pm}0.7067.09±0.34\mathbf{67.09{\pm}0.34}68.18±0.4068.18{\pm}0.40
35∘35^{\circ}44.0644.0667.20±0.5767.20{\pm}0.5765.87±0.4065.87{\pm}0.4062.10±2.5062.10{\pm}2.5066.47±0.6766.47{\pm}0.6766.35±0.6066.35{\pm}0.6066.92±0.37\mathbf{66.92{\pm}0.37}68.09±0.4068.09{\pm}0.40
45∘45^{\circ}40.9740.9767.05±0.5067.05{\pm}0.5065.88±0.3865.88{\pm}0.3859.90±3.0059.90{\pm}3.0066.42±0.7266.42{\pm}0.7266.41±0.5566.41{\pm}0.5566.88±0.35\mathbf{66.88{\pm}0.35}68.03±0.4068.03{\pm}0.40
60∘60^{\circ}35.9535.9566.90±0.4566.90{\pm}0.4565.83±0.3665.83{\pm}0.3657.80±3.2057.80{\pm}3.2066.20±0.6766.20{\pm}0.6766.31±0.4566.31{\pm}0.4566.92±0.37\mathbf{66.92{\pm}0.37}67.94±0.4067.94{\pm}0.40
Appendix
Table 20. Per-angle val mIoU on Stanford2D3D, 33 -seed mean ±\pm std (seeds {42,123,456}\{42,123,456\} , 1010 random SO(3)\mathrm{SO}(3) rotations ×3\times 3 repeats per cap) for our random-init ablations and the canonical EquiSSL ( ★\bigstar ). The SphereUFormer column is the single-seed published checkpoint. Bottom row is the 0∘ ⁣→ ⁣90∘0^{\circ}\!\to\!90^{\circ} drop. The C6C_{6} +area and C6C_{6} (no SSL) columns of this table are the canonical data source for the area-weighting ablation (Table 4 ); the main-body table presents only those two rows alongside the per-angle delta for the focused ablation read.
Label%NNNoneStdC4C_{4}C6C_{6}C4C_{4} SSL★ C6\bigstar\,C_{6} SSL
1%1\%1017.9217.9217.9817.9818.6218.6218.4918.4922.7722.7722.59\mathbf{22.59}
5%5\%5038.5038.5038.7138.7137.4737.4737.4737.4739.8539.8539.70\mathbf{39.70}
10%10\%10143.2243.2244.2144.2144.7344.7344.8544.8545.9245.9246.85\mathbf{46.85}
100%‡100\%^{\ddagger}101367.5867.5866.3566.3567.3167.3167.8567.8567.8067.8068.30\mathbf{68.30}
Appendix
Table 21. Label efficiency on Stanford2D3D — val mIoU (%) . Low-label rows ( 1%1\% , 5%5\% , 10%10\% ) random-init are seed- 4242 single runs; 100%100\% rows ( ‡\ddagger ) are 33 -seed means. SSL rows use the iBOT ++ MAE two-stage schedule (Table 9 ); the canonical EquiSSL row ( ★ C6 ⁣+ ⁣\bigstar\,C_{6}\!+\! SSL) is bolded.
Label%NNNoneStdC4C_{4}C6C_{6}C4C_{4} SSL★ C6\bigstar\,C_{6} SSL
1%1\%1013.8013.8013.7713.7712.2212.2212.4012.4014.8914.8914.79\mathbf{14.79}
5%5\%5021.2121.2120.9120.9118.8818.8818.8818.8820.1020.1020.05\mathbf{20.05}
10%10\%10124.1524.1524.1524.1524.4024.4024.5024.5025.0525.0525.20\mathbf{25.20}
100%‡100\%^{\ddagger}101331.4231.4231.1131.1130.5030.5031.2031.2029.9329.9331.35\mathbf{31.35}
Appendix
Table 22. Label efficiency on Stanford2D3D — test mIoU (%) , same row protocol as Table 21 . Test split has N=373N{=}373 samples (vs. 4040 for val), so single-seed test variance is correspondingly tighter.
ConfigurationRotation drop
Full EquiSSL0.8%0.8\%
−- SO(3) augmentation2.4%2.4\%
−- πR\pi_{R} teacher-token permutation2.8%2.8\%
−- GE-RPE (= Standard RPE ++ iBOT ++ MAE)8.6%8.6\%
Appendix
Table 23. Leave-one-out ablation on the three coupled levers of EquiSSL. Rotation drop @ θmax⁡=90∘\theta_{\max}=90^{\circ} on Stanford2D3D val. Architecture (GE-RPE C6C_{6} no-area) is held fixed except in the last row. The full-EquiSSL row reports the canonical 33 -seed mean ( {42,123,456}\{42,123,456\} , std ±0.38\pm 0.38 ); each leave-one-out variant is a seed- 4242 single run under the same training recipe as the canonical row.
Pretext objectiveVal mIoU @ 0∘0^{\circ}Rotation drop
iBOT ++ MAE (composite, canonical)68.3068.300.8%0.8\%
iBOT only ( recon weight=0\mathrm{recon\ weight}=0 )67.5567.551.0%1.0\%
MAE only ( distill weight=0\mathrm{distill\ weight}=0 )66.2066.201.5%1.5\%
Appendix
Table 24. iBOT vs MAE individual contribution at canonical (GE-RPE C6C_{6} no-area ++ πR\pi_{R} ++ SO(3) aug); rotation drop @ θmax⁡=90∘\theta_{\max}=90^{\circ} . “MAE only” zeros the iBOT distillation weight; “iBOT only” zeros the MAE recon weight. Both retain SO(3) aug and πR\pi_{R} ( πR\pi_{R} is structurally inactive in MAE-only).
Figure 15. 100100 -epoch iBOT ++ MAE pretraining loss curves; iBOT plateaus past epoch 6666 , MAE past epoch 4949 ; 200200 -epoch doubling adds ∼0.25\sim 0.25 pp val mIoU. Two-panel line plot of pretraining loss vs epoch. Left: iBOT patch-distillation cross-entropy drops from 9.2 to 0.39, plateau at epoch 66. Right: MAE pixel reconstruction loss drops from 0.88 to 0.13, plateau at epoch 49.
More in this topic

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 295 papers

Explore similar work

CardsList

  1. SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction

    Feb 26, 2026Qinfeng Zhu, Yunxi Jiang, Lei FanDepth EstimationVision Transformer

    University of Liverpool, United Kingdom. · CNRS, France. · Xi’an Jiaotong-Liverpool University, China.

  2. PanoWorld: Towards Spatial Supersensing in 360∘^\circ Panorama World

    May 13, 2026Changpeng Wang, Xin Lin, Junhan Liu +53D Spatial Reasoning3D Scene Understanding

    1Zhejiang University · University of California, San Diego · University of California, Irvine +1

  3. Unified Panoramic Geometry Estimation via Multi-View Foundation Models

    May 25, 2026Vukasin Bozic, Isidora Slavkovic, Dominik Narnhofer +43D Foundation Models3D ViTs

    ETH Zürich · Google · Athlence Sports +1