ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment

Alex Junho Lee

cs.CVOct 9, 2026

ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment

Authors: Alex Junho Lee

Organizations: School of Mechanical Systems Engineering, Sookmyung Women’s University, Seoul 04310, South Korea

Abstract

When a query must be matched against a database built with a different sensor or processed by a different encoder, the independently trained encoders impose distinct coordinate systems on the feature space. No canonical reference frame exists to which all encoders can be aligned, and no single training procedure has fully closed this gap. We show that the relative transformation between any two such systems is a spatially preserving rotation in whitened feature space, whose angular velocity exhibits white-noise characteristics: large in norm, temporally uncorrelated, and unpredictable from the features themselves. However, the cross-covariance between encoder outputs has low effective rank, constraining only a small fraction of the rotation parameters, therefore local alignment is tractable. ROSETTA is a training-free calibration layer that estimates the local rotation via an expectation-maximization procedure, requiring no coobservation at query time. We validate ROSETTA on three public driving datasets spanning LiDAR, camera, thermal, and event modalities in cross-modal, cross-device, cross-representation, and cross-backbone settings, with milliseconds overhead per query.

Figures & tables

Fig. 1: Two independently trained models map locations $P_{1},P_{2}$ on a smooth manifold ( top ) to different coordinates $X(P_{i})$ and $Y(P_{i})$ in separate feature spaces ( bottom ). The gray arrows indicate that the inter-model transformation $P_{i}\!(x_{i}{\to}y_{i})$ differs between locations: the rotation at $P_{1}$ does not equal the rotation at $P_{2}$ . Because $\boldsymbol{R}(\boldsymbol{x})\in\SO(K)$ varies with position, no single global rotation can align the two spaces.

Fig. 2: Overview of the Rosetta pipeline. (Left) Frozen encoders map each observation to

\boldsymbol{z}_{i},\boldsymbol{y}_{j}\in\mathbb{R}^{K}

; LiDAR and camera encoders are shown, but the method applies to any modality or backbone. (Center) A local Soft EM (Algorithm 1 ) computes soft weights

w_{j}

from a spatial prior

L_{ij}

, accumulates the cross-covariance

\boldsymbol{M}

, and recovers

\boldsymbol{R}\!\in\!\SO(K)

via SVD. Bottom: the cross-encoder similarity sharpens to diagonal after rotation. (Right) The rotated descriptor

\boldsymbol{R}\boldsymbol{z}_{i}\!\approx\!\boldsymbol{y}_{j}

yields cross-modal nearest-neighbor matches that enter a pose graph as loop-closure factors, producing the trajectory estimate

\hat{p}(t)

Panel of Figure 3 — Fig. 3: Cross-covariance analysis ( $W\!=\!20$ , $K\!=\!128$ ). Singular-value spectrum of $\mathbf{Z}^{\top}\!\mathbf{Y}$ on KITTI-360: correct pairings (blue) concentrate energy in $\sigma_{1}$ ; random pairings (orange) produce a flat spectrum. Distribution of the Procrustes residual $\|\mathbf{Z}{-}\mathbf{Y}\mathbf{R}^{\top}\|_{F}$ for correct (solid) and random (dashed) pairings on all three datasets. $d^{\prime}$ denotes the separation between the two distributions.

Panel of Figure 4 — Fig. 4: Cross-modal retrieval matrix on VIVID (MinkLoc3Dv2 $\to$ DINOv2-ViT-L). Each cell shows the fraction of queries from one spatial cluster retrieved from a given database cluster; a perfect retriever concentrates all mass on the diagonal. Raw descriptors yield 0.04 diagonal, comparable to random retrieval. After Rosetta alignment, 0.99 is on the diagonal.

Fig. 5: Effective rank

r_{\mathrm{eff}}

versus window coverage

L

. Colored lines: power-law fit

r_{\mathrm{eff}}=a\,L^{\beta}

. Gray dashed: number of paired observations

W=L/\delta

, where

\delta

is the spatial sampling interval. Since

\beta<1

W

grows faster than

r_{\mathrm{eff}}

, so local alignment is overdetermined for all tested

L

(Sec. III-D4 ).

Fig. 6: Per-frame cosine similarity $\cos(\boldsymbol{R}\boldsymbol{z}_{t},\boldsymbol{y}_{t})$ between the aligned query and the ground-truth database descriptor along the VIVID trajectory (Gaussian-smoothed, ${\pm}1\sigma$ shading). Local Soft EM (blue) maintains $\cos>0.8$ throughout. A single global $\boldsymbol{R}$ fitted to all co-observations (gray) oscillates between $0.3$ and $0.8$ . The MLP adapter [ 32 ] trained on the same environment (orange) tracks the local estimator in trained segments, but degrades in a different environment (dashed).

Dataset	Environment	Sensors
KITTI-360 [ 34 ]	Suburban	Velodyne (HDL-64E) PointGrey (stereo)
VIVID [ 35 ]	Campus	Ouster (OS1-64) PointGrey (fisheye) FLIR A65 (LWIR) DVXplorer (event)
HeLiPR [ 36 ]	Urban	Ouster (OS2-128) Velodyne (VLP-16) Livox (Avia)

TABLE I: Evaluation datasets and their sensor suites. The three datasets span suburban, campus, and urban environments with four modalities (LiDAR, RGB, thermal infrared, event).

Method	Super- vision	Learned module	Zero-shot pairing
VXP [ 14 ]	Paired	Both enc.	–
(LC) 2 [ 11 ]	Paired	Both enc.	–
MLP [ 32 ]	Paired	Projector	–
Rosetta	None	None	✓

TABLE II: Problem setting comparison. Unlike joint or supervised methods, Rosetta aligns independently trained, frozen encoders without any paired training data.

Method	KITTI	VIVID †	HeLiPR
(LC) 2 pretrained ‡ [ 11 ]	0.025	0.037	0.031
(LC) 2 fine-tuned §	0.509	0.505	0.652
Raw (no alignment)	0.305	0.058	0.014
MLP Adapter (cross-env)	0.023	0.061	0.032
Overfitting Procrustes ∗	0.998	0.857	0.649
MLP Adapter (same-env) ¶	0.988	0.901	0.830

TABLE III: Cross-modal retrieval (R@1@25 m, global NN). ∗ Requires test co-observations. † Cross-sequence. ‡ Pretrained [ 11 ] . § Fine-tuned per dataset. ¶ MLP trained on target environment.

Encoder	Sensor pair	Overfitting ∗	Ours
PCE	Ouster $\to$ Ouster (same-sensor)	0.967
	Ouster $\to$ Velodyne (DCC04)	0.649	0.681
	Ouster $\to$ Avia (DCC04)	0.652	0.695
	Ouster $\to$ Velodyne (KAIST04)	0.675	0.732
	Velodyne $\to$ Ouster (DCC04)	0.691	0.680
MinkLoc †	Ouster $\to$ Ouster (same-sensor)	0.998

TABLE IV: Cross-device retrieval on HeLiPR, Axis (ii) (R@1@25 m, global NN). ∗ Requires test co-observations. † MinkLoc3Dv2 (256-D; separate PCA).

Fig. 7: Qualitative cross-modal retrieval on VIVID (day2 LiDAR query $\to$ day1 camera database). Each row shows the query, the top-1 match without and with our alignment, and the ground truth. (a, b) Our alignment corrects a wrong raw match. (c) Failure: perceptual aliasing between visually similar scenes persists after alignment.

Pair	Overfitting ∗	Ours
$\mathcal{T}{\to}\mathcal{R}^{\dagger}$	0.776	0.858
$\mathcal{R}{\to}\mathcal{T}^{\dagger}$	0.873	0.886
$\mathcal{E}{\to}\mathcal{R}^{\dagger}$	0.769	0.737
$\mathcal{R}{\to}\mathcal{E}^{\dagger}$	0.751	0.756
$\mathcal{T}{\to}\mathcal{E}^{\dagger}$	0.609	0.744
$\mathcal{E}{\to}\mathcal{T}^{\dagger}$	0.739	0.734

TABLE V: Cross-modal retrieval on VIVID, Axis (i) (R@1@25 m, global NN). ∗ Requires test co-observations. † Cross-sequence.

Pair	Encoders	Overfitting ∗	Ours
Cross-sequence (day2 $\to$ day1):
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv3	0.859	0.909
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / PCE	0.711	0.677
$\mathcal{L}{\to}\mathcal{R}$	MinkLoc † / DINOv3	0.905	0.946
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / MinkLoc †	0.739	0.736
$\mathcal{E}{\to}\mathcal{R}$	Event DINOv3 / DINOv3	0.778	0.735

TABLE VI: Cross-representation retrieval on VIVID Campus, Axes (iii) & (iv) (R@1@25 m, global NN). ∗ Requires test co-observations. † MinkLoc3Dv2 (256-D; separate PCA).

Pair	Encoders	Overfitting ∗	MLP ×	Ours
Cross-representation (Axis iii):
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv3	0.963	0.013	0.945
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / PCE	0.868	0.007	0.966
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv2-ViT-L	0.965	0.002	0.965
$\mathcal{R}{\to}\mathcal{L}$	DINOv2-ViT-L / PCE	0.922	0.017	0.960
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv2-ViT-B	0.960	0.018	0.969

TABLE VII: Encoder-pair sweep on KITTI-360 (R@1@25 m, global NN). ∗ Requires test co-observations. × MLP trained on VIVID, transferred to KITTI.

Pair	Encoders	Overfitting ∗	MLP ×	Ours
Cross-representation (Axis iii):
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv3	0.611	0.007	0.758
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / PCE	0.612	0.007	0.856
$\mathcal{L}{\to}\mathcal{R}$	MinkLoc † / DINOv3	0.762	0.005	0.839
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / MinkLoc †	0.686	0.005	0.821
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv2-ViT-L	0.695	0.005	0.795

TABLE VIII: Cross-sequence retrieval on VIVID City (day2 query, day1 database; R@1@25 m, global NN). ∗ Requires test co-observations. × MLP trained on KITTI, transferred to VIVID City. † MinkLoc3Dv2 (256-D; separate PCA).

Approach	Inlier( $<$ 25 m)	Reason of failure
Diff. Procrustes adapter	0.07–0.09	SVD backward ill-cond.
Rotation prediction MLP	0.15–0.20	White-noise $\boldsymbol{\Omega}$
$\boldsymbol{R}$ -consistency reg.	0.076	Non-stationary process
Cross-traversal	0.418	Backbone repeatability

TABLE IX: Failed approaches on KITTI-360 (Inlier(

<

25 m)). Each method attempts to learn or predict

\boldsymbol{R}(x)

during training.

Dataset	Open-loop	Iso.	Aniso.
KITTI-360 (3.3 km)	20.8	3.0	0.9
VIVID (3.7 km)	22.8	2.1	1.3
HeLiPR DCC04 (5.4 km)	25.6	5.2	3.9
HeLiPR KAIST04 (5.8 km)	28.0	5.4	2.1

TABLE X: Cross-modal odometry (median position error, meters).

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 445 papers

CalibBEV: LiDAR-Camera Calibration via BEV Alignment
Aug 3, 2026Filippo D'Addeo, Lorenzo Cipelli, Adriano Cardace +3Cross-Modal Alignment Camera-LiDAR Fusion
University of Bologna, Department of Industrial Engineering, Italy. · University of Parma, Department of Engineering and Architecture, Italy. · Stanford University, USA. +1
Calousel: Extrinsic Calibration of Non-overlapping Multi-camera Systems from Pure Rotation
Jun 24, 2026Gwanhyeong Song, Chaehyeon Song, Ayoung KimMulti-View Geometry Camera Pose Estimation
Dept. of Mechanical Engineering, SNU, Seoul, S. Korea
PAGER: Partial-to-global Alignment via Geometric and Relational Distillation
Oct 1, 2026Akira-Miranda Adeyomi Adeniran-Lowe, Binod Singh, Lars Arnold Dethlefsen +23D Semantic Segmentation 3D Representation Learning
Technical University of Denmark · Pioneer Center for Artificial Intelligence

Aug 3, 2026J/K move · Enter open · S save

CalibBEV: LiDAR-Camera Calibration via BEV Alignment

Filippo D'Addeo, Lorenzo Cipelli, Adriano Cardace, Emanuele Ghelfi+2

University of Bologna, Department of Industrial Engineering, Italy. · University of Parma, Department of Engineering and Architecture, Italy. · Stanford University, USA.+1

Cross-Modal Alignment Camera-LiDAR Fusion Lidar-Camera Extrinsic Calibration

Open paper AskPDF arXiv

We present CalibBEV, a novel Bird's Eye View (BEV) alignment approach for LiDAR-camera calibration. Our method unifies LiDAR and camera data into a shared 3D spatial representation, enabling accurate and robust cross-modal calibration. CalibBEV extracts sensor-wise BEV features from each modality using domain-specific architectures and estimates the calibration matrix through a two-step alignment process. First, we perform an implicit alignment by regressing a coarse calibration matrix directly from the BEV features. To ease this alignment, we enforce semantic consistency between BEV representations across modalities using a contrastive loss inspired by CLIP, guiding both networks toward a unified feature space. In the second step, we leverage our BEV formulation to explicitly align the features of one modality with the other, refining the initial coarse estimate into a final, more accurate calibration matrix. CalibBEV significantly outperforms prior point-to-pixel matching methods, achieving state-of-the-art calibration accuracy. On the KITTI and nuScenes benchmarks, our method reduces the Relative Rotation Error (RRE) by 51% and 68%, and the Relative Translation Error (RTE) by 80% and 91%, respectively, compared to previous methods.

Abstract

Cross-Modal Alignment

Explore similar work

CalibBEV: LiDAR-Camera Calibration via BEV Alignment

Calousel: Extrinsic Calibration of Non-overlapping Multi-camera Systems from Pure Rotation

PAGER: Partial-to-global Alignment via Geometric and Relational Distillation