ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment

Alex Junho Lee

cs.CVOct 9, 2026

ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment

Authors: Alex Junho Lee

Organizations: School of Mechanical Systems Engineering, Sookmyung Women’s University, Seoul 04310, South Korea

Abstract

When a query must be matched against a database built with a different sensor or processed by a different encoder, the independently trained encoders impose distinct coordinate systems on the feature space. No canonical reference frame exists to which all encoders can be aligned, and no single training procedure has fully closed this gap. We show that the relative transformation between any two such systems is a spatially preserving rotation in whitened feature space, whose angular velocity exhibits white-noise characteristics: large in norm, temporally uncorrelated, and unpredictable from the features themselves. However, the cross-covariance between encoder outputs has low effective rank, constraining only a small fraction of the rotation parameters, therefore local alignment is tractable. ROSETTA is a training-free calibration layer that estimates the local rotation via an expectation-maximization procedure, requiring no coobservation at query time. We validate ROSETTA on three public driving datasets spanning LiDAR, camera, thermal, and event modalities in cross-modal, cross-device, cross-representation, and cross-backbone settings, with milliseconds overhead per query.

Figures & tables

Fig. 1: Two independently trained models map locations $P_{1},P_{2}$ on a smooth manifold ( top ) to different coordinates $X(P_{i})$ and $Y(P_{i})$ in separate feature spaces ( bottom ). The gray arrows indicate that the inter-model transformation $P_{i}\!(x_{i}{\to}y_{i})$ differs between locations: the rotation at $P_{1}$ does not equal the rotation at $P_{2}$ . Because $\boldsymbol{R}(\boldsymbol{x})\in\SO(K)$ varies with position, no single global rotation can align the two spaces.

Fig. 2: Overview of the Rosetta pipeline. (Left) Frozen encoders map each observation to

\boldsymbol{z}_{i},\boldsymbol{y}_{j}\in\mathbb{R}^{K}

; LiDAR and camera encoders are shown, but the method applies to any modality or backbone. (Center) A local Soft EM (Algorithm 1 ) computes soft weights

w_{j}

from a spatial prior

L_{ij}

, accumulates the cross-covariance

\boldsymbol{M}

, and recovers

\boldsymbol{R}\!\in\!\SO(K)

via SVD. Bottom: the cross-encoder similarity sharpens to diagonal after rotation. (Right) The rotated descriptor

\boldsymbol{R}\boldsymbol{z}_{i}\!\approx\!\boldsymbol{y}_{j}

yields cross-modal nearest-neighbor matches that enter a pose graph as loop-closure factors, producing the trajectory estimate

\hat{p}(t)

Panel of Figure 3 — Fig. 3: Cross-covariance analysis ( $W\!=\!20$ , $K\!=\!128$ ). Singular-value spectrum of $\mathbf{Z}^{\top}\!\mathbf{Y}$ on KITTI-360: correct pairings (blue) concentrate energy in $\sigma_{1}$ ; random pairings (orange) produce a flat spectrum. Distribution of the Procrustes residual $\|\mathbf{Z}{-}\mathbf{Y}\mathbf{R}^{\top}\|_{F}$ for correct (solid) and random (dashed) pairings on all three datasets. $d^{\prime}$ denotes the separation between the two distributions.

Panel of Figure 4 — Fig. 4: Cross-modal retrieval matrix on VIVID (MinkLoc3Dv2 $\to$ DINOv2-ViT-L). Each cell shows the fraction of queries from one spatial cluster retrieved from a given database cluster; a perfect retriever concentrates all mass on the diagonal. Raw descriptors yield 0.04 diagonal, comparable to random retrieval. After Rosetta alignment, 0.99 is on the diagonal.

Fig. 5: Effective rank

r_{\mathrm{eff}}

versus window coverage

L

. Colored lines: power-law fit

r_{\mathrm{eff}}=a\,L^{\beta}

. Gray dashed: number of paired observations

W=L/\delta

, where

\delta

is the spatial sampling interval. Since

\beta<1

W

grows faster than

r_{\mathrm{eff}}

, so local alignment is overdetermined for all tested

L

(Sec. III-D4 ).

Fig. 6: Per-frame cosine similarity $\cos(\boldsymbol{R}\boldsymbol{z}_{t},\boldsymbol{y}_{t})$ between the aligned query and the ground-truth database descriptor along the VIVID trajectory (Gaussian-smoothed, ${\pm}1\sigma$ shading). Local Soft EM (blue) maintains $\cos>0.8$ throughout. A single global $\boldsymbol{R}$ fitted to all co-observations (gray) oscillates between $0.3$ and $0.8$ . The MLP adapter [ 32 ] trained on the same environment (orange) tracks the local estimator in trained segments, but degrades in a different environment (dashed).

Dataset	Environment	Sensors
KITTI-360 [ 34 ]	Suburban	Velodyne (HDL-64E) PointGrey (stereo)
VIVID [ 35 ]	Campus	Ouster (OS1-64) PointGrey (fisheye) FLIR A65 (LWIR) DVXplorer (event)
HeLiPR [ 36 ]	Urban	Ouster (OS2-128) Velodyne (VLP-16) Livox (Avia)

TABLE I: Evaluation datasets and their sensor suites. The three datasets span suburban, campus, and urban environments with four modalities (LiDAR, RGB, thermal infrared, event).

Method	Super- vision	Learned module	Zero-shot pairing
VXP [ 14 ]	Paired	Both enc.	–
(LC) 2 [ 11 ]	Paired	Both enc.	–
MLP [ 32 ]	Paired	Projector	–
Rosetta	None	None	✓

TABLE II: Problem setting comparison. Unlike joint or supervised methods, Rosetta aligns independently trained, frozen encoders without any paired training data.

Method	KITTI	VIVID †	HeLiPR
(LC) 2 pretrained ‡ [ 11 ]	0.025	0.037	0.031
(LC) 2 fine-tuned §	0.509	0.505	0.652
Raw (no alignment)	0.305	0.058	0.014
MLP Adapter (cross-env)	0.023	0.061	0.032
Overfitting Procrustes ∗	0.998	0.857	0.649
MLP Adapter (same-env) ¶	0.988	0.901	0.830

TABLE III: Cross-modal retrieval (R@1@25 m, global NN). ∗ Requires test co-observations. † Cross-sequence. ‡ Pretrained [ 11 ] . § Fine-tuned per dataset. ¶ MLP trained on target environment.

Encoder	Sensor pair	Overfitting ∗	Ours
PCE	Ouster $\to$ Ouster (same-sensor)	0.967
	Ouster $\to$ Velodyne (DCC04)	0.649	0.681
	Ouster $\to$ Avia (DCC04)	0.652	0.695
	Ouster $\to$ Velodyne (KAIST04)	0.675	0.732
	Velodyne $\to$ Ouster (DCC04)	0.691	0.680
MinkLoc †	Ouster $\to$ Ouster (same-sensor)	0.998

TABLE IV: Cross-device retrieval on HeLiPR, Axis (ii) (R@1@25 m, global NN). ∗ Requires test co-observations. † MinkLoc3Dv2 (256-D; separate PCA).

Fig. 7: Qualitative cross-modal retrieval on VIVID (day2 LiDAR query $\to$ day1 camera database). Each row shows the query, the top-1 match without and with our alignment, and the ground truth. (a, b) Our alignment corrects a wrong raw match. (c) Failure: perceptual aliasing between visually similar scenes persists after alignment.

Pair	Overfitting ∗	Ours
$\mathcal{T}{\to}\mathcal{R}^{\dagger}$	0.776	0.858
$\mathcal{R}{\to}\mathcal{T}^{\dagger}$	0.873	0.886
$\mathcal{E}{\to}\mathcal{R}^{\dagger}$	0.769	0.737
$\mathcal{R}{\to}\mathcal{E}^{\dagger}$	0.751	0.756
$\mathcal{T}{\to}\mathcal{E}^{\dagger}$	0.609	0.744
$\mathcal{E}{\to}\mathcal{T}^{\dagger}$	0.739	0.734

TABLE V: Cross-modal retrieval on VIVID, Axis (i) (R@1@25 m, global NN). ∗ Requires test co-observations. † Cross-sequence.

Pair	Encoders	Overfitting ∗	Ours
Cross-sequence (day2 $\to$ day1):
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv3	0.859	0.909
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / PCE	0.711	0.677
$\mathcal{L}{\to}\mathcal{R}$	MinkLoc † / DINOv3	0.905	0.946
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / MinkLoc †	0.739	0.736
$\mathcal{E}{\to}\mathcal{R}$	Event DINOv3 / DINOv3	0.778	0.735

TABLE VI: Cross-representation retrieval on VIVID Campus, Axes (iii) & (iv) (R@1@25 m, global NN). ∗ Requires test co-observations. † MinkLoc3Dv2 (256-D; separate PCA).

Pair	Encoders	Overfitting ∗	MLP ×	Ours
Cross-representation (Axis iii):
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv3	0.963	0.013	0.945
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / PCE	0.868	0.007	0.966
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv2-ViT-L	0.965	0.002	0.965
$\mathcal{R}{\to}\mathcal{L}$	DINOv2-ViT-L / PCE	0.922	0.017	0.960
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv2-ViT-B	0.960	0.018	0.969

TABLE VII: Encoder-pair sweep on KITTI-360 (R@1@25 m, global NN). ∗ Requires test co-observations. × MLP trained on VIVID, transferred to KITTI.

Pair	Encoders	Overfitting ∗	MLP ×	Ours
Cross-representation (Axis iii):
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv3	0.611	0.007	0.758
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / PCE	0.612	0.007	0.856
$\mathcal{L}{\to}\mathcal{R}$	MinkLoc † / DINOv3	0.762	0.005	0.839
$\mathcal{R}{\to}\mathcal{L}$	DINOv3 / MinkLoc †	0.686	0.005	0.821
$\mathcal{L}{\to}\mathcal{R}$	PCE / DINOv2-ViT-L	0.695	0.005	0.795

TABLE VIII: Cross-sequence retrieval on VIVID City (day2 query, day1 database; R@1@25 m, global NN). ∗ Requires test co-observations. × MLP trained on KITTI, transferred to VIVID City. † MinkLoc3Dv2 (256-D; separate PCA).

Approach	Inlier( $<$ 25 m)	Reason of failure
Diff. Procrustes adapter	0.07–0.09	SVD backward ill-cond.
Rotation prediction MLP	0.15–0.20	White-noise $\boldsymbol{\Omega}$
$\boldsymbol{R}$ -consistency reg.	0.076	Non-stationary process
Cross-traversal	0.418	Backbone repeatability

TABLE IX: Failed approaches on KITTI-360 (Inlier(

<

25 m)). Each method attempts to learn or predict

\boldsymbol{R}(x)

during training.

Dataset	Open-loop	Iso.	Aniso.
KITTI-360 (3.3 km)	20.8	3.0	0.9
VIVID (3.7 km)	22.8	2.1	1.3
HeLiPR DCC04 (5.4 km)	25.6	5.2	3.9
HeLiPR KAIST04 (5.8 km)	28.0	5.4	2.1

TABLE X: Cross-modal odometry (median position error, meters).

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

Aug 3, 2026

CalibBEV: LiDAR-Camera Calibration via BEV Alignment

We present CalibBEV, a novel Bird's Eye View (BEV) alignment approach for LiDAR-camera calibration. Our method unifies LiDAR and camera data into a shared 3D spatial representation, enabling accurate and robust cross-modal calibration. CalibBEV extracts sensor-wise BEV features from each modality using domain-specific architectures and estimates the calibration matrix through a two-step alignment process. First, we perform an implicit alignment by regressing a coarse calibration matrix directly from the BEV features. To ease this alignment, we enforce semantic consistency between BEV representations across modalities using a contrastive loss inspired by CLIP, guiding both networks toward a unified feature space. In the second step, we leverage our BEV formulation to explicitly align the features of one modality with the other, refining the initial coarse estimate into a final, more accurate calibration matrix. CalibBEV significantly outperforms prior point-to-pixel matching methods, achieving state-of-the-art calibration accuracy. On the KITTI and nuScenes benchmarks, our method reduces the Relative Rotation Error (RRE) by 51% and 68%, and the Relative Translation Error (RTE) by 80% and 91%, respectively, compared to previous methods.

Filippo D'Addeo, Lorenzo Cipelli, Adriano Cardace +3

University of Bologna, Department of Industrial Engineering, Italy. · University of Parma, Department of Engineering and Architecture, Italy. · Stanford University, USA. +1

Cross-Modal Alignment Camera-LiDAR Fusion Lidar-Camera Extrinsic Calibration

Ask PDF

Jun 24, 2026

Calousel: Extrinsic Calibration of Non-overlapping Multi-camera Systems from Pure Rotation

Extrinsic calibration of multi-camera systems with non-overlapping FOVs has been a challenging problem in the robotics literature. Conventional target-based methods impose substantial target setup overhead, either deploying large calibration targets or requiring pre-measured multi-target poses. Motion-based approaches instead suffer from drift error, scale ambiguity, and motion degeneracy. Securing both accuracy and usability, we propose a novel calibration method that leverages pure rotational motion, requiring only a single static calibration board. The key idea is to make all cameras sequentially observe the same target under a shared geometric reference, even without overlapping views. To integrate these time-separated observations, we formulate the problem using a latent turntable frame and a 3D error on SE(3) within a global optimization framework. We validate the proposed method on both a controlled camera rig and a full-scale vehicle platform with heterogeneous cameras, and analyze robustness under non-ideal turntable motion. Extensive experiments show that our approach maintains competitive accuracy without specialized precision hardware, proving its strong suitability for realistic on-site deployments. Our code is publicly available here.

Gwanhyeong Song, Chaehyeon Song, Ayoung Kim

Dept. of Mechanical Engineering, SNU, Seoul, S. Korea

Multi-View Geometry Camera Pose Estimation Camera Calibration

Ask PDF

Oct 1, 2026

PAGER: Partial-to-global Alignment via Geometric and Relational Distillation

Pretrained 3D encoders are typically developed on globally reconstructed scenes expressed in a consistent world coordinate frame, whereas embodied systems must reason from partial, viewpoint-dependent observations in camera coordinates. We show that this shift from globally learned 3D feature spaces to realistic partial observations exposes a severe representation mismatch, which we find consistently across representative state-of-the-art encoders, including Sonata and Concerto. A frozen Sonata encoder with a global linear probe achieves 72.47 mIoU on full ScanNet scenes, but 2.57 mIoU on single-frame camera-coordinate inputs. Training-free gravity alignment recovers performance to 41.64 mIoU, showing that coordinate-frame mismatch is a dominant source of degradation but cannot be fully resolved through canonicalization alone. We introduce PAGER, a label-free adaptation method that aligns partial-view features with a frozen global 3D semantic space using only paired partial/global geometry. It learns lightweight adaptation modules while keeping the pretrained encoder and global segmentation probe frozen. Matched-point feature alignment anchors partial features to their global counterparts, while relational supervision preserves their similarity structure with respect to the global representation. Global geometry provides supervision only during training. Inference operates directly on the partial observation. Without partial-view labels, PAGER outperforms label-supervised PEFT on both Sonata and Concerto, and in zero-shot ScanNet

\rightarrow

ScanNet++ transfer surpasses fully fine-tuned Sonata (

53.93

vs.\

48.09

mIoU), suggesting that preserving the frozen global representation can improve cross-dataset transfer.

Akira-Miranda Adeyomi Adeniran-Lowe, Binod Singh, Lars Arnold Dethlefsen +2

Technical University of Denmark · Pioneer Center for Artificial Intelligence

3D Semantic Segmentation 3D Representation Learning Knowledge Distillation

Ask PDF

Abstract

Cross-Modal Alignment

Explore similar work

CalibBEV: LiDAR-Camera Calibration via BEV Alignment

Calousel: Extrinsic Calibration of Non-overlapping Multi-camera Systems from Pure Rotation

PAGER: Partial-to-global Alignment via Geometric and Relational Distillation