TartanMatch: Towards Universal Dense Correspondence Across Modalities

Hyeokjoon Kwon; Jiting Cai; Ruogu Li; Kritan Bhandari; Geethika Hemkumar; Parv Maheshwari; Yuheng Qiu; Yuchen Zhang

cs.CVOct 10, 2026

TartanMatch: Towards Universal Dense Correspondence Across Modalities

Authors: Hyeokjoon Kwon, Jiting Cai, Ruogu Li, Kritan Bhandari, Geethika Hemkumar, Parv Maheshwari, Yuheng Qiu, Yuchen Zhang, +2 more

Organizations: The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA · Kritan Bhandari is with Fisk University, Nashville, TN, USA

Abstract

Robotic platforms combine multi-modal sensors to operate across diverse conditions, but most dense correspondence methods are limited to RGB-RGB or a small set of modality pairs. Training a model for each modality pair scales poorly with the number of sensors and forgoes potential positive transfer across modality pairs. We introduce TartanMatch, a unified model for dense correspondence across all 25 ordered pairs among RGB, event, thermal, depth, and projected LiDAR. TartanMatch adapts an RGB-pretrained correspondence model to all modality pairs within one shared network using lightweight modality adaptation and a mixture of real and synthetic data. On zero-shot real-world benchmarks, TartanMatch reduces Endpoint Error (EPE) over the strongest baselines by 61.9% for cross-modal matching and 49.6% for same-modal matching. We further show that joint training across all modality pairs improves average matching accuracy over pair-specific training, especially on challenging cross-modal pairs. These results demonstrate that a single model achieves broad multi-modal coverage and state-of-the-art matching accuracy.

Figures & tables

Fig. 2: TartanMatch architecture. A modality-specific adaptation maps every sensor to the three-channel input the pretrained backbone expects: RGB and thermal enter directly, events are voxelized into $B{=}15$ temporal bins and mapped by a learned $1{\times}1$ convolution, and depth and projected LiDAR share a log-depth-plus-validity representation and a single learned $1{\times}1$ convolution. The UFM correspondence backbone (DINOv2 encoder, information-sharing transformer, DPT heads) is fully shared across all modality pairs.

Fig. 3: Ground-truth flow refinement on MTV. Each row warps the source onto the target using the flow label before and after refinement, for a thermal-to-thermal pair (top) and an RGB-to-thermal pair (bottom); insets magnify the outlined region. The misalignment left by the geometry-derived flow disappears after refinement.

Training mixture
Dataset	OF/WB	Source	Modalities	Mix ratio
TartanAirV2 [ 22 ]	OF	Synthetic	REDTL	36.3%
MegaDepth [ 25 ]	WB	Both	RDT	20.2%
DSEC [ 2 ]	OF	Real	RE	10.9%
TartanAirV2 WB [ 22 , 9 ]	WB	Synthetic	RT	9.7%
BlendedMVS [ 26 ]	WB	Synthetic	RD	4.3%

TABLE I: Datasets in the TartanMatch training mixture and the zero-shot evaluation benchmarks.

Dense matching
Method	RGB-anchored cross-modal pairs										Non-RGB cross-modal pairs								Inference Time
	RGB $\to$ Depth		RGB $\to$ Event		RGB $\to$ Thermal				RGB $\to$ LiDAR		Thermal $\to$ Event		Event $\to$ LiDAR		Thermal $\to$ LiDAR		Depth $\to$ Thermal
	DTU		DSERT-RoLL		MTV		DSERT-RoLL		DSERT-RoLL		DSERT-RoLL		DSERT-RoLL		DSERT-RoLL		MTV
	EPE $\downarrow$	5px $\downarrow$	EPE $\downarrow$	5px $\downarrow$	EPE $\downarrow$	5px $\downarrow$	EPE $\downarrow$	5px $\downarrow$	EPE $\downarrow$	5px $\downarrow$	EPE $\downarrow$	5px $\downarrow$	EPE $\downarrow$	5px $\downarrow$	EPE $\downarrow$	5px $\downarrow$	EPE $\downarrow$	5px $\downarrow$	ms $\downarrow$
MINIMA(RoMa) [ 15 ]	36.56	67.2	13.77	62.9	8.61	15.0	9.57	53.2	73.02	96.5	13.58	63.4	57.39	95.7	63.77	96.7	24.52	40.2	262.4
MatchAnything(RoMa) [ 16 ]	37.38	77.9	11.61	44.4	3.28	14.5	11.93	55.8	52.70	92.7	15.23	65.3	30.48	91.9	42.88	94.2	23.80	46.1	206.4

TABLE II: Cross-modal EPE (px),

>

5px outlier rate, and inference time on one H100 GPU

Table 5

Fig. 4: TartanMatch produces more accurate flow and covisibility than MINIMA on various pairs.

Dense matching
Method	RGB $\to$ RGB				Depth $\to$ Depth		Event $\to$ Event						Thermal $\to$ Thermal				LiDAR $\to$ LiDAR		Inference Time
	DTU		DSERT-RoLL		DTU		MVSEC 20Hz		MVSEC 45Hz		DSERT-RoLL		MTV		DSERT-RoLL		DSERT-RoLL		Inference Time
	EPE $\downarrow$	2px $\downarrow$	EPE $\downarrow$	2px $\downarrow$	EPE $\downarrow$	2px $\downarrow$	EPE $\downarrow$	2px $\downarrow$	EPE $\downarrow$	2px $\downarrow$	EPE $\downarrow$	2px $\downarrow$	EPE $\downarrow$	2px $\downarrow$	EPE $\downarrow$	2px $\downarrow$	EPE $\downarrow$	2px $\downarrow$	ms $\downarrow$
MINIMA(RoMa) [ 15 ]	12.28	30.9	4.09	29.6	29.86	73.9	2.06	32.6	1.92	30.4	4.80	44.8	1.07	4.3	6.52	43.3	13.21	81.8	260.2
MatchAnything(RoMa) [ 16 ]	12.49	32.3	3.95	27.4	26.54	71.9	1.84	31.4	1.45	23.2	4.80	44.4	0.56	3.6	6.83	43.7	13.78	82.1	206.9
RoMa [ 8 ]	11.52	29.5	4.04	28.0	37.26	80.7	1.33	21.7	0.96	10.4	6.19	47.2	1.32	4.6	6.96	43.7	13.01	78.4	191.3

TABLE V: Same-modal EPE (px),

>

2px outlier rate, and inference time on one H100 GPU

(a) Event representation
		EPE $\downarrow$
Setting	Test set	Cross-modality	Same-modality
Polarity-accumulated color image	DSERT-RoLL	10.56	3.44
Voxel volume (Ours)	DSERT-RoLL	7.46	2.46
(b) Depth representation
Color-mapped image	DTU	12.30	6.30

TABLE VI: Ablation studies. (a) trains on all event datasets; (b) trains on all depth datasets; (c) uses the full training mixture.

Pair	Single-pair	All-mod.	$\Delta$ (%)
Same-modality
R $\rightarrow$ R	0.82	1.00	$-22.0$
D $\rightarrow$ D	0.73	0.90	$-23.3$
E $\rightarrow$ E	1.29	1.22	$+5.4$
T $\rightarrow$ T	3.14	3.24	$-3.2$
L $\rightarrow$ L	7.84	5.00	$+36.2$

TABLE VII: EPE on the TartanAirV2. Single-modality-pair models use their best checkpoint per pair; the all-modality model is a single checkpoint.

\Delta

is the relative improvement of all-modality over single-pair.

Explore similar work

UniCorrn: Unified Correspondence Transformer Across 2D and 3D
May 5, 2026Prajnan Goswami, Tianye Ding, Feng Liu +1Image Matching Point Cloud Registration
Northeastern University · Adobe Research
TeaMatch: Teachable Cross-Modal Representation Learning for 2D-3D Matching
Aug 10, 2026Chongjian Wang, Junjie GaoCross-Modal Representation Learning Image Matching
Shandong Women’s University · Jinan, China · Shandong University of Science and Technology +1
SURE: Semi-dense Uncertainty-REfined Feature Matching
Mar 5, 2026Sicheng Li, Zaiwang Gu, Jie Zhang +3Image Matching
S. Li and X. Jiang are with the School of Electrical and Electronic Engineering, Nanyang Technological University (NTU), Singapore. · J. Zhang and S. Li are with Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore. · J. Cheng and Z. Gu are with the Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), Singapore. +1

May 5, 2026J/K move · Enter open · S save

UniCorrn: Unified Correspondence Transformer Across 2D and 3D

Prajnan Goswami, Tianye Ding, Feng Liu, Huaizu Jiang

Northeastern University · Adobe Research

Image Matching Point Cloud Registration Cross-Modal Feature Matching

Open paper AskPDF arXiv

Visual correspondence across image-to-image (2D-2D), image-to-point cloud (2D-3D), and point cloud-to-point cloud (3D-3D) geometric matching forms the foundation for numerous 3D vision tasks. Despite sharing a similar problem structure, current methods use task-specific designs with separate models for each modality combination. We present UniCorrn, the first correspondence model with shared weights that unifies geometric matching across all three tasks. Our key insight is that Transformer attention naturally captures cross-modal feature similarity. We propose a dual-stream decoder that maintains separate appearance and positional feature streams. This design enables end-to-end learning through stack-able layers while supporting flexible query-based correspondence estimation across heterogeneous modalities. Our architecture employs modality-specific backbones followed by shared encoder and decoder components, trained jointly on diverse data combining pseudo point clouds from depth maps with real 3D correspondence annotations. UniCorrn achieves competitive performance on 2D-2D matching and surpasses prior state-of-the-art by 8% on 7Scenes (2D-3D) and 10% on 3DLoMatch (3D-3D) in registration recall. Project website: https://neu-vi.github.io/UniCorrn

Abstract

Explore similar work

UniCorrn: Unified Correspondence Transformer Across 2D and 3D

TeaMatch: Teachable Cross-Modal Representation Learning for 2D-3D Matching

SURE: Semi-dense Uncertainty-REfined Feature Matching