Cross-Spectral Dense Correspondence for Multimodal Spectral Medical Imaging

Eric L. Wisotzky; Jost Triller; Simon W. Härtl; Oliver T. Bruns; Peter Eisert; Anna Hilsmann

cs.CVAug 28, 2026

Cross-Spectral Dense Correspondence for Multimodal Spectral Medical Imaging

Authors: Eric L. Wisotzky, Jost Triller, Simon W. Härtl, Oliver T. Bruns, Peter Eisert, Anna Hilsmann

Organizations: Vision & Imaging Technologies, Fraunhofer Heinrich Hertz Institute HHI, Berlin, Germany · Visual Computing, Humboldt University Berlin, Germany · Functional Imaging in Surgical Oncology, National Center for Tumordiseases, Dresden, Germany

Abstract

Precise dense correspondence is a fundamental prerequisite for multimodal spectral imaging systems that fuse disparate wavelength ranges for subsequent analysis in medical and scientific imaging. Corresponding image points are often observed with non-overlapping spectral sensitivities, leading to wavelength-dependent contrast changes, intensity inversions, and appearance shifts for which dense ground truth is difficult to obtain and conventional RGB-based training data provides only limited supervision. We address this data gap by introducing a sensor-agnostic cross-spectral modulation protocol on established correspondence benchmarks with intensity input projection, and by proposing a synthetic cross-spectral correspondence benchmark simulating physically plausible radiometric differences. Evaluation on several modern dense correspondence backbones trained with our unified cross-spectral protocol showed substantial improvements under severe spectral mismatch while maintaining performance on standard RGB benchmarks. Ablation experiments show that view-dependent channel selection and nonlinear radiometric transformations provide complementary robustness, indicating that the primary limitation of existing models is not their structural matching capacity but the mismatch between training distribution and spectral characteristics of the target image pair. Qualitative evaluations on heterogeneous medical spectral acquisition systems demonstrate the practical relevance of the proposed training data augmentation protocol as an enabler for spatially coherent spectral fusion in HSI workflows.

Figures & tables

	Chairs	Things	Things, Sintel, KITTI, HD1K
	(Stage 1)	(Stage 2)	(Stage 3)
Steps	120k	150k	200k
Learning Rate	2.5e-4	1.5e-4	1.5e-4
WDecay	1.0e-4	1.0e-5	1.0e-5
Gamma	0.8	0.8	0.85
Batch Size	8	8	8

Table 1: Training hyperparameters for our cross-spectral models.

Architecture	Variation	Sintel	Sintel	KITTI	Middlebury	ETH3D	InStereo2K
		clean	final
RAFT	original	0.139	0.153	0.904	0.296	0.403	7.183
RAFT	cross-spectral	0.158	0.177	1.022	0.308	0.395	5.984
DIP	original	0.121	0.161	1.762	0.260	0.381	5.941
DIP	cross-spectral	0.132	0.162	0.889	0.237	0.338	5.898
GMA	original	0.128	0.139	0.911	0.281	0.437	6.339

Table 2: All models are evaluated using images from original benchmark datasets (RGB for the original and intensity for the cross-spectral models). The reported metric is EPE with column-wise (i.e., per dataset) marked best and second best results.

Architecture	Variation	Chairs	Things	Things	Sintel	Sintel	KITTI	Middle-	ETH3D	InStereo2K
			clean	final	clean	final		bury
RAFT	original	15.619	28.581	30.268	2.473	2.870	17.797	14.947	7.193	29.145
RAFT	cross-spec	1.404	9.822	11.168	0.173	0.188	1.053	0.323	0.403	5.976
DIP	original	5.010	27.753	30.118	1.490	1.631	17.272	3.646	3.113	18.472
DIP	cross-spec	1.285	10.494	12.041	0.138	0.163	0.920	0.243	0.344	5.926
GMA	original	15.636	36.477	35.813	3.342	2.874	15.480	7.578	7.323	31.563

Table 3: The cross-spectral (cross-spec) and original models evaluated using modified evaluation images. The reported metric is EPE and best / second best result marked.

Architecture	Variation	Synth-Extreme	Synth-Perspective	Synth-Zero
RAFT	original	53.107	40.682	16.423
RAFT	cross-spectral	7.389	6.054	0.108
DIP	original	51.521	36.628	13.214
DIP	cross-spectral	8.491	7.757	0.015
GMA	original	67.632	53.030	27.697
GMA	cross-spectral	9.305	7.097	0.076

Table 4: Model comparison on Synth dataset regimes (preprocessed inputs). The best and second best result for each displacement regime (column) is marked.

Architec- ture	Variation	LWIR	LWIR	MWIR	NIR	NIR	NIR	SWIR	SWIR	UV
		vs.	vs.	vs.	vs.	vs.	vs.	vs.	vs.	vs.
		MWIR	SWIR	SWIR	SWIR	UV	VIS	UV	VIS	VIS
RAFT	original	18.635	33.585	39.942	21.584	21.501	14.284	28.007	25.393	17.512
RAFT	cross-spec	9.550	9.889	9.999	6.581	9.969	4.890	9.795	8.752	8.511
DIP	original	27.509	42.393	42.034	25.949	26.301	17.158	31.235	31.227	23.646
DIP	cross-spec	12.821	13.469	13.331	9.716	12.311	7.277	12.457	11.063	11.574

Table 5: Model comparison on Synth using different spectral camera pairings. The values are averaged over all three Synth displacement regimes. The reported metric is EPE (

\downarrow

), and the best / second best result for each wavelength-pairing are marked.

Training	Grayscale	Channel	Monotonic	Monotonic	EPE $\downarrow$
Configuration	Grayscale	Mixture	Increasing	Decreasing	EPE $\downarrow$
Standard RGB					47.5810
grayscale-same	$\checkmark$				32.0920
grayscale-different	$\checkmark$	$\checkmark$			16.0358
function-increase	$\checkmark$		$\checkmark$		25.9549
function-full	$\checkmark$		$\checkmark$	$\checkmark$	17.0292

Table 6: Ablation study of the cross-spectral training strategy for SEA-RAFT on the real-spectra Synth data. Results are averaged over all evaluated spectral pairings. The reported metric is EPE (

\downarrow

) and best / second best results are marked.

Input Variation		Identity	Permute	Monotonic	Monotonic	Mixed	Real-Spectral
Input Variation		Identity	Channels	Increasing	Decreasing	Monotonic	Synth
SEA-RAFT	original	1	0.9277	0.7813	0.7822	0.2540	0.3999
SEA-RAFT	cross-spectral	1	0.9864	0.9463	0.9384	0.9359	0.7537
RAFT	original	1	0.9553	0.8969	0.8793	0.1005	0.3581
RAFT	cross-spectral	1	0.9930	0.9279	0.9278	0.9223	0.5517
SKFlow	original	1	0.9443	0.8740	0.8723	0.1274	0.3688

Table 7: Cosine feature similarity across data processing variants and real-spectral Synth (last column) for different architectures (original and cross-spectral). The data modification by non-overlapping channel pairing is labeled as Permute , while the spectral response modulation using the function family is referred to as Monotonic Increase (no inversion), Monotonic Decrease (inversion), Mixed monotonic (random inversion).

Panel of Figure 1 — Figure 1: Different real-world data acquired with three different cross-spectral setups. All displacement maps are generated with the SEA-RAFT-based architecture (original and cross-spectral). The results of the other architectures can be found in the Supplementary Material. Top row: Surgical images acquired with an VIS-NIR MSI-stereo setup. Middle row: Tissue acquired with unrectified RGB-SWIR setup. Bottom row: Two surgical sub-views acquired with a HSI light-field setup. All displacement fields from the original SEA-RAFT architecture contain strong artifact or miss content, while the cross-spectral variant produces more spatially coherent fields with fewer obvious artifacts.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure A1 : Representative correspondence results for an RGB-SWIR image pair obtained with DUSt3R (left) and RoMa (right). Reliable matches are concentrated in locally distinctive image regions, while substantial parts of the scene remain without correspondence support.

Panel of Figure 10 — Figure A2 : Examples of a generated triplet of the own created synthetic cross-spectral dense correspondence dataset ( Synth-Extreme ). Left column: image $I_{L}$ Middle column: image $I_{R}$ Right column: displacement field $\mathbf{u}^{\prime}$ of the scene.

Panel of Figure 11 — Figure A3 : Examples of real RGB-SWIR radiometric relationships that are approximated by our functions. $1^{st}$ and $3^{rd}$ images show original SWIR image, while the $2^{nd}$ and $4^{th}$ images are the corresponding approximation using red and blue channel of the RGB capture, respectively.

Panel of Figure 12 — Figure A4 : A t-SNE visualization of a randomly chosen scene from the RGB-SWIR dataset. While the features show different clustering per view in the original model (a), all features are highly similar over different spectral modalities for the cross-spectral model (b) enabling better correspondence learning.

Panel of Figure 13 — Figure A5 : The qualitative results, i.e., displacement maps of the MSI stereo setup of the other utilized architectures of the same intraoperative scene as in Fig. 1 .

Panel of Figure 14 — Figure A6 : The qualitative results, i.e., displacement maps of the RGB-SWIR setup of the other utilized architectures of the same biomedical scene as in Fig. 1 .

Panel of Figure 15 — Figure A7 : The qualitative results, i.e., displacement maps of the HSI light-field setup of the other utilized architectures of the same surgical scene as in Fig. 1 .

Explore similar work

Bidirectional Cross-Attention Fusion of High-Resolution RGB and Low-Resolution Hyperspectral Inputs for Multimodal Semantic Segmentation
Mar 14, 2026Jonas V. Funk, Lukas Roming, Andreas Michel +4Semantic Segmentation Multimodal Fusion
KIT, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany · Fraunhofer IOSB, Fraunhofer Institute of Optronics, System Technologies and Image Exploitation, 76131 Karlsruhe, Germany
TartanMatch: Towards Universal Dense Correspondence Across Modalities
Oct 10, 2026Hyeokjoon Kwon, Jiting Cai, Ruogu Li +7Cross-Modal Feature Matching Unified Multimodal Models
The authors are with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA · Kritan Bhandari is with Fisk University, Nashville, TN, USA
SpectraDINO: Bridging the Spectral Gap in Vision Foundation Models via Lightweight Adapters
May 4, 2026Yagiz Nalcakan, Hyeongjin Ju, Incheol Park +3Cross-Modal Knowledge Distillation Multispectral Object Detection
Yonsei University, Seoul, Republic of Korea

Mar 14, 2026J/K move · Enter open · S save

Bidirectional Cross-Attention Fusion of High-Resolution RGB and Low-Resolution Hyperspectral Inputs for Multimodal Semantic Segmentation

Jonas V. Funk, Lukas Roming, Andreas Michel, Paul Bäcker+3

KIT, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany · Fraunhofer IOSB, Fraunhofer Institute of Optronics, System Technologies and Image Exploitation, 76131 Karlsruhe, Germany

Semantic Segmentation Multimodal Fusion Cross-Modal Attention

Open paper AskPDF arXiv

Multimodal semantic segmentation with heterogeneous sensors must reconcile complementary information across modalities that differ in spatial resolution and channel dimensionality. In particular, high-resolution RGB imaging provides detailed spatial structure but often fails to distinguish visually similar materials, whereas hyperspectral imaging (HSI) provides discriminative spectral signatures but at lower spatial resolution. We present Bidirectional Cross-Attention Fusion (BCAF), which aligns high-resolution RGB with low-resolution HSI at their native grids via localized, bidirectional cross-attention, avoiding pre-upsampling or early spectral collapse. BCAF uses two independent backbones: a standard Swin Transformer for RGB and an HSI-adapted Swin backbone that preserves spectral structure through 3D tokenization with spectral self-attention. Although our evaluation targets RGB-HSI fusion, BCAF is modality-agnostic and applies to co-registered RGB with lower-resolution, high-channel auxiliary sensors. On the benchmark SpectralWaste dataset, BCAF delivers strong performance, achieving 75.4% at 55 images/s. We further evaluate a novel industrial dataset: K3I-Cycling (first RGB subset already released on Fordatis). On this dataset, BCAF reaches 62.3% mIoU for material segmentation (paper, metal, plastic, etc.) and 66.2% mIoU for plastic-type segmentation (PET, PP, HDPE, LDPE, PS, etc.). These results show that preserving native-grid spatial detail and spectral structure improves multimodal segmentation under real-time constraints. Code and model checkpoints are publicly available at https://github.com/jonasvilhofunk/BCAF_2026.

Abstract

Explore similar work

Bidirectional Cross-Attention Fusion of High-Resolution RGB and Low-Resolution Hyperspectral Inputs for Multimodal Semantic Segmentation

TartanMatch: Towards Universal Dense Correspondence Across Modalities

SpectraDINO: Bridging the Spectral Gap in Vision Foundation Models via Lightweight Adapters