cs.CVMar 2, 2026

Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth

Authors: Andrew Wang, Mike Davies

Organizations: School of Engineering, University of Edinburgh

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multispectral demosaicing is crucial to reconstruct full-resolution spectral images from snapshot mosaiced measurements, enabling real-time imaging from neurosurgery to autonomous driving. Classical methods are blurry, while supervised learning requires costly ground truth (GT) obtained from slow line-scanning systems. We propose Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), a framework that learns multispectral demosaicing from mosaiced measurements alone. PEFD a) exploits the projective geometry of camera-based imaging systems to leverage a richer group structure than previous demosaicing methods to recover more null-space information, and b) learns efficiently without GT by adapting pretrained foundation models designed for 1-3 channel imaging. On surgical and automotive datasets, PEFD recovers fine details such as blood vessels and preserves spectral fidelity, substantially outperforming recent approaches, nearing supervised performance. Furthermore, the performance of PEFD is demonstrated on raw, unprocessed data from a commercial multispectral sensor. Code is at https://github.com/Andrewwango/pefd.

Figures & tables

Figure 1: Perspective-Equivariant Fine-tuning for Demosaicing (PEFD) recovers sharp multispectral images with correct colours without ground truth, compared to popular classical methods such as weighted bilinear demosaicing or learning from scratch.
Figure 2: Given a camera centre CC , camera systems in intraoperative (left) and automotive (right) imaging rotate freely about their axes, producing images related by perspective transformations.
Figure 3: Framework for perspective-equivariance fine-tuning for multispectral demosaicing, where fθ{f_{\theta}} is our adapted pretrained foundation model [ 47 ] to process multispectral data, y{\mathbf{y}} are the mosaiced measurements, A{\mathbf{A}} is the mosaicing operator, Tg{\mathbf{T}_{g}} is the parametrised perspective transformation and the loss depicts Eq. 3 .
Figure 4: Test set false-RGB reconstruction results on 3 example neurosurgical intraoperative images from HELICoiD [ 14 ] , showing mosaiced input y{\mathbf{y}} , reconstruction results for 9 comparison methods, our reconstruction with PEFD, and ground truth for reference.
Figure 5: Example spectral signatures from a sample image patch in each dataset (left: HELICoiD; right: HyKo).
Figure 6: Test set false-RGB reconstruction results on 3 example urban-driving images from the HyKo dataset [ 52 ] , showing mosaiced input y{\mathbf{y}} , reconstruction results for 9 comparison methods, our reconstruction with PEFD, and ground truth for reference.
Figure 7: Test-set false-RGB results of demosaicing raw imec CMV2K-SSM4x4-VIS sensor data captured during oral imaging [ 6 ] , centre-cropped to a 768×768768\times 768 ROI, showing sensor data y{\mathbf{y}} , interpolation and our method. Note no GT is available.
Table 1: Quantitative results on the HELICoiD [ 14 ] and HyKo [ 52 ] datasets. Best results in bold .
Table 2: Ablation study on the HELICoiD [ 14 ] and HyKo [ 52 ] datasets. Best self-supervised result in bold .
Figure 8: Ablation study on 3 example neurosurgical images from HELICoiD [ 14 ] (left) and 3 example urban-driving images from the HyKo dataset [ 52 ] (right).

Explore similar work

CardsList

  1. SegDem: Segmentation helps Demosaicing

    Aug 8, 2026Ping Chen, Xiangming Wang, Yongyong Chen +5Image ReconstructionTransfer Learning

    Harbin Institute of Technology, Shenzhen · Shanghai Jiao Tong University · Nanjing University (Suzhou)

  2. PolarAPP: Beyond Polarization Demosaicking for Polarimetric Applications

    Mar 24, 2026Yidong Luo, Chenggong Li, Yunfeng Song +4Multi-Task LearningPolarimetric Imaging

    Zhejiang University, Hangzhou, China · School of Automation, Central South University, Changsha, China · School of Engineering, Westlake University, Hangzhou, China +1

  3. Structural Guidance for Unified Joint Demosaicing and Denoising

    Aug 7, 2026Qixin Zheng, Ping Chen, Qiangqiang Shen +1Computational ImagingImage Restoration

    Harbin Institute of Technology (Shenzhen), Shenzhen, China