S3D-Mosaic: Subgradient-Guided Cross-Band Structure Transfer for Spectral Demosaicing

Weiqiang Zhao; Xian Li; Yuzhe Gui; Tianzhu Liu; Yanfeng Gu

cs.CVOct 17, 2026

S3D-Mosaic: Subgradient-Guided Cross-Band Structure Transfer for Spectral Demosaicing

Authors: Weiqiang Zhao, Xian Li, Yuzhe Gui, Tianzhu Liu, Yanfeng Gu

Organizations: School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin 150001, China

Abstract

Multispectral filter array (MSFA) imaging enables snapshot spectral acquisition, but its per-band sparse sampling causes severe spatial aliasing. Although missing spatial structures can be inferred across bands, wavelength-dependent contrast variations make direct cross-band transfer unreliable. Existing model-based methods lack explicit magnitude-invariant mechanisms, and learning-based methods require layout-specific training or per-image tuning, limiting generalization. To address these issues, we propose S3D-Mosaic, a model-based spectral demosaicing method built upon a subgradient-guided cross-band structure transfer framework. We first establish a subgradient-guided regularization framework with a generalized cosine-similarity interpretation, providing a convex mechanism for magnitude-invariant information transfer. Then we design a compact Directional Sparse Equivariant Difference Convolution (DSE-DConv) for band-level extraction of orientation-selective, high-frequency structure features at low computational cost. Instantiating the framework with DSE-DConv on spectrally adjacent bands yields a cross-band self-structure similarity regularizer for spectral demosaicing, which we optimize via a dual-scan ADMM algorithm that decouples the bidirectional dependency while propagating magnitude-robust structural information across bands. With DSE-DConv's two parameters trained once via self-supervision, S3D-Mosaic outperforms state-of-the-art unsupervised methods on CAVE and NTIRE 2022, and generalizes robustly to varying MSFA layouts, pan-mosaic fusion, and real-world UAV data with fixed reconstruction hyperparameters. The source code is available at https://github.com/bestwishes43/S3D-Mosaic.

Figures & tables

Fig. 1: Illustration of the proposed S3D-Mosaic pipeline: (i) a subgradient-guided optimization framework is developed to realize magnitude-invariant cross-band feature-direction transfer; (ii) the compact DSE-DConv operator is designed and self-supervisedly trained as a structural feature extractor; (iii) a cross-band self-structure similarity prior is instantiated for spectral demosaicing and optimized using a dual-scan ADMM algorithm.

Fig. 2: Illustration of the MSFA observation. Left: actual mosaic observation model from hyperspectral image (HSI) to raw; Middle: equivalent per-band subsampling model; Right: spatial aliasing caused by sparse sampling.

Fig. 3: Illustration of the DSE-DConv module and its adjoint operator. The right part shows the speed-optimized implementation with only $2HW$ multiplications.

Fig. 4: Visualization of feature direction similarity using DSE-DConv. a) Inter-band similarity; b) neighboring-band similarity. The visualization is based on scene 0901 of the NTIRE 2022 spectral demosaicing dataset.

Parameter	S3D-Mosaic	S3D-Mosaic ∗
ADMM iterations $T$	100	100
Initial penalty $\rho_{0}$	$1\times 10^{-3}$	$1\times 10^{-3}$
Initial structure weight $\mu_{0}$	$1\times 10^{-3}$	$1\times 10^{-3}$
Penalty growth factor $\gamma_{\rho}$	1.8	1.8
$\rho$ update interval	10	10
Z-subproblem FPI iterations $K_{z}$	45	45

TABLE I: Hyperparameter settings for the proposed methods. All parameters are fixed across datasets.

Metrics	Methods	balloons	beads	cd	chart	clay	cloth	bears	feathers	flowers	oil	AVG
PSNR ↑	WB	36.75	24.93	35.70	27.25	36.83	27.18	32.96	28.94	34.73	30.63	31.59
	ItSD	37.12	23.37	32.55	30.55	34.69	31.53	36.17	31.46	35.26	32.34	32.50
	PPID	38.89	26.24	36.60	29.00	38.34	28.54	35.61	31.00	37.12	31.05	33.24
	USD (TIP, 2024)	40.93	27.22	36.64	31.29	39.87	30.93	38.74	32.64	38.76	32.47	34.95
	UnNull (TIP, 2025)	43.66	31.28	36.41	34.93	42.89	34.44	41.63	37.38	41.01	34.55	37.82
	S3D-Mosaic	46.61	31.09	37.71	37.73	45.72	36.02	43.98	38.80	45.04	36.05	39.88

TABLE II: Quantitative comparison on the CAVE spectral demosaicing dataset. Best and second best are bold and underlined , respectively

Metrics	UnNull (TIP, 2025)	WB	ItSD	PPID	USD (TIP, 2024)	S3D-Mosaic	S3D-Mosaic*
PSNR ↑	29.68±4.10	35.00±3.54	34.96±3.37	37.71±3.53	43.39±3.58	43.97 ±3.91	44.75 ±4.10
SAM ↓	9.455±0.854	3.800±1.404	4.413±1.417	3.263±1.339	2.223±0.960	1.876 ±0.867	1.722 ±0.819
SSIM ↑	0.850±0.082	0.922±0.060	0.931±0.030	0.953±0.037	0.981±0.014	0.985 ±0.010	0.987 ±0.010

TABLE III: Quantitative comparison on the NTIRE 2022 spectral demosaicing dataset. Best and second best are bold and underlined , respectively.

Metrics	Methods	beans	oil	paints	photo	pompoms	apples	peppers	sponges	toys	superballs	spools	watercolors	AVG
PSNR ↑	PPID-PanGAN	32.37	34.90	32.22	38.66	33.81	39.34	38.76	31.19	33.46	35.36	35.38	32.77	34.85
	LSAN-WFANet	30.16	33.05	30.89	38.04	36.32	41.25	39.23	33.70	35.88	37.79	37.48	29.88	35.31
	LSAN-PanGAN	32.47	35.44	33.01	39.73	37.87	43.06	41.15	34.01	36.98	39.02	39.86	32.70	37.11
	EFN (TIP, 2026)	36.98	37.67	37.56	41.66	39.25	47.09	46.30	38.87	39.52	42.48	43.26	36.05	40.56
	EBVIF (TPAMI, 2026)	37.56	38.59	36.47	41.83	41.88	45.79	45.47	39.25	41.23	41.53	43.39	38.14	40.93
	S3D-Mosaic	39.84	39.81	39.74	42.51	42.84	50.08	49.91	42.33	44.16	46.04	45.50	40.39	43.60

TABLE IV: Quantitative comparison on the CAVE pan-mosaic fusion dataset. Best and second best are bold and underlined , respectively

LR	Feat.	Sub.	Initial	PSNR $\uparrow$	SAM $\downarrow$	SSIM $\uparrow$
			HS	16.43	76.398	0.115
✓			HS	16.43	76.398	0.115
	✓		HS	34.56	2.987	0.897
✓	✓		HS	34.64	2.968	0.898
	✓	✓	HS	43.97	1.876	0.985
✓	✓	✓	HS	44.75	1.722	0.987

TABLE V: Ablation of prior components under different initializations on the NTIRE 2022 validation set.

CD	DS	GE	PSNR $\uparrow$	SAM $\downarrow$	SSIM $\uparrow$	Mult. Comp.
			34.88	7.186	0.927	$72\,HW$
✓			22.02	21.883	0.634	$64\,HW$
	✓		32.12	10.274	0.880	$24\,HW$
		✓	25.48	19.971	0.645	$9\,HW$
✓	✓		39.70	4.692	0.972	$16\,HW$
✓		✓	21.59	23.885	0.594	$8\,HW$

TABLE VI: Ablation study on the central difference (CD), directional sparsity (DS), and group equivariance (GE) properties. Training is performed on NTIRE 2022 spectral demosaicing and testing on CAVE.

Fig. 5: False-color (R:570 nm, G:540 nm, B:454 nm) visual comparison of selected scenes on the real dataset. For each scene, the left column shows a zoomed region indicated in the raw image, and the right column displays the channel-averaged spatial frequency spectrum of the reconstructed scenes.

Fig. 6: The MSFA layout of the real dataset.

Methods	S01	S02	S03	S04	S05	S06
WB	7.143	6.944	7.978	8.032	7.562	7.826
ItSD	6.321	6.827	7.634	7.439	6.907	6.130
PPID	6.141	5.733	6.440	6.144	5.944	4.725
UnNull	6.257	6.154	6.368	5.525	4.804	4.531
S3D-Mosaic	4.378	4.725	4.941	4.816	4.289	3.834

TABLE VII: Quantitative comparison on the real dataset.

Demosaicing

See all 11 papers

Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth
Mar 2, 2026Andrew Wang, Mike DaviesDemosaicing Computational Imaging
School of Engineering, University of Edinburgh
SegDem: Segmentation helps Demosaicing
Aug 8, 2026Ping Chen, Xiangming Wang, Yongyong Chen +5Image Reconstruction Transfer Learning
Harbin Institute of Technology, Shenzhen · Shanghai Jiao Tong University · Nanjing University (Suzhou)
Structural Guidance for Unified Joint Demosaicing and Denoising
Aug 7, 2026Qixin Zheng, Ping Chen, Qiangqiang Shen +1Computational Imaging Image Restoration
Harbin Institute of Technology (Shenzhen), Shenzhen, China

Mar 2, 2026J/K move · Enter open · S save

Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth

Andrew Wang, Mike Davies

School of Engineering, University of Edinburgh

Demosaicing Computational Imaging Image Restoration Hyperspectral Imaging Equivariant Neural Networks

Open paper AskPDF arXiv

Multispectral demosaicing is crucial to reconstruct full-resolution spectral images from snapshot mosaiced measurements, enabling real-time imaging from neurosurgery to autonomous driving. Classical methods are blurry, while supervised learning requires costly ground truth (GT) obtained from slow line-scanning systems. We propose Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), a framework that learns multispectral demosaicing from mosaiced measurements alone. PEFD a) exploits the projective geometry of camera-based imaging systems to leverage a richer group structure than previous demosaicing methods to recover more null-space information, and b) learns efficiently without GT by adapting pretrained foundation models designed for 1-3 channel imaging. On surgical and automotive datasets, PEFD recovers fine details such as blood vessels and preserves spectral fidelity, substantially outperforming recent approaches, nearing supervised performance. Furthermore, the performance of PEFD is demonstrated on raw, unprocessed data from a commercial multispectral sensor. Code is at https://github.com/Andrewwango/pefd.

Abstract

Demosaicing

Explore similar work

Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth

SegDem: Segmentation helps Demosaicing

Structural Guidance for Unified Joint Demosaicing and Denoising