When Is the Expensive Method Worth It? A Content-Driven Analysis of Classical Bayer Demosaicing
Authors: Taranum Wasu, Harsh Kashyap, HS Pannu
Organizations: Research Student Thapar University, Patiala, India · Department of Computer Science Thapar Institute of Engineering and Technology Patiala, India
Demosaicing reconstructs a full-colour image from a single-sensor colour-filter-array (CFA) mosaic, where only one colour is measured per pixel. Classical methods make different assumptions about images, from simple averaging to directional edge-aware interpolation, and the directional methods cost markedly more compute. We ask a practical question that a per-image quality ranking does not answer: for a given image, is the expensive directional method worth its cost? Using a controlled simulate-reconstruct pipeline on 80 benchmark images from four public datasets (Kodak, McMaster, BSD100, Urban100), six classical methods, and image-content descriptors fixed before any results were seen, we show that the quality gain of the edge-aware method (Menon 2007) over the best cheap linear method is not constant. It ranges from −6.6 to +6.8 dB, and on 31% of images it is zero or negative, meaning a cheap method matches or beats the expensive one. The gain is also predictable from content: it correlates with mean image saturation at r=−0.86 (p<10−6). We then build an interpretable depth-2 decision tree that recommends the best cheap method from image descriptors, beating an always-pick-the-most-common baseline by 20.0 percentage points under leave-one-image-out validation. The analysis uses no deep learning, and every step can be inspected.
Figures & tables
Figure 1: Representative test images from each content category (Kodak and McMaster). The categories span the range of structure and colour that drives the per-image method comparison.
Parameter
Value
Datasets
Kodak + McMaster (IMAX) + BSD100 + Urban100
Total images
80
Kodak images
24 ( 768×512 )
McMaster images
18 ( 500×500 )
BSD100 images
20
Urban100 images
18
Table 1: Dataset Configuration
Figure 2: Genuine reconstructions of a repetitive/high-frequency crop (kodim19) by all six methods, with whole-image PSNR. Artifacts at the diagonal wall edge decrease from nearest-neighbour (coloured speckle, zippering) to the edge-aware method (sharpest, least false colour), matching the methods’ priors.
Method
PSNR
SSIM
ΔE
EdgePres.
Artifact
Nearest
25.36
0.899
5.67
0.910
28.55
Bilinear
28.80
0.944
3.83
0.960
17.49
Colour-diff.
29.03
0.955
3.47
0.974
15.70
Malvar
30.07
0.964
3.04
0.981
13.07
Alleysson
30.40
0.962
3.24
0.972
13.65
Edge-aware
31.78
0.966
2.49
0.986
10.48
Table 2: Mean Quality Over All 80 Images (clean)
Figure 3: The gain of the expensive directional method over the best cheap method is content-dependent and predictable from image saturation ( r=−0.86 , n=80 ). Below the dashed line, a cheap method is as good or better.
Descriptor
Pearson r
p
Saturation (mean)
−0.86
<10−6
Colour variance
−0.71
<10−6
Gradient magnitude (std)
+0.36
0.001
Saturation (std)
−0.42
<10−4
Table 3: Correlation of the “worth-it” gain with content descriptors ( n=80 )
Category
n
Mean gain (dB)
Repetitive / high-freq.
14
+1.57±1.40
Edge-heavy
15
+1.89±0.93
Smooth
15
+2.01±1.05
Fine texture
15
+1.32±0.79
High colour
21
−1.56±1.30
Table 4: “Worth-It” Gain by Content Category (mean ± 95% CI, dB)
Figure 4: Mean “worth-it” gain by content category (95% CI). The directional method earns its cost on structured images, not on colourful ones.
Metric
Bilinear
Colour-diff.
Malvar
Alleysson
PSNR
10
5
35
30
SSIM
3
10
50
17
ΔE
10
3
56
11
Edge pres.
0
18
61
1
Artifact
3
1
50
26
Table 5: Fast-Tier Wins per Metric (of 80 images)
Figure 5: Best fast-tier method per image (PSNR). The winner varies with content; Malvar and the frequency-domain Alleysson split most images. Bilinear is retained as the baseline floor.
Figure 6: The interpretable depth-2 recommender. Saturation and colour variance drive the choice among the cheap methods ( n=80 ).
Image demosaicing reconstructs a full-color image from incomplete color measurements produced by a sensor covered with a color filter array (CFA). Most existing methods formulate demosaicing as pixel-level reconstruction and mainly rely on local textures, cross-channel correlations, and low-level image statistics. Our core insight is that reconstruction and visual understanding can be viewed as complementary views of shared scene structure: both are grounded in the same underlying physical world, and therefore the structural and physical information inferred from an image should remain consistent across the two tasks. We instantiate this idea with instance segmentation and propose SegDem, a cross-task decoder representation transfer framework for demosaicing. SegDem first learns region- and boundary-aware representations through instance-aware structural pretraining and then transfers the decoder to RAW-conditioned reconstruction. Segmentation- and demosaicing-conditioned features are further anchored to a shared frozen DINOv2 representation space to preserve structural organization across tasks. We instantiate SegDem with convolutional, Transformer-based, and state-space backbones for unified Single- and Quad-Bayer demosaicing. Extensive experiments on synthetic, external, and challenging datasets demonstrate consistent improvements across different architectures and CFA layouts.
Ping Chen, Xiangming Wang, Yongyong Chen +5
Harbin Institute of Technology, Shenzhen · Shanghai Jiao Tong University · Nanjing University (Suzhou)
Multispectral demosaicing is crucial to reconstruct full-resolution spectral images from snapshot mosaiced measurements, enabling real-time imaging from neurosurgery to autonomous driving. Classical methods are blurry, while supervised learning requires costly ground truth (GT) obtained from slow line-scanning systems. We propose Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), a framework that learns multispectral demosaicing from mosaiced measurements alone. PEFD a) exploits the projective geometry of camera-based imaging systems to leverage a richer group structure than previous demosaicing methods to recover more null-space information, and b) learns efficiently without GT by adapting pretrained foundation models designed for 1-3 channel imaging. On surgical and automotive datasets, PEFD recovers fine details such as blood vessels and preserves spectral fidelity, substantially outperforming recent approaches, nearing supervised performance. Furthermore, the performance of PEFD is demonstrated on raw, unprocessed data from a commercial multispectral sensor. Code is at https://github.com/Andrewwango/pefd.
Multispectral filter array (MSFA) imaging enables snapshot spectral acquisition, but its per-band sparse sampling causes severe spatial aliasing. Although missing spatial structures can be inferred across bands, wavelength-dependent contrast variations make direct cross-band transfer unreliable. Existing model-based methods lack explicit magnitude-invariant mechanisms, and learning-based methods require layout-specific training or per-image tuning, limiting generalization. To address these issues, we propose S3D-Mosaic, a model-based spectral demosaicing method built upon a subgradient-guided cross-band structure transfer framework. We first establish a subgradient-guided regularization framework with a generalized cosine-similarity interpretation, providing a convex mechanism for magnitude-invariant information transfer. Then we design a compact Directional Sparse Equivariant Difference Convolution (DSE-DConv) for band-level extraction of orientation-selective, high-frequency structure features at low computational cost. Instantiating the framework with DSE-DConv on spectrally adjacent bands yields a cross-band self-structure similarity regularizer for spectral demosaicing, which we optimize via a dual-scan ADMM algorithm that decouples the bidirectional dependency while propagating magnitude-robust structural information across bands. With DSE-DConv's two parameters trained once via self-supervision, S3D-Mosaic outperforms state-of-the-art unsupervised methods on CAVE and NTIRE 2022, and generalizes robustly to varying MSFA layouts, pan-mosaic fusion, and real-world UAV data with fixed reconstruction hyperparameters. The source code is available at https://github.com/bestwishes43/S3D-Mosaic.
Weiqiang Zhao, Xian Li, Yuzhe Gui +2
School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin 150001, China