RAW-Domain Degradation Models for Realistic Smartphone Super-Resolution

Ali Mosleh; Faraz Ali; Fengjia Zhang; Stavros Tsogkas; Junyong Lee; Alex Levinshtein; Michael S. Brown

Figure 1: Our SR approach incorporates realistic SR kernels and sensor noise functions, accurately modeled for various modern smartphones. A RAW image captured with Pixel 6 Main camera is processed using a 4

\times

SR model trained on data generated with our realistic degradations. Our model recovers more details and structures compared to other baselines. Pixel 6’s specific degradations are not seen by the models during training.

Figure 2: Examples of calibration patterns and images used to approximate SR kernels. Gray-code patterns and color/gray patches displayed on a monitor, along with their RAW capturing with the target camera, are used for geometric and radiometric alignment and to form HR target images. A pair of

\mathbf{x}

and

\mathbf{y}

used in Eq. 3 for 4

\times

SR kernel estimation of Pixel 9 Pro Tele is illustrated.

Figure 3: Noise model calibration results for the S23U Main camera. Noise calibration is performed per color channel of the CFA at seven different ISO levels shown for the first green channel here. A curve is then fit to each noise parameter, enabling interpolation of noise variance for uncalibrated ISO levels.

Figure 4: Camera-specific SR results on data from four devices (from top to bottom: S23U Tele 1, S24U Tele 2, Pixel 9 Pro Main, and Pixel 9 Pro Tele). We compare the outputs of a RAW-to-RGB 4

\times

SR model trained on data synthesized with different degradation baselines and our own calibrated, camera-specific degradations. The model trained with our data produces sharper results, and recovers more structural details. For better visualization, the original LR capture is linearly demosaicked and naively upsampled by 4

\times

. All images are white-balanced in each row using similar gains and gamma correction to ensure a consistent visual comparison. The yellow dashed rectangle corresponds to the FOV of interest w.r.t 4

\times

zoom factor.

	S23U Tele 1				S24U Tele 2				Pixel 9 Pro Main				Pixel 9 Pro Tele
Method	No Ref. Metrics		Ref. Metrics		No Ref. Metrics		Ref. Metrics		No Ref. Metrics		Ref. Metrics		No Ref. Metrics		Ref. Metrics
	MTF50	MTF25	PSNR	SSIM	MTF50	MTF25	PSNR	SSIM	MTF50	MTF25	PSNR	SSIM	MTF50	MTF25	PSNR	SSIM
Bicubic	0.05	0.29	32.01	0.952	0.16	0.28	33.42	0.935	0.54	0.95	30.88	0.874	0.08	0.22	30.96	0.868
KernelGAN [ 5 ]	1.42	1.15	33.12	0.950	1.62	1.35	33.48	0.937	1.64	1.05	31.10	0.856	1.46	1.22	32.66	0.883
MANet [ 24 ]	0.31	0.89	33.42	0.959	0.29	0.48	33.47	0.935	1.66	1.24	33.14	0.889	0.38	0.62	32.93	0.886
Degradation-Transfer [ 14 ]	1.31	0.94	33.47	0.954	1.56	1.29	33.01	0.948	1.61	1.12	32.98	0.885	1.18	0.68	32.76	0.878

Table 1: Quantitative camera-specific SR results on data captures with four different cameras. The 4

\times

SR model trained with our calibrated, camera-specific degradations outperforms models trained with the baseline degradation pools, in terms of all metrics used.

Camera	Method	No Ref. Metrics		Ref. Metrics
Camera	Method	MTF50	MTF25	PSNR	SSIM
Pixel 6 Main	Real-ESRGAN [ 40 ]	0.38	0.50	31.75	0.884
	BSRAW [ 15 ]	0.97	0.82	28.79	0.713
	RAWSR [ 45 ]	0.96	0.90	28.99	0.790
	Ours (Cross-camera)	1.19	0.86	32.41	0.905
Mi11 Main	Real-ESRGAN [ 40 ]	0.33	0.40	35.08	0.914

Table 2: Quantitative cross-camera SR results on data captures with Pixel 6 Main and Mi 11 Main RAW cameras. Our 4

\times

SR model is trained on synthetically generated data using degradation models obtained from seven different cameras. The two test camera degradations are not seen by any of the models during training.

Figure 5: Cross-camera SR results on Mi 11 Main real RAW captures . We compare the outputs of a RAW-to-RGB

4\times

SR model, trained on data generated with different degradation modeling approaches. The degradations of the test camera are not seen by any of the models during training, yet the model trained with our synthetic data is better at recovering fine details such as building structures and text. The original LR capture is linearly demosaicked, and naive

4\times

upsampling is applied for visualization. All images are white-balanced with similar gains, and gamma corrected. The yellow dashed rectangle represents the FOV of interest relative to the

4\times

zoom factor.

Figure 6: Distributions of calibrated 4

\times

SR kernels for Pixel 6 Main and kernels from other degradation protocols. Pixel 6 kernels lie closer to those calibrated for other devices using our framework, hinting at similarities between RAW domains of different smartphones. For this visualization, we first run PCA [ 34 ] to reduce the dimension of 21

\times

21 kernels to 50 followed by t-SNE [ 39 ] to further reduce dimensions to 2.

Figure S1 : Our calibration setup and display prototype.

Camera	Optical Zoom	Focal Length	Sensor Size	Pixel Size	Sensor Resolution
S23+ Main	1x	23 mm	1/1.56”	1.0 $\mu$ m	4080x3060 (50MP)
S23+ Tele	3x	69 mm	1/3.94”	1.0 $\mu$ m	3648x2736
S23U Main	1x	24 mm	1/1.3”	0.6 $\mu$ m	4080x3060 (200MP)
S23U Tele 1	3x	69 mm	1/3.52”	1.12 $\mu$ m	3648x2736
S24U Tele 2	5x	111 mm	1/2.52”	0.7 $\mu$ m	4080x3060 (50MP)
Pixel 6 Main	1x	26 mm	1/1.31”	1.2 $\mu$ m	4080x3072 (50MP)

Table S1 : Specifications [ 35 ] of all the cameras used in our experiments. Note that some of the sensors are designed for binning and Android ProShot cannot access the unbinned version of the RAW captures. Sensor Resolution column indicates the captured RAW resolution in binned (regular Bayer) format, while the number in parentheses indicates the actual sensor resolution before binning.

Figure S2 : Display-to-sensor linearization function obtained for our display prototype and Pixel 9 Pro Main camera.

Figure S3 : Examples of camera-specific SR kernels we calibrate and add to the pool of SR degredations; Calibrated 4

\times

SR kernels for the center of FOV of S23P Tele camera.

Figure S4 : Our alignment method for producing paired evaluation data. After aligning the colour and brightness of the ground-truth image with a demosaiced crop of the monitor capture, we perform spatial alignment using homography and optical flow. The resulting aligned ground-truth image is well-suited for computing reference-based quantitative metrics, with the RAW capture crop serving as the input.

Figure S5 : Our MTF evaluation. The MTF measured on the LR image is used to calculate the relative MTF on the output of SR models. Here is the actual measurement for our model output in the Pixel 9 Pro Main experiment ( Tab. 1 of the main paper). The average of all values obtained over all the Siemens star patterns in the FOV is reported per model in our experiments in Sec. 4 .

Camera	Method	Ref. Metrics
Camera	Method	PSNR	SSIM
Pixel 6 Main	BSRAW [ 15 ]	28.75	0.877
Pixel 6 Main	Ours (Cross-camera)	29.14	0.913
Mi11 Main	BSRAW [ 15 ]	33.17	0.897
Mi11 Main	Ours (Cross-camera)	39.72	0.956

Table S2: Our 4

\times

SR model is trained on synthetically generated data using our degradation models obtained from seven different cameras. The performance is evaluated on RAW images captured from Pixel 6 Main and Mi 11 Main cameras, and compared with other baselines. The degradations of these two cameras are not included in the training data for any of the models.

Figure S6 : Additional qualitative results for 4

\times

SR on Pixel 6 Main from the experiments in Sec. 4.5 of the main paper. Note that Pixel 6 Main degradations are not explicitly included in the training data of any of the models. Model outputs are white-balanced and gamma-corrected better visualization.

Figure S7 : Additional qualitative results for Mi 11 Main 4

\times

SR results regarding the experiments presented in Sec. 4.5 of the paper. Note that Mi 11 Main degradations are not explicitly seen by any of the models during training. Model outputs are white-balanced and gamma-corrected similarly for better visualization.

RAW-Domain Degradation Models for Realistic Smartphone Super-Resolution

Abstract

Figures & tables

Image Super-Resolution

Degradation-Aware and Structure-Preserving Diffusion for Real-World Image Super-Resolution

Mind the Gap: Quantifying the Domain Gap in Cross-Sensor Diffusion Super-Resolution

Language-Assisted Super-Resolution from Real-World Low-Resolution Patches

Abstract

Image Super-Resolution

Explore similar work

Degradation-Aware and Structure-Preserving Diffusion for Real-World Image Super-Resolution

Mind the Gap: Quantifying the Domain Gap in Cross-Sensor Diffusion Super-Resolution

Language-Assisted Super-Resolution from Real-World Low-Resolution Patches