POS-ISP: Pipeline Optimization at the Sequence Level for Task-aware ISP

Jiyun Won; Heemin Yang; Woohyeok Kim; Jungseul Ok; Sunghyun Cho

cs.CVApr 8, 2026

POS-ISP: Pipeline Optimization at the Sequence Level for Task-aware ISP

Authors: Jiyun Won, Heemin Yang, Woohyeok Kim, Jungseul Ok, Sunghyun Cho

Organizations: POSTECH CSE1 · POSTECH GSAI2

Abstract

Recent work has explored optimizing image signal processing (ISP) pipelines for various tasks by composing predefined modules and adapting them to task-specific objectives. However, jointly optimizing module sequences and parameters remains challenging. Existing approaches rely on neural architecture search (NAS) or step-wise reinforcement learning (RL), but NAS suffers from a training-inference mismatch, while step-wise RL leads to unstable training and high computational overhead due to stage-wise decision-making. We propose POS-ISP, a sequence-level RL framework that formulates modular ISP optimization as a global sequence prediction problem. Our method predicts the entire module sequence and its parameters in a single forward pass and optimizes the pipeline using a terminal task reward, eliminating the need for intermediate supervision and redundant executions. Experiments across multiple downstream tasks show that POS-ISP improves task performance while reducing computational cost, highlighting sequence-level optimization as a stable and efficient paradigm for task-aware ISP. The project page is available at https://w1jyun.github.io/POS-ISP

Figures & tables

Figure 1: Overview of the proposed method. POS-ISP aims at constructing the ISP pipeline that best performs for the downstream task. The sequence predictor predicts the image processing module sequence based on the learned policy, and the parameter predictor estimates the corresponding parameters of each module.

Figure 2: Detailed architecture of sequence predictor. The sequence predictor predicts the image processing module sequence based on the learned policy.

Figure 3: Comparison of different ISP methods on object detection and instance segmentation tasks. Reference images are well-lit scenes from the LOD and LIS datasets, with brightness increased by $1.5\times$ for visualization. More results are in the supplementary material.

Figure 4: Qualitative comparison on image enhancement. We use the images retouched by Expert C from the Adobe FiveK dataset as ground truth. Our method more closely matches the brightness and color tones of the ground truth.

	LOD-Dark			LOD-All
Method	mAP @0.5:0.95	mAP @0.5	mAP @0.75	mAP @0.5:0.95	mAP @0.5	mAP @0.75
Input RAW	44.1	67.7	47.5	53.6	70.5	57.5
Camera ISP	37.6	55.4	41.6	48.8	64.5	52.2
DRL-ISP [ 12 ]	44.2	67.8	48.4	52.8	69.9	56.7
ReconfigISP [ 15 ]	43.7	66.7	47.8	51.1	68.5	54.8
AdaptiveISP [ 14 ]	47.2	71.4	51.7	56.1	72.8	60.6

Table 1: Quantitative comparison with object detection task. We highlight the best metrics as bold.

	LIS-Dark			LIS-All
Method	mAP @0.5:0.95	mAP @0.5	mAP @0.75	mAP @0.5:0.95	mAP @0.5	mAP @0.75
Input RAW	27.8	45.6	27.9	32.6	52.3	33.0
Camera ISP	20.1	35.1	20.0	30.4	48.9	31.0
DRL-ISP [ 12 ]	27.1	44.7	27.4	23.6	40.1	23.8
ReconfigISP [ 15 ]	24.2	40.8	24.5	31.1	51.2	31.0
AdaptiveISP [ 14 ]	25.2	42.3	25.2	32.4	52.3	32.5

Table 2: Quantitative comparison with instance segmentation task. We highlight the best metrics as bold.

Figure 5: Optimization behavior. (a) Task score on the test set over training progress. (b) (left) policy entropy convergence and (right) relative likelihood of the final pipeline.

Method	mAP @0.5:0.95	mAP @0.5	mAP @0.75
DRL-ISP [ 12 ]	44.00 $\pm$ 0.20	67.73 $\pm$ 0.47	47.77 $\pm$ 0.31
AdaptiveISP [ 14 ]	47.13 $\pm$ 0.25	71.17 $\pm$ 0.15	51.73 $\pm$ 0.57
Ours	47.80 $\pm$ 0.08	72.10 $\pm$ 0.16	52.67 $\pm$ 0.12

Table 3: Multi-seed quantitative comparison on the LOD-Dark object detection benchmark. We report mean

\pm

standard deviation over three seeds.

Method	Params (M)	MACs (M)	Peak GPU Memory (MB)	Runtime (ms)
DRL-ISP [ 12 ]	6.57	155.3	1013.9	15.71
AdaptiveISP [ 14 ]	7.18	70.2	39.6	12.72
Ours	0.53	15.1	14.4	1.55

Table 4: Comparison of computational efficiency. All results are measured on a single NVIDIA RTX 2080 Ti with inputs of resolution

512\times 512

. We excluded the module execution time when measuring the runtime. We highlight the best metrics as bold.

Sequence predictor	LOD-Dark			LIS-Dark
	mAP	mAP	mAP	mAP	mAP	mAP
	@0.5:0.95	@0.5	@0.75	@0.5:0.95	@0.5	@0.75
(a) Prob. table	47.5	71.4	52.1	31.3	50.9	31.9
(b) GRU (Ours)	47.8	72.1	52.8	32.1	51.8	32.1

Table 5: Ablation study on the effect of adding recurrent sequence estimation. We highlight the best metrics as bold.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Comparison of different ISP methods on object detection and instance segmentation tasks. Reference images are from the LOD-Normal and LIS-Normal datasets with brightness increased by $1.5\times$ for visualization.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Input RAW	9.18	0.195	0.585
DRL-ISP [ 12 ]	13.68	0.571	0.492
ReconfigISP [ 15 ]	17.28	0.818	0.216
AdaptiveISP [ 14 ]	22.73	0.908	0.105
Ours	23.11	0.923	0.097

Appendix

Table S1 : Quantitative comparison for full-reference image quality enhancement task. In the table, “Input RAW” denotes images that have undergone only the most basic preprocessing steps using Adobe Lightroom. We highlight the best metrics as bold.

Method	Error $\downarrow$				Accuracy $\uparrow$
Method	AbsRel	SqRel	RMS	RMSlog	$\delta<1.25$
Input RAW	0.293	2.560	8.73	0.398	0.493
DRL-ISP [ 12 ]	0.131	0.978	5.22	0.209	0.839
ReconfigISP [ 15 ]	0.154	1.128	5.76	0.235	0.788
AdaptiveISP [ 14 ]	0.131	0.971	5.23	0.207	0.840
Ours	0.128	0.947	5.13	0.205	0.847

Appendix

Table S2 : Quantitative comparison with depth estimation task. We highlight the best metrics as bold.

Method	Dataset	Object Detection (LOD)			Instance Segmentation (LIS)
		mAP @0.5:0.95	mAP @0.5	mAP @0.75	mAP @0.5:0.95	mAP @0.5	mAP @0.75
AdaptiveISP	Dark	47.2	71.4	51.7	25.2	42.3	25.2
Ours	Dark	47.8 $\pm$ 0.08	72.1 $\pm$ 0.16	52.7 $\pm$ 0.12	32.2 $\pm$ 0.05	51.9 $\pm$ 0.09	32.2 $\pm$ 0.05
AdaptiveISP	All	56.1	72.8	60.6	32.4	52.3	32.5
Ours	All	56.4 $\pm$ 0.20	73.1 $\pm$ 0.15	60.7 $\pm$ 0.21	34.9 $\pm$ 0.45	55.6 $\pm$ 0.66	34.8 $\pm$ 0.45

Appendix

Table S3 : Multi-seed results compared with the strongest baseline (AdaptiveISP [ 14 ] ). POS-ISP consistently outperforms AdaptiveISP across both data conditions and tasks, while maintaining low variance across seeds.

Figure S2 : Training loss trajectories of critic networks in AdaptiveISP [ 14 ] and DRL-ISP [ 12 ] . The graphs show unstable oscillations and spikes during actor–critic training.

	LOD-Dark (YOLOv3)			Runtime (ms)
Method	mAP @0.50:0.95	mAP @0.5	mAP @0.75	Prediction	End-to-end	Length
DRL-ISP [ 12 ]	44.2	67.8	48.4	15.71	26.85	3
AdaptiveISP [ 14 ]	47.1	71.4	51.7	12.72	$16.30$	5
Ours	47.8	72.1	52.8	1.55	$82.57$	10
Ours (w/ penalty)	47.7	72.0	52.2	1.55	$\mathbf{1.61}$	3

Appendix

Table S4 : Accuracy–runtime trade-off with runtime penalty on LOD-Dark. Object detection performance and corresponding runtime measured on a single NVIDIA RTX 2080 Ti with

512\times 512

input images.

Method	Prediction (ms / FPS)	End-to-end (ms / FPS)
AdaptiveISP [ 14 ]	29.0 / 34.48	115.0 / 8.70
Ours (w/ penalty)	7.21 / 138.70	32.4 / 30.86

Appendix

Table S5 : On-device runtime comparison on a Galaxy S10 CPU. Runtime is measured using ISP pipelines trained on the LOD-Dark dataset with

512\times 512

input images.

Method	Length Constraint	LOD-Dark			LIS-Dark
Method	Length Constraint	mAP @0.5:0.95	mAP @0.5	mAP @0.75	mAP @0.5:0.95	mAP @0.5	mAP @0.75
AdaptiveISP [ 14 ]	5	47.1	71.4	51.7	25.2	42.3	25.2
Ours	5	47.4	71.7	51.6	31.0	50.7	31.1
Ours	$\leq$ 10	47.8	72.1	52.8	32.1	51.8	32.1

Appendix

Table S6 : Ablation study on the impact of fixed and dynamic pipeline length. We highlight the best metrics as bold and the second best metrics as underline.

Method	LOD-All				LIS-All
	$P(\Delta<0)\downarrow$			Worst 5% Mean $\uparrow$	$P(\Delta<0)\downarrow$			Worst 5% Mean $\uparrow$
	All	Normal	Dark	Worst 5% Mean $\uparrow$	All	Normal	Dark	Worst 5% Mean $\uparrow$
DRL-ISP [ 12 ]	0.185	0.186	0.184	-0.28	0.423	0.441	0.405	-0.63
ReconfigISP [ 15 ]	0.252	0.229	0.274	-0.31	0.287	0.309	0.265	-0.38
AdaptiveISP [ 14 ]	0.135	0.138	0.132	-0.18	0.269	0.217	0.321	-0.30
Ours	0.123	0.138	0.108	-0.16	0.187	0.199	0.174	-0.25

Appendix

Table S7: Output-level robustness analysis using per-image

\Delta

[email protected] (w.r.t. input RAW).

P(\Delta<0)

denotes the fraction of images whose performance degrades after ISP, and Worst 5% Mean reports the average drop among the most severely degraded cases (AP in the [0,1] scale). Best results are in bold.

	LOD-Dark			LOD-All
Method	mAP @0.5:0.95	mAP @0.5	mAP @0.75	mAP @0.5:0.95	mAP @0.5	mAP @0.75
Input RAW	34.8	53.8	37.5	44.3	62.2	46.5
Camera ISP	26.9	41.4	28.9	40.1	56.2	42.4
DRL-ISP [ 12 ]	35.4	54.7	38.0	44.7	62.5	46.7
ReconfigISP [ 15 ]	33.4	52.0	35.9	14.0	20.7	14.6
AdaptiveISP [ 14 ]	36.8	56.1	40.2	47.3	65.4	49.9

Appendix

Table S8 : Quantitative comparison with object detection task with YOLOv13. In the table, “Input RAW” denotes images that have undergone only the most basic preprocessing steps: defective pixel correction, black level correction, lens shading correction, and demosaicking. We highlight the best metrics as bold.

Loss	Method	Error $\downarrow$			Accuracy $\uparrow$
Loss	Method	AbsDiff	AbsRel	RMS	$\delta<1.25$
RMSE	AdaptiveISP	2.574	0.131	5.22	0.840
RMSE	Ours	2.525	0.128	5.13	0.847
AbsDiff	AdaptiveISP	2.548	0.128	6.24	0.847
AbsDiff	Ours	2.504	0.127	5.15	0.850
AbsRel	AdaptiveISP	2.656	0.134	5.38	0.833

Appendix

Table S9 : Comparison under different training objectives. For each objective setting, POS-ISP consistently outperforms AdaptiveISP [ 14 ] across all metrics.

Parameter Predictor	LOD-Dark			LIS-Dark
Parameter Predictor	mAP @0.5:0.95	mAP @0.5	mAP @0.75	mAP @0.5:0.95	mAP @0.5	mAP @0.75
(a) SC	47.5 $\pm$ 0.26	71.5 $\pm$ 0.28	52.0 $\pm$ 0.42	31.6 $\pm$ 0.33	51.2 $\pm$ 0.42	31.3 $\pm$ 0.52
(b) IO (Ours)	47.8 $\pm$ 0.08	72.1 $\pm$ 0.16	52.7 $\pm$ 0.12	32.2 $\pm$ 0.05	51.9 $\pm$ 0.09	32.2 $\pm$ 0.05

Appendix

Table S10 : Comparison between sequence-conditioned (SC) and image-only (IO) parameter predictors. We report mean

\pm

standard deviation over three seeds.

Source Task (Sequence)	[email protected]:0.95	[email protected]	[email protected]
Image Enhancement	47.6	71.7	52.4
Instance Segmentation	47.1	71.4	50.9
Object Detection	48.0	72.2	52.8

Appendix

Table S11 : Cross-task evaluation. Module sequences from different source tasks, evaluated on object detection with parameters retrained for that task. We highlight the best metrics as bold.

Estimated from	Applied to	[email protected]:0.95	[email protected]	[email protected]
LOD-Normal	LOD-Dark	52.6	70.0	57.0
LOD-Dark	LOD-Dark	53.6	71.2	57.4

Appendix

Table S12 : Cross-exposure evaluation of ISP parameter transfer between different datasets. We highlight the best metrics as bold.

Figure S3 : Histogram comparisons of the predicted parameters across illumination domains.

Figure S4 : Qualitative examples of cross-domain parameter swapping. Reference images are well-lit scenes from the LOD-Normal dataset, with brightness increased by $1.5\times$ for visualization. ISP parameters are estimated from different domains and applied to LOD-Dark images.

Module	Equation	Parameter Type	# Params	Parameter Range
Exposure control	$\mathbf{I}^{\prime}=2^{\theta_{e}^{\prime}}\,\mathbf{I}$	EV shift $\theta_{e}^{\prime}$	1	$\theta_{e}^{\prime}\in[-3.5,3.5]$
Gamma correction	$\mathbf{I}^{\prime}=\mathbf{I}^{\,\theta_{g}^{\prime}}$	Gamma exponent $\theta_{g}^{\prime}$	1	$\theta_{g}^{\prime}\in[\tfrac{1}{3},3]$
Tone mapping	$\mathbf{I}^{\prime}=\frac{8}{Z}\sum_{i=1}^{8}\theta_{t,i}^{\prime}\,b_{i}(\mathbf{I})$	Segment weights $\theta_{t,i}^{\prime}$	8	$\theta_{t,i}^{\prime}\in[0.5,2.0]$
Contrast	$\mathbf{I}^{\prime}=(1-\theta_{c}^{\prime})\,\mathbf{I}+\theta_{c}^{\prime}\,\tilde{\mathbf{I}}$	Contrast factor $\theta_{c}^{\prime}$	1	$\theta_{c}^{\prime}\in[-1,1]$
Saturation	$\mathbf{I}^{\prime}=(1-\theta_{s})\,\mathbf{I}+\theta_{s}\,\mathbf{I}_{\mathrm{enh}}$	Blend factor $\theta_{s}$	1	$\theta_{s}\in[0,1]$
Desaturation	$\mathbf{I}^{\prime}=(1-\theta_{d})\,\mathbf{I}+\theta_{d}\,\mathbf{I}_{\mathrm{gray}}$	Blend factor $\theta_{d}$	1	$\theta_{d}\in[0,1]$

Appendix

Table S13 : Details of the adopted ISP modules.

Figure S5 : Qualitative comparison on the image enhancement task. We use the images retouched by Expert C from the Adobe FiveK dataset as ground truth. Error maps are also visualized to highlight pixel-wise differences from the ground truth.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

May 8, 2026

UniISP: A Unified ISP Framework for Both Human and Machine Vision

Compared to RGB images, raw sensor data provides a richer representation of information, which is crucial for accurate recognition, particularly under challenging conditions such as low-light environments. The traditional Image Signal Processing (ISP) pipeline generates visually pleasing RGB images for human perception through a series of steps, but some of these operations may adversely impact the information integrity by introducing compression and loss. Furthermore, in computer vision tasks that directly utilize raw camera data, most existing methods integrate minimal ISP processing with downstream networks, yet the resulting images are often difficult to visualize or do not align with human aesthetic preferences. This paper proposes UniISP, a novel ISP framework designed to simultaneously meet the requirements of both human visual perception and computer vision applications. By incorporating a carefully designed Hybrid Attention Module (HAM) and employing supervised learning, the proposed method ensures that the generated images are visually appealing. Additionally, a Feature Adapter module is introduced to effectively propagate informative features from the ISP stage to subsequent downstream networks. Extensive experiments demonstrate that our approach achieves state-of-the-art performance across various scenarios and multiple datasets, proving its generalizability and effectiveness.

Hanxi Li, Yao Cheng, Bo Zhang +1

Li Auto Inc.

Computational Imaging

Ask PDF

Jul 25, 2026

A Reference-Free Framework for Evaluating Single-Frame ISP Pipelines

Evaluating camera image signal processing (ISP) pipelines requires measuring low-level artifacts introduced by operations such as denoising, demosaicing, tone mapping, and compression. Blind image quality assessment (IQA) techniques can grade visual quality without a reference, but they typically focus on semantic and high-level visual cues or human perceptual scores rather than the low-level image-processing artifacts introduced by camera pipelines. In contrast, full-reference metrics such as PSNR and SSIM measure pixel-level differences and structural similarity, while LPIPS measures perceptual similarity in deep feature space. However, these metrics require perfectly aligned image pairs, which are difficult to collect in practical settings. We propose a reference-free learning framework that estimates full-reference image quality metrics from a processed sRGB image and its ISO metadata. Our method predicts a proxy sRGB reference, which is then compared with the processed image to compute PSNR, SSIM, and LPIPS in their standard full-reference form. Our experiments show that the proxy-reference model can be learned from synthetic data and applied to real camera data. We further show that lightweight LoRA fine-tuning enables efficient adaptation when ISP components or pipeline configurations are changed. The proposed method outperforms direct metric regression in estimating metric values and achieves higher agreement with full-reference rankings than conventional blind IQA methods. These results demonstrate the feasibility of reference-free estimation of full-reference metrics for practical camera-pipeline evaluation.

Yujin Cho, Sira Ferradans, Jean-Michel Morel +2

ENS Paris-Saclay, France · DXOMARK, France · Lingnan University, Hong Kong +2

Image Quality Assessment

Ask PDF

Jun 4, 2026

The Need for Neural ISP in the Small-Pixel Era: How Shrinking Pixels Push Optics to the Limit and Neural Restoration Pushes Back

Smartphone telephoto cameras are approaching a "telephoto physics wall": as pixel pitches shrink toward sub-0.5 micron, the optics remain limited by geometric aberrations, leading to diminishing returns on resolution. Traditional Image Signal Processors (ISPs) cannot eliminate these aberrations, because they operate through local, stage-wise processing with no explicit model of the underlying point spread function (PSF). We demonstrate how a learning-based Neural ISP for image restoration, trained on the underlying degradations, inverts what stage-wise pipelines cannot, turning small-pixel designs into a net advantage. We investigate this through a controlled simulation of a representative telephoto module, evaluating five configurations (0.35--0.75 micron pixel pitch). The aperture is scaled proportionally to keep per-pixel SNR and diffraction spot size fixed, thereby isolating geometric aberration and spatial sampling. While the traditional ISP improves only modestly with smaller pixels, the Neural ISP scales substantially: at 0.35 micron} it reaches 745 cycles/mm MTF50 (vertical), a 2.5--3x resolution improvement over the traditional ISP, and LPIPS improves significantly from 0.244 to 0.151 while traditional results stay comparatively flat. In a low-SNR extension (15 dB per-frame bursts at 0.35 micron), a multi-frame Neural ISP recovers performance close to the bright-light single-frame baseline, whereas a multi-frame traditional ISP shows no meaningful improvement -- indicating that traditional pipelines at small pixels are bottlenecked by uncorrected PSF blur rather than by noise. These results point to a design philosophy in which Neural ISPs enable high-resolution telephoto modules by correcting residual optical aberrations rather than requiring increasingly complex optics.

Jingxi Li, Neerja Aggarwal, Laurent Gudemann +4

Glass Imaging Inc.

Computational Imaging Image Restoration

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

UniISP: A Unified ISP Framework for Both Human and Machine Vision

A Reference-Free Framework for Evaluating Single-Frame ISP Pipelines

The Need for Neural ISP in the Small-Pixel Era: How Shrinking Pixels Push Optics to the Limit and Neural Restoration Pushes Back