Learning to read residual images for single-pixel photography

Hongjun An; Shuaizhe An; Tong Tian; Zhe Sun

Figure 1: Reading residual images for single-pixel photography. a , A digital micromirror device (DMD) displays binary coding patterns on the image of the scene, and a single-pixel detector records one bucket value per pattern. At 1,000 bucket values per frame, 0.24% of the pixel count, traditional correlation imaging (TCI) gives a noise-like scene correlation image (rendered schematic). b , The measurement residual

r

between the bucket values

A\hat{x}

predicted from the current estimate

\hat{x}

by the forward operator

A

and the acquired values

y

, centered within each frame (tilde) and correlated with the coding patterns

P_{m}

, is the residual image

g_{x}

. The transposed Jacobian of a pretrained video decoder

D

lifts it to

g_{z}

, whose decoder response resembles the true error. Measurement-Gradient-Conditioned Video Flow (MGV-Flow) reads it with the history frames for five Euler steps. Badges 1 to 3 mark the three contributions. c , Key results. (1) Peak signal-to-noise ratio (PSNR) of a pixel-space U-Net (UNet), frame-wise image-latent reconstruction (GILM) and video-latent inversion (VLI). (2) Time per five-frame clip. (3) Directional fidelity against PSNR for 30 coding configurations.

Figure 2: A noise-like residual image becomes a structured video correction through the pretrained decoder. a , The residual between predicted (

A\hat{x}

) and acquired (

y

) bucket values, centered within each frame (

\tilde{r}_{t,m}

) and correlated with the coding patterns

P_{m}

, gives

g_{x}

of equation ( 5 ), which the transposed Jacobian of the LeanVAE decoder

D

at the latent

z

lifts to

g_{z}

of equation ( 7 ). b , Ground truth

x^{\ast}

, scene correlation image of traditional correlation imaging (TCI), residual image

-g_{x}

, decoder response

J_{D}(z)(-g_{z})

and true error

x^{\ast}-\hat{x}

for windows of highest, median and lowest motion (frame 3, first of

K=5

Euler steps of Measurement-Gradient-Conditioned Video Flow). Corr is the Pearson correlation with the true error, and p99 the 99th percentile of absolute value that scales each tile. c , Band coherence of equation ( 8 ) with the true error for

-g_{x}

and the decoder responses of pretrained, re-initialized and weight-shuffled decoders. Lines are means over 15 window-frame samples and three seeds for controls, bands interquartile ranges, and the inset enlarges 0.04 to 0.25 cycles per pixel. d , Directional fidelity

C_{\tau}=\cos\angle(-g_{z},z^{\ast}-z_{\tau})

along a

K=10

trajectory with latent

z_{\tau}

at flow time

\tau

and ground-truth latent

z^{\ast}

, averaged over the 15 samples.

Figure 3: Measurement, spatial, and computational scaling across reconstruction spaces. a , Peak signal-to-noise ratio, PSNR, in decibels, dB, versus

M

, the coding patterns or bucket values per frame from equation ( 2 ). The

\times 10^{3}

scale expresses

M

in thousands. Methods are traditional correlation imaging, TCI, a convolutional neural network, CNN, a U-Net, UNet, GILM, the frame-wise image-latent method, and video-latent inversion, VLI. b , PSNR advantage of VLI over UNet and GILM across

M

and resolution. c , PFLOPs are

10^{15}

floating point operations per frame, GPU means graphics processing unit, and GiB means gibibyte. Percentages give reduction relative to GILM. The convergence time is defined by equation ( 19 ). d , Reconstructions at

128\times 72

px, where px means pixels, for two values of

M

. Quality is reported as each run’s best observed checkpoint averaged over four resolutions and five test videos.

Measurements per frame $M$ (1)	TCI	CNN	UNet	GILM	VLI
	PSNR (dB) SSIM (1)	PSNR (dB) SSIM (1)	PSNR (dB) SSIM (1)	PSNR (dB) SSIM (1)	PSNR (dB) SSIM (1)
1,000	11.99 / 0.058	11.89 / 0.045	11.59 / 0.033	16.44 / 0.154	18.02 / 0.346
2,000	12.10 / 0.062	12.04 / 0.051	16.53 / 0.163	18.01 / 0.276	18.96 / 0.376
3,000	12.18 / 0.065	12.07 / 0.053	18.01 / 0.238	18.53 / 0.312	19.38 / 0.398
4,000	12.10 / 0.066	12.16 / 0.056	18.95 / 0.305	19.00 / 0.325	19.70 / 0.423
5,000	12.18 / 0.068	12.30 / 0.062	19.68 / 0.344	19.40 / 0.346	20.04 / 0.443

Table 1: Reconstruction quality across the evaluated optical measurement range. Each entry reports mean peak signal-to-noise ratio, PSNR (dB), and structural similarity, SSIM, over four spatial resolutions and five test videos. Bold values indicate the highest PSNR or SSIM at each measurement count. For traditional correlation imaging (TCI), results at

M<10{,}000

were recomputed using prefixes of the same

10{,}000

measurement sequences.

Method	Compute (PFLOPs frame -1 )	Peak memory (GiB)	Conv. time, 3090 (s frame -1 )	Conv. time, 4090 (s frame -1 )
GILM	5.004	15.31	117.60	64.60
VLI	2.187	4.17	53.70	33.20
Reduction	56.3%	72.8%	54.4%	48.6%

Table 2: Computational cost of frame-wise image-latent and joint video-latent inversion at

856\times 480

and

M=10{,}000

. FLOPs are normalized per reconstructed frame. Peak memory denotes the incremental process-local graphics processing unit (GPU) allocation. Conv. time is the convergence time on an RTX 3090 or RTX 4090 GPU, as medians of repeated runs on the first five frames of one test video. Timing results were obtained from three repeated runs on an exclusively allocated GPU.

Figure 4: Reading versus consuming the residual image. a , Symbols follow equations ( 3 ) to ( 14 ) and ( 9 ). Measurement-Gradient-Conditioned Video Flow (MGV-Flow) predicts the velocity

v_{\theta}

at flow time

\tau

and latent

z_{\tau}

by reading the history

h

, the normalized physical direction

d_{\mathrm{phy}}

and the quality descriptor

q

, and one Euler step gives the next latent

z_{\tau+1/K}

. The network

f_{\phi}

takes a clip’s encoded five-frame history

h

as input, and video-latent inversion (VLI) optimizes

\phi

and the pretrained decoder

D

for every clip by backpropagating the single-pixel imaging (SPI) loss through the operator

A

with AdamW, Adam with decoupled weight decay. RMS is root mean square. History-conditioned flow matching (History-FM) omits

d_{\mathrm{phy}}

and

q

, and FlowDPS-inspired adds

\gamma d_{\mathrm{phy}}

to the velocity. b , Latency is in seconds, s, peak signal-to-noise ratio, PSNR, in decibels, dB, and

\times

is speedup. c , Median measurement normalized mean squared error, NMSE, from equation ( 18 ) and PSNR are shown for

K=5

, the Euler step count. d ,

\Delta_{\mathrm{zero}}

and

\Delta_{\mathrm{shuf}}

are correct-conditioning PSNR minus zero- or shuffled-conditioning PSNR. The scope is 21 latency cases and 105 conditioning cases per

K

.

Method	E2E latency (s clip -1 )	E2E latency (s frame -1 )	PSNR (dB)	SSIM (1)	Measurement NMSE (1)	Speedup ( $\times$ )
VLI 1,500 updates	397.99	79.60	18.12	0.323	0.00143	1.0
MGV-Flow, $K=5$	1.55	0.31	22.92	0.713	0.0210	256.4
MGV-Flow, $K=10$	3.04	0.61	22.56	0.702	0.0211	130.4

Table 3: Matched reconstruction latency at

856\times 480

and

M=1{,}000

. Latency and reconstruction metrics are medians over 21 paired cases after taking the median over three timing repeats within each case. All methods within a paired case used identical bucket values and were executed on the same exclusive RTX 4090 graphics processing unit (GPU). Speedup is the median of the case-wise end-to-end (E2E) latency ratio relative to VLI, video-latent inversion. PSNR is the peak signal-to-noise ratio, SSIM the structural similarity, and NMSE the normalized mean squared error.

Euler steps $K$ (1)	Correct PSNR (dB)	$\Delta_{\mathrm{zero}}$ (dB)	$\Delta_{\mathrm{shuf}}$ (dB)	Correct NMSE (1)	Zero NMSE (1)	Shuffled NMSE (1)
2	22.22	+1.24	+0.41	0.0328	0.0567	0.0484
3	22.10	+1.43	+0.58	0.0303	0.0605	0.0519
4	21.96	+1.52	+0.63	0.0295	0.0632	0.0541
5	21.86	+1.57	+0.68	0.0291	0.0651	0.0561
8	21.66	+1.63	+0.74	0.0287	0.0686	0.0596
10	21.58	+1.65	+0.76	0.0287	0.0700	0.0610

Table 4: Effect of target-consistent residual image conditioning over the validation cases. Peak signal-to-noise ratio (PSNR) values are means over 105 matched reconstruction cases at each integration budget.

\Delta_{\mathrm{zero}}

and

\Delta_{\mathrm{shuf}}

denote the PSNR difference of correct conditioning relative to zero and cross-video shuffled physics, respectively. Measurement normalized mean squared error (NMSE) compares the reconstructed video with the target bucket values.

Figure 5: When reading the residual image pays off. a , Motion score is the copy mean squared error, MSE, from the last history frame to five future frames. It is compared with the MGV-Flow PSNR difference versus History-FM, where MGV-Flow is Measurement-Gradient-Conditioned Video Flow, History-FM is history-conditioned flow matching, PSNR is peak signal-to-noise ratio in decibels, dB, and versus means that the named baseline is subtracted from MGV-Flow. Filled and open markers encode the displayed Euler step counts

K

. Annotations give the Spearman correlation

\rho_{\mathrm{s}}

and its two-sided

p

value. b , Mean PSNR difference of MGV-Flow versus History-FM in the high, middle, and low motion groups, with the grouping rule fixed in advance, across

K

. Bands are 95% bootstrap intervals over windows. c , The same for MGV-Flow versus FlowDPS-inspired, the explicit posterior-gradient flow baseline. The scope of b and c is 35 validation windows with three seeds each.

Figure 6: Rolling single-pixel photography of three 10 s high-motion clips. a , Every block of five target frames is reconstructed from its own

M=1{,}000

bucket values per frame with

K=5

Euler steps, taking the five preceding ground-truth frames as history, and one reconstructed frame is shown per second. Measurement-Gradient-Conditioned Video Flow (MGV-Flow) is compared with history-conditioned flow matching (History-FM) and with FlowDPS-inspired, the explicit posterior-gradient flow baseline. b , Peak signal-to-noise ratio (PSNR) in decibels (dB) of every reconstructed frame, with the clip mean in the legend. The periodic decline follows the position of a frame within its five-frame block.

Figure 7: Residual images produced by three coding configurations at

M=3{,}000

on one held-out window. Rows are the calibration-selected Walsh basis at 16 px, the median-ranked phase-speckle patterns at 32 px, and the low-frequency Fourier basis at 4 px, which has the lowest held-out mean directional fidelity of the 30 configurations. Columns show the first coding pattern, the residual image

-g_{x}

at the probe latent

z_{\mathrm{pr}}

of equation ( 15 ), the decoder response

J_{D}(z_{\mathrm{pr}})(-g_{z})

, the decoded latent correction

D(z^{\ast})-D(z_{\mathrm{pr}})

, which is the same in every row, and the reconstruction after the ten physics-only steps of equation ( 17 ).

C

is the directional fidelity of equation ( 16 ), Pearson is the correlation coefficient between decoder response and decoded latent correction, and PSNR is the peak signal-to-noise ratio of the reconstruction, all averaged over the five frames of the clip. Images show frame 3. Signed maps use a symmetric scale clipped at the 99th percentile of absolute value of each tile. The window is the lower median of the Walsh 16 px fidelity among the eight held-out windows.

Figure 8: Directional fidelity reveals coding-pattern utility. a , Directional fidelity is

C=\cos\angle(-g_{z}(z_{\mathrm{pr}}),z^{\ast}-z_{\mathrm{pr}})

, where

g_{z}

is the lifted residual image from equation ( 7 ),

z_{\mathrm{pr}}

is the probe latent at

\tau=0.5

from equation ( 15 ), and

z^{\ast}=E(x^{\ast})

is the ground-truth video latent. b , PSNR is peak signal-to-noise ratio in decibels, dB,

\rho_{\mathrm{s}}

is Spearman correlation, CI is confidence interval,

M

is patterns per frame,

\times 10^{3}

means thousands, and px means pixels. Saturated sets have exhausted their unique centered span at the shown

M

. c , LeanVAE, the Lean video variational autoencoder, is shown for Block Bernoulli, Walsh, Low-frequency Fourier, Haar, and Phase speckle patterns. d , These families are compared across LeanVAE, VidTok, Wan, and HunyuanVideo video autoencoder backends. VAE means variational autoencoder. e , First match is the smallest tested

M

where selected Walsh reaches the threshold. Coarse cells are spatial cells at the shown grain size. The scope is 30 configurations and eight held-out windows.

Figure 9: Physical single-pixel photography system, top view. A laser beam passes two lenses and a beam expander, is folded by a mirror onto a digital micromirror device (DMD) that displays the binary coding patterns, and is projected by two lenses onto the scene. A fifth lens collects the light from the scene onto the single-pixel detector, which records one bucket value per pattern. The red line marks the beam path.

Figure 10: Physical single-pixel photography reconstruction with 10,000 measured bucket values per frame. a , For each clip, the last frame of the ground-truth history, the ground-truth frames 6 to 10, and the frames that Measurement-Gradient-Conditioned Video Flow (MGV-Flow) predicts from the history and the measured bucket values of frames 6 to 10, with random seed 20261019 and

K=5

Euler steps. Tiles are rectified and contrast-stretched between the 1st and 99th percentiles. b , Equalized peak signal-to-noise ratio (PSNR) of each frame for MGV-Flow, averaged over three seeds, and for per-frame reconstruction by a convolutional neural network (CNN), UNet and GILM. Markers are the two clips and lines their mean. c , Root mean square (RMS) of the measurement residual

r

along the five Euler steps of MGV-Flow, normalized by its value at the first step, with the measured bucket values and with those of the other clip (

M=10{,}000

). Thin lines are the six runs and thick lines their means.

Method	Bucket values	$M$	PSNR	SSIM	Eq. PSNR	Eq. SSIM	Time	Memory
			(dB)		(dB)		(s)	(GiB)
CNN	measured	10,000	7.40	0.117	8.22	0.089	11.17	2.52
UNet	measured	10,000	9.03	0.191	8.06	0.085	18.28	2.55
GILM	measured	10,000	6.35	0.238	8.63	0.205	44.13	21.72
MGV-Flow	measured	10,000	12.08	0.514	10.25	0.328	7.85	2.76
MGV-Flow	measured	1,000	12.06	0.518	10.25	0.329

Table 5: Reconstruction of physical frames 6 to 10. MGV-Flow, Measurement-Gradient-Conditioned Video Flow, uses the ground-truth frames 1 to 5 as history and either the measured bucket values, the bucket values of the other clip, or no measurement input, with means over two clips and three seeds. CNN (convolutional neural network), UNet, and GILM reconstruct every frame independently from 10,000 bucket values, with means over the ten frames.

M

is the number of bucket values per frame. Metrics use the rectified support, and Eq. denotes independent histogram equalization of estimate and reference. Time is per frame for the per-frame methods and per five-frame clip for MGV-Flow, and time and peak graphics processing unit (GPU) memory were measured on an RTX 3090. PSNR is the peak signal-to-noise ratio and SSIM the structural similarity.

Figure 11: Imaging rate of sliding-window single-pixel photography versus compute. Filled markers are the median latencies measured on one RTX 4090 for

K=5

and

K=10

Euler steps at

M=1{,}000

, with the median peak signal-to-noise ratio (PSNR) in the legend. Open markers are theoretical values that assume latency inversely proportional to compute throughput, with the RTX 3090 scaled by its measured latency ratio to the RTX 4090 and the RTX 5090, H100 and B200 by nominal throughput ratios of 2.0, 3.1 and 6.8. The right axis gives the delay, the latency of one five-frame clip, and the dotted line marks 20 frames per second.

Learning to read residual images for single-pixel photography

Abstract

Figures & tables

Single-Pixel Imaging

PickMoment: Continuous-Time Single-Image-to-Video via Learning Deblurring and Blur-to-Video

HiResNets: Native Full-HD Video Recognition with Foveal Residual Streams

RIFT: Residual-Informed Flow Transfer for One-Step Generative Video Compression

Abstract

Single-Pixel Imaging

Explore similar work

PickMoment: Continuous-Time Single-Image-to-Video via Learning Deblurring and Blur-to-Video

HiResNets: Native Full-HD Video Recognition with Foveal Residual Streams

RIFT: Residual-Informed Flow Transfer for One-Step Generative Video Compression