Spatons: Spatiotonally Localized Building Blocks for Efficient XR Predistortion

Mohammadjavad Maheronnaghsh; Sushant Kondguli; Steve Mann; Andreas Moshovos

cs.CVOct 17, 2026

Spatons: Spatiotonally Localized Building Blocks for Efficient XR Predistortion

Authors: Mohammadjavad Maheronnaghsh, Sushant Kondguli, Steve Mann, Andreas Moshovos

Organizations: Department of Electrical and Computer Engineering, University of Toronto · Meta Reality Labs · Vector Institute

Abstract

We present Spatons, a family of compositional, spatially localized neural operators for computationally efficient XR (extended reality) predistortion and distortion compensation. XR displays (AR, VR, and MR) suffer from spatial (e.g., geometric) and tonal (e.g., colour) distortions. Frames can be pre-distorted so that the projected image looks as intended. Distortions vary with manufacturing tolerances and projection systems, motivating programmable compensation. Untethered, battery-operated XR systems also require low energy and latency. A Spaton is a hybrid patch-to-pixel block combining tiny Multi-Layer Perceptrons (MLPs) with fixed-function interpolation. Using Spatons, we build programmable, computationally efficient pipelines for varied distortion scenarios. On the full spatiotonal Scenario 5, Sequential-5' matches CompenNet++ in SSIM (0.944 vs. 0.944) and closely matches its PSNR (27.5 vs. 27.7 dB), with substantially lower complexity. We demonstrate a flexible Spaton solution with 6 times fewer FLOPs and 106 times fewer parameters than CompenNet++, and 4-10 times fewer memory transfers than UNet++. We characterize software implementations and analytical hardware estimates, including optimizations across target XR platforms. The distortions considered are drawn mostly from see-through, near-eye displays, where the spatial light path governs local intensity and spectral response. This early systems-level exploration uses closed-form differentiable distortions and one real projector-camera capture set for training and evaluation, rather than through-the-optics measurements. Testing on a wider range of real XR devices remains future work. Chromatic dispersion (color fringes) remains the main artifact where Spatons trail monolithic baselines.

Figures & tables

Panel of Figure 1 — Figure 1: Taxonomy of XR spatiotonal distortions. Spatial, tonal, and spatiotonal distortions are sibling families acting on coordinates, pixel values, or position-dependent tonal response.

Figure 3: The Sequential-5 ′ pipeline: a G-Res geometric predistortion stage followed by a Patch2Pixel-v3 tonal predistortion stage. RGB residuals are normalized to $[-2,+2]$ . Data types: int8 = 8-bit integer pixel index; FP32 = 32-bit float; RGB = one 3-channel pixel.

Table 4

Model	$h$	SSIM↑	PSNR↑	LPIPS↓	RMSE↓
Pure XR	–	.641	14.6	.635	.186
Sequential-5 ′ ( $p=3$ )	64,64	.944	27.5	.082	.043
Sequential-5 ′ ( $p=3$ )	64,48	.943	27.4	.085	.043
Sequential-5 ′ ( $p=3$ )	32,32	.939	27.1	.091	.044
Sequential-7 ′ ( $p=3$ )	64,64,64	.944	27.6	.083	.042
Sequential-9/10 (no patch-tonal)	64,64	.896	25.8	.172	.051

Table 3: Mixed geometric + non-geometric distortions, Scenario 5 (↑ higher better, ↓ lower better). Full

h

-sweep, all Sequential compositions, and the complete baseline ladder are in Table 24 ; Sequential 9/10 ablate the patch-tonal stage.

Model	Params	FLOPs (M)	Transfer (MB)	SSIM Sc5
CompenNet++	1.15M	8,526	162	.944
UNet++-Large	6.83M	12,129	1392	.868
Sequential-5 ′ ( $h{=}32,32$ )	3.4K	443	169	.939
Sequential-5 ′ ( $h{=}64,64$ )	10.8K	1,407	297	.944

Table 4: Model efficiency summary (

3{\times}256{\times}256

input). Full per-model breakdown in Appendix I (Table 17 ).

$h$	Device (Freq)	$N=1$	$N=2$	$N=4$	$N=8$	$N=16$	$N=32$
32	Lattice CrossLink-NX (200 MHz) Lattice Semiconductor (2024)	22.25	44.51	–	–	–	–
32	AMD Zynq ZU3EG (600 MHz) AMD (2025)	66.76	133.52	267.04	534.08	1068.15	–
64	Lattice CrossLink-NX (200 MHz) Lattice Semiconductor (2024)	7.72	15.44	–	–	–	–
64	AMD Zynq ZU3EG (600 MHz) AMD (2025)	23.16	46.33	92.66	185.32	370.64	–

Table 5: Peak FPS (throughput model above), assuming 16-PE MLP instances with 1 MAC/cycle/PE. Dashes indicate configurations exceeding device resources.

Appendix figures & tables50 assets

Supplementary material from the paper’s appendix.

Appendix

Distortion	Formula
Spatial
Keystone	Perspective transform.
Fisheye	$\vec{x}^{\prime}{=}\vec{x}/(1{-}ar^{2})$
Barrel/Pincushion	$\vec{x}^{\prime}{=}\vec{x}(1{\pm}ar^{2}{\pm}br^{4})$ ; $(+)$ =Barrel, $(-)$ =Pincushion
Mustache	$\vec{x}^{\prime}{=}\vec{x}(1{-}ar^{2}{+}br^{4})$
Tonal

Appendix

Table 6: AR distortions.

Model	Pipeline Definition
Sequential-1	G-Res $\to$ Patch2Pixel-v2 ( $p{=}5$ )
Sequential-2	Patch2Pixel-v2 ( $p{=}5$ ) $\to$ G-Res
Sequential-3	G-Res $\to$ C-Res $\to$ Patch2Pixel-v2 ( $p{=}5$ )
Sequential-4	C-Res $\to$ Patch2Pixel-v2 ( $p{=}5$ ) $\to$ G-Res
Sequential-5	G-Res $\to$ Patch2Pixel-v2 ( $p{=}3$ )
Sequential-5 ′	G-Res $\to$ Patch2Pixel-v3 ( $p{=}3$ )

Appendix

Table 7: Sequential model definitions.

Sc.	Type	Distortions
1	Non-Geometric	Color balance, Brightness (Br)&Contrast (Ct), elev. black, vignetting, blur, color fringes
2	Non-Geometric	Elev. black, Br $\pm$ -v2, contrast-v2, blur, color fringes, vignetting, WGNU-v2
3	Geometric	Keystone, fisheye
4	Geometric	Keystone, barrel, fisheye
5	Spatiotonal	Elev. black, Br $\pm$ -v2, contrast-v2, blur, vignetting, keystone, color fringes, WGNU-v2, barrel, fisheye

Appendix

Table 8: Summary of evaluation scenarios.

NN 1 model	Params	PSNR↑	SSIM↑	LPIPS↓	RMSE↓
CompenNet++	1,152,147	36.94	.978	.008	.014
Sequential-5 ′ ( $h{=}64$ )	10,847	28.81	.948	.035	.036

Appendix

Table 9: Two-phase Phase 1: forward-mimic fidelity on Scenario 5 (clean

\to

NN 1

\to

distorted).

Frozen NN 1	PSNR↑	SSIM↑	LPIPS↓	RMSE↓	$\Delta$ PSNR vs. Pure-AR
CompenNet++	26.59	.926	.110	.047	$+11.95$
Sequential-5 ′	26.66	.927	.112	.047	$+12.02$

Appendix

Table 10: Two-phase Phase 2 on Scenario 5: predistortion through the frozen NN 1 (clean

\to

NN 2

\to

{}_{1}^{\text{frozen}}

\to

clean). NN 2 is Sequential-5 ′ ,

8.5

K params.

Phase 1: forward mimic
Stage	Model	Params	PSNR↑	SSIM↑	LPIPS↓	RMSE↓
P1	CompenNet++ (black surface)	1,152,147	21.12	.742	.118	.088
P1	CompenNet++ (real surface)	1,152,147	23.92	.827	.066	.064
Phase 2: predistortion ( $\mathrm{NN}_{2}$ = Sequential-5 ′ , $8{,}511$ params)
P2	Pure AR (no $\mathrm{NN}_{2}$ ), black	–	10.28	.489	–	–
P2	Pure AR (no $\mathrm{NN}_{2}$ ), real	–	10.25	.487	–	–

Appendix

Table 11: Real projector–camera data ( lavender_np ). Top: Phase-1 forward mimics. Bottom: Phase-2 predistortion through each frozen mimic (

\mathrm{NN}_{2}

= Sequential-5 ′ ,

h{=}64/48

8{,}511

params).

Mode	Model	Params	PSNR↑	SSIM↑	LPIPS↓	RMSE↓	$\Delta$ PSNR
Known operator (analytic formulas)
Pure AR	(uncorrected)	–	13.09	.566	–	–	–
Formula	$\mathrm{NN}_{2}$	8,511	25.22	.915	.090	.055	$+12.1$
Phase 1: learned forward mimic
Mimic	CompenNet++	1,152,147	35.96	.970	.012	.016	–
Mimic	Sequential-5 ′	10,847	23.36	.872	.113	.068	–

Appendix

Table 12: Spatially non-uniform four-regime degradation (keystone/fisheye

\times

blur/contrast). Pure-AR is measured through each operator (rows grouped).

\mathrm{NN}_{2}

is Sequential-5 ′ ,

8{,}511

params. The frozen-Seq-5 ′ Phase-2 value is

28.865

3

dp.

Figure 4: Qualitative comparison of distortion compensation models under the full spatiotonal Scenario 5 pipeline, as mentioned in the paper. Each row corresponds to one model. Columns show: (1) the original/pre-distorted image input to the model, (2) the XR system output before ROI cropping, (3) the XR system output after ROI cropping (the quantity measured by all metrics), and (4) the absolute pixel difference between the corrected output and the expected image, where brighter regions indicate larger residual error. All difference maps share the same $[0,\,0.30]$ colormap scale.

Hyperparameter	Value
Batch size	3
Restart period	6 epochs
Warmup epochs	3
Training epochs	30
Dataset fraction	50% of FFHQ dataset

Appendix

Table 13: Hyperparameters of the FFHQ experiments. The rest of the hyperparameters are the same as the Images.cv dataset.

	Scenario 2 (Non-Geo)		Scenario 4 (Geo)
Model	S↑	P↑	S↑	P↑
Pure XR	.676	17.7	.744	15.2
G-Res (h=64)	-	-	.985 (+.241)	40.5 (+25.4)
G-Res (h=256)	-	-	.986 (+.242)	41.0 (+25.8)
Patch2Pixel-v2 (p=3, h=64)	.743 (+.066)	22.6 (+4.97)	-	-
Patch2Pixel-v2 (p=3, h=256)	.736 (+.060)	22.8 (+5.11)	-	-

Appendix

Table 14: FFHQ higher-resolution training results for Scenarios 2 and 4. Models are trained directly on FFHQ at

1024\times 1024

using the hyperparameters in Table 13. Values show improvement over baseline. Best in bold , second-best underlined .

	CelebA-HQ-256 Generalization
Model	S↑	P↑	L↓	R↓
Pure XR	.644	14.9	.565	.179
Sequential 1	.947 +.303	29.0 +14.1	.052 -.513	.035 -.144
Sequential 2	.946 +.302	28.9 +14.0	.060 -.505	.036 -.143
Sequential 3	.947 +.303	29.1 +14.2	.048 -.517	.035 -.144
Sequential 4	.946 +.302	29.0 +14.1	.057 -.508	.035 -.144

Appendix

Table 15: Mixed Geometric and Non-Geometric Distortions on CelebA-HQ-256 (↑ higher is better, ↓ lower is better). Models pretrained on Images.cv dataset, evaluated on CelebA-HQ-256 to assess generalization. Values show improvement over baseline. Best in bold , second-best underlined . Distortions are identical to those in Scenario 5 of the paper.

Component	Input/Output Dims	FLOPs	Memory Reads (bytes)	Memory Writes (bytes)
Conv2d	$(C_{in},H,W)\to(C_{out},H_{out},W_{out})$	$2\cdot H_{out}\cdot W_{out}\cdot C_{in}\cdot K^{2}\cdot C_{out}/groups$	$C_{in}\cdot H\cdot W\cdot 4+C_{out}\cdot\frac{C_{in}}{groups}\cdot K^{2}\cdot 4$	$C_{out}\cdot H_{out}\cdot W_{out}\cdot 4$
TransConv2d	$(C_{in},H,W)\to(C_{out},H_{out},W_{out})$	$2\cdot H_{out}\cdot W_{out}\cdot C_{in}\cdot K^{2}\cdot C_{out}/groups$	$C_{in}\cdot H\cdot W\cdot 4+\frac{C_{in}}{groups}\cdot C_{out}\cdot K^{2}\cdot 4$	$C_{out}\cdot H_{out}\cdot W_{out}\cdot 4$
Linear	$(B,in_{f})\to(B,out_{f})$	$2\cdot B\cdot in_{f}\cdot out_{f}$	$B\cdot in_{f}\cdot 4+in_{f}\cdot out_{f}\cdot 4$	$B\cdot out_{f}\cdot 4$
BatchNorm	$(C,H,W)\to(C,H,W)$	$2\cdot C\cdot H\cdot W$	$C\cdot H\cdot W\cdot 4+4\cdot C\cdot 4$	$C\cdot H\cdot W\cdot 4$
ReLU	$N\to N$	$N$	$N\cdot 4$	$N\cdot 4$
LeakyReLU	$N\to N$	$2\cdot N$	$N\cdot 4$	$N\cdot 4$

Appendix

Table 16: Base Component Rules for FLOPs, Memory Reads, and Memory Writes

Baseline Models
Model	Parameters	Size (MB)	Total FLOPs	Reads (MB)	Writes (MB)	Transfer (MB)
CompenNet++	1,152,147	4.40	8,525,512,704	85.16	77.25	162.41
UNet++ Large	6,825,011	26.04	12,129,370,112	723.35	669.12	1392.48
UNet++ Small	2,552,561	9.74	5,606,285,312	652.91	625.53	1278.44
UNet++ Medium	3,241,487	12.37	6,358,827,008	666.36	634.25	1300.61
Single-Stage Models

Appendix

Table 17: Model Summary: FLOPs, Memory, and Parameters

	Keystone		Fisheye		Pincushion		Barrel		Mustache
Model	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑
Pure XR	.541	13.2	.648	15.5	.479	14.3	.569	16.1	.516	14.7
G-Hybrid - H=16	.979 +.438	35.3 +22.1	.990 +.342	38.4 +23.0	.821 +.342	23.7 +9.4	.990 +.421	38.7 +22.6	.983 +.467	36.5 +21.8
G-Hybrid - H=64	.987 +.446	38.1 +24.9	.993 +.345	40.4 +25.0	.958 +.479	32.5 +18.2	.992 +.423	39.7 +23.6	.988 +.472	38.8 +24.2
G-Res	.987 +.446	38.2 +25.0	.993 +.345	40.6 +25.1	.958 +.479	32.5 +18.2	.992 +.423	39.7 +23.7	.988 +.472	39.0 +24.3
unet++ - Small	.785 +.244	22.7 +9.5	.850 +.202	23.8 +8.4	.726 +.247	21.2 +6.9	.857 +.288	23.5 +7.4	.794 +.278	22.6 +7.9

Appendix

Table 18: Geometric Distortions Performance (↑ higher is better, ↓ lower is better).

	WG-v1		WG-v2		WG-v3		ColBal		ElevBlk		Blur		ColFrg		Vign		Br&Ct
Model	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑
Pure XR	.996	34.4	.984	25.4	.976	23.4	.922	17.8	.971	29.5	.823	26.5	.746	22.6	.955	18.5	.492	9.4
C-Hybrid - H=16	.980 -.016	29.4 -5.0	.969 -.015	27.8 +2.4	.963 -.013	24.7 +1.3	.991 +.069	41.5 +23.7	.974 +.003	35.4 +5.9	.869 +.046	27.0 +0.5	.802 +.056	24.3 +1.7	.945 -.010	20.9 +2.4	.329 -.163	4.5 -4.9
C-Hybrid - H=64	.986 -.010	34.8 +0.4	.975 -.009	29.8 +4.4	.971 -.005	24.8 +1.4	.996 +.074	47.7 +29.9	.980 +.009	39.1 +9.6	.875 +.052	27.3 +0.8	.841 +.095	26.3 +3.7	.956 +.001	22.6 +4.1	.329 -.163	4.5 -4.9
Patch2Pixel (v1) - P=3	.986 -.010	34.3 -0.1	.986 +.002	33.8 +8.4	.978 +.002	25.2 +1.8	.999 +.077	52.6 +34.8	.979 +.008	38.1 +8.6	.876 +.053	27.1 +0.6	.800 +.054	24.9 +2.3	.961 +.006	22.9 +4.4	.329 -.163	4.5 -4.9
Patch2Pixel (v1) - P=5	.989 -.007	35.4 +1.0	.994 +.010	35.5 +10.1	.977 +.001	25.2 +1.8	.998 +.076	52.5 +34.7	.984 +.013	41.4 +11.9	.873 +.050	26.9 +0.4	.876 +.130	28.0 +5.4	.968 +.013	23.1 +4.6	.329 -.163	4.5 -4.9

Appendix

Table 19: Non-Geometric Distortions Part 1: WGNU Variations and Basic Distortions (↑ higher is better, ↓ lower is better).

	Br- v1		Br+ v1		Br- v2		Br+ v2		Br± v2		Br- v3		Br+ v3		Br± v3
Model	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑
Pure XR	.514	11.5	.752	11.3	.862	19.9	.927	19.9	.887	20.4	.851	19.4	.923	19.4	.896	20.4
C-Hybrid ( $h=16$ )	.942 +.428	20.3 +8.8	.853 +.101	20.1 +8.8	.946 +.084	25.6 +5.7	.943 +.016	25.5 +5.6	.884 -.003	21.0 +0.6	.943 +.092	24.6 +5.2	.943 +.020	25.2 +5.8	.872 -.024	20.0 -0.4
C-Hybrid ( $h=64$ )	.948 +.434	20.3 +8.8	.853 +.101	20.1 +8.8	.964 +.102	27.7 +7.8	.948 +.021	27.0 +7.1	.908 +.021	22.9 +2.5	.944 +.093	24.9 +5.5	.944 +.021	25.2 +5.8	.878 -.018	20.5 +0.1
Patch2Pixel-v1 ( $p=3$ )	.943 +.429	20.4 +8.9	.852 +.100	20.1 +8.8	.978 +.116	29.9 +10.0	.962 +.035	31.2 +11.3	.966 +.079	29.3 +8.9	.948 +.097	25.0 +5.6	.948 +.025	25.7 +6.3	.881 -.015	20.5 +0.1
Patch2Pixel-v1 ( $p=5$ )	.951 +.437	20.3 +8.8	.857 +.105	20.1 +8.8	.984 +.122	30.9 +11.0	.965 +.038	32.3 +12.4	.971 +.084	30.1 +9.7	.951 +.100	25.2 +5.8	.949 +.026	25.7 +6.3	.882 -.014	20.5 +0.1

Appendix

Table 20: Non-Geometric Distortions Part 2: Brightness Variations (↑ higher is better, ↓ lower is better).

	Ct+ v1		Ct- v1		Ct+ v2		Ct- v2		Ct+ v3		Ct- v3
Model	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑	S↑	P↑
Pure XR	.806	21.9	.857	18.9	.953	28.1	.949	28.4	.951	28.3	.952	28.2
C-Hybrid ( $h=16$ )	.993 +.187	36.7 +14.8	.882 +.025	21.3 +2.5	.940 -.013	26.1 -2.0	.941 -.008	26.5 -1.9	.940 -.011	26.3 -2.0	.941 -.011	26.2 -2.0
C-Hybrid ( $h=64$ )	.995 +.189	44.7 +22.8	.887 +.030	21.8 +2.9	.953 +0	28.8 +0.7	.950 +.001	28.4 +0	.945 -.006	27.9 -0.4	.944 -.008	27.5 -0.7
Patch2Pixel-v1 ( $p=3$ )	.998 +.192	50.8 +28.9	.889 +.032	22.1 +3.2	.970 +.017	30.5 +2.4	.965 +.016	29.3 +0.9	.950 -.001	28.1 -0.2	.947 -.005	28.2 +0
Patch2Pixel-v1 ( $p=5$ )	.999 +.193	53.1 +31.2	.892 +.035	22.0 +3.2	.955 +.002	28.9 +0.8	.973 +.024	32.3 +3.9	.949 -.002	28.4 +0.1	.949 -.003	28.2 +0

Appendix

Table 21: Non-Geometric Distortions Part 3: Contrast Variations (↑ higher is better, ↓ lower is better).

	Scenario 1		Scenario 2
Model	S↑	P↑	S↑	P↑
Pure XR	.416	8.3	.655	17.3
C-Hybrid ( $h=16$ )	.782 +.366	24.8 +16.5	.724 +.069	19.3 +2.0
C-Hybrid ( $h=64$ )	.633 +.217	8.7 +0.4	.798 +.143	21.5 +4.2
Patch2Pixel-v1 ( $p=3$ )	.620 +.204	8.7 +0.4	.797 +.142	21.6 +4.4
Patch2Pixel (v1) ( $p=5$ )	.615 +.199	8.7 +0.4	.843 +.188	22.7 +5.4

Appendix

Table 22: Non-Geometric Mixed Distortions (↑ higher is better, ↓ lower is better). Scenario 1: described in the paper.

	Scenario 3		Scenario 4
Model	S↑	P↑	S↑	P↑
Pure XR	.704	15.3	.650	14.9
G-Hybrid ( $h=16$ )	.982 +.278	35.8 +20.5	.968 +.318	32.9 +18.0
G-Hybrid ( $h=64$ )	.987 +.283	37.4 +22.1	.974 +.324	34.8 +19.9
G-Res	.987 +.283	37.5 +22.2	.975 +.325	34.9 +20.0
UNet++ - Small	.857 +.153	24.1 +8.9	.834 +.184	23.7 +8.8

Appendix

Table 23: Geometric Mixed Distortions (↑ higher is better, ↓ lower is better). Scenarios 3 & 4: described in the paper.

	Scenario 5
Hidden Dimensions	SSIM↑	PSNR↑	LPIPS↓	RMSE↓
Pure XR	.641	14.6	.635	.186
Sequential-1: G-Res $\rightarrow$ Patch2Pixel-v2 ( $p=5$ )
64, 64	.942 +.301	27.4 +12.7	.063 -.572	.043 -.143
Sequential-2: Patch2Pixel-v2 ( $p=5$ ) $\rightarrow$ G-Res
64, 64	.941 +.300	27.3 +12.7	.076 -.559	.043 -.143

Appendix

Table 24: Full Scenario 5 ablation (mixed geometric + non-geometric): all Sequential compositions and hidden dimensions. Subscript values are the improvement over the Pure XR baseline (“

+

” for SSIM/PSNR, “

-

” for LPIPS/RMSE).

Model	Scenario	Baseline (FP32)		Weight-Only Quantization			Weight+Activation Quantization
Model	Scenario	PSNR	SSIM	Bits	PSNR	SSIM	Bits (W+A)	PSNR	SSIM
Patch2Pixel-v2 (p=3)	1	27.69	0.8546	13b	27.68 (-0.01)	0.8539 (-0.0007)	13b+12b	27.65 (-0.04)	0.8510 (-0.0036)
Patch2Pixel-v3 (p=3)	1	27.52	0.8515	14b	27.42 (-0.10)	0.8515 (+0.0000)	14b+11b	27.39 (-0.13)	0.8474 (-0.0041)
Patch2Pixel-v2 (p=5)	2	23.26	0.8728	12b	23.26 (+0.00)	0.8725 (-0.0003)	12b+11b	23.24 (-0.02)	0.8684 (-0.0044)
Patch2Pixel-v2 (p=3)	2	23.05	0.8627	11b	23.00 (-0.05)	0.8626 (-0.0001)	11b+11b	22.98 (-0.07)	0.8556 (-0.0071)
Patch2Pixel-v3 (p=3)	2	22.99	0.8588	10b	22.91 (-0.08)	0.8580 (-0.0008)	10b+9b	22.88 (-0.11)	0.8518 (-0.0070)

Appendix

Table 25: Quantization Analysis Results (h=64)

G-Res Operation Modes
Mode	Description
mlp	Evaluates MLP on the full coordinate grid every frame.
lut	Precomputes the fixed G-Res warp grid once for a given resolution and replaces the per-frame MLP with a bilinear resample of the stored grid. Since G-Res depends only on $(x,y)$ , this is equivalent to mlp up to float32 rounding.
lutrevcache	Adds inter-frame reuse with a reverse LUT: for each input pixel, it stores the output pixels that read from it under the bilinear stencil. Only outputs whose source pixels changed since the previous frame are recomputed.
Patch2Pixel-v3 Operation Modes
mlp	Runs the per-pixel MLP on every output pixel of every frame, with no caching.

Appendix

Table 26: Design Space for Video Pre-distortion Exploration

Target	Compute	Dataset	$N$	Best config	G-Res (ms)	P2P (ms)	Total (ms)	$\mathrm{fps}_{\mathrm{real}}$	PSNR (dB)	SSIM
Ryzen 9 5900X	CPU	UGC	256	lutrevcache + mlp	1.23	27.48	28.71	34.90	26.40	0.896
Ryzen 9 5900X	CPU	UVG	1080	lut + mlp	5.53	487.16	492.69	2.03	23.52	0.838
RTX 3090	GPU	UGC	360	lut + mlp	1.15	1.03	2.18	530.37	25.44	0.877
RTX 3090	GPU	UVG	1080	lut + mlp	6.06	7.96	14.02	72.20	23.52	0.838
Raspberry Pi 5	CPU	UGC	256	lut + mlp	2.43	83.52	85.95	11.65	26.40	0.896
Raspberry Pi 5	CPU	UVG	1080	lut + mlp	23.66	1339.40	1363.06	0.73	23.51	0.838

Appendix

Table 27: Cross-platform video workload summary for Sequential 5 ′ . Latencies are per-frame medians of the predistortion compute. “Best config” minimizes

t_{\text{G-Res}}{+}t_{\text{P2P}}

Quantity	$N{=}256$	$N{=}360$	$N{=}1080$
Pixels per frame ( $3N^{2}$ )	196,608	388,800	3,499,200
ROI window	$179{\times}179$	$252{\times}252$	$755{\times}755$
ROI / total pixels	48.95%	49.00%	48.99%
Forward-LUT size	524 KB	1.0 MB	9.3 MB
Reverse-LUT size	1.3 MB	2.5 MB	23 MB
Reverse-LUT edges ( $4N^{2}$ )	262,144	518,400	4,665,600

Appendix

Table 28: Resolution-dependent quantities (one --input-size knob propagates everywhere).

#	G-Res mode	Patch2Pixel-v3 cache
1	mlp	none
2	mlp	center
3	mlp	3x3
4	lut	none
5	lut	center
6	lut	3x3

Appendix

Table 29: Nine video-runtime configurations (CPU and CUDA).

Panel of Figure 5 — Figure 5: Video runtime design space (self-contained schematic).

Category	Clips
Sports	24
Lecture	23
NewsClip	20
HowTo	20
LiveMusic	16
Gaming	16

Appendix

Table 30: YouTube-UGC 360p subset by category.

Group	Clip	Category	Mean change
Smooth	Lecture_360P-051a	Lecture	$1.157{\times}10^{-4}$
Smooth	LyricVideo_360P-5e87	LyricVideo	$1.361{\times}10^{-4}$
Medium	Gaming_360P-586d	Gaming	$1.758{\times}10^{-2}$
Medium	Vlog_360P-2973	Vlog	$1.835{\times}10^{-2}$
Fast	Vlog_360P-3121	Vlog	$1.386{\times}10^{-1}$
Fast	MusicVideo_360P-24b0	MusicVideo	$1.503{\times}10^{-1}$

Appendix

Table 31: Six motion-stratified YouTube-UGC benchmark clips.

Host	Arch	CPU / GPU	OMP CPU	OMP GPU
Ryzen 9 5900X	x86_64	12C/24T + RTX 3090 (sm_86)	16 / 32	14 / 16
Raspberry Pi 5	aarch64	4 $\times$ Cortex-A76	6 / 8	–
Jetson Orin Nano Super	aarch64	6 $\times$ Cortex-A78AE + Ampere iGPU (sm_87)	28 / 18	8 / 10

Appendix

Table 32: Execution targets. OMP columns are the per-target, per-dataset calibrated host thread counts (UGC / UVG).

Machine	CPU candidates	GPU host candidates
Ryzen 9 5900X	$2,4,\dots,32$ (16 values)	$2,4,\dots,32$ (16)
Raspberry Pi 5	$1,2,3,4,5,6,8,10$ (8)	—
Jetson Orin Nano Super	$2,4,\dots,28$ (14, extended)	$2,4,\dots,18$ (9)

Appendix

Table 33: OpenMP search ranges per machine.

Machine	Dataset	Res	#vids	Frames/vid
Ryzen 9 5900X	UGC	256/360	6	40
Ryzen 9 5900X	UVG	1080	2	15
Raspberry Pi 5	UGC	256	6	25
Raspberry Pi 5	UVG	1080	2	6
Jetson Orin Nano Super	UGC	256/360	6	40
Jetson Orin Nano Super	UVG	1080	2	15

Appendix

Table 34: Calibration workload (videos

\times

frames-per-video).

Exp	Host	Dataset	Kind	OMP	Margin
1	Ryzen 5900X	UGC	CPU	16	$+3.50\%$
2	Raspberry Pi 5	UGC	CPU	6	$+2.16\%$
3	Jetson Orin Nano Super	UGC	CPU	28	$+0.33\%$ (boundary)
4	Ryzen 5900X / RTX 3090	UGC	GPU	14	$+2.31\%$
5	Jetson iGPU	UGC	GPU	8	$+1.68\%$ (noise zone)
6	Ryzen 5900X	UVG	CPU	32	$+0.37\%$ (boundary)

Appendix

Table 35: Per-experiment OpenMP winners. “Margin” is the gap to the next-best thread count.

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	31.44	28.41	30.93	16.73	26.4007	0.8960
mlp	center	31.79	28.27	31.27	16.68	26.4004	0.8960
mlp	3x3	31.87	29.01	30.88	16.44	26.4007	0.8960
lut	none	2.45	27.96	30.84	33.06	26.4007	0.8960
lut	center	2.57	27.54	30.68	33.35	26.4004	0.8960
lut	3x3	2.52	28.23	30.77	32.65	26.4007	0.8960

Appendix

Table 36: Exp 1 — Ryzen 9 5900X CPU, UGC @256 (OMP 16).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	94.48	84.63	52.29	5.58	26.4007	0.8960
mlp	center	94.59	84.69	52.20	5.58	26.4004	0.8960
mlp	3x3	94.80	85.68	52.31	5.54	26.4007	0.8960
lut	none	2.43	83.52	52.10	11.65	26.4007	0.8960
lut	center	2.38	84.46	52.04	11.53	26.4004	0.8960
lut	3x3	2.47	85.42	52.03	11.39	26.4007	0.8960

Appendix

Table 37: Exp 2 — Raspberry Pi 5 CPU, UGC @256 (OMP 6).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	90.23	77.68	70.03	5.96	26.4007	0.8960
mlp	center	89.15	75.89	69.36	6.06	26.4004	0.8960
mlp	3x3	89.91	78.03	69.52	5.96	26.4007	0.8960
lut	none	2.85	80.26	69.10	12.04	26.4007	0.8960
lut	center	2.99	82.46	68.62	11.71	26.4004	0.8960
lut	3x3	3.15	82.90	67.76	11.62	26.4007	0.8960

Appendix

Table 38: Exp 3 — Jetson Orin Nano Super CPU, UGC @256 (OMP 28; boundary winner).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	2.21	1.01	56.26	335.99	25.4365	0.8774
mlp	center	2.22	1.74	56.02	268.09	25.4361	0.8774
mlp	3x3	2.23	2.00	56.11	247.75	25.4365	0.8774
lut	none	1.15	1.03	56.97	530.37	25.4365	0.8774
lut	center	1.14	1.77	56.79	378.99	25.4361	0.8774
lut	3x3	1.15	2.03	57.09	338.38	25.4365	0.8774

Appendix

Table 39: Exp 4 — RTX 3090 GPU, UGC @360 (host OMP 14).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	18.75	10.35	122.02	34.37	25.4550	0.8773
mlp	center	18.79	8.35	121.90	37.59	25.4547	0.8773
mlp	3x3	18.86	12.27	122.96	32.72	25.4550	0.8773
lut	none	3.94	10.78	119.76	68.02	25.4550	0.8773
lut	center	4.55	9.03	120.69	82.29	25.4547	0.8773
lut	3x3	4.23	13.15	120.53	61.87	25.4550	0.8773

Appendix

Table 40: Exp 5 — Jetson Orin Nano iGPU, UGC @360 (host OMP 8).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	574.08	484.67	433.96	0.94	23.5234	0.8380
mlp	center	575.34	497.90	447.88	0.93	23.5230	0.8380
mlp	3x3	576.15	504.89	447.88	0.93	23.5234	0.8380
lut	none	5.53	487.16	433.48	2.03	23.5234	0.8380
lut	center	5.46	500.15	450.39	1.98	23.5230	0.8380
lut	3x3	5.49	506.60	450.41	1.95	23.5234	0.8380

Appendix

Table 41: Exp 6 — Ryzen 9 5900X CPU, UVG @1080 (OMP 32; boundary winner).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	1526.53	1330.00	988.59	0.35	23.5135	0.8377
mlp	center	1527.62	1355.36	993.22	0.35	23.5131	0.8377
mlp	3x3	1524.51	1368.42	993.05	0.35	23.5135	0.8377
lut	none	23.66	1339.40	996.68	0.73	23.5135	0.8377
lut	center	23.43	1357.55	989.94	0.72	23.5131	0.8377
lut	3x3	23.15	1369.39	996.95	0.72	23.5135	0.8377

Appendix

Table 42: Exp 7 — Raspberry Pi 5 CPU, UVG @1080 (OMP 8).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	1116.79	984.18	1045.85	0.48	23.5135	0.8377
mlp	center	1115.58	1005.78	1060.20	0.47	23.5131	0.8377
mlp	3x3	1118.95	1019.58	1059.39	0.47	23.5135	0.8377
lut	none	18.20	994.59	1049.85	0.99	23.5135	0.8377
lut	center	17.41	1013.25	1062.38	0.97	23.5131	0.8377
lut	3x3	17.73	1030.74	1063.34	0.95	23.5135	0.8377

Appendix

Table 43: Exp 8 — Jetson Orin Nano Super CPU, UVG @1080 (OMP 18; true optimum after range extension).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	13.79	7.87	429.52	46.17	23.5234	0.8380
mlp	center	13.90	14.89	426.61	34.73	23.5230	0.8380
mlp	3x3	13.82	17.86	425.94	31.71	23.5234	0.8380
lut	none	6.06	7.96	441.54	72.20	23.5234	0.8380
lut	center	6.15	15.27	431.91	46.88	23.5230	0.8380
lut	3x3	6.13	17.95	433.75	41.57	23.5234	0.8380

Appendix

Table 44: Exp 9 — RTX 3090 GPU, UVG @1080 (host OMP 16).

G-Res	P2P	G ms	P ms	D ms	fps	PSNR	SSIM
mlp	none	113.09	72.65	1089.33	5.38	23.5135	0.8377
mlp	center	112.91	96.66	1084.57	4.77	23.5131	0.8377
mlp	3x3	113.31	140.83	1093.41	4.02	23.5135	0.8377
lut	none	20.21	68.07	1066.68	11.33	23.5135	0.8377
lut	center	21.84	83.52	1072.96	9.49	23.5131	0.8377
lut	3x3	22.45	125.18	1074.88	7.11	23.5135	0.8377

Appendix

Table 45: Exp 10 — Jetson Orin Nano iGPU, UVG @1080 (host OMP 10).

Target / dataset	mlp ms	lut ms	rev ms	mlp $\to$ lut
Ryzen / UGC256 (CPU)	31.44	2.45	1.23	$12.8{\times}$
Pi 5 / UGC256 (CPU)	94.48	2.43	2.57	$38.9{\times}$
Jetson / UGC256 (CPU)	90.23	2.85	2.59	$31.7{\times}$
Ryzen / UVG1080 (CPU)	574.08	5.53	28.77	$103.8{\times}$
Pi 5 / UVG1080 (CPU)	1526.53	23.66	53.91	$64.5{\times}$
Jetson / UVG1080 (CPU)	1116.79	18.20	57.02	$61.4{\times}$

Appendix

Table 46: G-Res latency by mode and mlp

\to

lut speedup.

Target / dataset	best G ms	P2P ms	P2P share
Ryzen / UGC256 (CPU)	1.23	27.48	95.7%
Pi 5 / UGC256 (CPU)	2.43	83.52	97.2%
Ryzen / UVG1080 (CPU)	5.53	487.16	98.9%
Pi 5 / UVG1080 (CPU)	23.66	1339.40	98.3%
RTX 3090 / UGC360 (GPU)	1.15	1.03	47.3%
RTX 3090 / UVG1080 (GPU)	6.06	7.96	56.8%

Appendix

Table 47: Best G-Res latency vs. Patch2Pixel-v3 ( none );

\mathrm{P2P}/(\mathrm{G}{+}\mathrm{P2P})

Target / dataset	sm lut	sm rev	$\Delta$	fa lut	fa rev	$\Delta$
Ryzen / UGC256	0.83	0.47	$-43\%$	3.03	2.34	$-23\%$
Pi 5 / UGC256	1.68	1.48	$-12\%$	2.53	3.78	$+49\%$
Jetson / UGC256	2.00	1.00	$-50\%$	3.30	4.11	$+25\%$
Ryzen / UVG1080	5.30	22.12	$+317\%$	5.76	35.42	$+515\%$
Pi 5 / UVG1080	21.42	48.00	$+124\%$	25.91	59.82	$+131\%$
Jetson / UVG1080	17.56	48.85	$+178\%$	18.84	65.18	$+246\%$

Appendix

Table 48: Reverse-cache vs. motion: per-video G-Res ms, lut

\to

rev (

\Delta

). “sm”/“fa”

=

smoothest/fastest clip.

Target / dataset	none	center	3x3	verdict
Ryzen / UGC256 (CPU)	27.96	27.54	28.23	noise
Pi 5 / UGC256 (CPU)	83.52	84.46	85.42	slight cost
Ryzen / UVG1080 (CPU)	487.16	500.15	506.60	slight cost
Pi 5 / UVG1080 (CPU)	1339.40	1357.55	1369.39	slight cost
RTX 3090 / UGC360 (GPU)	1.03	1.77	2.03	cache slower
RTX 3090 / UVG1080 (GPU)	7.96	15.27	17.95	cache slower

Appendix

Table 49: Patch2Pixel-v3 ms by cache mode (G-Res

=

lut ).

Platform / dataset	GPU G+P2P ms	CPU G+P2P ms	speedup
RTX 3090 / UVG1080	14.02	492.69	$35.1{\times}$
Jetson iGPU / UVG1080	88.28	1012.79	$11.5{\times}$
RTX 3090 / UGC360	2.18	cross-resolution (CPU @256)
Jetson iGPU / UGC360	14.72	cross-resolution (CPU @256)

Appendix

Table 50: GPU model-only ( lut + none ) vs. same-host CPU ( lut + none ).

Cell	Hosts	PSNR (dB)	SSIM
UGC @256 (CPU)	Exps 1,2,3	$26.40\pm 0.001$	$0.8960\pm 10^{-4}$
UGC @360 (GPU)	Exps 4,5	$25.44$ / $25.45$	$0.8774$ / $0.8773$
UVG @1080 (CPU+GPU)	Exps 6–10	$23.52$ / $23.51$	$0.838$

Appendix

Table 51: Quality by dataset/resolution cell.

Panel of Figure 6 — Figure 6: Sequential 5 ′ dataflow. Stage 1 is a LUT lookup plus bilinear interpolation; Stage 2 runs $N$ parallel Patch2Pixel MLPs. All buffers and weights are FP32.

Metric	Stage 1 (G-Res)	Stage 2 (Patch2Pixel)	Total
		$h=32\hskip 8.50012pth=64$	$h=32$ / $h=64$
Compute (FLOPs)	1.97 M	285.6 M 826.8 M	287.6 M / 828.8 M
Static (ROM)	512 KB	8.6 KB 24.9 KB	521 KB / 537 KB
Dynamic (RAM)	1,536 KB	1,536 KB 1,536 KB	2,304 KB + scratch
Transfer (Reads)	3,584 KB	777 KB 793 KB	4,361 KB / 4,377 KB per frame

Appendix

Table 52: Per-frame resource requirements for Sequential 5 ′ . All quantities assume FP32; no quantization is applied (cf. Sec. 5.6 ). Sizes use 1 KB

{=}1024

bytes. “Transfer (Reads)” counts read traffic only and excludes the intermediate- and output-frame writes (

2\times 768

KB), unlike the Transfer column of Table 4 , which is reads plus writes.

Neural Operators

Neural operators learn mappings between functions, enabling fast surrogates for PDEs and other continuous physical systems.

See all 313 papers

Oct 14, 2026

HRTF Upsampling and Personalization via Neural Operators

Head-related transfer function (HRTF) spatial upsampling from sparse measurements is a fundamental challenge in personalized binaural audio. We formulate HRTF upsampling as an operator learning problem and introduce SPARO, a neural operator that maps sparse measurements at a set of spatial directions to the HRTF field, a continuous representation that can be queried at arbitrary positions. SPARO projects the measurements onto a learned set of spherical latent modes, processes the latent modes with Transformer blocks, and decodes the resulting representations at query directions. On SONICOM, using the LAP24 measurement configuration, SPARO attains the best log-spectral distortion (LSD) and interaural level difference (ILD) error at every sparsity level from 3 to 100 measurements (18% lower LSD and 43% lower ILD error than the strongest baseline at 100 measurements) while remaining competitive in interaural time difference (ITD) error. It reconstructs a dense HRTF 15 to 373 times faster than the leading neural-field method RANF. We further validate the approach on HUTUBS, demonstrating its effectiveness across HRTF datasets.

Meenakshi Krishnan, You Zhang, Ramani Duraiswami +1

University of Maryland, College Park · Dolby Laboratories, Inc.

Spatial Audio Neural Operators

Ask PDF

Jun 13, 2026

SPARK: Spatial Policy-driven Adaptive Reinforcement learning for Knowledge distillation

Low-bit quantization enables deployment of image restoration (IR) networks on resource-constrained devices, but introduces rounding noise that disproportionately degrades high-frequency regions such as edges and fine textures. Existing knowledge distillation (KD) methods apply distillation signals uniformly across all spatial locations, overlooking the varying reconstruction difficulty across image regions. To address this, we propose SPARK (Spatial Policy-driven Adaptive Reinforcement Learning for Knowledge Distillation), a framework that adaptively allocates distillation effort using a lightweight reinforcement learning (RL) policy network. At each training step, a difficulty feature extractor computes four signals, namely Laplacian variance, pixel variance, student reconstruction error, and teacher-student knowledge gap, which are fed into a compact policy CNN that produces a stochastic spatial weight map to modulate the KD loss during quantization-aware training (QAT). SPARK is IR task-agnostic, adds no inference cost, and integrates into any existing QAT pipeline without architectural changes. Experiments on benchmark datasets demonstrate that SPARK consistently outperforms PTQ, QAT, and state-of-the-art (SOTA) KD approaches across multiple student architectures, achieving reconstruction quality closest to the full-precision teacher under significant computational constraints.

Mohamed Jismy Aashik Rasool, Shabir Ahmad, Gisong Oh +1

Department of IT Convergence Engineering, Gachon University, Sujeong-gu, Seongnam-si, Gyeonggi-do 461-701, Republic of Korea

RL for Knowledge Distillation Quantization-Aware Training Image Restoration Knowledge Distillation Low-Bit Quantization Quantization-Aware Distillation

Ask PDF

Apr 2, 2026

Setup-Independent Full Projector Compensation

Projector compensation seeks to correct geometric and photometric distortions that occur when images are projected onto nonplanar or textured surfaces. However, most existing methods are highly setup-dependent, requiring fine-tuning or retraining whenever the surface, lighting, or projector-camera pose changes. Progress has been limited by two key challenges: (1) the absence of large, diverse training datasets and (2) existing geometric correction models are typically constrained by specific spatial setups; without further retraining or fine-tuning, they often fail to generalize directly to novel geometric configurations. We introduce SIComp, the first Setup-Independent framework for full projector Compensation, capable of generalizing to unseen setups without fine-tuning or retraining. To enable this, we construct a large-scale real-world dataset spanning 277 distinct projector-camera setups. SIComp adopts a co-adaptive design that decouples geometry and photometry: A carefully tailored optical flow module performs online geometric correction, while a novel photometric network handles photometric compensation. To further enhance robustness under varying illumination, we integrate intensity-varying surface priors into the network design. Extensive experiments demonstrate that SIComp consistently produces high-quality compensation across diverse unseen setups, substantially outperforming existing methods in terms of generalization ability and establishing the first generalizable solution to projector compensation. The code and dataset are available on our project page: https://hai-bo-li.github.io/SIComp/

Haibo Li, Qingyue Deng, Jijiang Li +2

Southwest University · Westlake University

Computational Imaging

Ask PDF

Abstract

Neural Operators

Explore similar work

HRTF Upsampling and Personalization via Neural Operators

SPARK: Spatial Policy-driven Adaptive Reinforcement learning for Knowledge distillation

Setup-Independent Full Projector Compensation