Organizations: College of Computer and Information Science, Southwest University, Chongqing, China · School of Engineering, Westlake University, Hangzhou, Zhejiang, China
Full projector compensation adjusts a projector's input image to compensate for both geometric and photometric distortions introduced by the projection surface. Existing methods generally treat the viewing condition as a static camera viewpoint. When applied to a moving viewer, existing methods face two key challenges: (1) view-dependent methods require model retraining or recalibration to compensate for novel viewpoints; and (2) NeRF/GS-based view-agnostic methods use structure from motion (SfM) to calibrate the projector-camera systems (ProCams) with an unknown global scale, while view-aware tracking requires calibrating a camera oriented opposite to the projector, which introduces an additional calibration challenge. To address these challenges, we introduce VAComp, the first view-aware framework for full projector compensation for a moving viewer in a naked-eye setting. Using a projector, VAComp enables the viewer to perceive a virtual object that visually replaces the real object or to observe view-consistent modifications of the real object's appearance and material properties. To resolve the SfM scale ambiguity, we use metric calibration as a reference to recover the global scale; to calibrate the oppositely facing camera, we use two checkerboards, one facing the RGB camera and the other facing the depth camera. To balance compensation quality and runtime efficiency, we develop two non-iterative hybrid methods, VAComp-SD and VAComp-SI. Both combine 2D Gaussian-based view-conditioned geometric correction with learned photometric compensation, predicting each compensation image through a single forward pass without per-frame iterative optimization. Real-world experiments show that both methods achieve competitive compensation quality, with VAComp-SD enabling real-time updates and VAComp-SI operating at interactive rates. Project page: https://realkaizhou.github.io/VAComp/.
Figures & tables
Method
Viewer-tracking method
View-agnostic method
Textured-surface compensation
Real-time application
Direct-light optimization [ 57 ]
✓
✓
✗
✓
Homography-based [ 5 ]
✓
✓
✗
✓
Setup-dependent CNN [ 28 , 59 ]
✗
✗
✓
✓
Setup-independent CNN [ 36 ]
✗
✗
✓
✗
Geometry-attribute shading [ 27 ]
✗
✗
✓
✗
Path-tracing [ 37 ]
✗
✗
✓
✗
Table 1: Comparison of representative view-dependent projection mapping and projector compensation approaches with VAComp. In the table, ✗ and ✓ indicate the limitations and capabilities of different methods, respectively. The first two methods are view-dependent projection mapping approaches that adapt the projected content to the viewer’s viewpoint. Their photometric handling is nevertheless limited. Therefore, we use ✓ to distinguish these methods from fully view-agnostic methods.
Figure 1: View-aware full projector compensation (VAComp) pipeline. The uncompensated image is obtained by rendering the virtual object from the projector viewpoint and directly projecting the rendered image onto the projection surface. For our method, we first calibrate the projector, Kinect v2, RGB camera, and projection surface (see Section 2.2 ), and train GS-ProCams [ 14 ] as the scene-specific forward renderer and geometric compensator. At runtime, VAComp estimates the viewer’s eye pose and renders the viewer virtual object image. We use the geometric correction of GS-ProCams [ 14 ] to warp the desired image into the projector domain. According to the photometric compensation network used, GS-ProCams renders the corresponding surface-prior images, which are also geometrically corrected. The geometrically corrected desired image and surface priors are then processed by the corresponding network for photometric compensation. We use a generalized network diagram to represent both photometric compensation architectures: the pink dashed residual connection is present in PANet, whereas the green dashed residual connection is present in IVPCNet. In the diagram, “Surf.”, “Comp.”, and “Feat.” denote surface, compensated, and feature, respectively. VAComp-SD employs PANet from CompenHR [ 59 ] , whereas VAComp-SI employs IVPCNet from SIComp [ 36 ] .
Figure 2: Calibration pipeline. (a) We first calibrate the RGB camera and the projector. We then place one checkerboard on the RGB-camera side and another on the side opposite to the projector’s projection direction. The wide-angle RGB camera is positioned such that both checkerboards are visible, while the Kinect v2 is placed on the side opposite the projector with a clear view of the corresponding checkerboard. We then capture images using the RGB camera, the Kinect v2, and the wide-angle RGB camera. The numbers in the figure indicate the order of the calibration steps. (b) Following (a), we register the current RGB-camera pose to the COLMAP reconstruction and recover its metric scale. The resulting world coordinate system is defined with the RGB-camera pose in (a) as the origin, in which the virtual object is placed.
Figure 3: Visual comparisons of projector compensation for novel projections. Each row represents a different surface, with the label at the beginning of the row indicating the name of the corresponding case. The first two columns show the camera-captured projection surfaces and their uncompensated images, while the remaining columns present the viewer virtual object images together with the camera-captured compensated projections produced by different methods. For better visualization, the images in the second row have been cropped and enlarged.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
ΔE↓
Client FPS ↑
Repeat Ratio ↓
GS Train Time ↓
PCN Train Time ↓
VDPM-ERB [ 5 ]
14.3318
0.4340
0.3572
16.9982
42.35
0 repeats/frame
11.76 min
-
GS-Inverse
20.2499
0.6127
0.3190
9.4547
42.87
0 repeats/frame
11.54 min
-
GS-ProCams [ 14 ]
19.2644
0.6003
0.3138
9.6706
41.13
85 repeats/frame
11.91 min
-
VAComp-SD (Ours)
20.4819
0.6403
0.3032
9.1447
42.95
0 repeats/frame
11.76 min
8.13 min
VAComp-SI (Ours)
20.5432
0.6270
0.3122
9.8710
42.48
4 repeats/frame
11.76 min
0.00 min
Uncompensated
15.1919
0.4603
0.3828
16.4294
-
-
-
-
Table 2: Quantitative comparison of projector compensation on novel viewpoints with novel projections. Results are averaged over eight experimental cases derived from five different scenes. Each case contains 78 novel viewpoints. Client frames per second (FPS) is the client-side display refresh rate, and Repeat Ratio is the average number of repeated display frames per fresh compensation image. GS Train Time and PCN Train Time report the training times of the GS model and photometric compensation network (PCN) in minutes, respectively. In the PCN Train Time column, “-” indicates that the method does not use a PCN, whereas zero indicates that no per-setup PCN training is required. The best and second-best results are highlighted in bold and underlined , respectively.
Process
Processing time (ms)
Percentage
Eye tracking
8.5
36%
Compensation-image update
14.5
61%
Rendering virtual object
0.8
3%
Total latency
23.8
-
Table 3: Latency performance of the proposed framework. For each case-method pair, the stage-wise processing times are averaged over 1,000 display refresh frames. The reported values are then averaged across all evaluated methods and eight cases.
Figure 4: Client-side latency breakdown. The x-axis represents the eight evaluated cases, and the y-axis shows the processing time in milliseconds. The blue, orange, and green portions represent eye tracking, compensation-image update, and virtual object rendering, respectively. For cases whose qualitative results are not presented in the main paper, the corresponding results are provided in the supplementary material.
Figure 5: Augmented views under multiple viewpoints. Each row corresponds to the augmented result on a different projection surface, with the label on the left indicating the name of the corresponding case. From left to right, the first column shows the front-view camera-captured projection surfaces, while the remaining columns show the camera-captured projections under different viewpoints. The first row demonstrates the integration of a virtual object into a real scene with complex geometry and surface texture. The remaining rows illustrate how our framework enhances the appearance of existing real objects through material modification. All camera-captured compensated projections shown in these results are produced using compensation images generated by VAComp-SD. For clearer visualization, the images in the first two rows are cropped and enlarged.
Figure 6: Qualitative comparison of different projection strategies under viewpoint change for the first SAR application (real-object appearance enhancement). The images are captured at a specific viewpoint during multi-view motion. (a) Camera-captured projection surface. (b) Camera-captured uncompensated projection. (c) Camera-captured projection using static compensation under viewpoint mismatch (right-side compensation, left-side capture). (d) Camera-captured compensated projection using VAComp-SI. (e) Camera-captured compensated projection using VAComp-SD. (f) Viewer virtual object image at this viewpoint.
Figure 7: Qualitative comparison of projection strategies under dynamic viewpoint change for the second SAR application (virtual-object replacement). For clearer visualization, these images are cropped and enlarged. The images are captured at a specific viewpoint during multi-view motion. (a) Camera-captured projection surface. (b) Camera-captured uncompensated projection. (c) Camera-captured projection using static compensation under viewpoint mismatch (right-side compensation, left-side capture). (d) Camera-captured compensated projection using VAComp-SI. (e) Camera-captured compensated projection using VAComp-SD. (f) Viewer virtual object image at this viewpoint.
Figure 8: Comparison of pose assumptions for projector compensation using VAComp-SI. From left to right, the columns show the simulated projection surface, the viewer virtual object image, and the simulated compensated projections generated by VAComp-SI using the speckle-calibrated pose and the center-facing pose assumption, respectively. The bottom row shows magnified views of the regions highlighted by the orange boxes.
Figure 9: Failure case. VAComp-SD and VAComp-SI cannot reliably handle low-albedo regions or regions affected by specular highlights. These two cases respectively reflect the projector’s limited ability to compensate for low surface reflectance and the difficulty of reproducing view-dependent reflections.
Computational imaging combines optical and sensor design with algorithms to recover or infer scene information from indirect, compressed, or noisy measurements.
Projector compensation seeks to correct geometric and photometric distortions that occur when images are projected onto nonplanar or textured surfaces. However, most existing methods are highly setup-dependent, requiring fine-tuning or retraining whenever the surface, lighting, or projector-camera pose changes. Progress has been limited by two key challenges: (1) the absence of large, diverse training datasets and (2) existing geometric correction models are typically constrained by specific spatial setups; without further retraining or fine-tuning, they often fail to generalize directly to novel geometric configurations. We introduce SIComp, the first Setup-Independent framework for full projector Compensation, capable of generalizing to unseen setups without fine-tuning or retraining. To enable this, we construct a large-scale real-world dataset spanning 277 distinct projector-camera setups. SIComp adopts a co-adaptive design that decouples geometry and photometry: A carefully tailored optical flow module performs online geometric correction, while a novel photometric network handles photometric compensation. To further enhance robustness under varying illumination, we integrate intensity-varying surface priors into the network design. Extensive experiments demonstrate that SIComp consistently produces high-quality compensation across diverse unseen setups, substantially outperforming existing methods in terms of generalization ability and establishing the first generalizable solution to projector compensation. The code and dataset are available on our project page: https://hai-bo-li.github.io/SIComp/
Projector-camera (ProCams) systems achieve active scene perception and controllable appearance manipulation via structured illumination, serving as a core infrastructure for spatial augmented reality, projection mapping, and surface reflectance acquisition. Existing inverse-rendering methods for ProCams deliver high-fidelity results but rely on time-consuming per-scene optimization, while mainstream feed-forward 3D reconstruction models produce baked appearance that cannot adapt to spatially varying projector illumination. To resolve this accuracy-efficiency trade-off, we propose FF-ProCams, a Feed-Forward 3D Gaussian inverse-rendering framework for ProCams. A hybrid Mamba2-Transformer encoder aggregates cross-view geometric and photometric cues from sparse multi-view observations, and lightweight heads predict a relightable Gaussian representation in a single forward pass. We further design a projector-aware differentiable renderer to synthesize camera observations under arbitrary active illumination and ProCams poses. To enable feed-forward training, we construct a large-scale synthetic ProCams dataset covering diverse object geometries and surface materials. Experiments show FF-ProCams achieves high-fidelity projector-aware rendering, generalizes to unseen patterns, and supports novel projector-camera poses. Using only 8 input views, it outperforms optimization-based baselines with 297 views while reducing test-time reconstruction to 0.13 seconds (a three-to-five-order-of-magnitude speedup). The code and data are available at https://github.com/CPREgroup/FF-ProCams/.
Projection-based appearance editing changes selected visual attributes of a physical object through projection mapping (PM). To display a desired appearance, PM typically uses projector compensation, which computes a projector input that counteracts the geometric and photometric distortions of the scene. Compensation can only approximate this appearance within the available projection range, and it cannot revise the appearance itself. Pretrained LLM-based image editors provide a direct way to produce the desired appearance from a text prompt or reference image; projector compensation can then attempt to reproduce the edited result through projection. However, these editors do not have an accurate measure of the scene geometry or scene response. Therefore, their outputs may contain geometric misalignment, inconsistent shading, or changes outside the available projection range, which downstream projector compensation cannot correct. To address this problem, we introduce SREdit, an efficient response-guided framework for projection-based appearance editing. SREdit estimates a scene response map from the reconstructed scene that reflects how strongly projected illumination can affect different surface regions. SREdit uses the source appearance to constrain the early sampling process and preserve registration with the fixed physical object, while the scene response map adapts the amount of editing across surface regions. Experiments across two pretrained image editors show that SREdit improves projected appearance quality and reproduction fidelity. It adds less than one second per image with negligible VRAM overhead. Participants also consistently preferred the results produced with SREdit in our user study.
Yuqiao Xia, Yuchen Deng, Ziheng Chang +1
College of Computer and Information Science, Southwest University, Chongqing, China