Organizations: Graduate School of Engineering Science, The University of Osaka. · Graduate School of Information Science and Technology, Hokkaido University and Prometech CG Research.
Projection mapping (PM) enables augmented reality (AR) experiences without requiring users to wear head-mounted displays and supports multi-user interaction. It is regarded as a promising technology for a variety of applications in which users interact with content superimposed onto augmented objects in tabletop workspaces, including remote collaboration, healthcare, industrial design, urban planning, artwork creation, and office work. However, conventional PM systems often suffer from projection shadows when users occlude the light path. Prior approaches employing multiple distributed projectors can compensate for occlusion, but suffer from latency due to computational processing, degrading the user experience. In this research, we introduce a synthetic-aperture PM system that uses a significantly larger number of projectors, arranged densely in the environment, to achieve delay-free, shadowless projection for tabletop workspaces without requiring computational compensation. To address spatial resolution degradation caused by subpixel misalignment among overlaid projections, we develop and validate an offline blur compensation method whose computation time remains independent of the number of projectors. Furthermore, we demonstrate that our shadowless PM plays a critical role in achieving a fundamental goal of PM: altering material properties without evoking projection-like impression. Specifically, we define this perceptual impression as ``sense of projection (SoP)'' and establish a PM design framework to minimize the SoP based on user studies.
Figures & tables
Figure 1: Proposed shadow removal principle by synthetic aperture projection. (Top) In a single-projector setup, shadows are cast when a user’s hand or other object approaches the projection surface and blocks the light path. (Bottom) In a synthetic aperture projection setup (25 projectors in this example), even if some projectors are occluded, light from the remaining projectors still reaches the surface, effectively reducing shadow formation.
Figure 2: The prototype synthetic aperture projection system using 25 projectors arranged in a 5 × 5 grid. (a) A captured photograph of the ceiling-mounted projectors. (b) The system architecture that enables 2K video output to the 25 projectors.
Figure 3: Comparison of shadow removal performance between single-projector (left) and 25-projector (right) setups on the tabletop surface. (Top) The hand is positioned approximately 0.2 m above the surface. (Bottom) The hand touches four different locations on the surface.
Figure 4: Comparison of specular reflections between single-projector (left) and 25-projector (right) setups on the tabletop surface, from two different viewing directions.
Figure 5: Comparison of shadow removal performance between single-projector (top row) and 25-projector (bottom row) setups on the plaster bust surface. (1st column) Projections without the occluder. (2nd–4th columns) The finger touches different parts of the surface: the head (2nd column), the bust (3rd column), and the lower back of the head (4th column).
Figure 6: Pseudo-color visualization showing how many projectors illuminate different regions of the plaster bust surface.
Figure 7: Blur compensation results. From left to right: the target image, the projected result without blur compensation, the result using conventional blur compensation, and the result using our proposed method.
PSNR( ↑ )
SSIM( ↑ )
times [s]
w/o blur comp.
19.54
0.44
w/ conventional blur comp.
20.00
0.49
2732
Ours
20.05
0.50
109
Table 1: Similarity between the target appearance and projected results, and processing time.
Figure 8: Computation time for generating the compensation image.
Figure 9: Stimuli used in the first user study. Ten sheets of paper were placed on the tabletop surface: seven had printed text in various languages, and the remaining three were blank with projected text patterns instead.
Figure 10: Participants performing the task under the 25-projector condition in the first user study. The participant’s face has been intentionally blurred to protect their identity.
Figure 11: Average task completion time (left) and accuracy (right) for each condition in the first user study.
Figure 12: Spatial arrangements of projectors in the tested configurations. Red cells indicate the projectors used in each configuration.
Figure 13: Participants observing two projected images under different projector configurations, while placing their hands over each one.
Figure 14: Projection images used in the second user study.
Figure 15: Mean preference scores and statistical test results obtained using Scheffé’s method (Nakaya’s modification) for pairwise comparisons. Bars labeled with different letters are significantly different at p < .01. (a) Question 1: Which side appeared more as a pattern intrinsic to the surface? (b) Question 2: Which side appeared more as a projected pattern?
Computational imaging combines optical and sensor design with algorithms to recover or infer scene information from indirect, compressed, or noisy measurements.
Projection Mapping (PM) enables seamless superimposition of digital content onto real-world 3D objects, serving as a fundamental technique for immersive visualization, digital twins, and interactive art. Although text-to-image diffusion models have greatly facilitated customized content creation, directly integrating them into practical PM pipelines remains challenging due to the mismatch between idealized 2D generation and physical constraints. To bridge this gap, this paper formalizes two application-level generative paradigms: the cooperative paradigm (harmonizing generated semantics with physical attributes) and the adversarial paradigm (eliminating surface interference via radiometric compensation). Based on this, we propose ConPhyG, a unified controllable physically-guided generative multi-projection mapping framework that enables creators to interactively adjust physical constraints and flexibly switch generative paradigms. In cooperative mode, multi-dimensional physical priors (per-pixel gamut, depth, and edges) are injected into the diffusion process. In adversarial mode, the framework releases the generative potential and applies bounded numerical optimization for multi-projector radiometric compensation. It allows users to dynamically switch constraints to balance artistic freedom with physical feasibility. Furthermore, we extend ConPhyG to 360-degree multi-view consistent PM using a sequential generation strategy. Quantitative and qualitative evaluations on a real-world four-projector setup demonstrate that ConPhyG significantly outperforms state-of-the-art methods in geometric alignment, gamut utilization, and semantic fidelity.
Projector-camera (ProCams) systems achieve active scene perception and controllable appearance manipulation via structured illumination, serving as a core infrastructure for spatial augmented reality, projection mapping, and surface reflectance acquisition. Existing inverse-rendering methods for ProCams deliver high-fidelity results but rely on time-consuming per-scene optimization, while mainstream feed-forward 3D reconstruction models produce baked appearance that cannot adapt to spatially varying projector illumination. To resolve this accuracy-efficiency trade-off, we propose FF-ProCams, a Feed-Forward 3D Gaussian inverse-rendering framework for ProCams. A hybrid Mamba2-Transformer encoder aggregates cross-view geometric and photometric cues from sparse multi-view observations, and lightweight heads predict a relightable Gaussian representation in a single forward pass. We further design a projector-aware differentiable renderer to synthesize camera observations under arbitrary active illumination and ProCams poses. To enable feed-forward training, we construct a large-scale synthetic ProCams dataset covering diverse object geometries and surface materials. Experiments show FF-ProCams achieves high-fidelity projector-aware rendering, generalizes to unseen patterns, and supports novel projector-camera poses. Using only 8 input views, it outperforms optimization-based baselines with 297 views while reducing test-time reconstruction to 0.13 seconds (a three-to-five-order-of-magnitude speedup). The code and data are available at https://github.com/CPREgroup/FF-ProCams/.
Projection-based appearance editing changes selected visual attributes of a physical object through projection mapping (PM). To display a desired appearance, PM typically uses projector compensation, which computes a projector input that counteracts the geometric and photometric distortions of the scene. Compensation can only approximate this appearance within the available projection range, and it cannot revise the appearance itself. Pretrained LLM-based image editors provide a direct way to produce the desired appearance from a text prompt or reference image; projector compensation can then attempt to reproduce the edited result through projection. However, these editors do not have an accurate measure of the scene geometry or scene response. Therefore, their outputs may contain geometric misalignment, inconsistent shading, or changes outside the available projection range, which downstream projector compensation cannot correct. To address this problem, we introduce SREdit, an efficient response-guided framework for projection-based appearance editing. SREdit estimates a scene response map from the reconstructed scene that reflects how strongly projected illumination can affect different surface regions. SREdit uses the source appearance to constrain the early sampling process and preserve registration with the fixed physical object, while the scene response map adapts the amount of editing across surface regions. Experiments across two pretrained image editors show that SREdit improves projected appearance quality and reproduction fidelity. It adds less than one second per image with negligible VRAM overhead. Participants also consistently preferred the results produced with SREdit in our user study.
Yuqiao Xia, Yuchen Deng, Ziheng Chang +1
College of Computer and Information Science, Southwest University, Chongqing, China