ProCap: Projection-Aware Captioning for Spatial Augmented Reality

Zimo Cao; Yuchen Deng; Haibin Ling; Bingyao Huang

cs.CVApr 1, 2026

ProCap: Projection-Aware Captioning for Spatial Augmented Reality

Authors: Zimo Cao, Yuchen Deng, Haibin Ling, Bingyao Huang

Organizations: Southwest University · Westlake University

Abstract

Spatial augmented reality (SAR) directly projects digital content onto physical scenes using projectors, creating immersive experience without head-mounted displays. However, for SAR to support intelligent interaction, such as reasoning about the scene or answering user queries, it must semantically distinguish between the physical scene and the projected content. Standard Vision Language Models (VLMs) struggle with this virtual-physical ambiguity, often confusing the two contexts. To address this issue, we introduce ProCap, a novel framework that explicitly decouples projected content from physical scenes. ProCap employs a two-stage pipeline: first it visually isolates virtual and physical layers via automated segmentation; then it uses region-aware retrieval to avoid ambiguous semantic context due to projection distortion. To support this, we present RGBP (RGB + Projections), the first large-scale SAR semantic benchmark dataset, featuring 65 diverse physical scenes and over 180,000 projections with dense, decoupled annotations. Finally, we establish a dual-captioning evaluation protocol using task-specific tokens to assess physical scene and projection descriptions independently. Our experiments show that ProCap provides a robust semantic foundation for future SAR research. The source code, pre-trained models and the RGBP dataset are available on the project page: https://ZimoCao.github.io/ProCap/.

Figures & tables

Surface shape	Seen scenes	Unseen scenes
Planar	01, 02, 03, 04, 05, 06, 07, 08, 09, 21, 25, 28, 29, 37, 45, 47, 60	61, 63
Mildly curved	10, 11, 12, 13, 15, 17, 18, 19, 20, 23, 24, 32, 33, 34, 35, 39, 46, 49, 51, 52, 53, 55, 56	64, 65
Highly curved	14, 16, 22, 26, 27, 30, 31, 36, 38, 40, 41, 42, 43, 44, 48, 50, 54, 57, 58, 59	62

Table 1: Surface shape variations of the RGBP dataset

Figure 1: Configuration of the RGBP dataset capture environment.

Split	Physical scenes	Projected images per scene	Total pairs
Train	60	COCO train (2,000 ∗ )	118,287
		COCO val (160)
Eval.	60 (seen) + 5 (unseen)	nocaps val (300)	62,400
		WHOOPS! (500)
Total	65		180,678

Table 2: Composition of the proposed RGBP dataset. Training data are collected from 60 physical scenes using COCO train images as projected content; ∗ one COCO scene contains only 287 images. The evaluation set comprises 65 physical scenes, including 60 seen and 5 unseen, using COCO validation, nocaps, and WHOOPS! images as projected content. Total pairs = physical scenes

\times

projected images per scene.

Figure 2: RGBP scenes used to train and evaluate models. We show four representative scenes with projection, highlighted by the blue, yellow, red, and green boxes, respectively. Note that the projection masks are coarse and they do not full match the real projection regions (see \cref sec:projection_segmentation for details).

Figure 3: Overview of the proposed ProCap architecture. Given an observed image $\boldsymbol{I}$ containing both physical scene and projected content, a frozen vision transformer (ViT-g) backbone first extracts coarse features $\boldsymbol{Z}_{\text{c}}$ , which are refined into $\mathcal{U}(\boldsymbol{Z}_{\text{c}})$ by a feature refinement module $\mathcal{U(\cdot)}$ . A projection segmentation module $\mathcal{S}$ is employed to estimate a coarse projection mask $\boldsymbol{I}_{\text{m}}$ , enabling mask pooling to retain projection features as $\boldsymbol{Z}_{\text{p}}$ . The scene and projection features $\boldsymbol{Z}_{\text{c}}$ and $\boldsymbol{Z}_{\text{p}}$ are then processed by two specialized Q-Formers: a scene Q-Former and a projection Q-Former to obtain scene embeddings $\boldsymbol{Q}_{\text{s}}$ and projection embeddings $\boldsymbol{Q}_{\text{p}}$ , respectively. $\boldsymbol{Q}_{\text{p}}$ is further used to retrieve similar object names (semantic context) $\boldsymbol{N}$ from external semantic knowledge base $\mathcal{M}$ . The retrieved semantic context and $\boldsymbol{Q}_{\text{p}}$ together are then encoded by a knowledge Q-Former as $\boldsymbol{Q}_{\text{k}}$ . Finally, the scene embeddings $\boldsymbol{Q}_{\text{s}}$ , projection embeddings $\boldsymbol{Q}_{\text{p}}$ , and semantic context embeddings $\boldsymbol{Q}_{\text{k}}$ are projected into the embedding space of a frozen LLM decoder via a linear layer $\phi$ , conditioned to generate separate captions for both the physical scene and the projected content. Lock and fire symbols stand for frozen and trainable parameters, respectively.

Figure 4: Qualitative comparison of descriptive ability in complex SAR scenes. We evaluate ProCap ${}_{\text{OpenLLaMA-3B}}$ against FastVLM-7B [ 62 ] across three seen and three unseen scenes. The top panels compare descriptions of physical scenes, while the bottom panels focus on projected content. Incorrect descriptions are highlighted in red and correct ones in blue . Note that the model outputs are truncated by the max tokens limit.

Task	Method	COCO		nocaps val		WHOOPS!
		Test		Overall		Test
		C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
Scene captioning	FastVLM-0.5B [ 62 ]	1.72	9.20	1.60	9.47	1.59	10.17
	FastVLM-1.5B [ 62 ]	1.92	8.77	1.73	9.16	1.74	9.92
	FastVLM-7B [ 62 ]	2.31	10.31	2.21	10.79	2.21	11.98
	Qwen3-VL-2B-Instruct [ 6 ]	2.34	12.17	2.12	12.17	2.01	12.83

Table 3: Comparison on the 60 seen scenes with unseen projected content . CIDEr (C), SPICE (S) are reported. Results are averaged on all scenes. The best results are in bold , and the second best results are underlined . See the supplementary for the full table 1 1 1 Qwen3 (-VL) was released after the submission deadline of this work. Following reviewers’ suggestions, we additionally report its evaluation and fine-tuning results. However, it was not adopted as the backbone of our framework due to time constraints and will be explored in future work. .

Task	Method	COCO				nocaps val								WHOOPS!
		Test				In-domain		Near-domain		Out-domain		Overall		Test
		B@4 $\uparrow$	M $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
Scene captioning	FastVLM-0.5B [ 62 ]	2.16	10.98	1.44	5.42	1.42	5.20	1.50	4.96	1.90	6.62	1.32	5.58	1.02	6.78
	FastVLM-1.5B [ 62 ]	1.88	10.90	0.98	5.36	1.14	5.14	1.14	5.98	1.48	8.16	1.04	6.44	0.84	8.50
	FastVLM-7B [ 62 ]	2.10	11.64	1.24	6.00	1.60	6.30	1.48	6.02	1.88	7.80	1.38	6.72	1.34	8.66
	Qwen3-VL-2B-Instruct [ 6 ]	4.48	15.42	2.36	9.32	2.56	8.66	2.66	8.64	2.66	9.24	2.18	8.86	1.86	8.94

Table 4: Comparison on the 5 unseen scenes with unseen projected content . BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S) are reported. The best and the second best results are in bold and underlined , respectively. See the supplementary for the full table 1 1 footnotemark: 1 .

Task	Method	COCO		nocaps val		WHOOPS!
		Test		Overall		Test
		C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
Scene	ProCap ${}_{\text{TinyLlama-1.1B}}$	5.02	5.53	4.06	8.34	7.48	5.15
	ProCap ${}_{\text{TinyLlama-1.1B w/o mask}}$	2.32	6.36	1.78	6.50	6.85	7.00
Proj.	ProCap ${}_{\text{TinyLlama-1.1B}}$	46.90	12.60	32.36	5.10	15.23	4.57
	ProCap ${}_{\text{TinyLlama-1.1B w/o mask}}$	56.50	10.68	25.58	5.20	19.70	5.98

Table 5: Ablation study of projection segmentation module on the 5 unseen scenes with unseen projected content . CIDEr (C), SPICE (S) are reported. The best results are in bold .

Task	Method	COCO				nocaps val								WHOOPS!
		Test				In-domain		Near-domain		Out-domain		Overall		Test
		B@4 $\uparrow$	M $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
Scene	ProCap ${}_{\text{OpenLLaMA-3B}}$	1.50	11.73	3.78	7.77	4.88	7.16	4.96	7.28	5.56	7.14	4.02	7.10	5.73	6.60
Scene	ProCap ${}_{\text{OpenLLaMA-3B, w/o refinement}}$	0.25	10.60	3.68	7.78	4.56	6.10	4.34	6.28	4.62	6.46	3.60	6.28	5.52	5.73
	ProCap ${}_{\text{OpenLLaMA-3B, w/o projection Q-Former}}$	1.88	10.60	3.80	6.67	4.74	5.88	4.22	5.26	4.12	5.94	3.42	5.60	6.18	5.42
Projection	ProCap ${}_{\text{OpenLLaMA-3B}}$	19.90	21.22	66.03	14.90	58.22	8.66	48.00	7.34	27.10	5.20	44.72	7.06	24.65	6.80

Table 6: Ablation study of the feature refinement module and dual-captioning strategy on the 5 unseen scenes with unseen projected content . BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S) are reported. The best results are in bold .

Method	COCO		nocaps val		WHOOPS!
	Test		Overall		Test
	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
ProCap ${}_{\text{Vicuna-1.5-7B}}$	86.26	14.88	43.94	7.10	24.44	6.60
ProCap ${}_{\text{Vicuna-1.5-7B, w/o RAR}}$	67.98	11.64	36.28	5.48	20.48	5.50

Table 7: Ablation study of region-aware retrieval (RAR) on the 5 unseen scenes with unseen projected content . CIDEr (C), SPICE (S) are reported. The best results are in bold .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Training time	GPUs
ProCap ${}_{\text{TinyLlama-1.1B}}$ [ 70 ]	$\sim$ 4.5h	3 RTX4060Ti
ProCap ${}_{\text{OpenLLaMA-3B}}$ [ 19 ]	$\sim$ 4.5h	3 RTX4060Ti
ProCap ${}_{\text{OPT-2.7B}}$ [ 71 ]	$\sim$ 4.3h	3 RTX4060Ti
ProCap ${}_{\text{Llama-3.3-8B}}$ [ 2 ]	$\sim$ 4.2h	1 A100
ProCap ${}_{\text{Vicuna-1.5-7B}}$ [ 72 ]	$\sim$ 4.2h	1 A100
RGBP ${}_{\text{Qwen3-VL-8B-Instruct}}$ [ 6 ]	$\sim$ 6.0h	1 RTX PRO 6000 Blackwell

Appendix

Table 1: Methods on training time and used GPUs.

Figure 1: This figure shows the specific details of 60 seen scenes with projected content A (traffic lights) and projected content B (dolphins).

Task	Method	COCO				nocaps val								WHOOPS!
		Test				In-domain		Near-domain		Out-domain		Overall		Test
		B@4 $\uparrow$	M $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
Scene captioning	FastVLM-0.5B [ 62 ]	4.76	14.25	1.72	9.20	1.87	8.83	1.78	8.93	2.16	10.65	1.60	9.47	1.59	10.17
	FastVLM-1.5B [ 62 ]	4.75	14.15	1.92	8.77	1.98	8.44	1.96	8.64	2.33	10.42	1.73	9.16	1.74	9.92
	FastVLM-7B [ 62 ]	5.62	15.63	2.31	10.31	2.51	10.13	2.55	10.16	2.87	12.08	2.21	10.79	2.21	11.98
	Qwen2.5-VL-3B-Instruct [ 7 ]	7.95	18.56	2.20	11.99	2.28	11.93	2.29	11.86	2.58	12.85	2.00	12.21	1.99	13.04

Appendix

Table 2: Performance comparison on the 60 seen scenes with unseen projected content in main paper. Metrics are reported as BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S). Results are averaged on the 60 scenes, and the best results are in bold .

Task	Method	COCO				nocaps val								WHOOPS!
		Test				In-domain		Near-domain		Out-domain		Overall		Test
		B@4 $\uparrow$	M $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
Scene captioning	FastVLM-0.5B [ 62 ]	2.16	10.98	1.44	5.42	1.42	5.20	1.50	4.96	1.90	6.62	1.32	5.58	1.02	6.78
	FastVLM-1.5B [ 62 ]	1.88	10.90	0.98	5.36	1.14	5.14	1.14	5.98	1.48	8.16	1.04	6.44	0.84	8.50
	FastVLM-7B [ 62 ]	2.10	11.64	1.24	6.00	1.60	6.30	1.48	6.02	1.88	7.80	1.38	6.72	1.34	8.66
	Qwen2.5-VL-3B-Instruct [ 7 ]	4.40	14.08	2.14	7.50	2.36	7.50	2.46	7.30	2.52	8.58	2.10	7.78	1.82	8.94

Appendix

Table 3: Performance comparison on the 5 unseen scenes with unseen projected content in main paper. Metrics are reported as BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S). Results are averaged on the 5 scenes, and the best results are in bold .

Projection Content	COCO				nocaps val								WHOOPS!
	Test				In-domain		Near-domain		Out-domain		Overall		Test
	B@4 $\uparrow$	M $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
ProCap ${}_{\text{Vicuna-1.5-7B, seen scene}}$	24.58	21.05	78.99	13.83	55.19	8.32	39.75	6.68	22.85	4.75	39.93	6.59	22.33	6.02
ProCap ${}_{\text{Vicuna-1.5-7B, unseen scene}}$	26.20	22.08	86.26	14.88	58.70	8.84	43.46	7.18	27.32	5.30	43.94	7.10	24.44	6.60
ProCap ${}_{\text{Vicuna-1.5-7B, projected content}}$	26.20	23.00	89.80	15.60	63.6	9.40	45.60	7.90	18.70	3.90	43.50	7.10	27.60	7.40

Appendix

Table 4: Performance comparison on unseen projection captioning task on the 60 seen scenes (ProCap

{}_{\text{Vicuna-1.5-7B, seen scene}}

) and the 5 unseen scenes (ProCap

{}_{\text{Vicuna-1.5-7B, unseen scene}}

). In particular, we captioned projected content themselves (ProCap

{}_{\text{Vicuna-1.5-7B, projected content}}

) instead of in physical scenes. Metrics are reported as BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S). Results are averaged, and the best results are in bold .

Task	Method	COCO		nocaps val		WHOOPS!
		Test		Overall		Test
		C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$	C $\uparrow$	S $\uparrow$
Scene	Qwen3-VL-8B-Instruct ${}_{\text{w/~~ RGBP fine-tuned}}$	3.18	6.84	2.96	6.80	2.92	7.46
Scene	Qwen3-VL-8B-Instruct ${}_{\text{w/o RGBP fine-tuned}}$	0.48	9.84	0.38	9.58	0.32	10.20
Proj.	Qwen3-VL-8B-Instruct ${}_{\text{w/~~ RGBP fine-tuned}}$	133.50	22.18	106.38	13.64	87.10	17.34
Proj.	Qwen3-VL-8B-Instruct ${}_{\text{w/o RGBP fine-tuned}}$	4.96	15.32	8.08	13.54	5.80	13.18

Appendix

Table 5: Performance comparison on the additional 5 unseen scenes with unseen projected content, using an EPSON EB-C2050WN 3LCD projector (1600 × 1200) and a Nikon D3200 DSLR camera (1080 × 720). Metrics are reported as CIDEr (C), SPICE (S). Results are averaged on the 5 scenes, and the best results are in bold . “Proj.” denotes Projection.

Figure 2: Examples of projection mapping based on TOSHIBA TDP-T100C DLP projector (1024 $\times$ 768) and Nikon D3200 DSLR camera (1280 $\times$ 720) using ProCap ${}_{\text{Vicuna-1.5-7B}}$ . We highlight incorrect captioned objects in red and correct ones in blue .

Augmented Reality

See all 18 papers

LICOA: Learning Instruction-Conditioned Optical Actions for Mobile Spatial Augmented Reality
Oct 15, 2026Yuchen Deng, Shunkai Zhou, Ruixuan Zhu +2Augmented Reality Robot Learning
Southwest University · Peking University · Westlake University
SEGAR: Selective Enhancement for Generative Augmented Reality
Mar 25, 2026Fanjun Bu, Chenyang Yuan, Hiroshi YasudaVideo World Models Video Diffusion Models
Cornell University, Cornell Tech New York, NY, USA · Toyota Research Institute Cambridge, MA · Toyota Research Institute Los Altos, CA
SARAH: Spatially Aware Real-time Agentic Humans
Feb 20, 2026Evonne Ng, Siwei Zhang, Zhang Chen +2Human Motion Generation Co-Speech Gesture Generation
Meta Reality Labs, Redmond, WA, USA

Oct 15, 2026J/K move · Enter open · S save

LICOA: Learning Instruction-Conditioned Optical Actions for Mobile Spatial Augmented Reality

Yuchen Deng, Shunkai Zhou, Ruixuan Zhu, Haibin Ling+1

Southwest University · Peking University · Westlake University

Augmented Reality Robot Learning Language-Conditioned Robot Control

Open paper AskPDF arXiv

Spatial augmented reality (SAR) augments physical environments with projected content, but most systems assume that the projector pose, target region, or motion path has been fixed or specified. We instead make projected light an embodied action: given a language instruction and current visual observations, a system must decide how to move and where and at what scale to place the projection. We introduce LICOA to realize this capability. Its differential-drive chassis provides room-scale approach and repositioning, while a steerable mirror with two degrees of freedom (2DoF) and projection scale provide rapid local control of the beam. A shared interface maps synchronized external and onboard RGB cameras with robot control state to physical optical actions across pretrained action models. Because instruction-conditioned optical action lacks dedicated training data and a common evaluation protocol, we construct a matched simulation embodiment and a corpus with 150 tasks and 7,500 rendered episodes, then evaluate models trained only on this corpus in previously unseen real-world environments. Across 480 complete closed-loop real-world execution trials (rollouts) on 12 tasks, the best adapted action model achieves a 57.50% success rate and 40.64% success weighted by path length, compared with 8.33% and 5.15% for the same architecture trained from scratch. A further 150 real-world ablation rollouts support the need for both camera views and coordinated control of the chassis and mirror. These results establish a physical platform, shared perception and optical action interface, corpus, and real-world evaluation protocol for treating projected light as an embodied action.

Abstract

Augmented Reality

Explore similar work

LICOA: Learning Instruction-Conditioned Optical Actions for Mobile Spatial Augmented Reality

SEGAR: Selective Enhancement for Generative Augmented Reality

SARAH: Spatially Aware Real-time Agentic Humans