Spatial augmented reality (SAR) directly projects digital content onto physical scenes using projectors, creating immersive experience without head-mounted displays. However, for SAR to support intelligent interaction, such as reasoning about the scene or answering user queries, it must semantically distinguish between the physical scene and the projected content. Standard Vision Language Models (VLMs) struggle with this virtual-physical ambiguity, often confusing the two contexts. To address this issue, we introduce ProCap, a novel framework that explicitly decouples projected content from physical scenes. ProCap employs a two-stage pipeline: first it visually isolates virtual and physical layers via automated segmentation; then it uses region-aware retrieval to avoid ambiguous semantic context due to projection distortion. To support this, we present RGBP (RGB + Projections), the first large-scale SAR semantic benchmark dataset, featuring 65 diverse physical scenes and over 180,000 projections with dense, decoupled annotations. Finally, we establish a dual-captioning evaluation protocol using task-specific tokens to assess physical scene and projection descriptions independently. Our experiments show that ProCap provides a robust semantic foundation for future SAR research. The source code, pre-trained models and the RGBP dataset are available on the project page: https://ZimoCao.github.io/ProCap/.
Table 1: Surface shape variations of the RGBP dataset
Figure 1: Configuration of the RGBP dataset capture environment.
Split
Physical scenes
Projected images per scene
Total pairs
Train
60
COCO train (2,000 ∗ )
118,287
COCO val (160)
Eval.
60 (seen) + 5 (unseen)
nocaps val (300)
62,400
WHOOPS! (500)
Total
65
180,678
Table 2: Composition of the proposed RGBP dataset. Training data are collected from 60 physical scenes using COCO train images as projected content; ∗ one COCO scene contains only 287 images. The evaluation set comprises 65 physical scenes, including 60 seen and 5 unseen, using COCO validation, nocaps, and WHOOPS! images as projected content. Total pairs = physical scenes × projected images per scene.
Figure 2: RGBP scenes used to train and evaluate models. We show four representative scenes with projection, highlighted by the blue, yellow, red, and green boxes, respectively. Note that the projection masks are coarse and they do not full match the real projection regions (see \cref sec:projection_segmentation for details).
Figure 3: Overview of the proposed ProCap architecture. Given an observed image I containing both physical scene and projected content, a frozen vision transformer (ViT-g) backbone first extracts coarse features Zc , which are refined into U(Zc) by a feature refinement module U(⋅) . A projection segmentation module S is employed to estimate a coarse projection mask Im , enabling mask pooling to retain projection features as Zp . The scene and projection features Zc and Zp are then processed by two specialized Q-Formers: a scene Q-Former and a projection Q-Former to obtain scene embeddings Qs and projection embeddings Qp , respectively. Qp is further used to retrieve similar object names (semantic context) N from external semantic knowledge base M . The retrieved semantic context and Qp together are then encoded by a knowledge Q-Former as Qk . Finally, the scene embeddings Qs , projection embeddings Qp , and semantic context embeddings Qk are projected into the embedding space of a frozen LLM decoder via a linear layer ϕ , conditioned to generate separate captions for both the physical scene and the projected content. Lock and fire symbols stand for frozen and trainable parameters, respectively.
Figure 4: Qualitative comparison of descriptive ability in complex SAR scenes. We evaluate ProCap OpenLLaMA-3B against FastVLM-7B [ 62 ] across three seen and three unseen scenes. The top panels compare descriptions of physical scenes, while the bottom panels focus on projected content. Incorrect descriptions are highlighted in red and correct ones in blue . Note that the model outputs are truncated by the max tokens limit.
Task
Method
COCO
nocaps val
WHOOPS!
Test
Overall
Test
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
Scene captioning
FastVLM-0.5B [ 62 ]
1.72
9.20
1.60
9.47
1.59
10.17
FastVLM-1.5B [ 62 ]
1.92
8.77
1.73
9.16
1.74
9.92
FastVLM-7B [ 62 ]
2.31
10.31
2.21
10.79
2.21
11.98
Qwen3-VL-2B-Instruct [ 6 ]
2.34
12.17
2.12
12.17
2.01
12.83
Table 3: Comparison on the 60 seen scenes with unseen projected content . CIDEr (C), SPICE (S) are reported. Results are averaged on all scenes. The best results are in bold , and the second best results are underlined . See the supplementary for the full table 1 1 1 Qwen3 (-VL) was released after the submission deadline of this work. Following reviewers’ suggestions, we additionally report its evaluation and fine-tuning results. However, it was not adopted as the backbone of our framework due to time constraints and will be explored in future work. .
Task
Method
COCO
nocaps val
WHOOPS!
Test
In-domain
Near-domain
Out-domain
Overall
Test
B@4 ↑
M ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
Scene captioning
FastVLM-0.5B [ 62 ]
2.16
10.98
1.44
5.42
1.42
5.20
1.50
4.96
1.90
6.62
1.32
5.58
1.02
6.78
FastVLM-1.5B [ 62 ]
1.88
10.90
0.98
5.36
1.14
5.14
1.14
5.98
1.48
8.16
1.04
6.44
0.84
8.50
FastVLM-7B [ 62 ]
2.10
11.64
1.24
6.00
1.60
6.30
1.48
6.02
1.88
7.80
1.38
6.72
1.34
8.66
Qwen3-VL-2B-Instruct [ 6 ]
4.48
15.42
2.36
9.32
2.56
8.66
2.66
8.64
2.66
9.24
2.18
8.86
1.86
8.94
Table 4: Comparison on the 5 unseen scenes with unseen projected content . BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S) are reported. The best and the second best results are in bold and underlined , respectively. See the supplementary for the full table 1 1 footnotemark: 1 .
Task
Method
COCO
nocaps val
WHOOPS!
Test
Overall
Test
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
Scene
ProCap TinyLlama-1.1B
5.02
5.53
4.06
8.34
7.48
5.15
ProCap TinyLlama-1.1B w/o mask
2.32
6.36
1.78
6.50
6.85
7.00
Proj.
ProCap TinyLlama-1.1B
46.90
12.60
32.36
5.10
15.23
4.57
ProCap TinyLlama-1.1B w/o mask
56.50
10.68
25.58
5.20
19.70
5.98
Table 5: Ablation study of projection segmentation module on the 5 unseen scenes with unseen projected content . CIDEr (C), SPICE (S) are reported. The best results are in bold .
Task
Method
COCO
nocaps val
WHOOPS!
Test
In-domain
Near-domain
Out-domain
Overall
Test
B@4 ↑
M ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
Scene
ProCap OpenLLaMA-3B
1.50
11.73
3.78
7.77
4.88
7.16
4.96
7.28
5.56
7.14
4.02
7.10
5.73
6.60
ProCap OpenLLaMA-3B, w/o refinement
0.25
10.60
3.68
7.78
4.56
6.10
4.34
6.28
4.62
6.46
3.60
6.28
5.52
5.73
ProCap OpenLLaMA-3B, w/o projection Q-Former
1.88
10.60
3.80
6.67
4.74
5.88
4.22
5.26
4.12
5.94
3.42
5.60
6.18
5.42
Projection
ProCap OpenLLaMA-3B
19.90
21.22
66.03
14.90
58.22
8.66
48.00
7.34
27.10
5.20
44.72
7.06
24.65
6.80
Table 6: Ablation study of the feature refinement module and dual-captioning strategy on the 5 unseen scenes with unseen projected content . BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S) are reported. The best results are in bold .
Method
COCO
nocaps val
WHOOPS!
Test
Overall
Test
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
ProCap Vicuna-1.5-7B
86.26
14.88
43.94
7.10
24.44
6.60
ProCap Vicuna-1.5-7B, w/o RAR
67.98
11.64
36.28
5.48
20.48
5.50
Table 7: Ablation study of region-aware retrieval (RAR) on the 5 unseen scenes with unseen projected content . CIDEr (C), SPICE (S) are reported. The best results are in bold .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Training time
GPUs
ProCap TinyLlama-1.1B [ 70 ]
∼ 4.5h
3 RTX4060Ti
ProCap OpenLLaMA-3B [ 19 ]
∼ 4.5h
3 RTX4060Ti
ProCap OPT-2.7B [ 71 ]
∼ 4.3h
3 RTX4060Ti
ProCap Llama-3.3-8B [ 2 ]
∼ 4.2h
1 A100
ProCap Vicuna-1.5-7B [ 72 ]
∼ 4.2h
1 A100
RGBP Qwen3-VL-8B-Instruct [ 6 ]
∼ 6.0h
1 RTX PRO 6000 Blackwell
Appendix
Table 1: Methods on training time and used GPUs.
Figure 1: This figure shows the specific details of 60 seen scenes with projected content A (traffic lights) and projected content B (dolphins).
Task
Method
COCO
nocaps val
WHOOPS!
Test
In-domain
Near-domain
Out-domain
Overall
Test
B@4 ↑
M ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
Scene captioning
FastVLM-0.5B [ 62 ]
4.76
14.25
1.72
9.20
1.87
8.83
1.78
8.93
2.16
10.65
1.60
9.47
1.59
10.17
FastVLM-1.5B [ 62 ]
4.75
14.15
1.92
8.77
1.98
8.44
1.96
8.64
2.33
10.42
1.73
9.16
1.74
9.92
FastVLM-7B [ 62 ]
5.62
15.63
2.31
10.31
2.51
10.13
2.55
10.16
2.87
12.08
2.21
10.79
2.21
11.98
Qwen2.5-VL-3B-Instruct [ 7 ]
7.95
18.56
2.20
11.99
2.28
11.93
2.29
11.86
2.58
12.85
2.00
12.21
1.99
13.04
Appendix
Table 2: Performance comparison on the 60 seen scenes with unseen projected content in main paper. Metrics are reported as BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S). Results are averaged on the 60 scenes, and the best results are in bold .
Task
Method
COCO
nocaps val
WHOOPS!
Test
In-domain
Near-domain
Out-domain
Overall
Test
B@4 ↑
M ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
Scene captioning
FastVLM-0.5B [ 62 ]
2.16
10.98
1.44
5.42
1.42
5.20
1.50
4.96
1.90
6.62
1.32
5.58
1.02
6.78
FastVLM-1.5B [ 62 ]
1.88
10.90
0.98
5.36
1.14
5.14
1.14
5.98
1.48
8.16
1.04
6.44
0.84
8.50
FastVLM-7B [ 62 ]
2.10
11.64
1.24
6.00
1.60
6.30
1.48
6.02
1.88
7.80
1.38
6.72
1.34
8.66
Qwen2.5-VL-3B-Instruct [ 7 ]
4.40
14.08
2.14
7.50
2.36
7.50
2.46
7.30
2.52
8.58
2.10
7.78
1.82
8.94
Appendix
Table 3: Performance comparison on the 5 unseen scenes with unseen projected content in main paper. Metrics are reported as BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S). Results are averaged on the 5 scenes, and the best results are in bold .
Projection Content
COCO
nocaps val
WHOOPS!
Test
In-domain
Near-domain
Out-domain
Overall
Test
B@4 ↑
M ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
ProCap Vicuna-1.5-7B, seen scene
24.58
21.05
78.99
13.83
55.19
8.32
39.75
6.68
22.85
4.75
39.93
6.59
22.33
6.02
ProCap Vicuna-1.5-7B, unseen scene
26.20
22.08
86.26
14.88
58.70
8.84
43.46
7.18
27.32
5.30
43.94
7.10
24.44
6.60
ProCap Vicuna-1.5-7B, projected content
26.20
23.00
89.80
15.60
63.6
9.40
45.60
7.90
18.70
3.90
43.50
7.10
27.60
7.40
Appendix
Table 4: Performance comparison on unseen projection captioning task on the 60 seen scenes (ProCap Vicuna-1.5-7B, seen scene ) and the 5 unseen scenes (ProCap Vicuna-1.5-7B, unseen scene ). In particular, we captioned projected content themselves (ProCap Vicuna-1.5-7B, projected content ) instead of in physical scenes. Metrics are reported as BLEU@4 (B@4), METEOR (M), CIDEr (C), SPICE (S). Results are averaged, and the best results are in bold .
Task
Method
COCO
nocaps val
WHOOPS!
Test
Overall
Test
C ↑
S ↑
C ↑
S ↑
C ↑
S ↑
Scene
Qwen3-VL-8B-Instruct w/ RGBP fine-tuned
3.18
6.84
2.96
6.80
2.92
7.46
Qwen3-VL-8B-Instruct w/o RGBP fine-tuned
0.48
9.84
0.38
9.58
0.32
10.20
Proj.
Qwen3-VL-8B-Instruct w/ RGBP fine-tuned
133.50
22.18
106.38
13.64
87.10
17.34
Qwen3-VL-8B-Instruct w/o RGBP fine-tuned
4.96
15.32
8.08
13.54
5.80
13.18
Appendix
Table 5: Performance comparison on the additional 5 unseen scenes with unseen projected content, using an EPSON EB-C2050WN 3LCD projector (1600 × 1200) and a Nikon D3200 DSLR camera (1080 × 720). Metrics are reported as CIDEr (C), SPICE (S). Results are averaged on the 5 scenes, and the best results are in bold . “Proj.” denotes Projection.
Figure 2: Examples of projection mapping based on TOSHIBA TDP-T100C DLP projector (1024 × 768) and Nikon D3200 DSLR camera (1280 × 720) using ProCap Vicuna-1.5-7B . We highlight incorrect captioned objects in red and correct ones in blue .
Spatial augmented reality (SAR) augments physical environments with projected content, but most systems assume that the projector pose, target region, or motion path has been fixed or specified. We instead make projected light an embodied action: given a language instruction and current visual observations, a system must decide how to move and where and at what scale to place the projection. We introduce LICOA to realize this capability. Its differential-drive chassis provides room-scale approach and repositioning, while a steerable mirror with two degrees of freedom (2DoF) and projection scale provide rapid local control of the beam. A shared interface maps synchronized external and onboard RGB cameras with robot control state to physical optical actions across pretrained action models. Because instruction-conditioned optical action lacks dedicated training data and a common evaluation protocol, we construct a matched simulation embodiment and a corpus with 150 tasks and 7,500 rendered episodes, then evaluate models trained only on this corpus in previously unseen real-world environments. Across 480 complete closed-loop real-world execution trials (rollouts) on 12 tasks, the best adapted action model achieves a 57.50% success rate and 40.64% success weighted by path length, compared with 8.33% and 5.15% for the same architecture trained from scratch. A further 150 real-world ablation rollouts support the need for both camera views and coordinated control of the chassis and mirror. These results establish a physical platform, shared perception and optical action interface, corpus, and real-world evaluation protocol for treating projected light as an embodied action.
Yuchen Deng, Shunkai Zhou, Ruixuan Zhu +2
Southwest University · Peking University · Westlake University
Generative world models offer a compelling foundation for augmented-reality (AR) applications: by predicting future image sequences that incorporate deliberate visual edits, they enable temporally coherent, augmented future frames that can be computed ahead of time and cached, avoiding per-frame rendering from scratch in real time. In this work, we present SEGAR, a preliminary framework that combines a diffusion-based world model with a selective correction stage to support this vision. The world model generates augmented future frames with region-specific edits while preserving others, and the correction stage subsequently aligns safety-critical regions with real-world observations while preserving intended augmentations elsewhere. We demonstrate this pipeline in driving scenarios as a representative setting where semantic region structure is well defined and real-world feedback is readily available. We view this as an early step toward generative world models as practical AR infrastructure, where future frames can be generated, cached, and selectively corrected on demand.
Fanjun Bu, Chenyang Yuan, Hiroshi Yasuda
Cornell University, Cornell Tech New York, NY, USA · Toyota Research Institute Cambridge, MA · Toyota Research Institute Los Altos, CA
As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see https://evonneng.github.io/sarah/ for details.