LICOA: Learning Instruction-Conditioned Optical Actions for Mobile Spatial Augmented Reality

Yuchen Deng; Shunkai Zhou; Ruixuan Zhu; Haibin Ling; Bingyao Huang

cs.ROOct 15, 2026

LICOA: Learning Instruction-Conditioned Optical Actions for Mobile Spatial Augmented Reality

Authors: Yuchen Deng, Shunkai Zhou, Ruixuan Zhu, Haibin Ling, Bingyao Huang

Organizations: Southwest University · Peking University · Westlake University

Abstract

Spatial augmented reality (SAR) augments physical environments with projected content, but most systems assume that the projector pose, target region, or motion path has been fixed or specified. We instead make projected light an embodied action: given a language instruction and current visual observations, a system must decide how to move and where and at what scale to place the projection. We introduce LICOA to realize this capability. Its differential-drive chassis provides room-scale approach and repositioning, while a steerable mirror with two degrees of freedom (2DoF) and projection scale provide rapid local control of the beam. A shared interface maps synchronized external and onboard RGB cameras with robot control state to physical optical actions across pretrained action models. Because instruction-conditioned optical action lacks dedicated training data and a common evaluation protocol, we construct a matched simulation embodiment and a corpus with 150 tasks and 7,500 rendered episodes, then evaluate models trained only on this corpus in previously unseen real-world environments. Across 480 complete closed-loop real-world execution trials (rollouts) on 12 tasks, the best adapted action model achieves a 57.50% success rate and 40.64% success weighted by path length, compared with 8.33% and 5.15% for the same architecture trained from scratch. A further 150 real-world ablation rollouts support the need for both camera views and coordinated control of the chassis and mirror. These results establish a physical platform, shared perception and optical action interface, corpus, and real-world evaluation protocol for treating projected light as an embodied action.

Figures & tables

System / architecture	Projector origin mobility	Action components	Spatial instruction	Learned closed-loop control	Matched simulation
Fixed ProCams [ 30 , 17 , 45 , 12 , 13 ]	Fixed	No	Content prompt in some	No	Optics simulation in some
Fixed multi-projector systems [ 11 , 56 , 36 ]	Fixed	No	No	No	No
Fixed beam-steered [ 53 , 42 , 64 ]	Fixed	Optical steering only	Restricted speech / gesture in some	Rules; no learned action model	No
Arm-mounted [ 47 , 7 , 49 , 27 ]	Arm workspace	Arm only	Speech / gesture / tracking / script	Tracking / script; no learned action model	No
Mobile + fixed optics [ 19 , 24 , 61 ]	Room-scale	Chassis; fixed optical axis	Detected surface	Geometry; no learned action model	No
Mobile + steerable optics [ 63 , 69 , 57 , 44 ]	Room-scale	Chassis + PT / gimbal	Marker / path / script / search	Servo / planning / script; no learned action model	No

Table 1: Representative SAR architecture classes evaluated against the requirements of instruction-conditioned optical action. Cells summarize the range demonstrated by the cited systems in each class; an individual system may not support every listed capability. The comparison concerns physical projection placement rather than projected content quality.

Figure 1: The LICOA physical platform and its complementary motion scales. Setup : The mobile projection robot and tripod-mounted external RGB camera. Rear : The Kobuki differential-drive chassis, supporting frame, and integrated electronics for room-scale approach and repositioning. Front : The onboard RGB camera and projector assembly. Top : The NVIDIA Jetson Nano and embedded control electronics. 2DoF steerable mirror and projector : The mirror mounted above the projector redirects the beam locally without rotating the differential-drive chassis. Projection scale controls the footprint size.

Subsystem	Hardware	Role
Differential-drive chassis	Kobuki mobile chassis	Room-scale approach and repositioning
Onboard sensing	Microsoft Kinect v1	Robot-centric RGB observation
External sensing	Logitech C270	Global RGB observation
Embedded computing	NVIDIA Jetson Nano	Robot-side sensing and execution bridge
Projection	Custom laser projector	Optical output and footprint scaling
Optical steering	Custom 2DoF steerable mirror	Rapid local beam redirection

Table 2: Hardware roles in the physical LICOA platform. The chassis provides broad repositioning, while the mirror and projector provide local beam direction and scale control. Detailed interfaces, electronics, power distribution, and embedded control are provided in the Suppl. Material.

Figure 2: LICOA makes projected light executable through a shared perception and optical action interface. Top : The instruction $\ell$ , external and onboard RGB images, and five-dimensional robot control state pass through the encoders, backbone, and action module. Supervised fine-tuning adapts the action model to return $H$ -step action chunks. Each six-dimensional action contains commands for the differential-drive chassis, 2DoF steerable mirror, and projection scale, plus an episode termination output; any additional model output dimensions are excluded. Bottom : The host PC synchronizes the external camera stream and robot stream, sends an inference request, executes a short prefix of the returned action chunk, and replans from fresh observations. Simulation training and real-world deployment use the same observation and action interfaces.

Scenes $10$	External viewpoints/scene $5$	Tasks/external viewpoint $3$	Total tasks $150$
Source trajectories/task $10$	Appearances/trajectory $5$	Total episodes $7{,}500$

Table 3: Composition of the LICOA matched simulation corpus. We use the first

10

InteriorAgent scenes, with five external camera viewpoints and three spatial placement tasks per viewpoint, yielding

150

tasks. Each source trajectory is rendered with the original appearance and four randomized variants in lighting, wall, and floor materials. Trajectory density and task diversity analyses are provided in the Suppl. Material.

Figure 3: Per-task evidence for real-world deployment through the shared perception and optical action interface. Each action model is trained only on the matched simulation corpus and evaluated for $10$ rollouts on each of $12$ previously unseen real-world tasks. The instructions and target scenes are shown in reading order, and the bars report success rate. All tasks contribute equally to the overall SR in \cref tab:main_comparison.

Method	Success	SR ( $\%$ ) $\uparrow$	SPL ( $\%$ ) $\uparrow$	MT ( $\mathrm{s}$ ) $\downarrow$
$\pi_{0.5}$ -scratch	$10/120$	$8.33$	$5.15$	$47.65$
$\pi_{0.5}$ [ 3 ]	$69/120$	57.50	40.64	43.44
$\pi_{0}$ -FAST [ 52 ]	$21/120$	$17.50$	$15.81$	$139.38$
Fast-WAM [ 68 ]	$35/120$	$29.17$	$21.76$	$64.72$

Table 4: Real-world evaluation results for three pretrained action model families connected through the shared perception and optical action interface and a

\pi_{0.5}

-scratch control. SR is averaged over all

12

tasks (

10

rollouts per task;

120

rollouts per action model). SPL is averaged over the same

120

rollouts, while MT is averaged over successful rollouts only. The best and second-best results are shown in bold and underlined , respectively.

Config.	$\mathbf{I}^{o}$	$\mathbf{I}^{e}$	Chassis	Mirror	Success	SR $\uparrow$	SPL $\uparrow$	MT $\downarrow$
Full	✓	✓	✓	✓	$18/30$	$60.00$	$37.09$	$12.20$
W/o onb. cam.	$\times$	✓	✓	✓	$9/30$	$30.00$	$24.61$	$29.07$
W/o ext. cam.	✓	$\times$	✓	✓	$8/30$	$26.67$	$13.47$	$6.57$
Mirror only	✓	✓	$\times$	✓	$9/30$	$30.00$	$30.00$	$45.71$
Chassis only	✓	✓	✓	$\times$	$4/30$	$13.33$	$12.04$	$23.86$

Table 5: Camera and action component ablations for

\pi_{0.5}

on three novel real-world tasks (

10

rollouts per task;

30

per configuration). Camera variants are retrained without the indicated view, whereas actuation variants disable the indicated component only at deployment. All variants retain the robot control state and projection-scale command. The images below show Task

13

, “project white light onto the umbrella.”; Task

14

, “project white light onto the brown bag.”; and Task

15

, “project white light onto the white statue.”

Figure 4: Proof-of-concept applications of LICOA. Left : RGB colored point clouds reconstructed from structured light sequences for several surfaces and objects. Right, from top to bottom : desired projection content, compensated camera observations, and uncompensated camera observations on one textured surface. The compensation images are rectified to projector coordinates for visual comparison.

Figure 5: Qualitative rollouts of the full $\pi_{0.5}$ configuration with varied projection content. Each row shows selected frames from one rollout. From top to bottom, the instructions are “project circle light onto the wooden surface,” “project green light onto the wooden surface,” and “project image onto the wooden surface.”

Source	Output	Load	Purpose
Battery pack $1$ ( $6{,}000\,\mathrm{mAh}$ )	Chassis supply	Kobuki mobile chassis	Differential-drive locomotion
Battery pack $1$ ( $6{,}000\,\mathrm{mAh}$ )	$12\,\mathrm{V}/1.5\,\mathrm{A}$	Microsoft Kinect v1	Onboard RGB camera power; image data use USB
Battery pack $2$ ( $30{,}000\,\mathrm{mAh}$ )	$5\,\mathrm{V}/4\,\mathrm{A}$	NVIDIA Jetson Nano	Embedded computing
Battery pack $2$ ( $30{,}000\,\mathrm{mAh}$ )	$12\,\mathrm{V}/1\,\mathrm{A}$	Two SimpleFOC Mini drivers	Actuation of the 2DoF steerable mirror
Battery pack $2$ ( $30{,}000\,\mathrm{mAh}$ )	$5\,\mathrm{V}/2\,\mathrm{A}$	Custom laser projector	Projection
NVIDIA Jetson Nano	USB bus power	STM32F103C8T6	Controller logic power, separate from USB-to-UART data

Table S1: Power distribution of the physical LICOA platform. Capacities and output ratings are the nameplate specifications of the deployed supplies.

Figure S1: Electrical and control connections for the 2DoF steerable mirror and its pan-tilt servo assembly. The NVIDIA Jetson Nano mounted on the differential-drive chassis provides the USB-to-UART command and telemetry path, while a separate Jetson USB connection (not shown) supplies logic power to the STM32. The STM32 closes both motor loops locally using AS5600 encoder feedback and drives the two brushless direct-current (BLDC) driver boards with three-phase pulse-width modulation (PWM). The

12\,\mathrm{V}/1\,\mathrm{A}

rail from battery pack

2

supplies the two SimpleFOC Mini drivers and pan-tilt motors.

Setting	Configuration
Simulator	NVIDIA Isaac Sim 6.0.1; PhysX fixed-step simulation
Scene assets	First $10$ InteriorAgent USD indoor scenes; each a complete, large indoor environment; $1$ stage unit $=1\,\mathrm{m}$ ; frozen environment rigid bodies with static triangle-mesh collision proxies
Differential-drive chassis	Differential drive; wheel radius $0.035\,\mathrm{m}$ ; wheelbase $0.230\,\mathrm{m}$ ; $\lvert v\rvert\leq 0.7\,\mathrm{m/s}$ and $\lvert\omega\rvert\leq 3.14\,\mathrm{rad/s}$
2DoF steerable mirror and projection scale	Pan range $[-\pi/2,\pi/2]$ ; tilt range $[0,\pi/2]$ ; projection scale $[0,100]\%$
Virtual projector	Textured square RectLight; uniform white $256\!\times\!256$ source image; intensity $80$ ; throw-ratio parameter $6.0$
External and onboard RGB images	Onboard RGB camera and five fixed external camera viewpoints per scene; both streams rendered at $640\!\times\!480$

Table S2: Isaac Sim configuration used to match the LICOA physical embodiment and construct its matched simulation corpus.

Configuration	Initialization	Trainable scope	Native objective	AdamW $(\mathrm{lr},\boldsymbol{\beta},\mathrm{wd})$	Clip	Schedule
$\pi_{0.5}$	Pretrained	Full action model	Flow matching	$(5\!\times\!10^{-5},(0.9,0.95),10^{-10})$	$1$	$10{,}000$ -step warmup, then constant
$\pi_{0}$ -FAST	Pretrained	Full action model	FAST token causal cross-entropy (CE)	$(2.5\!\times\!10^{-5},(0.9,0.95),10^{-10})$	$1$	$1{,}000$ -step warmup, cosine to $2.5\!\times\!10^{-6}$
Fast-WAM	Pretrained	Video and action experts	$\mathcal{L}_{\mathrm{video}}+\mathcal{L}_{\mathrm{action}}$	$(10^{-4},(0.9,0.999),10^{-2})$	$10$	Zero-warmup cosine
$\pi_{0.5}$ -scratch	Random	Full action model	Flow matching	$(5\!\times\!10^{-5},(0.9,0.95),10^{-10})$	$1$	$10{,}000$ -step warmup, then constant

Table S3: Training recipes used for the main real-world comparison. All runs use one NVIDIA H200 GPU,

30{,}000

optimizer updates, batch size

32

, seed

0

, BF16 computation, gradient checkpointing, and a

16

-step action horizon. “Clip” denotes the global gradient norm threshold.

\mathrm{lr}

and

\mathrm{wd}

denote learning rate and weight decay, respectively.

Setting	Value
Compute	Single NVIDIA H200 GPU per run ( $143{,}771\,\mathrm{MiB}$ ); batch size $32$
Precision	BF16 mixed precision; gradient checkpointing; no compilation
Optimizer	AdamW, $\boldsymbol{\beta}=(0.9,0.95)$ , $\epsilon=10^{-8}$ , weight decay $10^{-10}$
Learning rate	$5\!\times\!10^{-5}$ ; $10{,}000$ -step linear warmup, then constant
Update budget	$30{,}000$ steps ( $960{,}000$ sample draws); gradient norm clipped to $1.0$
Logging	Log every $100$ steps; validate and save every $5{,}000$ steps

Table S4: Common offline training protocol.

Configuration	Initialization	Trainable params.	Ratio ( $\%$ )	Valid loss $\downarrow$	Peak memory ( $\mathrm{GiB}$ ) $\downarrow$	Runtime ( $\mathrm{h}$ ) $\downarrow$
Projection heads	Pretrained	$2{,}165{,}792$	$0.0523$	$0.3901$	$19.65$	$15.27$
Action expert	Pretrained	$693{,}422{,}112$	$16.7356$	$0.3030$	$21.79$	$15.34$
Full	Pretrained	$4{,}143{,}404{,}816$	$100.0000$	$0.3004$	$40.23$	$19.61$

Table S5: Parameter adaptation ablation for

\pi_{0.5}

. All runs use the full training split, the same

30{,}000

-update budget, and seed. Validation loss is measured at the common final update. Peak memory is the maximum logged CUDA tensor allocation, and runtime is the W&B wall-clock runtime.

Scale axis	Train tasks	Available eps./task	Train eps./task	Train episodes	Train frames	Eff. epochs	Valid loss $\downarrow$
Trajectory density	$150$	$10$	$5$	$750$	$95{,}445$	$10.06$	$0.8755$
Trajectory density	$150$	$20$	$15$	$2{,}250$	$274{,}845$	$3.49$	$0.5493$
Trajectory density	$150$	$30$	$25$	$3{,}750$	$456{,}510$	$2.10$	$0.4182$
Trajectory density	$150$	$40$	$35$	$5{,}250$	$641{,}470$	$1.50$	$0.3361$
Trajectory density	$150$	$50$	$45$	$6{,}750$	$823{,}440$	$1.17$	$0.3004$
Task diversity	$10$	$50$	$45$	$450$	$53{,}610$	$17.91$	$1.0454$

Table S6: Data scale ablation for

\pi_{0.5}

with full fine-tuning. “Available” includes the five held-out appearance variants per task. Every run uses the same validation set (

150

tasks,

750

episodes, and

96{,}510

frames) and

30{,}000

optimizer updates. The table reports the common endpoint.

Figure S2: Offline validation loss at the six common checkpoints from

5{,}000

30{,}000

updates. The adaptation scope panel uses the complete training split. The data scale panels use full fine-tuning and vary either the available episodes per task or the number of training tasks. All curves use the same fixed validation set; lower is better. Curves combine distinct line styles and markers so that they remain identifiable without color.

Augmented Reality

See all 18 papers

ProCap: Projection-Aware Captioning for Spatial Augmented Reality
Apr 1, 2026Zimo Cao, Yuchen Deng, Haibin Ling +1Augmented Reality Image Captioning
Southwest University · Westlake University
SARI: Phase-Split Sim-Real Co-Training for Contact-Rich Manipulation
Oct 2, 2026Xingxin He, Yuxuan Jiang, Haonan Zhang +5Contact-Rich Robotic Manipulation Sim-to-Real Transfer
The Hong Kong University of Science and Technology · BYD Company Limited
StereoVLA: Enhancing Vision-Language-Action Models with Stereo Vision
Dec 26, 2025Shengliang Deng, Mi Yan, Yixin Zheng +7Visual Spatial Reasoning Robotic Manipulation
Galbot · The University of Hong Kong · CFCS, School of CS, Peking University +3

Apr 1, 2026J/K move · Enter open · S save

ProCap: Projection-Aware Captioning for Spatial Augmented Reality

Zimo Cao, Yuchen Deng, Haibin Ling, Bingyao Huang

Southwest University · Westlake University

Augmented Reality Image Captioning

Open paper AskPDF arXiv

Spatial augmented reality (SAR) directly projects digital content onto physical scenes using projectors, creating immersive experience without head-mounted displays. However, for SAR to support intelligent interaction, such as reasoning about the scene or answering user queries, it must semantically distinguish between the physical scene and the projected content. Standard Vision Language Models (VLMs) struggle with this virtual-physical ambiguity, often confusing the two contexts. To address this issue, we introduce ProCap, a novel framework that explicitly decouples projected content from physical scenes. ProCap employs a two-stage pipeline: first it visually isolates virtual and physical layers via automated segmentation; then it uses region-aware retrieval to avoid ambiguous semantic context due to projection distortion. To support this, we present RGBP (RGB + Projections), the first large-scale SAR semantic benchmark dataset, featuring 65 diverse physical scenes and over 180,000 projections with dense, decoupled annotations. Finally, we establish a dual-captioning evaluation protocol using task-specific tokens to assess physical scene and projection descriptions independently. Our experiments show that ProCap provides a robust semantic foundation for future SAR research. The source code, pre-trained models and the RGBP dataset are available on the project page: https://ZimoCao.github.io/ProCap/.

Abstract

Augmented Reality

Explore similar work

ProCap: Projection-Aware Captioning for Spatial Augmented Reality

SARI: Phase-Split Sim-Real Co-Training for Contact-Rich Manipulation

StereoVLA: Enhancing Vision-Language-Action Models with Stereo Vision