3D Pose Estimation in Projection Space via Pixel-level Visible Light Communication
Authors: Michikuni Eguchi, Takefumi Hiraki
Organizations: University of Tsukuba, 1-2 Kasuga, Tsukuba, Ibaraki 305-8550 Japan · Cluster Metaverse Lab, 8-9-5 Nishigotanda, Shinagawa, Tokyo 140-0031 Japan
Projector-based augmented reality enables natural interaction between users and devices by overlaying digital information onto the physical world through projected imagery. A key technical requirement of such systems is accurate positional alignment between the projector and the devices. Pixel-level Visible Light Communication (PVLC) achieves this alignment without external sensing infrastructure by embedding invisible spatial codes into each projected pixel through high-speed brightness modulation that preserves image quality. However, the information that PVLC transmits is inherently planar. PVLC has therefore been limited to devices moving on a flat surface and cannot estimate the 3D pose of devices in three-dimensional space, such as aerial robots. To extend PVLC-based pose estimation to three dimensions, we formulate it as a Perspective-n-Point (PnP) problem. The 2D pixel positions that four coplanar photosensors on the device receive via PVLC serve as the image points, and their known 3D layout provides the object points. Experiments show that the proposed system estimates 3D pose with a latency of approximately 8ms and a 3D position RMSE of 19.2mm after Kalman filtering. These results confirm the first realization of real-time 3D pose estimation coordinated with high-quality projected image content without external sensing infrastructure.
Figures & tables
50 mm/s
100 mm/s
200 mm/s
400 mm/s
3D [mm]
Angle [deg]
3D [mm]
Angle [deg]
3D [mm]
Angle [deg]
3D [mm]
Angle [deg]
PVLC (Ours) 160 mm
13.5±7.3
7.4±5.5
14.2±7.9
7.4±5.5
15.3±8.5
7.2±5.3
17.9±10.0
6.8±4.9
PVLC (Ours) 80 mm
23.4±13.9
21.6±15.5
25.1±15.2
21.3±15.4
25.3±14.9
21.9±15.8
28.1±17.1
22.9±16.2
AR 160 mm
7.0±2.6
2.4±0.7
8.0±2.8
2.5±0.9
14.0±5.3
3.4±1.5
26.7±13.5
6.1±2.9
AR 80 mm
9.8±3.7
4.5±3.3
11.5±4.4
4.7±3.3
19.7±9.0
7.0±4.2
30.3±17.6
11.7±7.0
TABLE I: Pose estimation error values of the proposed PVLC and the AR marker for different movement speeds and marker sizes. For each condition, the values are averaged over all trajectory patterns. Values are presented as RMSE ± SD.
3D error [mm]
Angle error [deg]
Processing time [ms]
HD
23.0±13.2
22.2±15.8
0.22±0.06
IPPE
22.8±13.2
24.2±16.4
0.01±0.01
P3P
29.6±18.2
24.4±15.1
0.02±0.00
EPnP
121.0±62.3
50.7±15.4
0.03±0.01
TABLE II: Pose estimation error values and solver processing time comparing PnP solvers. The errors were aggregated over five laps at all movement speeds along the trajectory shown in Fig. IV-C with the 80 mm photosensor layout. Values are presented as RMSE ± SD.
3D error [mm]
2D ( x,y ) error [mm]
Angle error [deg]
Raw signal
25.4±15.4
10.2±6.6
22.0±15.9
Kalman-filtered signal
19.2±12.1
9.0±5.9
16.0±9.2
TABLE III: Pose estimation error values of the raw PVLC signal and the Kalman-filtered signal with the 80 mm photosensor layout, aggregated over all trajectory patterns and movement speeds. Values are presented as RMSE ± SD.
Layout
Bound
Measured
3D
80 mm
11.5 mm
23.4±13.9 mm
160 mm
5.8 mm
13.5±7.3 mm
2D
80 mm
0.54 mm
8.5±5.0 mm
160 mm
0.54 mm
6.8±3.9 mm
Angle
80 mm
0.67∘
21.6∘±15.5∘
160 mm
0.34∘
7.4∘±5.5∘
TABLE IV: Theoretical lower bounds and measured pose estimation errors of the proposed PVLC at 50 mm/s. Measured values are presented as RMSE ± SD.