3D Pose Estimation in Projection Space via Pixel-level Visible Light Communication
Authors: Michikuni Eguchi, Takefumi Hiraki
Organizations: University of Tsukuba, 1-2 Kasuga, Tsukuba, Ibaraki 305-8550 Japan · Cluster Metaverse Lab, 8-9-5 Nishigotanda, Shinagawa, Tokyo 140-0031 Japan
Projector-based augmented reality enables natural interaction between users and devices by overlaying digital information onto the physical world through projected imagery. A key technical requirement of such systems is accurate positional alignment between the projector and the devices. Pixel-level Visible Light Communication (PVLC) achieves this alignment without external sensing infrastructure by embedding invisible spatial codes into each projected pixel through high-speed brightness modulation that preserves image quality. However, the information that PVLC transmits is inherently planar. PVLC has therefore been limited to devices moving on a flat surface and cannot estimate the 3D pose of devices in three-dimensional space, such as aerial robots. To extend PVLC-based pose estimation to three dimensions, we formulate it as a Perspective-n-Point (PnP) problem. The 2D pixel positions that four coplanar photosensors on the device receive via PVLC serve as the image points, and their known 3D layout provides the object points. Experiments show that the proposed system estimates 3D pose with a latency of approximately 8ms and a 3D position RMSE of 19.2mm after Kalman filtering. These results confirm the first realization of real-time 3D pose estimation coordinated with high-quality projected image content without external sensing infrastructure.
Figures & tables
50 mm/s
100 mm/s
200 mm/s
400 mm/s
3D [mm]
Angle [deg]
3D [mm]
Angle [deg]
3D [mm]
Angle [deg]
3D [mm]
Angle [deg]
PVLC (Ours) 160 mm
13.5±7.3
7.4±5.5
14.2±7.9
7.4±5.5
15.3±8.5
7.2±5.3
17.9±10.0
6.8±4.9
PVLC (Ours) 80 mm
23.4±13.9
21.6±15.5
25.1±15.2
21.3±15.4
25.3±14.9
21.9±15.8
28.1±17.1
22.9±16.2
AR 160 mm
7.0±2.6
2.4±0.7
8.0±2.8
2.5±0.9
14.0±5.3
3.4±1.5
26.7±13.5
6.1±2.9
AR 80 mm
9.8±3.7
4.5±3.3
11.5±4.4
4.7±3.3
19.7±9.0
7.0±4.2
30.3±17.6
11.7±7.0
TABLE I: Pose estimation error values of the proposed PVLC and the AR marker for different movement speeds and marker sizes. For each condition, the values are averaged over all trajectory patterns. Values are presented as RMSE ± SD.
3D error [mm]
Angle error [deg]
Processing time [ms]
HD
23.0±13.2
22.2±15.8
0.22±0.06
IPPE
22.8±13.2
24.2±16.4
0.01±0.01
P3P
29.6±18.2
24.4±15.1
0.02±0.00
EPnP
121.0±62.3
50.7±15.4
0.03±0.01
TABLE II: Pose estimation error values and solver processing time comparing PnP solvers. The errors were aggregated over five laps at all movement speeds along the trajectory shown in Fig. IV-C with the 80 mm photosensor layout. Values are presented as RMSE ± SD.
3D error [mm]
2D ( x,y ) error [mm]
Angle error [deg]
Raw signal
25.4±15.4
10.2±6.6
22.0±15.9
Kalman-filtered signal
19.2±12.1
9.0±5.9
16.0±9.2
TABLE III: Pose estimation error values of the raw PVLC signal and the Kalman-filtered signal with the 80 mm photosensor layout, aggregated over all trajectory patterns and movement speeds. Values are presented as RMSE ± SD.
Layout
Bound
Measured
3D
80 mm
11.5 mm
23.4±13.9 mm
160 mm
5.8 mm
13.5±7.3 mm
2D
80 mm
0.54 mm
8.5±5.0 mm
160 mm
0.54 mm
6.8±3.9 mm
Angle
80 mm
0.67∘
21.6∘±15.5∘
160 mm
0.34∘
7.4∘±5.5∘
TABLE IV: Theoretical lower bounds and measured pose estimation errors of the proposed PVLC at 50 mm/s. Measured values are presented as RMSE ± SD.
Estimating a camera pose from 3D-2D correspondences - Perspective-n-Point - is one of the most fundamental problems in computer vision. Existing methods either solve a minimal problem or replace the objective with an easier one. Surprisingly, no scalable certifiable solver for the actual Maximum Likelihood Estimation (MLE) exists. We propose PnP*, the first practical solver that guarantees efficient computation and returns a certifiable globally optimal solution. We prove that the underlying convex relaxation is exact under moderate noise, which to our knowledge is the first stability guarantee for any relaxation for the MLE of PnP. Furthermore, we reduce the relaxation by exploiting chordal sparsity pattern and quotient ring reduction such that the relaxation is tractable and scalable. The stability and scalability results are verified numerically, as PnP* returns a provably correct camera pose at over 10 Hz.
Bang-Shien Chen, Caitrin E. Eaton, David M. Rosen
Northeastern University, Boston, MA 02115, USA. · Draper, Cambridge, MA 02139, USA.
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.
Digital fringe projection (DFP) enables micrometer-level 3D reconstruction, yet extending it to large-scale mapping remains challenging because six-degree-of-freedom pose estimation often cannot match the reconstruction's precision. Conventional iterative closest point (ICP) registration becomes inefficient on multi-million-point clouds and typically relies on downsampling or feature-based selection, which can reduce local detail and degrade pose precision. Drift-correction methods improve long-term consistency but do not resolve sampling sensitivity in dense DFP point clouds.We propose a high-precision pose estimation method that augments a moving DFP system with a fixed, intrinsically calibrated global projector. Using the global projector's phase-derived pixel constraints and a PnP-style reprojection objective, the method estimates the DFP system pose in a fixed reference frame without relying on deterministic feature extraction, and we experimentally demonstrate sampling invariance under coordinate-preserving subsampling. Experiments demonstrate sub-millimeter pose accuracy against a reference with quantified uncertainty bounds, high repeatability under aggressive subsampling, robust operation on homogeneous surfaces and low-overlap views, and reduced error accumulation when used to correct ICP-based trajectories. The method extends DFP toward accurate 3D mapping in quasi-static scenarios such as inspection and metrology, with the trade-off of time-multiplexed acquisition for the additional projector measurements.
Sehoon Tak, Keunhee Cho, Sangpil Kim +1
Department of Mechanical Engineering, Yonsei University, Seoul 03722, South Korea. · Department of Artificial Intelligence, Korea University, Seoul