Towards an End-to-End System for 3D Tracking of Physical Objects in Virtual Immersive Environments

Stanisław Knapiński; Maciej Grzeszczuk; Barbara Karpowicz; Pavlo Zinevych; Wieslaw Kopec

cs.HCMar 30, 2026

Towards an End-to-End System for 3D Tracking of Physical Objects in Virtual Immersive Environments

Authors: Stanisław Knapiński, Maciej Grzeszczuk, Barbara Karpowicz, Pavlo Zinevych, Wieslaw Kopec

Organizations: Polsko-Japońska Akademia Technik Komputerowych w Warszawie, Koszykowa 86, Warszawa, Polska

Abstract

This work aims to establish an end-to-end system for tracking of physical 3D objects for virtual reality (VR) applications. We focus on training applications requiring real-time tracking of the position of small physical objects and their reflection in VR space. Out goal is to perform object tracking in a "plug and play" manner, without using complex systems with quite large tracking devices or manually implementing object tracking. We therefore propose a system for object tracking via fiducial markers alongside a software harness, to enable fast and efficient designation of objects to be tracked and data streaming solution for end-use applications. The system utilizes AruCo, AprilTag and an original Colored Control Points based fiducial system. It allows for easy tag detection and use of object position data, which are crucial for immersive training environments based on VR and eXtended Reality (XR). We evaluate various tag sizes, detection distances, and different camera devices against the theoretical limits. In effect, we create a complete solution for implementing marker-based, real-to-virtual object position mapping for various applications.

Figures & tables

Figure 1: Object detection systems. Source: Own elaboration, [ 24 , 10 , 3 , 13 ]

Figure 2: Example workflow with the object detection software. Source: own elaboration

Figure 3: Tracking Configuration UI. Source: own elaboration

Figure 6: Block diagram of the Colored Points algorithm, illustrating the single-pass clustering and topology-based pose estimation pipeline. Source: own elaboration.

Figure 7: AprilTags detected as Unity objects. Source: own elaboration.

Figure 8: Timeline of detection rate. Note: Colored Points on Generic Webcam not included since they had 0% detection rate. Source: own elaboration.

Figure 9: Average detection rate. Source: own elaboration.

Figure 10: Maximum distance from camera for various marker sizes. Source: own elaboration.

Device	Raspberry Pi Camera Module 3		Generic WebCam
Detection Method	AprilTag (4mm)	Colored Point	AprilTag (4mm)
max. normal angle hor. dev.	$\approx 170\degree$	$\approx 45\degree$	N/A
max. normal angle ver. dev.	$\approx 45\degree$	$\approx 45\degree$	N/A
max. distance	$\approx 46\text{cm}$	$\approx 40\text{cm}$	$\approx 15\text{cm}$
high movement speed det.	YES	YES	NO
% of time detected	73.8%	79.3%	14.0%

Table 1: Video test results at 1080p. Source: own elaboration.

Device	Raspberry Pi Camera Module 3		Generic Webcam
Detection Method	AprilTag	Colored Points	AprilTag	Colored Points
stable det. at $d\leq 25\text{cm}$	$\geq 2\text{mm}$	$\geq 1\text{mm}$	$\geq 7\text{mm}$	$\geq 4\text{mm}$
stable det. at $d\leq 50\text{cm}$	$\geq 4\text{mm}$	$\geq 2\text{mm}$	$\geq 12\text{mm}$	$\geq 5\text{mm}$

Table 2: Marker Sizes for stable detection at various distance thresholds at 1080p. Source: own elaboration.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 175 papers

BlobBoards: Robust Markers for Accurate Pose
Aug 28, 2026James Pritts, Till Sittart, Hendrik Sauer +4Object Pose Estimation Camera Pose Estimation
Marine Data Science Kiel University
SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild
Mar 30, 2026Patrick Rim, Kevin Harris, Braden Copple +8Hand-Object Interaction Human-Object Interaction
Meta Reality Labs · Yale University
GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction
May 21, 2026Corentin Dumery, David Colmenares, Alexander Fix +3Gaze Estimation Synthetic Data Generation
Meta Reality Labs, Redmond, USA · EPFL, Lausanne, Switzerland · Work done during an internship at Meta

Aug 28, 2026J/K move · Enter open · S save

BlobBoards: Robust Markers for Accurate Pose

James Pritts, Till Sittart, Hendrik Sauer, Silja Janßen+3

Marine Data Science Kiel University

Object Pose Estimation Camera Pose Estimation Object Detection

Open paper AskPDF arXiv

We propose BlobBoards, a fiducial marker system comprising a dense, multi-scale field of Gaussian blobs and a feature-based pipeline for joint detection, identification, and pose estimation. Each board is registered from hundreds of blob features whose dense spatial coverage constrains pose, while multiple scales preserve detectability across large changes in focal length, distance, and obliquity. Learned local descriptors are matched to the reference pattern and spatially verified, so the correspondences determine pose and certify identity. Against motion-capture ground truth, BlobBoards achieve median translation errors of 3.6-5.0 mm, reducing AprilTag's median translation error by 89% on small boards and 70% on large ones. They also produce far fewer large-rotation failures than state-of-the-art tag systems. BlobBoards achieve the highest detection rate, 80% versus 74% for AprilTag and 58% for ArUco, with the largest margin on the smallest markers. Under 50% occlusion, they still detect 69% of boards with essentially unchanged median translation error, while AprilTag and ArUco detect none. In experiments BlobBoards give state-of-the-art detection rate, pose accuracy and occlusion robustness.

Abstract

Visual Object Tracking

Explore similar work

BlobBoards: Robust Markers for Accurate Pose

SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild

GazePrior: Zero-Shot AR/VR Eye Tracking via Learned 3D Gaze Reconstruction