cs.CVNov 13, 2026

Toward On-Device Markerless 6DoF Tracking of a Physical Manipulative on a Mixed-Reality Headset: A Preliminary Study

Authors: Guangxi Feng, Philip Yasskin, Wei Yan

Organizations: Department of Architecture, Texas A&M University, College Station, Texas · Department of Mathematics, Texas A&M University, College Station, Texas

Abstract

AR-Classroom uses physical models and virtual overlays to help learners explore spatial transformations. To align these overlays on a standalone headset, we developed a markerless six-degree-of-freedom(6DoF) tracker for a LEGO airplane on the Meta Quest 3. A compact YOLO model trained on synthetic Unity renders predicts 20 keypoints from dynamic RGB crops resized to 128 by 128 pixels. A perspective-n-point solver estimates the airplane's pose from these predictions and the corresponding 3D landmarks. In a paired evaluation on 1,000 synthetic validation images, cropping guided by ground-truth boxes increased the detection rate on annotated images from 32.5 to 46.2 percent. Mean keypoint F1 increased from 42.9 to 44.6 percent, with lower keypoint precision in the cropped view. In preliminary headset tests, 2 Hz was the highest tested tracking target at which the application maintained approximately 72 frames per second. A headset demonstration illustrates virtual overlay alignment with the physical airplane before and after a manual rotation. Quantitative 6DoF pose accuracy on real headset footage will be measured in future work.

Figures & tables

Explore similar work

CardsList
  1. MuCHMDPoser: Multiview RGB-Camera Full-Body Pose Tracking for Virtual Reality

    Nov 2, 2026Antoine Maiorca, George Fletcher, Thierry Ravet +1Virtual Reality3D Human Pose Estimation

  2. ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

    Jul 7, 2026Ruihang Zhang, Felix Taubner, Pooja Ravi +2Object Pose EstimationVideo-to-Video Translation

  3. ReVA-HPE: Relative 6DoF Head Pose Estimation with Synthetic Pose Supervision and Personalized Virtual Anchors

    Nov 2, 2026Vasiliki Vasileiou, Panagiotis P. Filntisis, Petros Maragos +1Relative Pose Estimation