Breaking the Sub-Millimeter Barrier: Eyeframe Acquisition from Color Images
Authors: Manel Guzmán, Antonio Agudo
Organizations: Manel Guzmán is with the Horizons Optical, Sant Cugat del Vallès, Spain. · Antonio Agudo is with the Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Barcelona, Spain.
Eyeframe lens tracing is an important process in the optical industry that requires sub-millimeter precision to ensure proper lens fitting and optimal vision correction. Traditional frame tracers rely on mechanical tools that need precise positioning and calibration, which are time-consuming and require additional equipment, creating an inefficient workflow for opticians. This work presents a novel approach based on artificial vision that utilizes multi-view information. The proposed algorithm operates on images captured from an InVision system. The full pipeline includes image acquisition, frame segmentation to isolate the eyeframe from background, depth estimation to obtain 3D spatial information, and multi-view processing that integrates segmented RGB images with depth data for precise frame contour measurement. To this end, different configurations and variants are proposed and analyzed on real data, providing competitive measurements from still color images with respect to other solutions, while eliminating the need for specialized tracing equipment and reducing workflow complexity for optical technicians.
Figures & tables
Fig. 1: Invision Head. The acquisition system consists of four calibrated cameras and two sources of light: visible and IR. Just the 1296 × 1296 RGB images are considered in this paper.
Fig. 2: Color and depth information to obtain measurements. The figure shows a particular image acquisition, its corresponding depth information; as well as the same information after applying our segmentation model to infer the region of interest (this represents our input data).
Fig. 3: Main architecture for eye-frame trace. Multi-view RGB and depth channels are processed at the same networks.
Metric
Tiny
Small
Base+
Large
Size [Mb] ↓
38.9
46.0
80.8
224.4
Speed [Hz] ↑
47.2
43.3
34.8
24.2
IoU ↑
0.902
0.922
0.951
0.961
TABLE I: Segmentation evaluation for several configurations. The table reports model size (in Mb), inference speed (in Hz), and IoU, for different configurations.
Fig. 4: Qualitative evaluation of our segmentation strategy with the base+ model for different users. The glass segmentation applying DeepLabv3+ [ 9 ] and ours is displayed in red and blue, respectively. In last column, a zooming plot is shown. Best viewed in color.
Metric
DeepLabV3 [ 9 ]
Small
Base+
Train Loss (Dice loss) ↓
0.367
0.362
0.348
Test Loss (Dice loss) ↓
0.367
0.363
0.348
IoU ↑
0.931
0.938
0.958
TABLE II: Final Segmentation results after finetuning. Three configurations are finetuned on the custom dataset. The table reports the train and test losses as well as the corresponding IoU.
Fig. 5: Qualitative results on depth estimation. From left to right: Test input image, depth prediction using ViT-Small, ViT-Base, and ViT-Large, respectively.
TABLE III: Quantitative evaluation on frame tracer for several modalities and model size. The table reports the minimum, maximum and median error values of frame radius measurement, all of them in millimeters. The one-head approach is considered.
TABLE IV: Fusion Strategy Evaluation for trace measurements. The table reports the frame tracer results for different fusion strategies. Minimum, maximum and median error values of frame radius measurement are included, all of them in millimeters. The one-head approach and model “S” are considered.
Fig. 6: Qualitative evaluation of our approach. In all rows, the same information is provided. From left to right: an arbitrary image acquisition, the predicted depth, the predicted segmentation mask and the final trace output plot trace. In this last plot, the scale is in hundredths of a millimeters. Ground truth and our prediction are represented in blue and orange, respectively. Best viewed in color.
Retinal image-based eye tracking is widely used in ophthalmic imaging and vision science, and is a promising path to deliver higher gaze accuracy than the pupil- and cornea-based approaches commonly used in modern AR/VR devices. Nevertheless, existing retinal tracking algorithms still primarily rely on classical template-matching registration, which can be insufficiently robust to retinal feature variability and real-world imaging conditions. In this work, we propose a novel weakly-supervised, learning-based framework for robust retinal eye tracking. Initial studies demonstrate high accuracy, achieving the 95th-percentile gaze error < 0.45 deg across a cohort of 6 participants.
Bo Wen, Dillon Lohr, Yatong An +8
Meta Reality Labs · Independent Researcher · University of California, San Diego
Image-guided surgery (IGS) depends on accurate tracking of surgical instruments to provide real-time navigation relative to anatomical structures. Commercial stereo infrared trackers are accurate but prone to occlusion and cost-prohibitive for many settings. This work presents LARK, a multi-camera optical tracking system using commodity RGB hardware and multi-view redundancy and fusion. We develop and evaluate two complete tracking methods: multi-view monocular pose fusion and multi-view triangulation. Both methods are assessed under varying occlusion levels using a precision-machined grid and an anatomical head phantom, and compared against a gold-standard stereo infrared system. With five cameras and adaptive Kalman filtering, LARK achieves median target registration errors of 0.64 mm for point localization with triangulation and 0.73 mm for trajectory tracking with pose fusion on the machined grid. Camera-subset experiments show graceful degradation in adaptive pose-fusion accuracy as fewer views remain available. With tracking hardware costing under $1,000 USD, LARK provides a low-cost platform for image-guided surgery research. Hardware designs and software are publicly available at https://nist.mni.mcgill.ca/software/ , and datasets at https://nist.mni.mcgill.ca/data/ .
George Sideris, Justin Cree, Andrew Stirling +3
Department of Mechanical Engineering, McGill University, Montreal, Canada · Montreal Neurological Institute, McGill University, Montreal, Canada
Eye tracking (ET) plays a critical role in augmented and virtual reality applications. However, rapidly deploying high-accuracy, on-device gaze estimation for new products remains challenging because hardware configurations (e.g., camera placement, camera pose, and illumination) often change across device generations. Visual foundation models (VFMs) excel on natural-image benchmarks and offer a promising path to rapid training and deployment; yet, we find that off-the-shelf VFMs still struggle to reach high accuracy on specialized near-eye infrared images. To close this gap, we introduce DistillGaze, a framework that distills a VFM using labeled synthetic data and unlabeled real data for rapid, high-accuracy on-device gaze estimation. DistillGaze proceeds in two stages. First, we adapt a VFM into a domain-specialized teacher using synthetic gaze labels and unlabeled real images. Synthetic data provide scalable, high-quality gaze supervision, while unlabeled real data bridges the synthetic-to-real domain gap. Second, we train an on-device student from both teacher guidance and self-training. Evaluated on a large-scale crowd-sourced dataset spanning more than 2,000 participants, DistillGaze reduces median gaze error by 58.6% relative to synthetic-only baselines while maintaining a lightweight 256K-parameter model suitable for real-time on-device deployment. More broadly, DistillGaze offers an efficient path to training and deploying ET models that adapt to hardware changes, and a recipe for combining synthetic supervision with unlabeled real data in on-device regression tasks.