Screen localization and pointing estimation are key to low-cost interactive devices. Yet developing and evaluating these algorithms requires realistic data: synthetic captures cannot fully reproduce the optical distortion, rolling shutter, motion blur, and display processing of a physical acquisition, and most existing datasets provide static images rather than the video needed to assess continuous pointing. In this paper, we introduce the DABACO Dataset. Developed within the DABACO (Dispositivo Apuntador de BAjo COste, or Low-Cost Pointing Device) project, this dataset supports the development and evaluation of screen detection and camera-based pointing algorithms for embedded systems. It comprises video sequences captured with multiple low-cost camera sensors, including monochrome global-shutter and color rolling-shutter modules, on embedded platforms such as the Raspberry Pi 4B and ESP32-S3. We present a systematic annotation pipeline combining temporary visual watermarking, optical-flow tracking, manual verification, and marker removal. Both the original marked captures and the marker-free images, together with explicit corner annotations and modification masks, are released to support auditing and the study of potential reconstruction bias. In addition, we release an open-source evaluation toolkit with two reference baselines: a classical screen-detection pipeline based on edge and contour geometry, and a closed-vocabulary, off-the-shelf YOLOv8 segmentation model evaluated without dataset-specific training. Both are evaluated on a general-purpose computer using Intersection over Union, corner error, and pointing error, establishing initial reference results for future embedded implementations. Overall, DABACO addresses a gap in existing resources and is intended to support the development and evaluation of new low-cost screen-localization and pointing systems.
Figures & tables
Dataset
Video
Temporal
Display
Homo-
Corners
Embedded
Pointing
Eval.
graphy
code
HPatches [ 15 ]
Sequence
Yes
No
Yes
No
No
No
Yes
HEB/Pi3D [ 16 ]
No
No
No
Yes
No
No
No
Yes
Screen Detection YOLO [ 17 ]
No
No
Yes
No
No
No
No
No
Screen Detection v2 [ 18 ]
No
No
Yes
No
No
No
No
No
PlanarTrack [ 19 , 20 ]
Yes
Yes
Partial
Derivable
Yes
No
No
Yes
Table 1: Comparison between related resources and DABACO. Column criteria are defined in the text.
Figure 1: Overview of the Ground Truth Generation Pipeline.
Figure 2: Visual progression of a frame through the four stages of the Ground Truth Generation Pipeline.
Figure 3: The Interactive Verification editor’s graphical user interface. The left panel shows the system’s output during automatic pre-annotation. The right panel illustrates the manual review process, where annotators utilize the integrated 4.5× zoom loupe to perform fine-grained coordinate corrections.
Annotation Source
Frames
Percentage
Direct Marker Detection (Anchors)
16,993
52.08%
Optical Flow Propagation
14,158
43.39%
Manual Correction
489
1.50%
Discarded / Invalid Geometry
989
3.03%
Total Processed Frames
32,629
100.00%
Table 2: Frame annotation origins in the initial release.
Figure 4: Visual comparison of Telea inpainting across representative capture scenarios. Top row: original crops containing the ArUco markers. Bottom row: corresponding inpainted regions. Left (a, d): Seamless marker removal on uniform background ( esp32_ov3660_scrA_v2_m2vga , frame 170). Middle (b, e): Stretched pixel structure forming a square of four converging triangles ( rpi4_ov5647_scrA_v1_m1fhd , frame 94). Right (c, f): Inward bleeding of the dark screen bezel creating an artificial vignetting corner cut under reduced scale ( rpi4_ov9281_scrC_v1_m3hifps , frame 267).
Camera
Type
High-Quality (HQ)
Medium-Quality (MQ)
High-Speed (HS)
OV9281
Mono, Global
1280×800 @30, Auto
1280×800 @30, 1/1000s
640×400 @100, Auto
OV5647
Color, Rolling
1080p@30, Auto
960p@30, 1/500s
720p@60, 1/1000s
ESP32-S3
IoT, Color
800×600 @17.4, Auto
640×480 @24, 1/125s
320×240 @27.6, Auto
Table 3: Operational modes per camera: the three exposure and resolution configurations (HQ, MQ, HS).
Camera
Display
V1 (Animation)
V2 (Gameplay)
V3 (Presentation)
OV9281
Monitor
HQ
MQ
HS
TV
MQ
HS
HQ
Laptop
HS
HQ
MQ
OV5647
Monitor
HQ
MQ
HS
TV
MQ
HS
HQ
Laptop
HS
HQ
MQ
Table 4: Balanced Latin Square assignment of the three modes to each display–content combination.
Figure 5: Block diagram of the classical computer vision baseline for screen detection.
Algorithm
Coverage (%)
False-Lock (%)
Corner Error (%)
IoU (%)
Pointing Error (% – all)
Pointing Error (% – valid)
Time-to-Lock (frames)
Classical
92.63
58.79
12.52 (11.44)
52.98 (53.24)
28.74 (23.30)
23.07
39.7
YOLOv8 Seg.
86.97
52.31
7.89 (6.39)
58.84 (63.16)
90.56 (30.72)
89.15
25.4
Table 5: Aggregate Baseline Performance on the DABACO Dataset (Mean values, with medians in parentheses). The “all frames” column reports the mean over all evaluated frames, including the default penalty for missing predictions. The “valid preds.” column reports the mean restricted to frames for which a valid prediction was returned. Median and P95 values for both aggregation protocols are available in the accompanying benchmark report.
Display
Method
Coverage (%)
False-Lock (%)
IoU (%)
Pointing Error (% – all frames)
Monitor
Classical
96.71
62.09
58.76
21.09
Monitor
YOLOv8
97.13
35.35
76.99
9.83
TV
Classical
89.86
37.08
67.56
19.23
TV
YOLOv8
84.32
38.33
68.59
20.10
Laptop
Classical
91.31
77.19
32.64
45.91
Laptop
YOLOv8
79.47
83.27
30.93
241.76
Table 6: Baseline performance by display type (Mean values reported). Pointing error can exceed 100% for valid but geometrically unstable homography predictions.