cs.LGMar 27, 2026

Evaluating Interactive 2D Visualization as a Sample Selection Strategy for Biomedical Time-Series Data Annotation

Authors: Einari Vaaras, Manu Airaksinen, Okko Räsänen

Organizations: Signal Processing Research Centre, Tampere University, Tampere, 33720, Finland · BABA Center, Department of Physiology, University of Helsinki, Helsinki, 00029, Finland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reliable machine-learning models in biomedical settings depend on accurate labels, yet annotating biomedical time-series data remains challenging. Algorithmic sample selection may support annotation, but evidence from studies involving real human annotators is scarce. Consequently, we compare three sample selection methods for annotation: random sampling (RND), farthest-first traversal (FAFT), and a graphical user interface-based method enabling exploration of complementary 2D visualizations (2DVs) of high-dimensional data. We evaluated the methods across four classification tasks in infant motility assessment (IMA) and speech emotion recognition (SER). Twelve annotators, categorized as experts or non-experts, performed data annotation under a limited annotation budget, and post-annotation experiments were conducted to evaluate the sampling methods. Across all classification tasks, 2DV performed best when aggregating labels across annotators. In IMA, 2DV most effectively captured rare classes, but also exhibited greater annotator-to-annotator label distribution variability resulting from the limited annotation budget, decreasing classification performance when models were trained on individual annotators' labels; in these cases, FAFT excelled. For SER, 2DV outperformed the other methods among expert annotators and matched their performance for non-experts in the individual-annotator setting. A failure risk analysis revealed that RND was the safest choice when annotator count or annotator expertise was uncertain, whereas 2DV had the highest risk due to its greater label distribution variability. Furthermore, post-experiment interviews indicated that 2DV made the annotation task more interesting and enjoyable. Overall, 2DV-based sampling appears promising for biomedical time-series data annotation, particularly when the annotation budget is not highly constrained.

Figures & tables

Figure 1: An overview of the study. 1(a) : Overall study design. 1(b) : An example of different sampling strategies. 1(c) : Model performance evaluation procedure.
Figure 2: An example screenshot of the TSExplorer GUI as used for posture annotation for IMA in the present study. On the top left of the GUI, there is a video widget for video playback, and a scatter plot visualizing the entire dataset is located on the right. On the bottom left, there are two plots visualizing the accelerometer (right) and gyroscope (left) signals from the MAIJU-DS dataset. From top to bottom, the panels show the x, y, and z components for the right arm, left arm, right leg, and left leg. Users can select samples from the scatter plot for annotation in any order, and the annotation can be performed either via a drop-down menu or through keyboard shortcuts. The color of each data point in the scatter plot reflects its current label (green denoting unlabeled samples), and the active sample is highlighted with a large yellow ring. The color code as well as the keyboard shortcut for each class can be seen in the scatter plot legend.
Table 1: A summary of the datasets (IMA: MAIJU-DS; SER: NICU-A) used in the present experiments.
Figure 3: The combined label distributions (±SD) for all annotators in terms of IMA (top left: posture; top right: movement) and SER (bottom left: valence; bottom right: arousal). Each label-wise histogram group is organized from left to right in the following order: MAIJU-DS or NICU-A GS reference distribution, RND, FAFT, and 2DV.
Figure 4: The mean classification results for IMA when training models with annotator-wise labels separately, organized by annotator group (solid line: expert annotators; dotted line: non-expert annotators) and by classification task (left: posture; right: movement). A topline performance level (black horizontal line), derived from a model trained using MAIJU-DS labels, is shown for comparison.
Figure 5: The mean classification results for SER when training models with annotator-wise labels separately, organized by annotator group (solid line: expert annotators; dotted line: non-expert annotators) and by classification task (left: valence; right: arousal). A reference performance level (black horizontal line), derived from a model trained using the labeled subset of the NICU-A training set, is shown for comparison.
Figure 6: The classification results for IMA when training models with the labels of either three annotators (top row: expert and non-expert annotators) or six annotators (bottom row: all annotators), organized by classification task (left column: posture; right column: movement). A topline performance level (black horizontal line), derived from a model trained using MAIJU-DS labels, is shown for comparison.
Figure 7: The classification results for SER when training models with the labels of either three annotators (top row: expert and non-expert annotators) or six annotators (bottom row: all annotators), organized by classification task (left column: valence; right column: arousal). A reference performance level (black horizontal line), derived from a model trained using the labeled subset of the NICU-A training set, is shown for comparison.
TaskAnnotator group# AnnotatorsMetrics usedRanked methods (combined risk score)
IMAExpert1cov, mod, disFAFT (7.0) == RND (7.0) >> 2DV (13.0)
IMAExpert2cov, disFAFT (5.0) == RND (5.0) >> 2DV (8.0)
IMAExpert3cov, mod, disFAFT (7.0) == RND (7.0) >> 2DV (10.0)
IMANon-expert1cov, mod, disFAFT (7.0) >> RND (8.0) >> 2DV (14.0)
IMANon-expert2cov, disFAFT (5.0) == RND (5.0) >> 2DV (9.0)
IMANon-expert3cov, mod, disFAFT (7.0) == RND (7.0) >> 2DV (10.0)
Table 2: Combined failure risk score ranking for IMA (top) and SER (bottom). Lower scores indicate lower estimated risk, and each individual score is summed across classification tasks and different failure risk metrics. Metric abbreviations: cov = rare class coverage failures; mod = model performance failures; dis = label distribution instability.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: An example screenshot of the TSExplorer GUI as used for valence annotation for SER in the present study. On the left of the GUI, there is an audio widget used for audio playback, and a scatter plot visualizing the entire dataset is located on the right.
Figure 9: The combined IMA label distributions (±SD) for expert annotators (top row), non-expert annotators (middle row), and all annotators (bottom row) in terms of posture (left column) and movement (right column) labels. Each label-wise histogram group is organized from left to right in the following order: MAIJU-DS reference distribution, RND, FAFT, and 2DV.
Figure 10: The combined SER label distributions (±SD) for expert annotators (top row), non-expert annotators (middle row), and all annotators (bottom row) in terms of valence (left column) and arousal (right column) labels. Each label-wise histogram group is organized from left to right in the following order: NICU-A GS set reference distribution, RND, FAFT, and 2DV.
More in this topic

Human-in-the-Loop Annotation

Human-in-the-loop annotation combines human judgment and AI assistance to create, review, and improve labeled data.

See all 135 papers

Explore similar work

CardsList
Aug 31, 2026

TSExplorer: An interactive data annotation and exploration tool for time-series data

We present TSExplorer, a cross-platform tool for interactive annotation and exploration of time-series data. The tool enables users to inspect high-dimensional datasets through multiple complementary 2D visualizations derived from high-dimensional feature representations. TSExplorer is designed as a general-purpose research tool supporting a wide range of workflows, including exploratory data analysis, annotation of unlabeled or partially-labeled datasets, comparison of feature representations, and post-hoc inspection and refinement of existing labels with interactive visual feedback.

Einari Vaaras, Manu Airaksinen, Okko Räsänen

Signal Processing Research Centre, Tampere University, Finland · BABA Center, Department of Physiology, University of Helsinki, Finland

Data VisualizationVisual Analytics
AskPDF
Feb 26, 2026

Interactive Medical-SAM2 GUI: A Napari-based semi-automatic annotation tool for medical images

Interactive Medical-SAM2 GUI is an open-source desktop application for semi-automatic annotation of 2D and 3D medical images. Built on the Napari multi-dimensional viewer, box/point prompting is integrated with SAM2-style propagation by treating a 3D volume as a slice sequence, enabling mask propagation from sparse prompts using Medical-SAM2 on top of SAM2. Voxel-level annotation remains essential for developing and validating medical imaging algorithms, yet manual labeling is slow and expensive for 3D scans, and existing integrations frequently emphasize per-slice interaction without providing a unified, cohort-oriented workflow for navigation, propagation, interactive correction, and quantitative export in a single local pipeline. To address this practical limitation, a local-first Napari workflow is provided for efficient 3D annotation across multiple studies using standard DICOM series and/or NIfTI volumes. Users can annotate cases sequentially under a single root folder with explicit proceed/skip actions, initialize objects via box-first prompting (including first/last-slice initialization for single-object propagation), refine predictions with point prompts, and finalize labels through prompt-first correction prior to saving. During export, per-object volumetry and 3D volume rendering are supported, and image geometry is preserved via SimpleITK. The GUI is implemented in Python using Napari and PyTorch, with optional N4 bias-field correction, and is intended exclusively for research annotation workflows. The code is released on the project page: https://github.com/SKKU-IBE/Medical-SAM2GUI/.

Woojae Hong, Jong Ha Hwang, Jiyong Chung +3

Department of Biomechatronic Engineering, Sungkyunkwan University, Suwon, Gyeonggi, Republic of Korea · Department of Neurosurgery, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Republic of Korea

Interactive Medical Image SegmentationHuman-in-the-Loop AnnotationMedical Image Analysis3D Medical Image Segmentation
AskPDF
Jul 28, 2026

AnnoBench: A Benchmark for Visualization Annotation Generation

Annotation is among the most demanding visualization tasks to automate, as it simultaneously requires correctly navigating visual, semantic, and stylistic constraints. Failure to meet any of these conditions severely undermines the utility of an annotation, rendering it challenging to read, inaccurate, or visually discordant. Despite a growing body of annotation tools and automations, no existing benchmark or evaluation framework tests whether these conditions are met because of their scope and annotation not being the focus of their studies. We introduce AnnoBench, a benchmark for visualization annotation that materializes the inherent challenges of this domain in a structured and testable manner. AnnoBench pairs visualizations from professional data journalism and visualization galleries with annotation tasks, spanning four representation formats, five chart description conditions, and two prompt specification levels. The benchmark is executed via VLM-as-a-judge, using models aligned with manual human assessment. We evaluate the benchmark via four one-factor-at-a-time experiments, exploring the effects of input representation, semantic context, and prompt specificity, and model selection on annotation quality. This work provides a foundation for advancing annotation automation, tooling, and visualization-generation pipelines.

Md Rahat-uz-Zaman, Md Dilshadur Rahman, Andrew McNutt +1

University of Utah

Data VisualizationAutomated Evaluation
AskPDF