Conformal Annotation with Randomized Expert Review: Application to Online Cell-Type Labeling

Xuhe Zhang; Zhi Huang; Ying Jin

stat.MEOct 29, 2026

Conformal Annotation with Randomized Expert Review: Application to Online Cell-Type Labeling

Authors: Xuhe Zhang, Zhi Huang, Ying Jin

Organizations: Applied Mathematics and Computational Science, University of Pennsylvania · Department of Pathology and Laboratory Medicine, University of Pennsylvania · Department of Statistics and Data Science, University of Pennsylvania

Abstract

Artificial intelligence (AI) models can accelerate cell-type annotation in digital pathology, but safely scaling their use requires determining, with error control, which model-generated labels can be accepted without expert review. This is challenging in evolving workflows because cell morphology and type composition may change across tissue regions and slides, making one-time calibration unreliable. We propose Conformal Annotation with Randomized Expert review under Distribution shift (CARED), a distribution-free framework that leverages randomized expert audit to achieve long-run error control for AI-assisted annotation under distribution shift. For each incoming batch, CARED uniformly selects a subset of cases for expert review and uses the acquired labels to calibrate later decisions. Among unreviewed cases, CARED determines a subset of them to safely skip expert labeling, while suggesting candidate labels for the others for downstream resolution. We prove that this workflow provides long-run control of cumulative error over both the full stream and the cells outside of the audit sample, without stationarity or exchangeability assumptions. We further enhance the framework by introducing class-conditional guarantees to ensure error control within important but rare cell types, together with a utility module to target more automatic resolutions, smaller candidate sets, and greater biological coherence, without compromising validity. Experiments on pancreas, lung, and ovary annotation streams, together with a stream constructed from real expert annotations, show that CARED controls the cumulative errors while a substantial fraction of cells (up to 70%) bypass expert review.

Figures & tables

Panel of Figure 1 — Figure 1: Evidence of workflow shift in image-based cell-type annotation. A pancreas whole-slide image containing more than $10^{5}$ cells is partitioned into 206 approximately equal-sized spatial fractions. (a) Spatial locations of cells, with color indicating the order in which the fractions enter the annotation workflow. (b) Empirical $\alpha$ -quantiles of the NuClass score assigned to the true class, shown separately by cell type for $\alpha\in\{0.05,0.10,0.20\}$ . The dashed rose curve in each panel (right vertical axis) shows the corresponding cell-type proportion. Its irregular and, at times, abrupt changes reveal shifts in cell-type composition, while the variation in the solid quantile curves reveals within-cell-type score-distribution changes over the workflow.

Figure 2: Schematic of the CARED workflow. (a) Incoming cell batches are processed sequentially: CARED forms prediction sets, cells with set size at most one skip expert labeling, and a random audit set is reviewed to update the conformal level for subsequent batches. (b) Zoom-in of the prediction set generation process highlighted by the gray dashed box in panel (a), where prior review information updates the conformal level and the scoring model with the utility module produces prediction sets for the incoming batch.

Panel of Figure 3 — Figure 3: Validity and efficiency of CARED across three tissue-specific tasks using the global NuClass score source (Section 4 ). The $x$ -axis indicates the fraction of processed batches in the stream. (a) Cumulative annotation error (solid lines) and miscoverage (dashed lines) for the three tasks; the dotted horizontal line denotes the target error level $\alpha=0.1$ . (b) Fraction of cases bypassing expert review among all incoming ones for the three tasks at $\alpha=0.1$ .

Tissue cohort	# Cells	# Batches	Cell type (fractions %)
Pancreas	103,166	206	Acinar (12.8), Ductal (45.2), Endocrine (2.0), Endothelial (8.6), Fibroblast-like (27.6), T cell (3.8)
Lung	61,490	123	Endothelial (27.6), Epithelial malignant (61.8), T cell (5.9), Tumor-associated fibroblast (4.7)
Ovary	150,748	301	Endothelial (5.0), Epithelial malignant (85.3), Macrophage (9.7)

Table 1: Data used in the case study. Class proportions are computed over all cells in each cohort.

Figure 4: Interface for one pancreas batch in the partial-review workflow. The left panel shows a local histology field with reviewed cells highlighted by class-specific colored outlines. After clicking on a reviewed cell, the right panel displays its annotation information, including the default label, current label, and conformal prediction set. This example illustrates a reviewed cell whose label has been modified.

Panel of Figure 5 — Figure 5: Cumulative error trajectories under the global NuClass score for three batch-release schemes. Rows correspond to the error-guided, adjacent, and random schemes, while columns correspond to pancreas, lung, ovary, and marginal results. The tissue-specific panels report class-wise cumulative error over time; the marginal panels report cumulative error by tissue against the relative time $t/T$ . All experiments use target error level $\alpha=0.1$ and reviewed fraction $p=0.1$ . Dashed horizontal lines mark the target error level.

Panel of Figure 6 — Figure 6: Comparison of final cumulative error on pancreas tissue using the global score source. Each experiment randomly selects 10 batches as calibration data for the baselines and as the initial batches for CARED. (a) Distribution of final cumulative marginal error over 200 repeated random experiments, where the dashed vertical line marks the target error level $\alpha=0.1$ . (b) Empirical distributions of final class-wise error over the same 200 experiments for class-conditional conformal prediction and our method, shown separately by cell type. The dashed horizontal line is the target level $\alpha=0.1$ .

Panel of Figure 7 — Figure 7: Final skipped ratios using the global NuClass score. Rows correspond to tissues, while columns correspond to batch-release schemes. Each panel reports the marginal skipped ratio (bold) together with ratios stratified by true label. Here, EM denotes Epithelial Malignant and TAF denotes Tumor Associated Fibroblast. All experiments use target level $\alpha=0.1$ and reviewed fraction $p=0.1$ .

Panel of Figure 8 — Figure 8: Utility of the raw and transformed scores over time on pancreas tissue. Rows correspond to the batch-release schemes, while columns correspond to the metrics. Higher skipped ratio and structural coherence and smaller prediction sets indicate better utility (indicated by the arrows in panel captions).

Panel of Figure 9 — Figure 9: Results with real expert annotations on the colon dataset. (a) Cumulative annotation error (solid curves) and cumulative miscoverage (dashed curves) under the global, local, and gated NuClass score sources, with reviewed fraction $p=0.1$ and target error level $\alpha=0.1$ ; the black dotted horizontal line marks this target. (b) Cumulative skipped ratio under the same setting.

Panel of Figure 10 — Figure 10: Sensitivity analysis of marginal cumulative error on pancreas tissue with the global NuClass score source. Rows correspond to the error-guided, adjacent, and random batch release schemes, while columns vary the reviewed fraction $p$ , the random seed, and the target error level $\alpha$ . Dashed horizontal lines indicate the target error level used in each setting. In each column, only the parameter named in the column heading is varied; the other two retain their default values, $p=0.1$ , seed $0$ , and $\alpha=0.1$ , used in Sections 4.2 and 4.3 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Name and notation	Choice used in the experiments
Held-out tissues	Pancreas, ovary, and lung
Label space for pancreas	Acinar, Ductal, Endocrine, Endothelial, Fibroblast-like, T Cell
Subtree partition for pancreas	$\mathcal{S}_{1}=\{\mathrm{Acinar},\ \mathrm{Ductal},\ \mathrm{Endocrine}\}$ ; $\mathcal{S}_{2}=\{\mathrm{Endothelial},\ \mathrm{Fibroblast\text{-}like}\}$ ; $\mathcal{S}_{3}=\{\mathrm{T\ Cell}\}$
Label space for ovary	Endothelial, Epithelial Malignant, Macrophage
Subtree partition for ovary	$\mathcal{S}_{1}=\{\mathrm{Endothelial},\ \mathrm{Macrophage}\}$ ; $\mathcal{S}_{2}=\{\mathrm{Epithelial\ Malignant}\}$
Label space for lung	Endothelial, Epithelial Malignant, T Cell, Tumor Associated Fibroblast

Appendix

Table 2: Details and parameters in the main experiments.

Panel of Figure 11 — Figure 11: Online level trajectories $\alpha_{k,t}$ under the error-guided batch-release scheme using the global NuClass score. Panels (a)–(c) correspond to pancreas, lung, and ovary, respectively; each curve represents a cell type. The curves are smoothed with a window size of 7.

Panel of Figure 12 — Figure 12: Additional cumulative conformal error trajectories under the error-guided batch-release scheme. Rows correspond to the local and gated NuClass scores, while columns show pancreas, lung, ovary, and marginal results. The first three columns report class-wise errors within each tissue; the marginal panels compare the marginal errors of the three tissues against the fraction of processed batches. Dashed horizontal lines mark the target error level $\alpha=0.1$ .

Panel of Figure 13 — Figure 13: Additional cumulative utility comparison under the error-guided batch-release scheme using the global NuClass score. Rows correspond to the skipped ratio, average prediction set size, and structural coherence, while columns report lung and ovary. Solid blue curves use the utility-aware transformed score, while dashed orange curves use the raw score without the utility module. Higher skipped ratio and structural coherence and smaller prediction sets indicate better utility.

Panel of Figure 14 — Figure 14: Class-wise sensitivity analysis of final cumulative empirical error on pancreas tissue with the global score source. Rows correspond to the error-guided, adjacent, and random batch release schemes, while columns vary the reviewed fraction $p$ , the random seed, and the target error level $\alpha$ . Dashed horizontal lines indicate the target error level used in each setting, and percentages in the shared legend denote the empirical fraction of each cell type in the stream.

Panel of Figure 15 — Figure 15: Sensitivity analysis of utility metrics on the six-class pancreas annotation task with the global score source. Rows correspond to the error-guided, adjacent, and random batch release schemes, while columns vary the utility learning rate $\eta$ , the random seed, and the target error level $\alpha$ . Each panel compares the utility metrics obtained from the utility-aware transformed score and the raw score. The shared legend identifies the four utility metrics.

Figure 16: Comparison between constant and decaying class-wise step sizes on the pancreas experiment with the global NuClass score. The curves report per-batch all-cell deviations $|\mathrm{BatErr}_{k,t}^{D}-\alpha|$ for each class under the constant schedule used in the main experiments and the decaying schedule $\gamma_{k,t}=\gamma_{k}t^{-2/3}$ . Shaded intervals indicate which schedule is closer to the target error level $\alpha=0.1$ at each time.

Figure 17: Sensitivity of cumulative class-wise error trajectories to the scale of the constant step sizes on the pancreas experiment with the global NuClass score. Each curve uses the same class-wise stepsize rule as in the main experiments, multiplied by the factor shown in the legend. The dashed line marks the target error level $\alpha=0.1$ .

Human-in-the-Loop Annotation

Human-in-the-loop annotation combines human judgment and AI assistance to create, review, and improve labeled data.

See all 136 papers

From Model Uncertainty to Human Attention: Localization-Aware Visual Cues for Scalable Annotation Review
May 12, 2026Moussa Kassem Sbeyti, Joshua Holstein, Philipp Spitzer +2Uncertainty Quantification Human-in-the-Loop Annotation
1Scientific Computing Center, Karlsruhe Institute of Technology. · Institute for Information Systems, Karlsruhe Institute of Technology.
Vision-Language Models as Zero-Annotation Oracles in Histopathology
Jun 15, 2026Vishal Jain, Giorgio Buzzanca, Sarah Cechnicka +6Computational Pathology Histopathology Image Segmentation
Imperial College London, London, UK · Leiden University Medical Center, Leiden, The Netherlands · KU Leuven, Leuven, Belgium +3
Reliable AI-Assisted Decisions via Selective Auditing
Oct 22, 2026Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani +1AI-Assisted Decision Making Human Oversight of AI
Departments of Statistics and Management Science & Engineering, Stanford University

May 12, 2026J/K move · Enter open · S save

From Model Uncertainty to Human Attention: Localization-Aware Visual Cues for Scalable Annotation Review

Moussa Kassem Sbeyti, Joshua Holstein, Philipp Spitzer, Nadja Klein+1

1Scientific Computing Center, Karlsruhe Institute of Technology. · Institute for Information Systems, Karlsruhe Institute of Technology.

Uncertainty Quantification Human-in-the-Loop Annotation

Open paper AskPDF arXiv

High-quality labeled data is essential for training robust machine learning models, yet obtaining annotations at scale remains expensive. AI-assisted annotation has therefore become standard in large-scale labeling workflows. However, in tasks where model predictions carry two independent components, a class label and spatial boundaries, a model may classify an object with high confidence while mislocalizing it. Existing AI-assisted workflows offer annotators no signal about where spatial errors are most likely. Without such guidance, humans may systematically underinspect subtly misplaced boxes. We address this by studying the effect of visualizing spatial uncertainty via a purpose-built interface. In a controlled study with 120 participants, those receiving uncertainty cues achieve higher label quality while being faster overall. A box-level analysis confirms that the cues redirect annotator effort toward high-uncertainty predictions and away from well-localized boxes. These findings establish localization uncertainty as a lever to improve human-in-the-loop annotation. Code is available at https://mos-ks.github.io/MUHA/.

Abstract

Human-in-the-Loop Annotation

Explore similar work

From Model Uncertainty to Human Attention: Localization-Aware Visual Cues for Scalable Annotation Review

Vision-Language Models as Zero-Annotation Oracles in Histopathology

Reliable AI-Assisted Decisions via Selective Auditing