Conformal Annotation with Randomized Expert Review: Application to Online Cell-Type Labeling
Authors: Xuhe Zhang, Zhi Huang, Ying Jin
Organizations: Applied Mathematics and Computational Science, University of Pennsylvania · Department of Pathology and Laboratory Medicine, University of Pennsylvania · Department of Statistics and Data Science, University of Pennsylvania
Artificial intelligence (AI) models can accelerate cell-type annotation in digital pathology, but safely scaling their use requires determining, with error control, which model-generated labels can be accepted without expert review. This is challenging in evolving workflows because cell morphology and type composition may change across tissue regions and slides, making one-time calibration unreliable. We propose Conformal Annotation with Randomized Expert review under Distribution shift (CARED), a distribution-free framework that leverages randomized expert audit to achieve long-run error control for AI-assisted annotation under distribution shift. For each incoming batch, CARED uniformly selects a subset of cases for expert review and uses the acquired labels to calibrate later decisions. Among unreviewed cases, CARED determines a subset of them to safely skip expert labeling, while suggesting candidate labels for the others for downstream resolution. We prove that this workflow provides long-run control of cumulative error over both the full stream and the cells outside of the audit sample, without stationarity or exchangeability assumptions. We further enhance the framework by introducing class-conditional guarantees to ensure error control within important but rare cell types, together with a utility module to target more automatic resolutions, smaller candidate sets, and greater biological coherence, without compromising validity. Experiments on pancreas, lung, and ovary annotation streams, together with a stream constructed from real expert annotations, show that CARED controls the cumulative errors while a substantial fraction of cells (up to 70%) bypass expert review.
Figures & tables
Figure 1: Evidence of workflow shift in image-based cell-type annotation. A pancreas whole-slide image containing more than 105 cells is partitioned into 206 approximately equal-sized spatial fractions. (a) Spatial locations of cells, with color indicating the order in which the fractions enter the annotation workflow. (b) Empirical α -quantiles of the NuClass score assigned to the true class, shown separately by cell type for α∈{0.05,0.10,0.20} . The dashed rose curve in each panel (right vertical axis) shows the corresponding cell-type proportion. Its irregular and, at times, abrupt changes reveal shifts in cell-type composition, while the variation in the solid quantile curves reveals within-cell-type score-distribution changes over the workflow.
Figure 2: Schematic of the CARED workflow. (a) Incoming cell batches are processed sequentially: CARED forms prediction sets, cells with set size at most one skip expert labeling, and a random audit set is reviewed to update the conformal level for subsequent batches. (b) Zoom-in of the prediction set generation process highlighted by the gray dashed box in panel (a), where prior review information updates the conformal level and the scoring model with the utility module produces prediction sets for the incoming batch.
Figure 3: Validity and efficiency of CARED across three tissue-specific tasks using the global NuClass score source (Section 4 ). The x -axis indicates the fraction of processed batches in the stream. (a) Cumulative annotation error (solid lines) and miscoverage (dashed lines) for the three tasks; the dotted horizontal line denotes the target error level α=0.1 . (b) Fraction of cases bypassing expert review among all incoming ones for the three tasks at α=0.1 .
Table 1: Data used in the case study. Class proportions are computed over all cells in each cohort.
Figure 4: Interface for one pancreas batch in the partial-review workflow. The left panel shows a local histology field with reviewed cells highlighted by class-specific colored outlines. After clicking on a reviewed cell, the right panel displays its annotation information, including the default label, current label, and conformal prediction set. This example illustrates a reviewed cell whose label has been modified.
Figure 5: Cumulative error trajectories under the global NuClass score for three batch-release schemes. Rows correspond to the error-guided, adjacent, and random schemes, while columns correspond to pancreas, lung, ovary, and marginal results. The tissue-specific panels report class-wise cumulative error over time; the marginal panels report cumulative error by tissue against the relative time t/T . All experiments use target error level α=0.1 and reviewed fraction p=0.1 . Dashed horizontal lines mark the target error level.
Figure 6: Comparison of final cumulative error on pancreas tissue using the global score source. Each experiment randomly selects 10 batches as calibration data for the baselines and as the initial batches for CARED. (a) Distribution of final cumulative marginal error over 200 repeated random experiments, where the dashed vertical line marks the target error level α=0.1 . (b) Empirical distributions of final class-wise error over the same 200 experiments for class-conditional conformal prediction and our method, shown separately by cell type. The dashed horizontal line is the target level α=0.1 .
Figure 7: Final skipped ratios using the global NuClass score. Rows correspond to tissues, while columns correspond to batch-release schemes. Each panel reports the marginal skipped ratio (bold) together with ratios stratified by true label. Here, EM denotes Epithelial Malignant and TAF denotes Tumor Associated Fibroblast. All experiments use target level α=0.1 and reviewed fraction p=0.1 .
Figure 8: Utility of the raw and transformed scores over time on pancreas tissue. Rows correspond to the batch-release schemes, while columns correspond to the metrics. Higher skipped ratio and structural coherence and smaller prediction sets indicate better utility (indicated by the arrows in panel captions).
Figure 9: Results with real expert annotations on the colon dataset. (a) Cumulative annotation error (solid curves) and cumulative miscoverage (dashed curves) under the global, local, and gated NuClass score sources, with reviewed fraction p=0.1 and target error level α=0.1 ; the black dotted horizontal line marks this target. (b) Cumulative skipped ratio under the same setting.
Figure 10: Sensitivity analysis of marginal cumulative error on pancreas tissue with the global NuClass score source. Rows correspond to the error-guided, adjacent, and random batch release schemes, while columns vary the reviewed fraction p , the random seed, and the target error level α . Dashed horizontal lines indicate the target error level used in each setting. In each column, only the parameter named in the column heading is varied; the other two retain their default values, p=0.1 , seed 0 , and α=0.1 , used in Sections 4.2 and 4.3 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Name and notation
Choice used in the experiments
Held-out tissues
Pancreas, ovary, and lung
Label space for pancreas
Acinar, Ductal, Endocrine, Endothelial, Fibroblast-like, T Cell
Endothelial, Epithelial Malignant, T Cell, Tumor Associated Fibroblast
Appendix
Table 2: Details and parameters in the main experiments.
Figure 11: Online level trajectories αk,t under the error-guided batch-release scheme using the global NuClass score. Panels (a)–(c) correspond to pancreas, lung, and ovary, respectively; each curve represents a cell type. The curves are smoothed with a window size of 7.
Figure 12: Additional cumulative conformal error trajectories under the error-guided batch-release scheme. Rows correspond to the local and gated NuClass scores, while columns show pancreas, lung, ovary, and marginal results. The first three columns report class-wise errors within each tissue; the marginal panels compare the marginal errors of the three tissues against the fraction of processed batches. Dashed horizontal lines mark the target error level α=0.1 .
Figure 13: Additional cumulative utility comparison under the error-guided batch-release scheme using the global NuClass score. Rows correspond to the skipped ratio, average prediction set size, and structural coherence, while columns report lung and ovary. Solid blue curves use the utility-aware transformed score, while dashed orange curves use the raw score without the utility module. Higher skipped ratio and structural coherence and smaller prediction sets indicate better utility.
Figure 14: Class-wise sensitivity analysis of final cumulative empirical error on pancreas tissue with the global score source. Rows correspond to the error-guided, adjacent, and random batch release schemes, while columns vary the reviewed fraction p , the random seed, and the target error level α . Dashed horizontal lines indicate the target error level used in each setting, and percentages in the shared legend denote the empirical fraction of each cell type in the stream.
Figure 15: Sensitivity analysis of utility metrics on the six-class pancreas annotation task with the global score source. Rows correspond to the error-guided, adjacent, and random batch release schemes, while columns vary the utility learning rate η , the random seed, and the target error level α . Each panel compares the utility metrics obtained from the utility-aware transformed score and the raw score. The shared legend identifies the four utility metrics.
Figure 16: Comparison between constant and decaying class-wise step sizes on the pancreas experiment with the global NuClass score. The curves report per-batch all-cell deviations ∣BatErrk,tD−α∣ for each class under the constant schedule used in the main experiments and the decaying schedule γk,t=γkt−2/3 . Shaded intervals indicate which schedule is closer to the target error level α=0.1 at each time.
Figure 17: Sensitivity of cumulative class-wise error trajectories to the scale of the constant step sizes on the pancreas experiment with the global NuClass score. Each curve uses the same class-wise stepsize rule as in the main experiments, multiplied by the factor shown in the legend. The dashed line marks the target error level α=0.1 .
High-quality labeled data is essential for training robust machine learning models, yet obtaining annotations at scale remains expensive. AI-assisted annotation has therefore become standard in large-scale labeling workflows. However, in tasks where model predictions carry two independent components, a class label and spatial boundaries, a model may classify an object with high confidence while mislocalizing it. Existing AI-assisted workflows offer annotators no signal about where spatial errors are most likely. Without such guidance, humans may systematically underinspect subtly misplaced boxes. We address this by studying the effect of visualizing spatial uncertainty via a purpose-built interface. In a controlled study with 120 participants, those receiving uncertainty cues achieve higher label quality while being faster overall. A box-level analysis confirms that the cues redirect annotator effort toward high-uncertainty predictions and away from well-localized boxes. These findings establish localization uncertainty as a lever to improve human-in-the-loop annotation. Code is available at https://mos-ks.github.io/MUHA/.
Moussa Kassem Sbeyti, Joshua Holstein, Philipp Spitzer +2
1Scientific Computing Center, Karlsruhe Institute of Technology. · Institute for Information Systems, Karlsruhe Institute of Technology.
Foreground segmentation is the critical first step of every computational pathology pipeline, yet existing methods rely on hand-tuned heuristics or supervised models that overfit to narrow stain and scanner distributions, failing silently on specialised stains such as Jones silver or Elastica van Gieson. We propose a coarse-to-fine approach that recasts foreground segmentation as a visual perception task and leverages general-purpose vision-language models (VLMs) as zero-annotation oracles. Our key insight is that tissue-versus-background discrimination is a natural-image recognition problem, not a histopathological one, so VLMs trained on internet-scale corpora generalise where domain-specific models cannot. We introduce Leica-75, a benchmark of 75 renal transplant whole-slide images spanning three stain families. On Leica-75, our method achieves the highest segmentation quality on out-of-distribution stains (Dice 0.858 +/- 0.027 on Jones, 0.853 +/- 0.041 on EVG) with 7x lower cross-stain variance than the best supervised baseline, while remaining competitive on in-distribution H&E. Few-shot prompting with automatically curated exemplars (Auto-context) rescues hard cases on Stress-32 (n=32), a curated stress-test subset (Dice 0.470 to 0.819 for the 2B model). VLM-based annotation review matches human expert consensus (kappa=0.989 for blur detection; mean precision/recall grading accuracy 0.708 vs. human 0.646 for segmentation mask review). The resulting pseudo-labels are used to distil lightweight student models that are as performant as the teacher model while running for a fraction of the cost. Our framework provides a principled, scalable solution to a persistent infrastructure bottleneck in digital pathology.
Vishal Jain, Giorgio Buzzanca, Sarah Cechnicka +6
Imperial College London, London, UK · Leiden University Medical Center, Leiden, The Netherlands · KU Leuven, Leuven, Belgium +3
We consider online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score estimating the trustworthiness of the decision. The decision and confidence score need not be produced by the same model: for example, one agent may propose a decision, while a separate verifier agent evaluates it and outputs a correctness or safety score. Based on this information, we must decide whether to accept the AI decision automatically or consult an expert, such as a human auditor, who reviews the proposal and either accepts or corrects it. We formulate this task as an online constrained optimization problem in which the goal is to minimize the frequency of expert auditing while guaranteeing that the fraction of AI errors that go unaudited stays below a user-defined threshold. We derive a family of algorithms with provable error-control guarantees with no assumptions, including a method that is optimally efficient: it asymptotically attains the best achievable auditing rate for both adversarial and i.i.d. data streams. Experiments on screening harmful requests to large language models, qualitative text annotation, and data labeling show that our methods achieve valid error control in practice while substantially reducing the need for human auditing compared to baselines.
Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani +1
Departments of Statistics and Management Science & Engineering, Stanford University