cs.AINov 2, 2026

JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias

Authors: Fengxu Yang, Minghui Liwang, Yang Lu, Bo Han, Xianbin Wang, Seyyedali Hosseinalipour

Organizations: Guohao School, Tongji University, Shanghai, China · Department of Control Science and Engineering and the National Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai, China · School of Informatics, Xiamen University, Fujian, China · Department of Computer Science, Hong Kong Baptist University, Hong Kong, China · 

Abstract

High-quality annotations are essential for training and evaluating artificial intelligence (AI) systems. Quality control (QC) often acquires additional human or AI judgments before accepting a label. Nevertheless, shared biases can cause correlated judgments, leading conventional methods to mistake repeated agreement for independent evidence of correctness. To address this challenge, we propose JABER (Joint Alias-Preserving Belief for Evidence Routing), a bias-aware sequential QC framework that maintains a joint Bayesian posterior over label correctness and shared biases to guide evidence acquisition and termination decisions. Building on this posterior, we develop a finite-horizon algorithm that selects evidence-acquisition and termination actions to minimize expected QC loss under resource constraints. Our theoretical analysis shows that conditional-independence updating can become arbitrarily confident in an incorrect label under persistent shared bias. We further identify when repeated inspections cannot distinguish a correct label from an incorrect label supported by shared bias, when obtaining a new label can resolve this ambiguity, and why modeling the dependence between label correctness and shared bias can be necessary for QC decisions. Our experiments show that JABER reduces expected QC loss by an average of 4.65% compared with a method maintaining separate label correctness and bias probabilities when shared bias makes repeated inspections unreliable. Further, JABER's finite-horizon planning, which evaluates outcomes of successive QC actions, reduces expected QC loss by an average of 10.05% compared with a one-step lookahead method. In real-API experiments on FEVER, JABER achieves zero labeling errors while reducing QC costs by 2.67% and 16.99% compared with marginal-based belief modeling and one-step lookahead methods, respectively.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. The geometry of AI validation: From structural blindness to reusable audits

    Aug 21, 2026Ricardo FitasAutomated EvaluationAlgorithmic Auditing

  2. Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation

    Apr 12, 2025Hsuan Wei Liao, Christopher Klugmann, Daniel Kondermann +1Human-in-the-Loop AnnotationAnnotator Disagreement

  3. Reliable AI-Assisted Decisions via Selective Auditing

    Oct 22, 2026Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani +1AI-Assisted Decision MakingHuman Oversight of AI