JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias
Authors: Fengxu Yang, Minghui Liwang, Yang Lu, Bo Han, Xianbin Wang, Seyyedali Hosseinalipour
Organizations: Guohao School, Tongji University, Shanghai, China · Department of Control Science and Engineering and the National Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai, China · School of Informatics, Xiamen University, Fujian, China · Department of Computer Science, Hong Kong Baptist University, Hong Kong, China · Department of Electrical and Computer Engineering, Western University, Ontario, Canada · Department of Electrical Engineering, University at Buffalo-SUNY, Buffalo, NY USA
High-quality annotations are essential for training and evaluating artificial intelligence (AI) systems. Quality control (QC) often acquires additional human or AI judgments before accepting a label. Nevertheless, shared biases can cause correlated judgments, leading conventional methods to mistake repeated agreement for independent evidence of correctness. To address this challenge, we propose JABER (Joint Alias-Preserving Belief for Evidence Routing), a bias-aware sequential QC framework that maintains a joint Bayesian posterior over label correctness and shared biases to guide evidence acquisition and termination decisions. Building on this posterior, we develop a finite-horizon algorithm that selects evidence-acquisition and termination actions to minimize expected QC loss under resource constraints. Our theoretical analysis shows that conditional-independence updating can become arbitrarily confident in an incorrect label under persistent shared bias. We further identify when repeated inspections cannot distinguish a correct label from an incorrect label supported by shared bias, when obtaining a new label can resolve this ambiguity, and why modeling the dependence between label correctness and shared bias can be necessary for QC decisions. Our experiments show that JABER reduces expected QC loss by an average of 4.65% compared with a method maintaining separate label correctness and bias probabilities when shared bias makes repeated inspections unreliable. Further, JABER's finite-horizon planning, which evaluates outcomes of successive QC actions, reduces expected QC loss by an average of 10.05% compared with a one-step lookahead method. In real-API experiments on FEVER, JABER achieves zero labeling errors while reducing QC costs by 2.67% and 16.99% compared with marginal-based belief modeling and one-step lookahead methods, respectively.
Figures & tables
图 1: Overview of JABER framework for sequential annotation QC under shared bias. The top panel introduces a motivating preference-annotation example in which an annotator initially favors a longer but incorrect response, potentially due to verbosity bias. The left panel illustrates JABER \CJK@punctchar \CJK@uniPunct 0”80”99s three phases: constructing possible observation outcomes and updating the joint belief over label correctness and shared bias (Phase 1), evaluating the expected losses of feasible QC actions through finite-horizon planning (Phase 2), and executing the selected action followed by belief updating and replanning (Phase 3). The right panel shows how repeated agreement driven by shared bias can lead conventional conditional-independence updating to accept the incorrect label, whereas JABER preserves competing explanations and uses a more informative QC action to identify the correct label.
K
b0
JABER
Marginal- Projection
CI Updating
Loss ↓
False accept (%) ↓
Loss ↓
False accept (%) ↓
Loss ↓
False accept (%) ↓
Original
2
0.50
7.95
26.43
8.29
21.48
8.22
22.14
2
0.60
7.24
23.52
7.30
22.43
7.24
23.52
2
0.70
6.00
30.00
6.25
17.82
6.31
18.51
3
0.50
8.23
28.51
8.57
23.59
8.54
24.37
表 I: Expected total QC loss and false-accept rate for JABER and its updating ablations in controlled synthetic experiments. Bold entries mark the lowest displayed value for each metric. K denotes the number of shared-bias types, and b0 is the initial correctness probability.
图 2: JABER ’s percentage reduction in expected total QC loss relative to Marginal-Projection ; higher is better. Both panels use Risk-sensitive costs and acquisition budget 8. Each cell or point averages the loss reduction over b0∈{0.50,0.60,0.70} . (a) Single-bias configurations vary the bias shifts affecting Inspect and Relabel , with at most six continuing steps. (b) A two-bias configuration varies the available continuing steps; the dashed line marks the first nonzero gain at three steps.
Method
Errors ↓ / 40
Mean API fee ↓
Mean total QC loss ↓
JABER
0
0.8845
0.8845
Marginal-Projection
0
0.9088
0.9088
CI Updating
1
0.5837
2.0507
One-step lookahead
0
1.0656
1.0656
Best precommitted sequence
0
0.8845
0.8845
Development-selected fixed escalation rule
0
1.2719
1.2719
表 II: Primary FEVER results on 40 evaluation claims. Errors count incorrect final labels; no listed method leaves a claim unresolved. Mean API fees and total QC losses are in 10−3 Chinese yuan (CNY). Total loss adds a penalty of 0.058678 CNY per erroneous output to API fees.
图 3: QC action counts on the 40 primary FEVER evaluation claims. Segment labels give counts by action; labels at bar ends give totals. Accept requires no API call and is omitted. Fixed escalation denotes the development-selected rule in Table II .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Y,y∗
Binary label space and true label.
y^init,y^t
Initial and currently held labels.
x,zt
Task attributes; held-label correctness.
K,ξk,ξ
Number of bias types, activity indicator, and bias vector.
P0,Pt
Initial and current joint beliefs.
bt,λt
Correctness probability and log-odds.
Appendix
表 III: Key notation
K
Weights
ΔL↓
ΔLact↓
ΔLerr↓
ΔLU↓
2
Original
−0.214
−1.267
+1.215
−0.161
3
Original
−0.156
−0.464
+0.364
−0.055
2
Risk-sensitive
−0.842
+0.038
−0.880
0
3
Risk-sensitive
−1.192
−0.103
−1.090
0
Appendix
表 IV: Decomposition of the expected QC loss difference between JABER and Marginal-Projection . Entries are JABER minus Marginal-Projection , averaged equally over b0∈{0.50,0.60,0.70} . ΔL is the total loss difference; ΔLact , ΔLerr , and ΔLU are its action-cost, erroneous-output, and unresolved-outcome components. K denotes the number of shared-bias types. Negative values indicate lower loss for JABER .
Original
Risk-sensitive
Baseline
Loss gain ↑
Δ FA ↓
Loss gain ↑
Δ FA ↓
CLARA-based Query/Stop
2.15%
−2.09 pp
1.58%
−1.96 pp
Raykar-style
5.26%
−0.95 pp
7.62%
−3.05 pp
Fixed-1
5.17%
−0.14 pp
22.72%
−2.45 pp
Fixed-3
15.29%
+1.32 pp
12.35%
−0.99 pp
Fixed-5
28.29%
+2.18 pp
9.88%
−0.13 pp
Appendix
表 V: Single-bias alias-stress comparisons, averaged over b0∈{0.50,0.60,0.70} by exact enumeration. Positive relative loss gain and negative false-accept difference ( Δ FA, in percentage points) favor JABER . All rows compare baselines in the matched Query/Stop task.
Data-generating link
Mean loss reduction ↑
Original
Risk-sensitive
Sigmoid (matched control)
2.51%
4.77%
Probit
2.30%
5.81%
Centered complementary log-log
2.63%
5.90%
Appendix
表 VI: Mean loss reduction of JABER relative to Marginal-Projection under observation-link misspecification.
Method
E/U
Mean API fee ↓
Mean QC loss ↓
Paired 95% interval
JABER
0/0
0.8845
0.8845
–
Marginal-Projection
0/0
0.9088
0.9088
[−0.0312,−0.0174]
CI Updating
1/0
0.5837
2.0507
[−4.1551,0.5174]
One-step lookahead
0/0
1.0656
1.0656
[−0.2809,−0.0934]
Best precommitted sequence
0/0
0.8845
0.8845
[0.0000,0.0000]
Development-selected fixed escalation rule
0/0
1.2719
1.2719
[−0.7024,−0.1134]
Appendix
表 VII: Complete primary FEVER comparison. Fees, losses, and intervals are in 10−3 CNY. E/U denotes erroneous outputs/unresolved cases out of 40. Intervals are for JABER minus the comparator’s total QC loss.
Mean total QC loss ↓ [E/U counts]
Method
Half error
Double error
Lower error
JABER
0.8845 [0/0]
0.9590 [0/0]
0.7506 [1/0]
Marginal-Projection
0.9088 [0/0]
0.9668 [0/0]
0.6290 [1/1]
CI Updating
1.3172 [1/0]
3.5176 [1/0]
0.7154 [1/3]
One-step lookahead
1.0656 [0/0]
1.9720 [0/0]
0.9348 [1/0]
Best precommitted sequence
0.8845 [0/0]
0.9011 [0/0]
0.7506 [1/0]
Appendix
表 VIII: FEVER sensitivity to terminal-loss weights. Each cell reports mean total QC loss in 10−3 CNY [E/U counts out of 40]. The half-error, double-error, and lower-error settings use the loss pairs specified in the text; API fees are unchanged. Primary results are reported in Table VII .
图 4: HealthContradict reference-assisted Relabel cost sensitivity. The horizontal axis gives the Relabel cost; the vertical axis gives mean total QC loss over 40 gold-NO samples with two episodes per sample, with lower values indicating better performance. Risk-sensitive weights are used, with Inspect cost normalized to 1. The vertical line marks the calibration-selected cR=4 .
Dataset
Reference label
Action mapping
Data and evaluation split
LLMBar
Human response-pair preference.
Initial choice: GRM-Llama-3.2-3B ; six further scoring models provide Inspect judgments.
Model/order selection: 209 training and 82 validation records; test: 98 adversarial and 30 natural pairs.
PPE-IFEval
Programmatic instruction checks.
Initial model: InternLM2-7B ; Inspect : its 20B and 1.8B variants; Blind Relabel : cross-family Gemma .
512 instructions, split by instruction: 1,540 development and 520 validation episodes; 495 complete records from 500 held-out episodes.
AI systems increasingly search among candidate answers and deploy the highest-scoring one. Increasing search changes which errors matter, so a precise evaluation at one computation budget can leave another budget unresolved. We connect this information gap to the cost of closing it. For independent best-of-n search, aggregate reliability measurements identify deployment only through the directions they observe; we derive an exact ambiguity frontier when only small search widths are audited. Retaining candidate ranks and truth labels enables a constructive alternative: one audit can estimate reliability across all widths up to N. With known score percentiles, the minimax worst-coordinate mean squared error scales as (1 + log N)/T + N/M, capped at a constant, for expected budgets of T truth labels and M candidate observations. Matching lower bounds allow adaptive label acquisition, establishing that the distinct label and candidate costs are intrinsic to this experiment. An explicit design attains this order; a complementary record-based procedure supplies simultaneous guarantees without a known score distribution. Retrospective mathematical-reasoning and code-generation analyses show why search-dependent validation matters. In held-out CodeRM pools, a shared audit reduces the 95th-percentile maximum error across 100 widths by 58% and 40% relative to uniform labeling. These results turn structural ambiguity into a quantitative prescription for reusable validation.
Ricardo Fitas
Technical University of Darmstadt, 64289 Darmstadt, Germany
Data annotation is an essential but costly aspect of developing machine learning-based technology, requiring multiple human workers to repeatedly label each data point in a data set. We explore the tradeoff between annotation accuracy and cost by identifying minority reports---worker responses disagreeing with the reference majority-vote label---that may indicate unnecessary redundancy, and can therefore be proactively pruned before they are executed. We develop a theoretical framework for choosing the number of repeated annotations required for a task and pruning policies at a target label quality. We estimate minority-report risk on two computer vision datasets re-annotated through a professional platform, then evaluate a rolling policy on the assignment stream to prune over 35% of repeats while preserving more than 99.9% of reference majority-vote labels.
Hsuan Wei Liao, Christopher Klugmann, Daniel Kondermann +1
Telfer School of Management, University of Ottawa · Quality Match · NVIDIA
We consider online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score estimating the trustworthiness of the decision. The decision and confidence score need not be produced by the same model: for example, one agent may propose a decision, while a separate verifier agent evaluates it and outputs a correctness or safety score. Based on this information, we must decide whether to accept the AI decision automatically or consult an expert, such as a human auditor, who reviews the proposal and either accepts or corrects it. We formulate this task as an online constrained optimization problem in which the goal is to minimize the frequency of expert auditing while guaranteeing that the fraction of AI errors that go unaudited stays below a user-defined threshold. We derive a family of algorithms with provable error-control guarantees with no assumptions, including a method that is optimally efficient: it asymptotically attains the best achievable auditing rate for both adversarial and i.i.d. data streams. Experiments on screening harmful requests to large language models, qualitative text annotation, and data labeling show that our methods achieve valid error control in practice while substantially reducing the need for human auditing compared to baselines.
Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani +1
Departments of Statistics and Management Science & Engineering, Stanford University