JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias
Authors: Fengxu Yang, Minghui Liwang, Yang Lu, Bo Han, Xianbin Wang, Seyyedali Hosseinalipour
Organizations: Guohao School, Tongji University, Shanghai, China · Department of Control Science and Engineering and the National Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai, China · School of Informatics, Xiamen University, Fujian, China · Department of Computer Science, Hong Kong Baptist University, Hong Kong, China · Department of Electrical and Computer Engineering, Western University, Ontario, Canada · Department of Electrical Engineering, University at Buffalo-SUNY, Buffalo, NY USA
High-quality annotations are essential for training and evaluating artificial intelligence (AI) systems. Quality control (QC) often acquires additional human or AI judgments before accepting a label. Nevertheless, shared biases can cause correlated judgments, leading conventional methods to mistake repeated agreement for independent evidence of correctness. To address this challenge, we propose JABER (Joint Alias-Preserving Belief for Evidence Routing), a bias-aware sequential QC framework that maintains a joint Bayesian posterior over label correctness and shared biases to guide evidence acquisition and termination decisions. Building on this posterior, we develop a finite-horizon algorithm that selects evidence-acquisition and termination actions to minimize expected QC loss under resource constraints. Our theoretical analysis shows that conditional-independence updating can become arbitrarily confident in an incorrect label under persistent shared bias. We further identify when repeated inspections cannot distinguish a correct label from an incorrect label supported by shared bias, when obtaining a new label can resolve this ambiguity, and why modeling the dependence between label correctness and shared bias can be necessary for QC decisions. Our experiments show that JABER reduces expected QC loss by an average of 4.65% compared with a method maintaining separate label correctness and bias probabilities when shared bias makes repeated inspections unreliable. Further, JABER's finite-horizon planning, which evaluates outcomes of successive QC actions, reduces expected QC loss by an average of 10.05% compared with a one-step lookahead method. In real-API experiments on FEVER, JABER achieves zero labeling errors while reducing QC costs by 2.67% and 16.99% compared with marginal-based belief modeling and one-step lookahead methods, respectively.
Figures & tables
图 1: Overview of JABER framework for sequential annotation QC under shared bias. The top panel introduces a motivating preference-annotation example in which an annotator initially favors a longer but incorrect response, potentially due to verbosity bias. The left panel illustrates JABER \CJK@punctchar \CJK@uniPunct 0”80”99s three phases: constructing possible observation outcomes and updating the joint belief over label correctness and shared bias (Phase 1), evaluating the expected losses of feasible QC actions through finite-horizon planning (Phase 2), and executing the selected action followed by belief updating and replanning (Phase 3). The right panel shows how repeated agreement driven by shared bias can lead conventional conditional-independence updating to accept the incorrect label, whereas JABER preserves competing explanations and uses a more informative QC action to identify the correct label.
K
b0
JABER
Marginal- Projection
CI Updating
Loss ↓
False accept (%) ↓
Loss ↓
False accept (%) ↓
Loss ↓
False accept (%) ↓
Original
2
0.50
7.95
26.43
8.29
21.48
8.22
22.14
2
0.60
7.24
23.52
7.30
22.43
7.24
23.52
2
0.70
6.00
30.00
6.25
17.82
6.31
18.51
3
0.50
8.23
28.51
8.57
23.59
8.54
24.37
表 I: Expected total QC loss and false-accept rate for JABER and its updating ablations in controlled synthetic experiments. Bold entries mark the lowest displayed value for each metric. K denotes the number of shared-bias types, and b0 is the initial correctness probability.
图 2: JABER ’s percentage reduction in expected total QC loss relative to Marginal-Projection ; higher is better. Both panels use Risk-sensitive costs and acquisition budget 8. Each cell or point averages the loss reduction over b0∈{0.50,0.60,0.70} . (a) Single-bias configurations vary the bias shifts affecting Inspect and Relabel , with at most six continuing steps. (b) A two-bias configuration varies the available continuing steps; the dashed line marks the first nonzero gain at three steps.
Method
Errors ↓ / 40
Mean API fee ↓
Mean total QC loss ↓
JABER
0
0.8845
0.8845
Marginal-Projection
0
0.9088
0.9088
CI Updating
1
0.5837
2.0507
One-step lookahead
0
1.0656
1.0656
Best precommitted sequence
0
0.8845
0.8845
Development-selected fixed escalation rule
0
1.2719
1.2719
表 II: Primary FEVER results on 40 evaluation claims. Errors count incorrect final labels; no listed method leaves a claim unresolved. Mean API fees and total QC losses are in 10−3 Chinese yuan (CNY). Total loss adds a penalty of 0.058678 CNY per erroneous output to API fees.
图 3: QC action counts on the 40 primary FEVER evaluation claims. Segment labels give counts by action; labels at bar ends give totals. Accept requires no API call and is omitted. Fixed escalation denotes the development-selected rule in Table II .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Y,y∗
Binary label space and true label.
y^init,y^t
Initial and currently held labels.
x,zt
Task attributes; held-label correctness.
K,ξk,ξ
Number of bias types, activity indicator, and bias vector.
P0,Pt
Initial and current joint beliefs.
bt,λt
Correctness probability and log-odds.
Appendix
表 III: Key notation
K
Weights
ΔL↓
ΔLact↓
ΔLerr↓
ΔLU↓
2
Original
−0.214
−1.267
+1.215
−0.161
3
Original
−0.156
−0.464
+0.364
−0.055
2
Risk-sensitive
−0.842
+0.038
−0.880
0
3
Risk-sensitive
−1.192
−0.103
−1.090
0
Appendix
表 IV: Decomposition of the expected QC loss difference between JABER and Marginal-Projection . Entries are JABER minus Marginal-Projection , averaged equally over b0∈{0.50,0.60,0.70} . ΔL is the total loss difference; ΔLact , ΔLerr , and ΔLU are its action-cost, erroneous-output, and unresolved-outcome components. K denotes the number of shared-bias types. Negative values indicate lower loss for JABER .
Original
Risk-sensitive
Baseline
Loss gain ↑
Δ FA ↓
Loss gain ↑
Δ FA ↓
CLARA-based Query/Stop
2.15%
−2.09 pp
1.58%
−1.96 pp
Raykar-style
5.26%
−0.95 pp
7.62%
−3.05 pp
Fixed-1
5.17%
−0.14 pp
22.72%
−2.45 pp
Fixed-3
15.29%
+1.32 pp
12.35%
−0.99 pp
Fixed-5
28.29%
+2.18 pp
9.88%
−0.13 pp
Appendix
表 V: Single-bias alias-stress comparisons, averaged over b0∈{0.50,0.60,0.70} by exact enumeration. Positive relative loss gain and negative false-accept difference ( Δ FA, in percentage points) favor JABER . All rows compare baselines in the matched Query/Stop task.
Data-generating link
Mean loss reduction ↑
Original
Risk-sensitive
Sigmoid (matched control)
2.51%
4.77%
Probit
2.30%
5.81%
Centered complementary log-log
2.63%
5.90%
Appendix
表 VI: Mean loss reduction of JABER relative to Marginal-Projection under observation-link misspecification.
Method
E/U
Mean API fee ↓
Mean QC loss ↓
Paired 95% interval
JABER
0/0
0.8845
0.8845
–
Marginal-Projection
0/0
0.9088
0.9088
[−0.0312,−0.0174]
CI Updating
1/0
0.5837
2.0507
[−4.1551,0.5174]
One-step lookahead
0/0
1.0656
1.0656
[−0.2809,−0.0934]
Best precommitted sequence
0/0
0.8845
0.8845
[0.0000,0.0000]
Development-selected fixed escalation rule
0/0
1.2719
1.2719
[−0.7024,−0.1134]
Appendix
表 VII: Complete primary FEVER comparison. Fees, losses, and intervals are in 10−3 CNY. E/U denotes erroneous outputs/unresolved cases out of 40. Intervals are for JABER minus the comparator’s total QC loss.
Mean total QC loss ↓ [E/U counts]
Method
Half error
Double error
Lower error
JABER
0.8845 [0/0]
0.9590 [0/0]
0.7506 [1/0]
Marginal-Projection
0.9088 [0/0]
0.9668 [0/0]
0.6290 [1/1]
CI Updating
1.3172 [1/0]
3.5176 [1/0]
0.7154 [1/3]
One-step lookahead
1.0656 [0/0]
1.9720 [0/0]
0.9348 [1/0]
Best precommitted sequence
0.8845 [0/0]
0.9011 [0/0]
0.7506 [1/0]
Appendix
表 VIII: FEVER sensitivity to terminal-loss weights. Each cell reports mean total QC loss in 10−3 CNY [E/U counts out of 40]. The half-error, double-error, and lower-error settings use the loss pairs specified in the text; API fees are unchanged. Primary results are reported in Table VII .
图 4: HealthContradict reference-assisted Relabel cost sensitivity. The horizontal axis gives the Relabel cost; the vertical axis gives mean total QC loss over 40 gold-NO samples with two episodes per sample, with lower values indicating better performance. Risk-sensitive weights are used, with Inspect cost normalized to 1. The vertical line marks the calibration-selected cR=4 .
Dataset
Reference label
Action mapping
Data and evaluation split
LLMBar
Human response-pair preference.
Initial choice: GRM-Llama-3.2-3B ; six further scoring models provide Inspect judgments.
Model/order selection: 209 training and 82 validation records; test: 98 adversarial and 30 natural pairs.
PPE-IFEval
Programmatic instruction checks.
Initial model: InternLM2-7B ; Inspect : its 20B and 1.8B variants; Blind Relabel : cross-family Gemma .
512 instructions, split by instruction: 1,540 development and 520 validation episodes; 495 complete records from 500 held-out episodes.