When Disagreement Labels Do Not Identify F1 Rankings
Organizations: Faculty of Engineering, University of Deusto Avda. de las Universidades, 24, 48007 Bilbao, Spain
Abstract
Agreement labels cancel from an accuracy comparison but can change an F1 ranking. We specialize partial identification to an explicit audit after labeling disagreements: two agreement counts give exact bounds, certifying a ranking or a choice within a specified F1 tolerance under every completion. Across 3,200 frozen model pairs, shared completions tighten bounds and a 0.01 tolerance reduces labeling costs. A zero-loss (weak-winner) control attributes most savings to positive tolerance, not ties. Guarantees concern the evaluated batch, not future data.
Figures & tables
| Initial | Additional | Total | ||||
|---|---|---|---|---|---|---|
| Cohort | Policy | Exact | Exact | |||
| All pairs | Uniform | 54.5 | 88.21 | 35.58 | 142.70 | 90.07 |
| All pairs | Alternating | 54.5 | 80.64 | 31.29 | 135.13 | 85.78 |
| All pairs | Magnitude | 54.5 | 66.67 | 20.10 | 121.16 | 74.59 |
| Validation | Uniform | 30.7 | 89.07 | 25.56 | 119.79 | 56.28 |
| Validation | Alternating | 30.7 | 77.58 | 16.34 | 108.30 | 47.07 |
Pairwise Ranking
Pairwise ranking infers ordered preferences from comparisons between items, systems, or responses.
See all 66 papers