Counting Under Disagreement: Dollar-Weighted Measurement, Selective Machine Labelling and Budgeted Human Review from Disagreeing AI Readers of Administrative Text
Organizations: Ascendr, PBC
Abstract
We study AI-assisted measurement when one text label applies to many administrative records, the target is a monetary total, and the human readings that anchor the machines disagree. The testbed is 121,070 grant-purpose texts from US tax filings (799,675 records, $126.4B), labelled by a rule-based classifier and three language models, with a designed sample of 651 texts read independently by three people under one codebook. We state exact conditions for a fixed-predictor difference estimator and for a cross-fitted predictor whose folds are assigned to the whole frame before sampling, under Poisson sampling, and assess the implemented fixed-size procedures by simulation: 8 predictors whose own totals span $11.2-15.0B agree to within $0.4B once rectified. Two rules that accept machine labels at a stated dollar-weighted error, weighted conformal p-values under a dollar measure and Learn-then-Test on the design estimate of the error, are evaluated on nested held-out folds and on re-draws of the design with synthetic truths; their error control in deployment is not established, and the count-based rule accepts no text under a dollar-drawn sample. In a replay that charges every human label, a two-phase review policy whose calibrator is fitted on its own first phase reduces the RMSE of the total by 28% at a budget of 100 texts relative to dollar-proportional sampling, while pure Neyman allocation fails with an uncalibrated predictor. Independent readers shift the estimated total by $3.93B, 2.2 times the within-reader sampling interval, and institution-level intervals under-cover at every size. Every headline result is re-run under each reader. The results identify both the benefits and the limits of using learned predictions to decide what a person should verify.
Figures & tables
| ; , | texts (shared evidence units); the human sample and its size |
|---|---|
| ; ; | sampling size (all external dollars); domain dollars; domain dollars in tax year |
| ; ; | reference label; reading of machine reader ; features |
| ; , ; | vote-pattern stratum; its domain dollars and sampling size; its PPS draws |
| ; ; | inclusion probability; design weight; inclusion indicator |
| ; | a prediction of ; the residual at |
| dollars text stands for in the population (its likelihood-ratio weight under the dollar measure) |
| Task | training and tuning labels | prediction of a text |
|---|---|---|
| T1 estimation (Section 6 ) | the sampled texts outside the text’s frame fold; ridge strength chosen by inner 5-fold cross-validation within those texts | the model of the text’s own frame fold, whether or not the text was sampled |
| T2 held-out evaluation (Section 7 ) | in each of 100 splits, the four calibration folds of only; strength re-chosen on them; nulls and risk estimates from them | calibration texts: cross-fitted within the calibration folds; held-out fold and unsampled population: one model fitted on all calibration folds |
| T2 stress tests (Appendix B ) | none refitted: the T1 calibrator’s scores are held fixed (fixed-score tests) | as T1 |
| T3 protected replay (Section 8 ) | two thirds of by stratum, strength a fixed constant | one model fitted on the two thirds scores the replay third |
| T3 designs with every label charged (Section 8.4 ) | only the labels the policy has bought so far; strength a fixed constant; constant mean label below five labels | one model fitted on the bought labels scores every unread text before the next wave is drawn |
| Institutions (Section 10 ) | the T1 calibrator, held fixed in the simulation | as T1 |
| Readers | same status ( ) | same answer on research or not | |||||
|---|---|---|---|---|---|---|---|
| texts | design-weighted | texts | design-weighted | dollar-weighted | |||
| reader 1 vs reader 2 | 63% | 59% [42%, 75%] | 0.36 | 84% | 82% [66%, 96%] | 94% [91%, 96%] | 0.51 |
| reader 1 vs reader 3 | 70% | 67% [49%, 84%] | 0.48 | 87% | 84% [68%, 98%] | 96% [93%, 97%] | 0.64 |
| reader 2 vs reader 3 | 61% | 63% [46%, 79%] | 0.34 | 82% | 94% [91%, 97%] | 92% [89%, 95%] | 0.51 |
| all three readers | 50% | 45% [30%, 62%] | 0.39 | 77% | 80% [64%, 94%] | 91% [87%, 94%] | 0.56 |
| Removed | log-loss (texts) | AUC (texts) | log-loss (dollars) | vs full [95% CI] | AUC (dollars) |
|---|---|---|---|---|---|
| none (full model) | 0.397 | 0.784 | 0.098 | – | 0.987 |
| Rules | 0.388 | 0.788 | 0.100 | [-0.001, +0.005] | 0.986 |
| Claude Sonnet 5.5 | 0.374 | 0.742 | 0.102 | [-0.000, +0.010] | 0.986 |
| Gemini 2.5 Flash | 0.405 | 0.755 | 0.097 | [-0.002, +0.001] | 0.987 |
| GPT-5 mini | 0.394 | 0.767 | 0.095 | [-0.007, -0.000] | 0.987 |
| DS posteriors | 0.391 | 0.790 | 0.096 | [-0.003, -0.001] | 0.987 |
| Reader A | Reader B | A wrong | B wrong | both wrong | if independent | error correlation |
|---|---|---|---|---|---|---|
| Rules | Claude Sonnet 5.5 | 22% | 20% | 18% | 4.4% | 0.82 |
| Rules | Gemini 2.5 Flash | 22% | 21% | 19% | 4.6% | 0.86 |
| Rules | GPT-5 mini | 22% | 21% | 18% | 4.5% | 0.84 |
| Claude Sonnet 5.5 | Gemini 2.5 Flash | 20% | 21% | 18% | 4.2% | 0.83 |
| Claude Sonnet 5.5 | GPT-5 mini | 20% | 21% | 17% | 4.1% | 0.81 |
| Gemini 2.5 Flash | GPT-5 mini | 21% | 21% | 19% | 4.3% | 0.90 |
| Text feature | share of texts | share of disagreed dollars | LLMs disagree (coef.) | human vs LLM majority (coef.) |
|---|---|---|---|---|
| scholarship / tuition / aid | 17% | 4% | -0.50 | -0.23 |
| ‘research’ | 9% | 3% | -0.25 | -0.15 |
| log rows covered | – | – | -0.15 | -0.13 |
| education / community / arts | 14% | 9% | -0.14 | -0.08 |
| exempt-purpose boilerplate | 1% | 0% | -0.12 | -0.43 |
| log dollars | – | – | +0.12 | -0.01 |
| reader 1 | reader 2 | reader 3 | majority of three | |
|---|---|---|---|---|
| calibrator log-loss, texts | 0.397 | 0.213 | 0.223 | 0.243 |
| calibrator AUC, texts | 0.784 | 0.957 | 0.938 | 0.946 |
| calibrator log-loss, dollars | 0.098 | 0.152 | 0.125 | 0.079 |
| classic total ($B) | 11.29 [10.3, 12.2] | 10.06 [9.1, 11.1] | 13.99 [13.0, 15.0] | 11.55 [10.7, 12.4] |
| dw-PPI total, ($B) | 11.29 [10.4, 12.1] | 10.12 [9.1, 11.1] | 14.00 [13.0, 15.0] | 11.56 [10.8, 12.3] |
| ; ESS gain | 0.80; 1.18 | 0.65; 1.02 | 0.60; 1.01 | 0.85; 1.23 |
| Policy | |||||
|---|---|---|---|---|---|
| dollar-PPS, no model (classic) | 3.30 (0.92) | 1.98 (0.98) | 1.67 (0.95) | 1.34 (0.97) | 0.80 (0.97) |
| phase 2 / later batches by active mixed 50/50 with dollar-PPS | |||||
| two-phase, phase 1 = 25% by dollar-PPS | 2.72 (0.92) | 1.43 (0.95) | 1.04 (0.93) | 0.83 (0.94) | 0.48 (0.94) |
| two-phase, phase 1 = 50% by dollar-PPS | 2.67 (0.95) | 1.59 (0.93) | 1.11 (0.96) | 0.87 (0.95) | 0.55 (0.92) |
| two-phase, phase 1 = 75% by dollar-PPS | 3.73 (0.95) | 2.02 (0.95) | 1.48 (0.94) | 1.09 (0.94) | 0.68 (0.94) |
| batch-sequential, 4 batches, refit each, last batch | 3.60 (0.92) | 2.08 (0.94) | 1.42 (0.93) | 1.02 (0.93) | 0.61 (0.93) |
| Model | readings compared | weighting | readings | within-text share of variance [95% CI] | mean research share |
|---|---|---|---|---|---|
| Gemini 2.5 Flash | wrappers | texts | 5 | 2.7% [1.2%, 6.1%] | 21% |
| Gemini 2.5 Flash | repeat run | texts | 2 | 2.7% [0.6%, 7.3%] | 21% |
| Gemini 2.5 Flash | wrappers | dollars | 5 | 2.5% [1.5%, 3.7%] | 14% |
| Gemini 2.5 Flash | repeat run | dollars | 2 | 1.2% [0.5%, 2.2%] | 14% |
| GPT-5 mini | wrappers | texts | 3 | 0.7% [0.3%, 2.0%] | 22% |
| GPT-5 mini | repeat run | texts | 2 | 1.4% [0.5%, 3.4%] | 22% |
| Institution | domain $B | sampled texts | rules plug-in | calibrated plug-in | direct dw-PPI (design s.e.) | composite (working-model s.e.) | |
|---|---|---|---|---|---|---|---|
| Johns Hopkins University | 5.41 | 138 | 0.41 | 0.42 | 0.58 (0.10) | 0.53 (0.09) | 0.72 |
| Stanford University | 3.87 | 140 | 0.65 | 0.58 | 0.47 (0.04) | 0.49 (0.04) | 0.88 |
| Harvard University | 3.25 | 146 | 0.67 | 0.40 | 0.73 (0.14) | 0.59 (0.10) | 0.57 |
| University of California-San Francisco | 2.47 | 117 | 0.64 | 0.59 | 0.62 (0.08) | 0.61 (0.06) | 0.54 |
| Columbia University in the City of New York | 1.65 | 128 | 0.30 | 0.25 | 0.25 (0.01) | 0.25 (0.01) | 0.91 |
| University of California-Berkeley | 1.62 | 107 | 0.26 | 0.23 | 0.38 (0.12) | 0.27 (0.06) | 0.24 |
| Research dollars | PPS texts | rules | calibrated | direct | composite | cov. direct | cov. composite | ||
|---|---|---|---|---|---|---|---|---|---|
| A: Bernoulli(calibrated p) | |||||||||
| 1B | 29 | 52.0 | 30% | 11% | 25% | 10% | 0.98 / 0.98 | 0.70 / 0.70 | 0.34 |
| 100M | 91 | 25.9 | 35% | 21% | 70% | 16% | 0.80 / 0.80 | 0.58 / 0.57 | 0.46 |
| 10M | 219 | 9.1 | 45% | 34% | 188% | 34% | 0.40 / 0.47 | 0.27 / 0.33 | 0.66 |
| under $1M | 1049 | 3.9 | 100% | 82% | 519% | 82% | 0.26 / 0.42 | 0.21 / 0.36 | 0.81 |
| B: Sonnet’s label is the truth | |||||||||
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Design and prediction construction | bias | ||
|---|---|---|---|
| one of two (fixed size) | |||
| fixed predictor f = (1/2, 1/2) | 1.0 | 1.0000 | +0.0000 |
| fixed predictor f = (1/2, 1/2) | 0.5 | 1.0000 | +0.0000 |
| frame-fold predictor (every unit) | 1.0 | 1.5000 | +0.5000 |
| frame-fold predictor (every unit) | 0.5 | 1.2500 | +0.2500 |
| sampled/unsampled switch | 1.0 | 1.2500 | +0.2500 |
| , dollar measure | unweighted share | |
|---|---|---|
| 0.01 | 0.001 | 0.018 |
| 0.02 | 0.015 | 0.051 |
| 0.05 | 0.034 | 0.103 |
| 0.10 | 0.071 | 0.140 |
| 0.20 | 0.192 | 0.234 |
| 0.30 | 0.291 | 0.308 |
| Model | 3-class log-loss | accuracy (texts) | accuracy (dollars) | Brier for U | AUC for U |
|---|---|---|---|---|---|
| Panel calibrator | 0.971 | 58% | 86% | 0.202 | 0.738 |
| Panel + text calibrator | 0.908 | 68% | 88% | 0.178 | 0.695 |
| Gradient boosting | 1.018 | 65% | 87% | 0.196 | 0.661 |
| Text features only | 0.944 | 49% | 85% | 0.195 | 0.628 |
| DS posterior | 4.031 | 53% | 81% | 0.425 | 0.614 |
| DS-mix posterior | 3.259 | 54% | 81% | 0.401 | 0.567 |
| Vote-pattern stratum | texts | dollars ($B) | expected research ($B) | expected unclear ($B) | mean |
|---|---|---|---|---|---|
| conflict | 628 | 0.3 | 0.0 | 0.0 | 0.13 |
| lean_N | 17,411 | 7.4 | 0.1 | 0.9 | 0.03 |
| lean_R | 6,108 | 3.3 | 1.1 | 1.0 | 0.41 |
| unanimous_N | 54,877 | 65.4 | 0.1 | 2.1 | 0.00 |
| unanimous_R | 16,092 | 11.9 | 8.9 | 1.7 | 0.70 |
| unanimous_U | 25,954 | 13.8 | 0.9 | 5.1 | 0.09 |
| Sensitivity Specificity | 0.90 | 0.95 | 0.98 | 1.00 |
|---|---|---|---|---|
| 0.85 | 1.43 | 7.73 | 11.14 | 13.28 |
| 0.90 | 1.34 | 7.27 | 10.51 | 12.55 |
| 0.95 | 1.26 | 6.87 | 9.94 | 11.89 |
| 1.00 | 1.19 | 6.51 | 9.44 | 11.29 |
| Policy | texts read | words read | RMSE ($B) |
|---|---|---|---|
| word budget 705 (what dollar-PPS spends on 100 texts) | |||
| dollar-PPS | 100 | 709 | 1.36 |
| dollar-PPS, cost-adjusted (m / sqrt c) | 151 | 704 | 1.28 |
| active mixed 50/50 with dollar-PPS | 92 | 704 | 1.02 |
| active mixed, cost-adjusted (m sqrt(v / c)) | 142 | 705 | 0.92 |
| word budget 1,504 (what dollar-PPS spends on 200 texts) | |||
| Question | total-only allocation | minimax over questions | ||||
|---|---|---|---|---|---|---|
| RMSE ($B) | relative | variance ratio | RMSE ($B) | relative | variance ratio | |
| total | 0.75 | 6.5% | 1.00 | 0.81 | 6.9% | 1.16 |
| 2020 | 0.25 | 11.7% | 7.40 | 0.15 | 7.2% | 4.20 |
| 2021 | 0.24 | 10.5% | 3.90 | 0.25 | 11.1% | 4.20 |
| 2022 | 0.20 | 8.8% | 3.63 | 0.24 | 10.5% | 4.20 |
| Model | comparison | same status | texts |
|---|---|---|---|
| claude-fable-5-1 | codebook v1.0 vs v1.1 (same prompt) | 93.7% | 300 |
| claude-fable-5-1 | repeat run, same prompt and codebook | 96.3% | 300 |
| gpt-5 | codebook v1.0 vs v1.1 (same prompt) | 95.0% | 300 |
| gpt-5 | repeat run, same prompt and codebook | 96.0% | 300 |
| gemini-2.5-pro | codebook v1.0 vs v1.1 (same prompt) | 90.0% | 300 |
| Tax year | Rules | Claude Sonnet 5.5 | Gemini 2.5 Flash | GPT-5 mini | Calibrated panel (OOF) |
|---|---|---|---|---|---|
| 2020 | +0.2 | +1.1 | +2.3 | +3.0 | -1.1 |
| 2021 | +1.5 | +1.2 | +2.3 | +3.1 | -0.4 |
| 2022 | +1.5 | +1.9 | +2.9 | +3.8 | +0.1 |
| 2023 | +2.7 | +2.0 | +3.8 | +4.7 | +0.6 |
| 2024 | +1.1 | +0.2 | +1.2 | +2.7 | -0.1 |
| Stratum | 1 vs 2 | 1 vs 3 | 2 vs 3 | all three | all three, same status | |
|---|---|---|---|---|---|---|
| unanimous_R | 136 | 59% | 76% | 62% | 49% | 43% |
| lean_R | 93 | 74% | 71% | 71% | 58% | 27% |
| conflict | 32 | 91% | 88% | 84% | 81% | 34% |
| lean_N | 92 | 95% | 96% | 92% | 91% | 53% |
| unanimous_U | 137 | 91% | 86% | 82% | 80% | 34% |
| unanimous_N | 161 | 99% | 100% | 99% | 99% | 84% |
| Machine reader | reader 1 | reader 2 | reader 3 | majority | unanimous | ||||
|---|---|---|---|---|---|---|---|---|---|
| texts | dollars | texts | dollars | texts | dollars | texts | dollars | texts | |
| Rules | 82% | 94% | 78% | 92% | 84% | 95% | 83% | 95% | 90% |
| Claude Sonnet 5.5 | 85% | 95% | 82% | 93% | 86% | 95% | 87% | 96% | 93% |
| Gemini 2.5 Flash | 81% | 95% | 81% | 93% | 84% | 95% | 84% | 96% | 91% |
| GPT-5 mini | 81% | 94% | 79% | 92% | 84% | 95% | 84% | 95% | 89% |
| LLM majority (2 of 3) | 84% | 95% | 83% | 93% | 87% | 96% | 87% | 96% | 94% |
Selective Prediction
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.
See all 318 papers