Human Responses to Algorithmic Correction: Evidence from Centre Court
Authors: David Almog, Romain Gauriot, Lionel Page, Daniel Martin
Organizations: Kellogg School of Management, Northwestern University · Deakin University · University of Queensland · University of California, Santa Barbara
We study how algorithmic review changes the initial decisions that a system is designed to correct. Using Hawk-Eye data from professional tennis, we compare umpire decisions with and without review against a common accuracy benchmark. Under review, initial errors were 8.6 percentage points higher just outside the line and lower for comparably close balls inside. We find no evidence that average initial accuracy deteriorated. Within matches, reversals were followed by larger adjustments than upheld calls, consistent with adaptation to correction. Overall, correction offsets worse initial decisions for incorrect in calls and reinforces better initial decisions for incorrect out calls. Whether review lowers organizational costs depends jointly on these changes and the costs of remedying corrected errors. A conditional illustration shows how utilization spreads fixed review costs across calls.
Figures & tables
Before Hawk-Eye review
After Hawk-Eye review
Total
Post - Pre
(PostHK=0)
(PostHK=1)
[Std. diff.]
A. Sample counts
Players
71
161
174
Tournament editions
7
28
35
Matches
109
589
698
Points
15,435
83,902
99,337
Table 1: Summary statistics for the consolidated data set. The last column reports the difference in the sample characteristics before and after Hawk-Eye review. Bracketed values are absolute standardized differences.
Figure 1: The figure plots the probability that a ball is initially called “in” against its signed distance from the line. Negative distances indicate balls landing out and positive distances balls landing in. Each marker reports the mean call-in rate for observations within 10 mm of the indicated distance. For example, the marker at zero includes balls landing between -10 and 10 mm from the line. Bars denote 95% confidence intervals. Curves are locally weighted scatterplot smoothing (LOWESS) fits, estimated separately before and after the introduction of Hawk-Eye review. The vertical dashed line denotes the court boundary.
Figure 2: Estimated change in the incorrect-call rate after the introduction of Hawk-Eye review, by proximity to the line. Each dot is the coefficient on the interaction between a distance-bin indicator and PostHK , which equals 1 when Hawk-Eye review is active. The regressions include point and match controls. Standard errors are clustered at the tournament-edition level.
Figure 3: The figure reports coefficients from a linear probability model estimated on post-Hawk-Eye calls for balls landing within 20 mm of the line. The dependent variable equals one if the ball is initially called in. Panel (a) reports responses to prior successful challenges that reversed the initial call. Panel (b) reports responses to prior unsuccessful challenges that upheld the initial call. The four cumulative feedback measures are included jointly in the specification. For each series, the omitted category is zero prior events of that type within the match. Counts are capped at three, with the final category pooling three or more events. The specification includes match fixed effects and point-level controls. Standard errors are clustered at the match level. Bars denote 95% confidence intervals.
Figure 4: Estimated change in the incorrect-call rate after the introduction of Hawk-Eye review, by proximity to the line and tournament stage. Matches at the final, semifinal, and quarterfinal stages are analyzed separately from earlier-stage matches. Each dot represents the coefficient on the interaction between a distance-bin indicator and PostHK .
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
Incorrect call (Serves)
Incorrect call (Non-serves)
PostHK
-0.006
-0.002
0.000
0.000
-0.024 ∗∗
-0.021 ∗∗
-0.023 ∗∗
-0.023 ∗
(0.010)
(0.009)
(0.009)
(0.009)
(0.010)
(0.010)
(0.010)
(0.012)
Point controls
X
X
X
X
X
X
Match controls
X
X
X
X
Cluster level
Tournament
Tournament
Table 2: OLS regressions of umpire mistakes for balls bouncing within 100 mm of the line separately for serves and non-serves. PostHK=1 if Hawk-Eye review is active. Point controls include distance fixed effects (by 20 mm bins), speed (whether it is below or above the median), score, game, set, and an indicator if the point played is in the tie-break stage. Match controls include round and tournament tier.
Table 3: OLS regressions of umpire mistakes (separately for serves and non-serves). PostHK=1 if the Hawk-Eye review system is active. Point controls include distance fixed effects (by 20 mm bins), speed (whether it is below or above the median), score, game, set, and an indicator if the point played is in the tie-break stage. Match controls include round and tournament tier.
Quantity
Incorrect in
Incorrect out
Total
Without review: initial errors, Ej0
5.61
8.28
13.88
With review: initial errors, Ej1
6.19
6.27
12.47
With review: corrected errors, Cj
2.09
1.35
3.45
With review: remaining errors, Ej1−Cj
4.10
4.92
9.02
Reduction in remaining errors, Ej0−Ej1+Cj
1.50
3.36
4.86
Initial mistakes corrected, Cj/Ej1 (%)
33.8
21.6
27.6
Table 4: Error accounting for calls within 100 mm of the line.
Illustrative utilization
Minimum w if k=0
Minimum w if k=w
PlayReplay: 200 matches per court-year
$13.16
$45.20
PlayReplay: 1,000 matches per court-year
$2.63
$9.04
Historical Hawk-Eye: 25 matches per court-week
1,316–1,645
4,520–5,650
Table 5: Error-loss thresholds for covering reported system charges.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Iteration
Correct Merge
False Positives
Efficacy
Challenges Left
1
98
2
98%
44
2
25
2
93%
17
3
5
0
100%
12
4
1
0
100%
11
5
0
0
−
11
6
0
0
−
11
Appendix
Table A.1 : Performance for each iteration of the merging algorithm. Out of the 144 challenges audited, only one remained unmerged.
Tournament
Start Month
Category
Court Type
Number of Matches
2005
2006
2007
2008
Marseille
Feb
International (250)
Hard
25
Rotterdam
Feb
International Gold (500)
Hard
15
26
Dubai
Mar
International Gold (500)
Hard
19
Las Vegas
Mar
International (250)
Hard
19
Indian Wells
Mar
Masters (1000)
Hard
17
27
Appendix
Table C.1 : Information on the 35 tournament editions in the consolidated data set. The numbers in each cell indicate the matches available in our data set. The color represents the review regime, with blue indicating tournament editions without Hawk-Eye review and red indicating tournament editions with Hawk-Eye review.
Estimate
Clustered SE
Wild-bootstrap p
95% CI
Closest outside bin [−20,0)
0.086
0.034
0.052
[−0.001,0.163]
First inside bin [0,20)
-0.041
0.034
0.256
[−0.114,0.040]
Second inside bin [20,40)
-0.077
0.027
0.028
[−0.136,−0.013]
Cross-boundary contrasts
Closest outside − first inside
0.127
0.060
0.076
[−0.019,0.259]
Closest outside − second inside
0.163
0.049
0.018
[0.048,0.270]
Appendix
Table C.2: Wild-Cluster-Bootstrap Inference for the Main Distance-Bin Estimates. The first three rows report estimated changes in the incorrect-call rate after the introduction of Hawk-Eye review for the indicated 20 mm distance bins. The remaining rows report contrasts between these estimates. Standard errors are clustered at the tournament-edition level. Wild-cluster-bootstrap inference is based on 9,999 replications clustered at the tournament-edition level. For the final row, the reported estimate compares the closest outside bin with the average of the first two inside bins. The corresponding wild-bootstrap test is implemented as 2β[−20,0)−β[0,20)−β[20,40)=0 . Its confidence set is divided by two so that it is expressed in the same units as the reported contrast.
Cross-boundary
Clustered
Wild
Tournament
Specification
contrast
SE
p
editions
Panel A. Same-event specification
Full sample
0.145
0.052
0.034
35
Same events + event fixed effects
0.102
0.063
0.152
17
Panel B. Leave-one-event-out estimates
Omitted event
Contrast
Clustered SE
95% CI
Appendix
Table C.3: Same-Event and Leave-One-Event-Out Robustness. The cross-boundary contrast is defined as the estimated change in the incorrect-call rate for the closest outside bin, [−20,0) , minus the average estimated change for the first two inside bins, [0,20) and [20,40) . The same-event specification restricts the sample to named events observed under both review regimes and includes named-event fixed effects. Toronto and Montreal are treated as a single event, the Canada Open, because the tournament alternates between the two host cities. Standard errors are clustered at the tournament-edition level. Wild-cluster-bootstrap inference is based on 9,999 replications clustered at the tournament-edition level. Panel B re-estimates the same-event specification after omitting each named event in turn. The leave-one-event-out estimates are reported to assess sensitivity to individual events rather than for separate hypothesis testing.
Specification
Cross-boundary contrast
Clustered SE
95% CI
Baseline
0.145
0.052
[0.040,0.251]
+ Serve/non-serve control
0.144
0.052
[0.038,0.250]
+ Flexible speed controls
0.144
0.052
[0.039,0.250]
+ Hitter and receiver fixed effects
0.136
0.053
[0.028,0.244]
Appendix
Table C.4: Robustness to Shot and Player Composition. The cross-boundary contrast is defined as the estimated change in the incorrect-call rate for the closest outside bin, [−20,0) , minus the average estimated change for the first two inside bins, [0,20) and [20,40) . The baseline specification includes 20 mm distance-bin fixed effects, an above-median-speed indicator, score, game, set, tie-break status, round, and tournament tier. Controls are added sequentially across rows: the second row adds serve status, the third replaces the above-median-speed indicator with flexible controls for ball speed, and the fourth additionally includes hitter and receiver player fixed effects. Standard errors are clustered at the tournament-edition level.
Specification
Asymmetry
Clustered SE
p -value
Panel A. Flexible effects of prior reversals
One prior reversal
0.125
0.061
0.040
Two prior reversals
0.178
0.102
0.081
Three or more prior reversals
0.156
0.158
0.324
Joint symmetry test
—
—
0.151
Panel B. Linear cumulative-reversal specification
Appendix
Table C.5: Asymmetry in Responses to Prior Reversals. The table tests whether the behavioral response following an out-to-in reversal is larger in absolute magnitude than the response following an in-to-out reversal. The asymmetry estimate is the sum of the coefficients on the corresponding out-to-in and in-to-out reversal variables. A positive estimate therefore indicates a larger response following an out-to-in reversal. Panel A allows separate effects after one, two, and three-or-more prior reversals. The joint symmetry test evaluates the null that the three corresponding asymmetry estimates are jointly equal to zero ( F(3,567)=1.78 ). Panel B enters the cumulative number of reversals of each type linearly and reports the difference in response magnitude associated with one additional prior reversal. Standard errors are clustered at the match level.
Comparison
Estimate
Clustered SE
p -value
IN → OUT reversal minus upheld IN
-0.067
0.038
0.078
OUT → IN reversal minus upheld OUT
0.115
0.040
0.004
Joint equality with upheld challenges
—
—
0.004
Difference in absolute incremental effects
0.049
0.053
0.363
Appendix
Table C.6: The table compares estimated associations of subsequent calls with prior reversals and with unsuccessful challenges involving the same initial-call direction. A negative estimate in the first row indicates an additional shift toward calling balls out following an IN-to-OUT reversal. A positive estimate in the second row indicates an additional shift toward calling balls in following an OUT-to-IN reversal. All four cumulative feedback variables are included simultaneously in the same within-match specification used in Section 3.3. Standard errors are clustered at the match level. The joint test evaluates whether the two reversal coefficients equal their corresponding upheld-challenge coefficients. The final row tests whether the incremental associations are equal in absolute magnitude.
(1)
(2)
(3)
(4)
Incorrect call
PostHK
-0.014 ∗∗
-0.011
-0.011
-0.011
(0.007)
(0.007)
(0.007)
(0.008)
Point controls
X
X
X
Match controls
X
X
Cluster level
Tournament
Appendix
Table C.7: OLS regressions of umpire mistakes for balls bouncing within 100 mm of the line. PostHK=1 if Hawk-Eye review is active. Point controls include 20 mm distance-bin fixed effects, an above-median-speed indicator, score, game, set, and a tie-break indicator. Match controls include round and tournament tier.
Figure C.1 : The location of the umpiring crew: S = the serve-line umpire, RB = the right baseline umpire, RN = the right near long-line umpire, RC = the right center-line umpire, RF = the right far-side long-line umpire. The left side of the court has corresponding line umpires, except for the serve-line umpire, who moves to the left side when the right side player has the serve.
Figure C.2 : Call-in rates before and after the introduction of Hawk-Eye review for balls landing within 20 mm of the line, regardless of which side of the line they actually bounced on. The solid lines report fitted values from a linear probability model in which the probability of calling a ball in is allowed to vary linearly with month, separately before and after Hawk-Eye review. The dashed lines indicate 95% confidence intervals. The regression controls for game, set, point, tiebreak, round, tournament tier, and shot speed. Standard errors are clustered at the match level.
Figure C.3 : Empirical distributions of signed distance from the line and ball speed before and after Hawk-Eye review.
Figure C.4 : Subperiod analysis of changes in the incorrect-call rate after the introduction of Hawk-Eye review, by proximity to the line. Matches with review are grouped into tournaments from 2006 through the first half of 2007 and tournaments from the second half of 2007 through 2008. Each dot represents the coefficient on the interaction between a distance-bin indicator and PostHK .
Figure C.5 : An analog of Figure 2, using a one-treatment-at-a-time approach. We estimate the effect for each distance bin separately to avoid contamination across bins.
Figure C.6 : Estimated change in umpire performance after the introduction of Hawk-Eye review, across 20 mm distance bins and by tournament category. Matches are grouped into higher-tier tournaments (Masters 1000) and lower-tier tournaments (ATP 500 and 250). Each dot represents the coefficient on the interaction between a distance-bin indicator and PostHK , estimated separately by tournament category.
Figure C.7 : Speed distribution for all the balls that bounced within 100 mm of the line, separated by non-serves and serves.
Figure C.8 : Incorrect call rates by proximity to the line (separately for serves and non-serves). Each dot is the rate of incorrect calls for a bin of 20 mm, and dots to the left of the dashed line represent bins out of bounds, and those to the right of the dashed line represent bins in bounds.
Actual location
Initial call
Without review
With review
Corrected with review
In
In
1,216
6,664
0
In
Out
217
890
192
Out
In
147
879
297
Out
Out
1,042
5,757
0
Total
2,622
14,190
489
Appendix
Table D.1: Initial calls by actual location and subsequent correction.
Quantity and conditioning
Without review
With review
pr=Pr(ball in∣initial call in)
0.89214967
0.88346812
qr=Pr(ball in∣initial call out)
0.17235902
0.13389499
mr=Pr(ball in)
0.54652937
0.53234672
λr=Pr(initial call in)
0.51983219
0.53157153
ηI=Pr(corrected∣ball in, initial call out)
0
0.21573034
ηO=Pr(corrected∣ball out, initial call in)
0
0.33788396
Appendix
Table D.2: Pooled probabilities implied by Table D.1 .
Automated checking pipelines increasingly place one language model as the checker and another (or the same one) as the fixer. We ask whether that wiring changes what the checker reports. Measuring false alarms on human-verified-correct ProcessBench traces with the present task held byte-identical, we find that a completed audit -> repair episode already in the model's context lowers false alarms in 15 of 15 model x wording combinations, by 2.8 to 11.5 percentage points against a length-matched non-audit control, a 9 to 25% reduction relative to that control. The direction contradicts what the accumulated-message literature predicts: an episode whose audit reported an error lowers false alarms further still, at all five wordings on the model where that manipulation lands cleanly, though a negativity asymmetry predicts more flagging. Decomposing the episode finds repair content and audit verdict complementary: different components carry the effect on different model families. Signal-detection analysis locates the change in the threshold rather than in discrimination -- the criterion moves in 15 of 15 combinations and survives correction in 13 while d' survives in none, though the d' test is half as sensitive by construction -- and a hand audit of 50 false alarms finds 82% simply wrong, so at this operating point the shift need not be harmful. With reasoning enabled the effect keeps its relative size on both models tested, and the threshold reading holds there too.
Parsa Mazaheri, Kasra Mazaheri
University of California, Santa Cruz · Massachusetts Institute of Technology
How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation within 1pp in 2018 and 2021-2025. Longitudinally, we find no robust time trend in reviewer noise over 2017-2025. The high accepted-paper flip rates of 2020 and 2021 have distinct mechanisms: the 2020 four-point scale compressed scores (23.7% of papers had zero within-paper variance), and a counterfactual shows coarsening the scale raises disagreement by about 7pp; 2021 instead combined the lowest signal-to-noise ratio in the sample with the most threshold-crowded acceptances. For the LLM era, a 2023 breakpoint test on within-paper score variance finds no break, but the design has almost no power, and no post-2022 review text or confidence data exist, so no LLM attribution is attempted.
The Duckworth-Lewis-Stern (DLS) method has been the international standard for revising target scores in rain-interrupted limited-overs cricket since 1999. Despite over two decades of operational use, no large-scale empirical audit of its prediction bias has been published. We conduct such an audit on 8,150 international matches (3,095 ODIs, 5,055 T20Is) from Cricsheet, generating 233,550 synthetic interruption scenarios with temporal splits. We document two structured biases. First, DLS prediction error spans a 137-run range across (overs-remaining, wickets-lost) match-state buckets. Second, DLS exhibits a gender-differential bias on ODIs that has not previously been quantified: on the training split, mean over-prediction is +1.51 runs for men but +7.63 runs for women, a gap of +6.13 runs (F = 195.16, p < 10^-43). We benchmark DLS against five modern alternatives: Bi-LSTM, XGBoost, an enriched XGBoost variant, a deep context-aware model, and a stacking ensemble, and propose DLS-Cal, a lightweight interpretable calibration layer (27K parameters) outputting a state-conditioned correction added to DLS. DLS-Cal reduces absolute bias by 31% on ODI and 19% on T20I, and a gender-aware variant reduces women's ODI residual bias from +6.19 to +0.65 runs while leaving men's calibration unchanged. We release code, models, and data.