Human Responses to Algorithmic Correction: Evidence from Centre Court

David Almog; Romain Gauriot; Lionel Page; Daniel Martin

cs.LGJan 30, 2024

Human Responses to Algorithmic Correction: Evidence from Centre Court

Authors: David Almog, Romain Gauriot, Lionel Page, Daniel Martin

Organizations: Kellogg School of Management, Northwestern University · Deakin University · University of Queensland · University of California, Santa Barbara

Abstract

We study how algorithmic review changes the initial decisions that a system is designed to correct. Using Hawk-Eye data from professional tennis, we compare umpire decisions with and without review against a common accuracy benchmark. Under review, initial errors were 8.6 percentage points higher just outside the line and lower for comparably close balls inside. We find no evidence that average initial accuracy deteriorated. Within matches, reversals were followed by larger adjustments than upheld calls, consistent with adaptation to correction. Overall, correction offsets worse initial decisions for incorrect in calls and reinforces better initial decisions for incorrect out calls. Whether review lowers organizational costs depends jointly on these changes and the costs of remedying corrected errors. A conditional illustration shows how utilization spreads fixed review costs across calls.

Figures & tables

A. Sample counts
	Before Hawk-Eye review	After Hawk-Eye review	Total	Post - Pre
	(PostHK=0)	(PostHK=1)		[Std. diff.]
Players	71	161	174
Tournament editions	7	28	35
Matches	109	589	698
Points	15,435	83,902	99,337

Table 1: Summary statistics for the consolidated data set. The last column reports the difference in the sample characteristics before and after Hawk-Eye review. Bracketed values are absolute standardized differences.

Figure 1: The figure plots the probability that a ball is initially called “in” against its signed distance from the line. Negative distances indicate balls landing out and positive distances balls landing in. Each marker reports the mean call-in rate for observations within 10 mm of the indicated distance. For example, the marker at zero includes balls landing between -10 and 10 mm from the line. Bars denote 95% confidence intervals. Curves are locally weighted scatterplot smoothing (LOWESS) fits, estimated separately before and after the introduction of Hawk-Eye review. The vertical dashed line denotes the court boundary.

Figure 2: Estimated change in the incorrect-call rate after the introduction of Hawk-Eye review, by proximity to the line. Each dot is the coefficient on the interaction between a distance-bin indicator and $PostHK$ , which equals 1 when Hawk-Eye review is active. The regressions include point and match controls. Standard errors are clustered at the tournament-edition level.

Panel of Figure 3 — Figure 3: The figure reports coefficients from a linear probability model estimated on post-Hawk-Eye calls for balls landing within 20 mm of the line. The dependent variable equals one if the ball is initially called in. Panel (a) reports responses to prior successful challenges that reversed the initial call. Panel (b) reports responses to prior unsuccessful challenges that upheld the initial call. The four cumulative feedback measures are included jointly in the specification. For each series, the omitted category is zero prior events of that type within the match. Counts are capped at three, with the final category pooling three or more events. The specification includes match fixed effects and point-level controls. Standard errors are clustered at the match level. Bars denote 95% confidence intervals.

Figure 4: Estimated change in the incorrect-call rate after the introduction of Hawk-Eye review, by proximity to the line and tournament stage. Matches at the final, semifinal, and quarterfinal stages are analyzed separately from earlier-stage matches. Each dot represents the coefficient on the interaction between a distance-bin indicator and $PostHK$ .

	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)
	Incorrect call (Serves)				Incorrect call (Non-serves)
PostHK	-0.006	-0.002	0.000	0.000	-0.024 ∗∗	-0.021 ∗∗	-0.023 ∗∗	-0.023 ∗
	(0.010)	(0.009)	(0.009)	(0.009)	(0.010)	(0.010)	(0.010)	(0.012)
Point controls		X	X	X		X	X	X
Match controls			X	X			X	X
Cluster level				Tournament				Tournament

Table 2: OLS regressions of umpire mistakes for balls bouncing within 100 mm of the line separately for serves and non-serves.

PostHK=1

if Hawk-Eye review is active. Point controls include distance fixed effects (by 20 mm bins), speed (whether it is below or above the median), score, game, set, and an indicator if the point played is in the tie-break stage. Match controls include round and tournament tier.

Table 3: OLS regressions of umpire mistakes (separately for serves and non-serves).

PostHK=1

if the Hawk-Eye review system is active. Point controls include distance fixed effects (by 20 mm bins), speed (whether it is below or above the median), score, game, set, and an indicator if the point played is in the tie-break stage. Match controls include round and tournament tier.

Quantity	Incorrect in	Incorrect out	Total
Without review: initial errors, $E_{j}^{0}$	5.61	8.28	13.88
With review: initial errors, $E_{j}^{1}$	6.19	6.27	12.47
With review: corrected errors, $C_{j}$	2.09	1.35	3.45
With review: remaining errors, $E_{j}^{1}-C_{j}$	4.10	4.92	9.02
Reduction in remaining errors, $E_{j}^{0}-E_{j}^{1}+C_{j}$	1.50	3.36	4.86
Initial mistakes corrected, $C_{j}/E_{j}^{1}$ (%)	33.8	21.6	27.6

Table 4: Error accounting for calls within 100 mm of the line.

Illustrative utilization	Minimum $w$ if $k=0$	Minimum $w$ if $k=w$
PlayReplay: 200 matches per court-year	$13.16	$45.20
PlayReplay: 1,000 matches per court-year	$2.63	$9.04
Historical Hawk-Eye: 25 matches per court-week	$1,316–$ 1,645	$4,520–$ 5,650

Table 5: Error-loss thresholds for covering reported system charges.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Iteration	Correct Merge	False Positives	Efficacy	Challenges Left
1	98	2	$98\%$	44
2	25	2	$93\%$	17
3	5	0	$100\%$	12
4	1	0	$100\%$	11
5	0	0	$-$	11
6	0	0	$-$	11

Appendix

Table A.1 : Performance for each iteration of the merging algorithm. Out of the 144 challenges audited, only one remained unmerged.

Tournament	Start Month	Category	Court Type	Number of Matches
				2005	2006	2007	2008
Marseille	Feb	International (250)	Hard				25
Rotterdam	Feb	International Gold (500)	Hard			15	26
Dubai	Mar	International Gold (500)	Hard				19
Las Vegas	Mar	International (250)	Hard				19
Indian Wells	Mar	Masters (1000)	Hard	17		27

Appendix

Table C.1 : Information on the 35 tournament editions in the consolidated data set. The numbers in each cell indicate the matches available in our data set. The color represents the review regime, with blue indicating tournament editions without Hawk-Eye review and red indicating tournament editions with Hawk-Eye review.

	Estimate	Clustered SE	Wild-bootstrap $p$	95% CI
Closest outside bin $[-20,0)$	0.086	0.034	0.052	$[-0.001,\ 0.163]$
First inside bin $[0,20)$	-0.041	0.034	0.256	$[-0.114,\ 0.040]$
Second inside bin $[20,40)$	-0.077	0.027	0.028	$[-0.136,\ -0.013]$
Cross-boundary contrasts
Closest outside $-$ first inside	0.127	0.060	0.076	$[-0.019,\ 0.259]$
Closest outside $-$ second inside	0.163	0.049	0.018	$[0.048,\ 0.270]$

Appendix

Table C.2: Wild-Cluster-Bootstrap Inference for the Main Distance-Bin Estimates. The first three rows report estimated changes in the incorrect-call rate after the introduction of Hawk-Eye review for the indicated 20 mm distance bins. The remaining rows report contrasts between these estimates. Standard errors are clustered at the tournament-edition level. Wild-cluster-bootstrap inference is based on 9,999 replications clustered at the tournament-edition level. For the final row, the reported estimate compares the closest outside bin with the average of the first two inside bins. The corresponding wild-bootstrap test is implemented as

2\beta_{[-20,0)}-\beta_{[0,20)}-\beta_{[20,40)}=0

. Its confidence set is divided by two so that it is expressed in the same units as the reported contrast.

	Cross-boundary	Clustered	Wild	Tournament
Specification	contrast	SE	$p$	editions
Panel A. Same-event specification
Full sample	0.145	0.052	0.034	35
Same events + event fixed effects	0.102	0.063	0.152	17
Panel B. Leave-one-event-out estimates
Omitted event	Contrast	Clustered SE	95% CI

Appendix

Table C.3: Same-Event and Leave-One-Event-Out Robustness. The cross-boundary contrast is defined as the estimated change in the incorrect-call rate for the closest outside bin,

[-20,0)

, minus the average estimated change for the first two inside bins,

[0,20)

and

[20,40)

. The same-event specification restricts the sample to named events observed under both review regimes and includes named-event fixed effects. Toronto and Montreal are treated as a single event, the Canada Open, because the tournament alternates between the two host cities. Standard errors are clustered at the tournament-edition level. Wild-cluster-bootstrap inference is based on 9,999 replications clustered at the tournament-edition level. Panel B re-estimates the same-event specification after omitting each named event in turn. The leave-one-event-out estimates are reported to assess sensitivity to individual events rather than for separate hypothesis testing.

Specification	Cross-boundary contrast	Clustered SE	95% CI
Baseline	0.145	0.052	$[0.040,\ 0.251]$
+ Serve/non-serve control	0.144	0.052	$[0.038,\ 0.250]$
+ Flexible speed controls	0.144	0.052	$[0.039,\ 0.250]$
+ Hitter and receiver fixed effects	0.136	0.053	$[0.028,\ 0.244]$

Appendix

Table C.4: Robustness to Shot and Player Composition. The cross-boundary contrast is defined as the estimated change in the incorrect-call rate for the closest outside bin,

[-20,0)

, minus the average estimated change for the first two inside bins,

[0,20)

and

[20,40)

. The baseline specification includes 20 mm distance-bin fixed effects, an above-median-speed indicator, score, game, set, tie-break status, round, and tournament tier. Controls are added sequentially across rows: the second row adds serve status, the third replaces the above-median-speed indicator with flexible controls for ball speed, and the fourth additionally includes hitter and receiver player fixed effects. Standard errors are clustered at the tournament-edition level.

Panel A. Flexible effects of prior reversals
Specification	Asymmetry	Clustered SE	$p$ -value
One prior reversal	0.125	0.061	0.040
Two prior reversals	0.178	0.102	0.081
Three or more prior reversals	0.156	0.158	0.324
Joint symmetry test	—	—	0.151
Panel B. Linear cumulative-reversal specification

Appendix

Table C.5: Asymmetry in Responses to Prior Reversals. The table tests whether the behavioral response following an out-to-in reversal is larger in absolute magnitude than the response following an in-to-out reversal. The asymmetry estimate is the sum of the coefficients on the corresponding out-to-in and in-to-out reversal variables. A positive estimate therefore indicates a larger response following an out-to-in reversal. Panel A allows separate effects after one, two, and three-or-more prior reversals. The joint symmetry test evaluates the null that the three corresponding asymmetry estimates are jointly equal to zero (

F(3,567)=1.78

). Panel B enters the cumulative number of reversals of each type linearly and reports the difference in response magnitude associated with one additional prior reversal. Standard errors are clustered at the match level.

Comparison	Estimate	Clustered SE	$p$ -value
IN $\rightarrow$ OUT reversal minus upheld IN	-0.067	0.038	0.078
OUT $\rightarrow$ IN reversal minus upheld OUT	0.115	0.040	0.004
Joint equality with upheld challenges	—	—	0.004
Difference in absolute incremental effects	0.049	0.053	0.363

Appendix

Table C.6: The table compares estimated associations of subsequent calls with prior reversals and with unsuccessful challenges involving the same initial-call direction. A negative estimate in the first row indicates an additional shift toward calling balls out following an IN-to-OUT reversal. A positive estimate in the second row indicates an additional shift toward calling balls in following an OUT-to-IN reversal. All four cumulative feedback variables are included simultaneously in the same within-match specification used in Section 3.3. Standard errors are clustered at the match level. The joint test evaluates whether the two reversal coefficients equal their corresponding upheld-challenge coefficients. The final row tests whether the incremental associations are equal in absolute magnitude.

	(1)	(2)	(3)	(4)
	Incorrect call
PostHK	-0.014 ∗∗	-0.011	-0.011	-0.011
	(0.007)	(0.007)	(0.007)	(0.008)
Point controls		X	X	X
Match controls			X	X
Cluster level				Tournament

Appendix

Table C.7: OLS regressions of umpire mistakes for balls bouncing within 100 mm of the line.

PostHK=1

if Hawk-Eye review is active. Point controls include 20 mm distance-bin fixed effects, an above-median-speed indicator, score, game, set, and a tie-break indicator. Match controls include round and tournament tier.

Figure C.1 : The location of the umpiring crew: S = the serve-line umpire, RB = the right baseline umpire, RN = the right near long-line umpire, RC = the right center-line umpire, RF = the right far-side long-line umpire. The left side of the court has corresponding line umpires, except for the serve-line umpire, who moves to the left side when the right side player has the serve.

Figure C.2 : Call-in rates before and after the introduction of Hawk-Eye review for balls landing within 20 mm of the line, regardless of which side of the line they actually bounced on. The solid lines report fitted values from a linear probability model in which the probability of calling a ball in is allowed to vary linearly with month, separately before and after Hawk-Eye review. The dashed lines indicate 95% confidence intervals. The regression controls for game, set, point, tiebreak, round, tournament tier, and shot speed. Standard errors are clustered at the match level.

Panel of Figure 20 — Figure C.3 : Empirical distributions of signed distance from the line and ball speed before and after Hawk-Eye review.

Figure C.4 : Subperiod analysis of changes in the incorrect-call rate after the introduction of Hawk-Eye review, by proximity to the line. Matches with review are grouped into tournaments from 2006 through the first half of 2007 and tournaments from the second half of 2007 through 2008. Each dot represents the coefficient on the interaction between a distance-bin indicator and $PostHK$ .

Figure C.5 : An analog of Figure 2, using a one-treatment-at-a-time approach. We estimate the effect for each distance bin separately to avoid contamination across bins.

Figure C.6 : Estimated change in umpire performance after the introduction of Hawk-Eye review, across 20 mm distance bins and by tournament category. Matches are grouped into higher-tier tournaments (Masters 1000) and lower-tier tournaments (ATP 500 and 250). Each dot represents the coefficient on the interaction between a distance-bin indicator and $PostHK$ , estimated separately by tournament category.

Figure C.7 : Speed distribution for all the balls that bounced within 100 mm of the line, separated by non-serves and serves.

Panel of Figure 25 — Figure C.8 : Incorrect call rates by proximity to the line (separately for serves and non-serves). Each dot is the rate of incorrect calls for a bin of 20 mm, and dots to the left of the dashed line represent bins out of bounds, and those to the right of the dashed line represent bins in bounds.

Actual location	Initial call	Without review	With review	Corrected with review
In	In	1,216	6,664	0
In	Out	217	890	192
Out	In	147	879	297
Out	Out	1,042	5,757	0
Total		2,622	14,190	489

Appendix

Table D.1: Initial calls by actual location and subsequent correction.

Quantity and conditioning	Without review	With review
$p_{r}=\Pr(\text{ball in}\mid\text{initial call in})$	0.89214967	0.88346812
$q_{r}=\Pr(\text{ball in}\mid\text{initial call out})$	0.17235902	0.13389499
$m_{r}=\Pr(\text{ball in})$	0.54652937	0.53234672
$\lambda_{r}=\Pr(\text{initial call in})$	0.51983219	0.53157153
$\eta_{I}=\Pr(\text{corrected}\mid\text{ball in, initial call out})$	0	0.21573034
$\eta_{O}=\Pr(\text{corrected}\mid\text{ball out, initial call in})$	0	0.33788396

Appendix

Table D.2: Pooled probabilities implied by Table D.1 .

Human-AI Decision Making

Human-AI decision making studies how people and AI systems share, support, or delegate decisions across tasks and institutions.

See all 173 papers

Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency
Aug 17, 2026Parsa Mazaheri, Kasra MazaheriLanguage Model Error Detection Bias in LLM-as-a-Judge
University of California, Santa Cruz · Massachusetts Institute of Technology
The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)
Oct 5, 2026Feilian HuangInter-Rater Reliability Annotator Disagreement
Independent researcher
A Fairness Audit of the Duckworth-Lewis-Stern Method: Format-Specific and Gender-Differential Bias, with an Interpretable Calibration Layer for Cricket Target Revision
Sep 4, 2026Soumyadeep RoyAlgorithmic Bias
St. Xavier’s College (Autonomous), Kolkata, India

Aug 17, 2026J/K move · Enter open · S save

Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency

Parsa Mazaheri, Kasra Mazaheri

University of California, Santa Cruz · Massachusetts Institute of Technology

Language Model Error Detection Bias in LLM-as-a-Judge LLM Evaluation Language Model Auditing

Open paper AskPDF arXiv

Automated checking pipelines increasingly place one language model as the checker and another (or the same one) as the fixer. We ask whether that wiring changes what the checker reports. Measuring false alarms on human-verified-correct ProcessBench traces with the present task held byte-identical, we find that a completed audit -> repair episode already in the model's context lowers false alarms in 15 of 15 model x wording combinations, by 2.8 to 11.5 percentage points against a length-matched non-audit control, a 9 to 25% reduction relative to that control. The direction contradicts what the accumulated-message literature predicts: an episode whose audit reported an error lowers false alarms further still, at all five wordings on the model where that manipulation lands cleanly, though a negativity asymmetry predicts more flagging. Decomposing the episode finds repair content and audit verdict complementary: different components carry the effect on different model families. Signal-detection analysis locates the change in the threshold rather than in discrimination -- the criterion moves in 15 of 15 combinations and survives correction in 13 while d' survives in none, though the d' test is half as sensitive by construction -- and a hand audit of 50 false alarms finds 82% simply wrong, so at this operating point the shift need not be harmful. With reasoning enabled the effect keeps its relative size on both models tested, and the threshold reading holds there too.

Abstract

Human-AI Decision Making

Explore similar work

Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency

The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

A Fairness Audit of the Duckworth-Lewis-Stern Method: Format-Specific and Gender-Differential Bias, with an Interpretable Calibration Layer for Cricket Target Revision