We study how to combine a human binary call (a referee's offside or onside decision) with an AI numerical measurement (VAR's estimate of an attacker's position relative to the offside line). We derive an explicit maximum-likelihood rule that determines when the referee's call should stand and when a conflicting VAR measurement should overturn it. Our model treats the true offside margin as fixed and unknown and assumes independent additive Gaussian errors in the human's perception and the AI measurement. When both errors have zero means and known standard deviations σH and σAI, respectively, the reversal threshold is 2/πσAI2/σH. An onside call is overturned precisely when VAR's measured offside margin exceeds this threshold---not merely because VAR places the attacker beyond the offside line. We also derive the rule when either source has a systematic bias. We prove that the maximum-likelihood estimate of the true margin exists and is unique, while the final binary decision requires only a threshold comparison, without computing that estimate numerically. We illustrate the rule through numerical examples and show how to estimate the error parameters from calibration data. The same analysis applies beyond soccer to binary classification problems in which a human call and an AI measurement satisfy the stated model. This paper is intended for educational purposes: we aim to illustrate how undergraduate students and advanced high school students familiar with basic probability and calculus can investigate research questions in AI and its applications.
Figures & tables
Figure 1: The 3D reconstruction of Khalilzadeh’s late winner. According to VAR, the tip of his foot was in an offside position, disallowing the goal.
Figure 2: The signal used by VAR to support its claim that the ball touched Matanović’s hair. The Croatian federation filed a complaint with FIFA challenging VAR’s decision to overturn the human ruling.
Figure 3: Left: Alexander Zverev photographs a ball mark with his phone during the match to challenge an electronic line call that ruled Davidovich Fokina’s shot in. Right: the photograph, which Zverev later posted on Instagram to support his claim that the ball had landed out.
Figure 4: Human offside-call probabilities from ( 3.5 ). A larger σH makes the transition more gradual. A nonzero mH shifts the 50% crossover point from θ=0 to θ=−mH . The parameters are illustrative.
Figure 5: Log-likelihoods for y=1 cm, mH=mAI=0 , and σH=σAI=2 cm. Each curve has been shifted to have maximum zero. Changing the human call changes the sign of θML even though the observed AI measurement is unchanged.
Figure 6: Decision error probabilities for mH=mAI=0 and σH=σAI=2 cm. The human call H and the AI decision T(Y) have the same error probability.
Human call
z (cm)
θML (cm)
Final decision
Action
Offside
−0.20
0.1162
Offside
Retain
Offside
−0.50
−0.1763
Onside
Overturn
Onside
0.20
−0.1162
Onside
Retain
Onside
0.50
0.1763
Offside
Overturn
Table 1: Maximum-likelihood decisions for mH=mAI=0 , σH=10 cm, and σAI=2 cm. Numerical estimates are rounded to four decimal places.
Figure 7: Decision boundaries for ∣z∣=0.05 , 0.10 , 0.20 , and 0.40 cm, with both means zero and hz<0 . For each curve, we overturn the human decision below the curve and retain it above.
Figure 8: The decision boundary in ( 4.2 ). Both means are zero, and hz<0 . Above the surface, we overturn the human decision; below it, we retain it. On the surface, the estimated true margin is zero and the final decision is onside.
We study how algorithmic review changes error composition using Hawk-Eye records from professional tennis. Within 20 mm of the line, mistaken "in" call rates increase by 8.6 percentage points, while mistaken "out" call rates decrease by 4.1 points. Within matches, calls shift toward avoiding previously overturned errors. We find no evidence that initial accuracy deteriorates within 100 mm of the line, despite concerns that workers may rely on correction as a safety net. We derive an investment framework incorporating human responses alongside correction benefits and remedy costs, showing that holding human decisions fixed can misstate the organizational returns to review.
David Almog, Romain Gauriot, Lionel Page +1
Harvard Business School · Deakin University · University of Queensland +1
AI systems increasingly assist decision making by producing cheap factor-level assessments of complex inputs, but these assessments can be biased or incomplete. We study selective human evaluation: given AI signals, which factors should be dispatched to human experts? We formulate this human--AI collaboration problem as a factor-level information-acquisition problem. Under squared loss, the optimal dispatching rule maximizes a contextual reward; under a linear model, this reward admits a closed-form decomposition into two interpretable terms: predictive relevance and residual uncertainty in the human evaluation after conditioning on the AI signal. We instantiate the framework on peer review, decomposing papers into ten aspects and evaluating up to 3,408 ICLR submissions across three LLMs and multiple regression heads. In this retrospective proxy, where human aspect signals are extracted from existing reviews, predictions that query three of the ten aspects under the optimal dispatching rules match the full ten-aspect human benchmark. The finding holds when the human signal comes from a single noisy reviewer, and across review years. These results position principled dispatching as a practical building block for scalable human--AI decision systems. We also discuss safeguards under which such a rule can serve as decision support that leaves the final decision with a human.
When does an AI assistant improve human decision-making, and when does it hurt? We argue that the answer depends not just on AI accuracy but also on how much its evidence overlaps with the human's and how the human combines the two. A worker makes a binary decision by observing her own signal and receiving a recommendation from an AI. We develop a human behavioral model with two primitives: her awareness of the overlap and her trust in the AI. Our analysis gives a closed-form phase diagram over AI capability and overlap, with regions of complementarity, impairment, and automation. A worker who neglects the overlap (correlation neglect) can be made less accurate by a more capable AI, double-counting the shared evidence. Trust reshapes the regions - undertrust may compensate for correlation neglect, so that the trust level maximizing accuracy depends on the deployment parameters. Empirically, we identify the two model parameters by pairing the same workers with different AIs, and break the accuracy loss into a correlation-neglect cost and an undertrust cost, one tied to reducing overlap on the AI side, the other to calibrating trust on the human side. On ChatBench, AI assistance raises accuracy by 21 points, yet workers apply only about a quarter of the Bayes-rational weight to AI information; GPT-4o falls in the automation region and Llama-3.1-8b stays at the complementarity boundary; and calibrating trust recovers about 2.6 points for GPT-4o but almost nothing for Llama-3.1-8b.