Counting the Bandages: On measuring the time that AI saves

Solomon Shalom Lijo

cs.AIOct 23, 2026

Counting the Bandages: On measuring the time that AI saves

Authors: Solomon Shalom Lijo

Organizations: Independent Researcher

Abstract

I consider what it means for AI assistance to save time. I contrast a stopwatch view, on which time saved is whatever the clock records, with a ledger view, on which it is a counterfactual that has to be read next to the errors the work lets through, and I argue for the latter. The argument has four parts. I define a model-free pair of quantities, time saved and excess escaped defects, and one decision number built from them: the break-even defect cost. I give a baseline model of checking in which, until the model is good enough not to check, a calibrated worker's escaped-defect rate does not depend on how good the model is; checking time grows with the logarithm of the model's error rate; and over-trust buys minutes on the clock at a larger cost in total minutes. I show what randomized studies can and cannot identify, and I replicate METR's developer study from its public data: the same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them. Finally, I restate the published record, with stated bounds in place of imported point assumptions. In GDPval's own review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x; the consultants outside the frontier were faster, and more often wrong. Last, I test two of the model's predictions in a small pre-specified experiment on AI reviewers checking AI drafts of the same kind of brief, built from SEC filings made after the models' training data were collected. Told the drafter's true error rate, the reviewer checked more as it rose, but all-or-nothing, switching near where its prompt's costs break even. Told that the drafter erred ten times less often than it did, it checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none: faster and worse, entirely through reduced effort.

Figures & tables

Figure 1: What a worker who checks optimally does as the model improves (

\kappa=5

R=10

c=600

minutes, so

q^{\dagger}\approx 0.85\%

). (a) Checking and repair time is zero below

q^{\dagger}

and grows with

\ln q

above it; the over-trusting worker is never slower. (b) Escaped defects are pinned at

q^{\dagger}

for the calibrated worker. The over-trusting worker matches them below

q^{\dagger}

, lets through everything (escapes equal

q

) between

q^{\dagger}

and

10q^{\dagger}

, and lets through ten times as many above

10q^{\dagger}

. Generated by reproduce.py .

Check	Result
TSB’s adjustment on observed times	With completion times that include repairs, TSB’s reliability-adjusted score minus true time saved is $-0.100$ , $-0.298$ , $-0.699$ for draft error rates $0.1$ , $0.3$ , $0.7$ : minus the realised draft error rate, exactly.
Frontier, winner’s curse	200 tasks, all with true reliability $0.8$ ; $k=10$ , $r^{\star}=0.9$ . Labelled reliable: 37.5% (exact: 37.6%). Their reliability, same drafts: $0.929$ . Cross-fitted: $0.800$ .
Frontier, identification	Beta(8, 2) reliabilities, infinitely many tasks, $k=10$ . Identified set for the share $\geq 0.9$ : $[0.082,0.407]$ , stable to $0.001$ across grids of 2,001 to 8,001 points; truth $0.225$ .
Makarov bounds	Closed form matches an exact transport linear programme over all couplings, on 20 random data sets with ties (unit tests) and on METR’s data ( metr_reanalysis.json ).
Interval coverage	16 workers $\times$ 15 tasks, worker-level effect s.d. $0.3$ , 1,000 replications. Nominal 95%: task bootstrap 87.8%, cluster bootstrap 91.0%, jackknife- $t$ 95.0%. With 60 workers: 89.0%, 93.6%, 94.9%.
Three summaries	Proposition 4 (iii) with $10^{6}$ draws: geometric mean ratio $1.105$ , mean of ratios $1.209$ , ratio of means $0.896$ , each within $0.001$ of its closed form.

Table 1: Simulation checks (seeds fixed; reproduce.py , about two minutes).

Estimand	Forecast-adjusted	Unadjusted
Geometric mean ratio	$+18.8\%$ $[-0.2,\ +41.6]$	$+23.0\%$ $[+0.8,\ +50.0]$
	METR: $[+1.6,\ +39.0]^{a}$
Ratio of means	$+24.0\%$ $[+0.4,\ +53.2]$ (PPML)	$+33.7\%$ $[+8.7,\ +64.4]$
Share of issues on which AI helped: Makarov bounds $[2.6\%,\ 82.1\%]^{b}$
Post-review fix time, AI minus no AI: $+5.2$ min, cluster bootstrap $[-0.6,\ +12.1]^{c}$
a METR’s published developer-clustered interval (normal critical values), reproduced exactly. b Plug-in bounds from the two arms’ distributions of log(time/forecast); sampling error not shown, and conditioning on the forecast could only narrow them. c On the 230 issues where post-review time was recorded.

Table 2: One data set, several headlines: METR’s early-2025 study ( Becker et al., 2025 ) , 246 issues, 16 developers. Positive means more time with AI. Intervals are leave-one-developer-out jackknife with

t_{15}

critical values unless marked.

Study	Time	Quality	Region
Peng et al. (2023) , Copilot	$-55.8\%$ time $[-89,-21]$	task success $+7$ pp, n.s.	dominant
Noy & Zhang (2023) , writing	$-40\%$ time	grade $+18\%$	dominant
Dell’Acqua et al. (2026) , inside	$-25.1\%$ time	quality score $+30$ – $34\%$	dominant
Dell’Acqua et al. (2026) , outside	$-30\%$ / $-18\%$ time	wrong answers $+24.5$ / $+13.9$ pp	trade-off; $\lambda^{\star}\approx 48$ min
Brynjolfsson et al. (2025) , less-skilled	$+30\%$ resolutions/hour	speed and quality both up	dominant
Brynjolfsson et al. (2025) , most-skilled	small or no gain	small decline	trade-off or dominated

Table 3: Published results as (time, quality) pairs. Regions, as defined in Section 2 , use point estimates; “n.s.” marks a quality difference that was not statistically significant. Sources report different quality measures, so compare regions, not magnitudes.

Hypothesis (experiment, $n$ )	Estimate	Verdict
H1. Detection is concave in effort, A2 (E1; 25 defective drafts per budget)	Caught 0, 32, 48, 100% with 0, 1, 3, 9 lookups. Exponential fit preferred to linear (AIC 75.0 vs 77.8; $\kappa=3.1$ lookups)	confirmed
H2. Effort rises with $\ln q$ , P1 (E2; 120 reviews, $q$ stated truthfully)	$+2.2$ lookups $[1.8,\ 2.5]$ per unit of $\ln q$ ; $p<.001$	confirmed
H2b. Effort is graded, not all-or-nothing (E2)	96% $[91,\ 99]$ of reviews made 0 or 9 lookups; the switch lies between $q=2\%$ and $5\%$	all-or-nothing
H3. Escapes flat in $q$ where checked, P2 (E2, $q\geq 10\%$ )	No escapes at any of these levels; slope $0$ $[0,\ 0]$	estimate only
H4a. Over-trust checks less, P4 (E3; 40 paired drafts, true $q=30\%$ , told $3\%$ )	$-7.2$ lookups $[-8.1,\ -6.0]$ ; $p<.001$	confirmed
H4b. …and lets more through, P4 (E3)	Escapes $22.5\%$ vs $0\%$ : $+22.5$ points $[10,\ 35]$ ; $p<.001$	confirmed

Table 4: Pre-specified hypotheses and results, AI reviewers ( claude-sonnet-5-5 ) checking AI drafts. Brackets are 95% cluster-bootstrap intervals.

p

-values are bootstrap, Holm-corrected across the three primary tests (2,000 resamples, so

p<.001

is the floor).

Figure 2: AI reviewers checking AI drafts. (a) Seeded defects caught under a fixed number of lookups, with both pre-specified fits; nine lookups check every line. (b) Lookups chosen when told the drafter’s true error rate: each grey dot is one review (jittered), black dots are means. The mean rises with

\ln q

, but almost every review checks nothing or everything, switching near the break-even rate

q_{bb}

that the prompt’s costs imply. (c) Same drafts, true error rate 30%: a reviewer told 3% looks less and lets more through. The two “told 3%” arms are each shifted

0.3

lookups sideways so both are visible (their means are

1.85

and

2.05

). Bars are 95% cluster-bootstrap intervals. Generated from the run’s logs by reviewlab/figure.py .

1	Name the summary (ratio of means, geometric mean ratio, or other), and the populations of tasks and workers it averages over.
2	Use observed times from both arms. Self-reports are not a substitute: METR’s developers misjudged their own speedup by about 40 points.
3	Audit final deliverables blind, in both arms, and report $\delta$ with an interval.
4	When $\tau$ and $\delta$ have the same sign, report $\lambda^{\star}$ .
5	If instrumented, report checking time and repair events ( Mozannar et al., 2024 , coded as in) .
6	Report inference cost in money, in its own column.
7	Cluster inference by worker; with few workers, use a small-sample method such as the jackknife- $t$ .

Table 5: Reporting checklist for claims that AI saves time.

Contribution	Closest prior work, and what is new
Break-even defect cost $\lambda^{\star}$ and the three regions	The incremental cost-effectiveness ratio and the cost-effectiveness plane ( Black, 1990 ) ; team utility with a cost per mistake ( Bansal et al., 2021 ) ; verification friction ( Kamara, 2026 ) . New is only the application to time-saved claims.
$\varepsilon^{\star}=\min(q,q^{\dagger})$ ; checking time linear in $\ln q$	Koopman’s optimal search effort, of which this is the one-cell case ( Koopman, 1957 ) ; effort-optimal verification ( Huang et al., 2026 ) . New is the reading for AI: better models buy speed, not quality.
Over-trust loss $\kappa(m-1-\ln m)$ , its asymmetry, the clock/ledger split	Huang et al. (2026) : optimistic beliefs cause over-delegation, shown qualitatively and in utility rather than minutes.
Certification cost $\approx 3/q^{\dagger}$	The rule of three ( Hanley & Lippman-Hand, 1983 ) ; new is tying it to the checking threshold.
Identified set for “share of tasks reliable”	Standard moment arguments for binomial mixtures; new is the application to frontier statistics.
Makarov bounds, PPML and jackknife- $t$ on METR’s data	METR’s appendix, which reports log-time OLS, the raw ratio of means and other small-sample corrections ( Becker et al., 2025 ) .

Table 6: Contributions, and the nearest thing that already existed.

Human Oversight of AI

Human oversight of AI studies how people supervise, verify, and control AI decisions, especially as systems become more autonomous and consequential.

See all 184 papers

Aug 9, 2026

AI Evaluation Should Measure Verification Cost, Not Correctness Alone

The reliability of AI generative models is typically measured by output correctness, yet in practice it depends on the effort required to verify those outputs. We argue that current evaluation metrics overlook a critical failure mode: Verification-Cost Errors (VCEs), defined as incorrect input-output pairs that a declared fraction of the verifier population fails to identify within the verification budget available in a given deployment context. Unlike standard notions of "hallucination", VCEs are defined operationally, by the failure of correct identification within budget rather than by any property of the output itself. Plausibility and authoritative presentation are hypothesised contributors to that failure, not defining conditions. To capture this asymmetry, we introduce the notion of verification cost relative to a deployment budget as an operational dimension that current evaluation does not routinely capture. The quantity is presented as a conceptual instrument rather than a finalized metric. Evidence from code generation and multi-modal document understanding shows that high benchmark accuracy can mask significant verification effort in practice. We therefore take the position that correctness alone is insufficient as a measure of reliability. AI evaluation should explicitly account for verification cost, reflecting whether errors can be detected under realistic resource constraints.

Viviana Crescitelli, Generoso Immediato, Fabio Persia +1

Hitachi, Ltd., Tokyo, Japan · Hitachi Rail, Naples, Italy · Department of Information Engineering, Computer Science and Mathematics, University of L’Aquila, Italy

Language Model Generation Evaluation Generative AI Evaluation

Ask PDF

Jul 25, 2026

The Checking Problem: What must be true before AI ships in a regulated firm

Enterprise AI programmes stall at a rate that is widely quoted and poorly explained. This paper measures the mechanism. Six document-heavy workflows of the kind performed daily in regulated financial services were run across four model families and three tool configurations, three times each, producing 5,093 scored output elements across 72 configurations. Each configuration was assessed twice: against a demonstration bar, being a single correct run on a single case, and against a production bar requiring sustained accuracy, reproducibility across repeats, verifiable attribution, and a confidence signal that carries information. 57 of 72 configurations cleared the demonstration bar and 32 cleared the production bar, a survival rate of 56.1%. The paper then computes the review burden each configuration imposes, estimated out of sample rather than with hindsight. A tool that states no confidence requires review of 100% of its output, because it offers a reviewer no basis for triage. Requiring the tool to cite its sources and state a confidence reduces that to 49% while holding the residual error tolerance in 17 of 20 configurations. Adding a self-verification pass costs 2.3 times the latency of the plain configuration, reaches 44%, and is the only configuration that fails to hold the error tolerance. The practical implication is that the value of an AI workflow is set less by how often it is right than by how much of it a human must still check, and that the second property is measurable and rarely measured.

Prerit Ahuja

Independent Researcher

Human-in-the-Loop Evaluation Human Oversight of AI Generative AI Evaluation

Ask PDF

Jun 5, 2026

Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models

Many efforts to ensure frontier AI models are safe rely on monitoring their chain-of-thought (CoT) reasoning. If models become able to perform sufficiently complex reasoning internally, without explicit thinking tokens, this would undermine such oversight. We measure how well frontier models reason without CoT across a suite of over 30,000 questions spanning 43 benchmarks in domains including math, coding, puzzles, causality, theory-of-mind, and strategic reasoning. To compare models against humans, we estimate the

50\%

-task-completion time horizon (TH): the human time required for tasks a model completes with

50\%

success rate. We complement this with a

50\%

reasoning token horizon: the minimum number of o3-mini reasoning tokens needed for tasks a model solves with

50\%

success rate. We find that the no-CoT

50\%

TH of frontier models has been doubling roughly every year over the past six years, with GPT-5.5's TH reaching over 3 minutes and reasoning token horizon exceeding 1,500 tokens. Our median estimates predict that frontier no-CoT THs could exceed 7 minutes by 2028, and 25 minutes by 2030, though these projections carry substantial uncertainty. We recommend frontier developers track this explicitly.

Dewi Gould, Francis Rhys Ward, Anders Cairns Woodruff +18

Redwood Research · Astra Fellows Program · Aether Research +10

LLM Evaluation AI Safety Evaluation LLM Reasoning Large Reasoning Models

Ask PDF

Abstract

Human Oversight of AI

Explore similar work

AI Evaluation Should Measure Verification Cost, Not Correctness Alone

The Checking Problem: What must be true before AI ships in a regulated firm

Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models