Late but Valuable: Why Deadlines Change the Measured Value of LLM Reasoning
Organizations: Ray and Stephanie Lane Computational Biology Department, Kenneth C. Griffin School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA · Arm, Inc.
Abstract
Large language models (LLMs) increasingly plan and run experiments in automated laboratories. These experiments keep running while the model reasons and run on schedules with hard deadlines. If an LLM-generated answer arrives late, the system executes a safe fallback action instead, such as rejecting a sample. This creates a systems problem that standard serving metrics do not capture. An answer that matches the fallback adds no value, and answers that differ from the fallback may require more reasoning time. Reporting answer quality and latency separately can therefore overstate the value of reasoning. We test this on synthetic sample decisions, where the best choice is known exactly and a runtime enforces the deadline. We compare one LLM's fast mode, which answers directly, with its thinking mode, which reasons first. In a prespecified replay of 1,200 new decisions, thinking answers that missed a deadline were worth about five times as much as on-time ones. Multiplying the on-time rate by the mean answer value predicted about twice the value that thinking delivered. This prediction overstated thinking's advantage over fast mode by 0.0241 per decision, and by 0.0191 in a repeat on 300 new decisions. Both 95% confidence intervals excluded zero. In a single-GPU Qwen3-8B experiment using simultaneous request bursts, raising the number of requests from 8 to 56 reduced the share of thinking answers that met a 45 s deadline from 79.2% to 1.4%, although total output tripled. Evaluations of LLM agents for laboratories should therefore measure each answer's value and arrival time together and score the action that actually runs.
Figures & tables
| study | instances and actions | what is measured | evidence status |
|---|---|---|---|
| decision quality (§ 4 ) | 1,200 terminal instances; continuation offered | terminal decision regret by effort level | prespecified tests; regret breakdown descriptive |
| deadline execution (§ 5 ) | 600 instances per clock; accept or reject only; a new response at each deadline | how thinking’s advantage in realized utility changes between 2.1 and 60 s | one prespecified test per clock; other deadlines descriptive |
| prospective replay (§ 6 ) | 1,200 instances; accept or reject only; one 60 s-prompt answer per mode | how much separate summaries overstate thinking’s delivered posterior expected advantage | one prespecified test; pooled ranking secondary |
| Sonnet check (§ 4 , § 5 ) | 1,200 and 600 instances | regret contrast and dispatch-clock interaction | two prespecified tests |
| contrast | effect | CI | adj. | result |
|---|---|---|---|---|
| low fast | .002 | detected | ||
| medium low † | .88 | not detected | ||
| high medium | .60 | not detected |
| Pooled answer diagnostics | ||||
|---|---|---|---|---|
| mode | on time | late | ||
| fast | 0.9625 | 0.0501 | 0.0504 | 0.0444 |
| low | 0.4525 | 0.1605 | 0.0509 | 0.2511 |
| 8 requests | 56 requests | |
| burst start to dispatch (s) | 0.13 | 0.71 |
| dispatch to first token (s) | 1.95 | 9.33 |
| generating the answer (s) | 37.29 | 77.37 |
| total time to receipt (s) | 39.37 | 87.41 |
| output tokens per answer | 1,359 | 1,430 |
| speed per answer (tokens/s) | 35.8 | 17.4 |
| main claim | study | independent unit | primary endpoint | evidence status | location |
|---|---|---|---|---|---|
| Low-effort thinking improves returned terminal decisions; higher effort adds no detected improvement | terminal decision quality (1,200 new instances) | instance | terminal decision regret, thinking minus fast; medium minus low; high minus medium | confirmatory (four-test and two-test families) | A.4 |
| Deadlines limit how much of the improvement is realized in execution | dispatch-clock deadline execution (600 new instances) | instance | growth in thinking’s realized-utility advantage from 2.1 to 60 s | confirmatory (four-test family) | A.5 |
| Thinking’s advantage grows with the deadline when preparation time counts; both modes’ utility is near fallback utility at 2.1 s | evidence-onset deadline execution (600 new instances) | instance | growth in thinking’s realized-utility advantage from 2.1 to 60 s | confirmatory (four-test family) | A.5 |
| Overprediction of thinking’s advantage recurs on new instances | replay replication (300 new instances) | instance | at 2.1 s | confirmatory (single test); planned after the original analysis | A.7 |
| Separate summaries overestimate thinking’s delivered expected advantage over fast mode | prospective fixed-answer replay (1,200 new instances) | instance, within 12 groups | overprediction difference at 2.1 s | confirmatory (single test) | A.6 |
| Scope: a second hosted model | Sonnet 5.5 regret and dispatch-clock studies | instance | regret contrast; deadline interaction | confirmatory (two-test family); neither contrast detected | A.4 |
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
| term | meaning |
|---|---|
| terminal observation | an observation requiring a final decision: there is not enough time to continue in the decision-quality study, and continuation is excluded from the available actions in the deadline and replay studies |
| fallback | the preset action executed when an answer is late or unusable: reject for stopping and scientific faults, quarantine for physical faults; in the separate output-constraint protocol, a rejected proposal leaves the process waiting |
| posterior expected utility | the exactly computed expected value of an action given the evidence |
| simulator-realized utility | the value an action produced in the simulated run |
| task family | one of the two stopping processes (optical-fiber draw or quantum-dot synthesis) |
| cell | one experimental condition, such as one mode at one deadline |
| quantity | definition |
|---|---|
| optimum , rate | ; ; uniform over 21 optima 2 rates |
| quality at position | ; good iff |
| quality observation | |
| residual observation | |
| optical-fiber measurement / next parameters | |
| quantum-dot measurement / low resolution / tune |
| process | action | duration (s) | transition |
|---|---|---|---|
| both | accept; reject | 0; 0 | Terminal disposition, with the same charged 10 s in-module removal |
| optical fiber | next_parameters | 5 | Set position to parameter; preserve latent model and epoch |
| quantum dot | low_resolution | 68 | Preserve quality, position and epoch; take another measurement |
| quantum dot | high_resolution | 308 | Preserve quality, position and epoch; use high-resolution likelihood |
| quantum dot | tune | 103 | Set position to parameter; preserve latent model and epoch (35 s tuning then 68 s scan) |
| deadline (s) | mode | latency (s) | executed action | realized utility | answer used / late / goal |
|---|---|---|---|---|---|
| 2.1 | fast | 0.929 | reject | yes / no / no | |
| 2.1 | low | 3.652 | reject | no / yes / no | |
| 60 | fast | 1.082 | reject | yes / no / no | |
| 60 | low | 5.163 | accept | 0.9999 | yes / no / yes |
| cases | contrast | effect | CI | adjusted | result |
|---|---|---|---|---|---|
| stopping: 1,200 new, unselected instances | low fast | .002 | detected | ||
| faults: fixed 128-case model, 3 responses each | low fast | .002 | detected | ||
| stopping: the same 1,200 instances | medium low † | .88 | not detected | ||
| stopping: the same 1,200 instances | high medium | .60 | not detected |
| component | fast | low |
|---|---|---|
| continued; accepting was optimal | 0.189 (265) | 0.043 (56) |
| accepted; rejecting was optimal | 0.078 (94) | 0.005 (13) |
| rejected; accepting was optimal | 0.000 (0) | 0.009 (22) |
| continued; rejecting was optimal | 0.001 (767) | 0.001 (441) |
| total regret | 0.268 | 0.057 |
| optimal terminal action returned | 74 | 668 |
| study | task (instances) | contrast | effect | CI | Bonferroni | result |
|---|---|---|---|---|---|---|
| initial effort sweep | fault (128 cases 3) | low fast | .002 | detected | ||
| initial effort sweep | stopping (36 terminal 3) | low fast | .070 | not established | ||
| prior-sensitivity panel | fault, no-fault prior .99 | medium fast | .002 | detected | ||
| prior-sensitivity panel | stopping (240 new, unselected) | medium fast | .002 | detected |
| deadline | mode | utility | goal completion | fallback | occupancy |
|---|---|---|---|---|---|
| 2.1 s | fast | 0.0016 | 0.3% | 96.5% | 12.07 |
| 2.1 s | low | 0.0016 | 0.2% | 98.8% | 12.09 |
| 10 s | fast | 0.070 | 9.5% | 0.0% | 15.15 |
| 10 s | low | 0.155 | 16.7% | 4.3% | 16.64 |
| 60 s | fast | 0.032 | 6.7% | 0.0% | 15.16 |
| 60 s | low | 0.175 | 19.2% | 0.0% | 16.66 |
| deadline (s) | within groups | pooled |
|---|---|---|
| 2.1 | ||
| 4.0 | ||
| 6.0 | ||
| 10.0 | ||
| 60.0 |
| deadline (s) | block 1 [95% CI] | block 2 [95% CI] | difference [95% CI] | approx. |
|---|---|---|---|---|
| 2.1 | 0.667 | |||
| 4.0 | 0.910 | |||
| 6.0 | 0.640 | |||
| 10.0 | 0.670 | |||
| 60.0 | 1.000 |
| mode | deadline (s) | |||
|---|---|---|---|---|
| fast | 2.1 | |||
| fast | 4.0 | |||
| fast | 6.0 | |||
| fast | 10.0 | |||
| fast | 60.0 | |||
| low | 2.1 |
| deadline (s) | within-group separate | joint | pooled separate | signs agree |
|---|---|---|---|---|
| 2.1 | yes | |||
| 4.0 | no | |||
| 6.0 | yes | |||
| 10.0 | yes | |||
| 60.0 | yes |
| deadline (s) | fast | low | |
|---|---|---|---|
| 2.1 | |||
| 4.0 | |||
| 6.0 | |||
| 10.0 | |||
| 60.0 |
| mode | deadline (s) | timely | late | invalid | accepts, timely/late | mean , timely | mean , late |
|---|---|---|---|---|---|---|---|
| fast | 2.1 | 1155 | 45 | 0 | 110/2 | 0.0504 | 0.0444 |
| fast | 4.0 | 1183 | 17 | 0 | 111/1 | 0.0500 | 0.0588 |
| fast | 6.0 | 1196 | 4 | 0 | 112/0 | 0.0503 | 0.0000 |
| fast | 10.0 | 1200 | 0 | 0 | 112/0 | 0.0501 | n/a |
| fast | 60.0 | 1200 | 0 | 0 | 112/0 | 0.0501 | n/a |
| low | 2.1 | 543 | 657 | 0 | 45/186 | 0.0509 | 0.2511 |
| quantity at 2.1 s | original (1,200 instances) | replication (300 instances) |
|---|---|---|
| share of thinking answers on time | 0.4525 | 0.4367 |
| mean gain of on-time thinking answers | 0.0509 | 0.0657 |
| mean gain of late thinking answers | 0.2511 | 0.2064 |
| on-time thinking answers with no thinking tokens | 96.1% | 92.4% |
| overprediction , fast mode | ||
| overprediction , thinking |
| original run | replication | |||
|---|---|---|---|---|
| deadline | reject | accept | reject | accept |
| 2.1 s | ||||
| 4 s | ||||
| 6 s | ||||
| 10 s | ||||
| original run | replication | |||||||
|---|---|---|---|---|---|---|---|---|
| thinking tokens | answers | gain | on time at 2.1 s | share accepting | answers | gain | on time at 2.1 s | share accepting |
| 0 | 540 | 0.049 | 96.7% | 9% | 125 | 0.069 | 96.8% | 7% |
| 1 to 150 | 206 | 0.015 | 10.2% | 1% | 58 | 0.000 | 17.2% | 0% |
| 151 to 400 | 306 | 0.278 | 0% | 29% | 80 | 0.171 | 0% | 18% |
| more than 400 | 148 | 0.526 | 0% | 63% | 37 | 0.573 | 0% | 70% |
| run | mode | action | answers | median arrival (s) | share on time |
|---|---|---|---|---|---|
| original | fast | accept | 112 | 1.02 | 0.982 |
| original | fast | reject | 1,088 | 0.98 | 0.960 |
| original | thinking | accept | 231 | 4.61 | 0.195 |
| original | thinking | reject | 969 | 1.99 | 0.514 |
| replication | fast | accept | 28 | 0.98 | 0.964 |
| replication | fast | reject | 272 | 0.98 | 0.971 |
| state | prior (percent) | plate unseated | spill | missing cap | wrong orientation | tray jam | control anomaly |
|---|---|---|---|---|---|---|---|
| 55 | 12 | 8 | 15 | 18 | 7 | 8 | |
| 12 | 82 | 8 | 15 | 18 | 7 | 35 | |
| 9 | 72 | 64 | 15 | 18 | 7 | 82 | |
| 8 | 12 | 8 | 70 | 18 | 7 | 58 | |
| 10 | 12 | 8 | 15 | 62 | 7 | 22 | |
| 6 | 12 | 8 | 15 | 18 | 77 | 67 |
| family | action | ||||||
|---|---|---|---|---|---|---|---|
| physical | continue | 8 | |||||
| physical | reseat | 5 | 7 | 7 | 6 | ||
| physical | quarantine | ||||||
| scientific | accept | 10 | 4 | ||||
| scientific | reject | 0 | 0 | 0 | 0 | 0 | 0 |
| state | ||||||
|---|---|---|---|---|---|---|
| 72 | 10 | 4 | 6 | 5 | 3 | |
| 10 | 68 | 8 | 4 | 7 | 3 | |
| 6 | 16 | 66 | 6 | 3 | 3 | |
| 10 | 5 | 6 | 69 | 6 | 4 | |
| 12 | 12 | 3 | 5 | 64 | 4 | |
| 8 | 8 | 4 | 4 | 6 | 70 |
| mixture | 10 | 30 | 60 | 300 | 1,800 |
|---|---|---|---|---|---|
| main comparison | .01 | .04 | .10 | .35 | .50 |
| pause tolerant | 0 | 0 | .05 | .25 | .70 |
| rapid monitoring | .10 | .20 | .30 | .30 | .10 |
| panel | practical rules | without thinking | thinking |
|---|---|---|---|
| fault, boundary-enriched (120) | counting 1.63; threshold 1.89 | 3.08 | 0.054 |
| fault, no-fault prior 0.99 | posterior threshold 0.005; counting 3.68 | 4.57 | 0.015 |
| stopping, terminal (240 new) | rule 0.037; majority 0.152; reject 0.202 | 0.225 | 0.054 |
| allocation | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| .093 | .196 | .285 | .061 | .147 | .167 | .305 | .196 | .148 | .120 | .314 | .333 |
| scored mode | executed utility | timely and valid | submission | TTFT | completion | receipt | capped time | late | token-cap truncation | regret ( ) | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 8 | fast | .2811 | 100% | .131 | 1.52 | 2.43 | 2.57 | 2.57 | 0 | 0 | .0345 (72) |
| 8 | thinking | .1944 | 79.2% | .128 | 1.95 | 39.24 | 39.37 | 31.51 | 14 | 1 | .0260 (71) |
| 56 | fast | .2811 | 100% | .711 | 8.42 | 13.47 | 14.18 | 14.18 | 0 | 0 | .0345 (72) |
| 56 | thinking | 1.4% | .711 | 9.33 | 86.69 | 87.41 | 44.91 | 71 | 0 | .0481 (72) |
| mode | until sent | to first token | generation | total | tokens | tokens/s | valid and on time at 45 s | 90th-percentile arrival | |
|---|---|---|---|---|---|---|---|---|---|
| fast | 8 | 0.13 | 1.52 | 0.91 | 2.57 | 7 | – | 100% | 2.7 |
| fast | 56 | 0.71 | 8.42 | 5.04 | 14.18 | 7 | – | 100% | 17.7 |
| thinking | 8 | 0.13 | 1.95 | 37.29 | 39.37 | 1,359 | 35.8 | 79.2% | 69.5 |
| thinking | 56 | 0.71 | 9.33 | 77.37 | 87.41 | 1,430 | 17.4 | 1.4% | 126.5 |
| scored mode | waiting | to first token | generation | ms per token | tokens per request | burst output (tokens/s) | |
|---|---|---|---|---|---|---|---|
| fast | 8 | 0.13 | 1.28 | 8.31 | 27.5 | 306 | 59 |
| thinking | 8 | 0.14 | 1.25 | 36.48 | 28.0 | 1,310 | 140 |
| fast | 56 | 5.81 | 2.26 | 65.49 | 53.8 | 1,220 | 396 |
| thinking | 56 | 5.81 | 2.25 | 74.56 | 55.6 | 1,342 | 438 |
| running requests | 1 | 2 | 3–4 | 7–8 | 9–16 | 17–24 | 25–32 | 41–48 | 49–56 |
|---|---|---|---|---|---|---|---|---|---|
| server output (tokens/s) | 39 | 77 | 126 | 261 | 350 | 483 | 606 | 799 | 938 |
| per request (tokens/s) | 39.4 | 38.3 | 35.9 | 34.8 | 28.0 | 23.6 | 21.3 | 18.0 | 17.9 |
| configuration | value (all) | hard accuracy | median latency (s) | 90th-percentile latency (s) | $/call | ||||
|---|---|---|---|---|---|---|---|---|---|
| Haiku fast | 0.099 | 0.28 | 0.92 | 1.14 | 0.00026 | 0.097 | 0.099 | 0.099 | 0.099 |
| Haiku thinking | 0.258 | 0.28 | 1.57 | 3.66 | 0.00037 | 0.062 | 0.221 | 0.258 | 0.258 |
| Sonnet fast | 0.307 | 0.17 | 1.37 | 2.05 | 0.00523 | 0.284 | 0.300 | 0.307 | 0.307 |
| Opus | 0.454 | 0.88 | 4.68 | 10.01 | 0.01851 | 0.047 | 0.161 | 0.439 | 0.454 |
| Opus accelerated | 0.461 | 0.91 | 2.08 | 4.34 | 0.03814 | 0.217 | 0.438 | 0.457 | 0.461 |
| Fable | 0.420 | 0.60 | 3.28 | 20.56 | 0.05202 | 0.177 | 0.204 | 0.282 | 0.420 |
| configuration | easy | conflicting | hard | weighted value | easy time (s) | conflicting time (s) | hard time (s) |
|---|---|---|---|---|---|---|---|
| Haiku fast | 0.00/ 0.00 | 0.29/0.29 | 0.01/0.01 | 0.048 | 0.9 | 0.9 | 0.9 |
| Haiku thinking | 0.01/0.08 | 0.13/0.65 | 0.04/0.05 | 0.034 | 1.5 | 2.5 | 1.3 |
| Sonnet fast | 0.11/0.11 | 0.79/0.84 | 0.04/ 0.03 | 0.224 | 1.4 | 1.4 | 1.4 |
| Opus | 0.02/0.10 | 0.12/0.88 | 0.00/0.34 | 0.037 | 2.8 | 4.3 | 7.8 |
| Opus accelerated | 0.08/0.11 | 0.57/0.88 | 0.01/0.38 | 0.163 | 1.3 | 1.9 | 3.2 |
| Fable | 0.10/0.10 | 0.43/0.76 | 0.00/ 0.02 | 0.156 | 1.3 | 2.8 | 10.2 |
| cheaper | more expensive | [95% CI] | ($) [95% CI] |
|---|---|---|---|
| Haiku fast | Sonnet fast | 0.187 [0.150, 0.224] | 0.027 [0.022, 0.033] |
| Haiku fast | Opus accelerated | 0.120 [0.084, 0.157] | 0.315 [0.239, 0.454] |
| Haiku fast | Fable | 0.080 [0.044, 0.118] | 0.643 [0.435, 1.185] |
| Haiku thinking | Sonnet fast | 0.222 [0.187, 0.256] | 0.022 [0.019, 0.026] |
| Haiku thinking | Opus accelerated | 0.155 [0.123, 0.187] | 0.244 [0.200, 0.308] |
| Haiku thinking | Fable | 0.115 [0.083, 0.147] | 0.448 [0.344, 0.635] |
| option | cost per decision ($) | value | added value per added $ |
|---|---|---|---|
| reject without a model | 0 | 0 | none |
| Haiku fast | 0.00026 | 0.097 | 371 |
| Haiku thinking only when Haiku fast disagrees with the latest-reading rule | 0.00041 | 0.108 | 75 |
| concurrent pair, Haiku thinking, else Haiku fast | 0.00063 | 0.151 | 188 |
| Sonnet fast | 0.00523 | 0.284 | 29 |
| concurrent pair, Sonnet fast, else Haiku fast | 0.00549 | 0.289 | 20 |
| comparison | model and inference setup | study size and cost |
|---|---|---|
| terminal and fault decisions | Haiku 5.5, hosted | 3,168 scored calls, $1.73 |
| medium and high effort | Haiku 5.5, hosted | 2,400 scored calls |
| evidence-onset evaluation | Haiku 5.5, simulator | 3,600 executed decisions, $1.08, no failures |
| dispatch-clock evaluation | Haiku 5.5, simulator | 6,000 executed decisions |
| prospective replay | Haiku 5.5, recorded answers | 1,200 new instances, 2,400 scored calls |
| concurrent versus separate dispatch | Haiku 5.5, hosted | 1,200 scored calls |
| study | what was reproduced | difference |
|---|---|---|
| terminal and fault comparisons | estimates, intervals, and all 3,236 records | |
| medium and high effort | estimates and intervals, on the same instances | 0 |
| evidence-onset evaluation | all 3,756 records against the simulator | 0 |
| dispatch-clock evaluation | interval, and all 6,000 scored records | |
| concurrent selection | rule, selected records, interval and | 0 |
| latency check | intervals, schedule, 1,256 requests and cost | 0 |