math.OCAug 25, 2026

The Shadow Price of Intelligence: Quality Degradation in LLM Inference as a Supply Chain Problem

Authors: Elioth Sanabria

Organizations: Department of Decision and Technology Analytics Lehigh University - College of Business

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language model providers are compute constrained, and a common response to congestion is to degrade service. A degraded answer fails with some probability, and a failed answer either returns as a retry or departs as churn, destroying LTV on an unaccounted ledger. We model inference allocation as a newsvendor whose stockout cost is churned lifetime value, a geometric retry multiplier in which the recycled product is dissatisfaction, and a two-regime transient fluid queue whose arrival rate is made endogenous by retries. Statically, there is a regime in which a cheaper model saves energy per initiated task while consuming strictly more capacity per initiated task, so the discount inverts when capacity binds. Dynamically, a reactive throttle fired during a surge can cross an ignition threshold beyond which it manufactures more traffic than it sheds, and a release rule set below the degraded equilibrium converts a transient surge into a permanent degraded regime. With heterogeneous customers, throttling is a transportation problem in retry-inflated load whose optimal policy rations intelligence by critical ratio, and whose dual, the shadow price of intelligence, prices a marginal query by class and by hour. Under congestion, throttling is not a cost lever but a demand lever.

Figures & tables

Figure 1 : The two frontiers (schematic). The posted frontier, per raw query, is monotone: weaker is cheaper, which is what every pricing page implies. The realized frontier, per initiated task, carries the multiplier and the churn charge of ( 5 ) and bends back up as djρd_{j}\rho grows; past the energy break-even the ordering of the tiers inverts. The red arrow is the inversion at the distilled model: the vertical gap between its posted and realized cost, the retry storm plus the churn, is the cost of confusing a query with an answer.
provideralways strongestreactive thresholdscheduledindex rulefluid DPtreebest/DP
Anthropicdiv †$991.88M 9vdiv †$1.17B 3v$9.40M$9.44MNA
OpenAI$117.70M 24v$8.73M 21v$61.84M 17v$1.59M$1.78M$1.78M0.89 ×\times
Google$7.26B 3v$1.75B 2v$7.26B 3v$7.26B 3v$9.21M$8.87MNA
DeepSeekdiv †$27.87M 23vdiv †$2.03M$2.03M$2.03M1.00 ×\times
Kimi$3.44M$3.44M$1.62B 12v$3.44M$3.44M$3.44M1.00 ×\times
Table 1 : Twenty-four-hour DES-estimated economic cost by policy and provider (30 replications; mean; lower is better), with the per-attempt SLA screen of Section 5.2 applied to the DP’s action menu. Demand is scaled to each provider’s own throughput, so dollar figures compare across policies within a row, not across providers. Baselines: always strongest serves every query at tier 0; the reactive threshold degrades everyone when total backlog crosses a trigger, releasing below a second threshold; scheduled degrades everyone during the published peak windows; the index rule (hourly) re-solves the class-aware transportation problem ( 8 ) hourly against the forecast. All are tuned per instance as in Section 5.2 . † div : the DES-estimated cost exceeds the 1000×1000\times reporting cutoff under the policy’s own retry feedback (Proposition 4 ); no finite estimate is reported. kv : the policy’s DES-estimated \mboxP(Wattempt>300 s)\mbox{\sf P}(W_{\mathrm{attempt}}>300\,\mathrm{s}) exceeds 0.050.05 in kk of 24 hours (point estimates), and the policy is marked infeasible. The final column reports the best feasible baseline as a multiple of the DP; where no baseline is feasible it reports NA. Latency is DES-estimated throughout; no policy is certified feasible by construction.
Figure 2 : Policy-conditioned sensitivities π^x(t)\hat{\pi}_{x}(t) , five providers: finite differences of the twenty-four-hour cost through the closed-form fluid forward pass under the DP’s own actions; dotted verticals mark hours where the optimal action switches, where the sensitivity may kink. These sensitivities agree with the optimized-value derivative ( 9 ) when the entire optimal action sequence and its feasibility remain unchanged under the perturbation; they may differ at action-switch hours. Top panels : the specialized class’s sensitivity decomposed into the direct cost of one more initiated task at the hour’s action, Mj(x)(cj+dj(x)(1−ρx)prℓx)M_{j}(x)\bigl(c_{j}+d_{j}(x)(1{-}\rho_{x})p_{r}\ell_{x}\bigr) with the peak electricity multiplier, and the congestion rent above it. Bottom panels : the rent by class, in dollars. The direct floors differ across classes at every hour, so the flat relative price of one that any uniform policy implicitly charges is wrong around the clock, off peak by 3.9 ×\times on Anthropic; and the pool charges a near-common rent when capacity binds. Rent panels are clipped where a single hour dominates, the value printed at the marker: on Google the marginal casual query at the surge carries 145ofservice−levelknock−ons,thesensitivityatthecapacitywall.DeepSeekandKimiholdoneactionallday,thestaticoptimum:noswitches,andrentmovesonlywithload.Sensitivitiesinthelasthoursarebiasedlowbythefinitehorizon,andtherentreachesthelatencyconstraintthroughthesmooth145 of service-level knock-ons, the sensitivity at the capacity wall. DeepSeek and Kimi hold one action all day, the static optimum: no switches, and rent moves only with load. Sensitivities in the last hours are biased low by the finite horizon, and the rent reaches the latency constraint through the smooth C_{\mathrm{SLA}}$ term rather than a hard wall.
providerfluid (s)Euler (s)speedupDES policy-value gap
Anthropic0.433479 ×\times-3.68%
OpenAI2.24275123 ×\times-12.15%
Google1.29414321 ×\times+2.11%
DeepSeek0.181475 ×\times+0.00%
Kimi0.05474 ×\times+0.00%
Table 2 : Runtime and DES-estimated accuracy of the fluid solver against a brute-force competitor: the identical dynamic program, grid, action set, and cost function, with the closed-form transition replaced by a converged Euler integration of the same fluid dynamics at the per-provider stable step. The last column is the DES-estimated policy-value gap, 100×(Euler−fluid)/Euler100\times(\mathrm{Euler}-\mathrm{fluid})/\mathrm{Euler} from the two greedy policies’ twenty-four-hour costs (30 replications each): negative means the Euler-solved policy is cheaper under simulation. It replaces the fluid-rollout policy-value comparison; pointwise Q-regret statistics are in Appendix C .
Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5 : One transition of the backward induction, magnified. The induction fills the same T×∣A1∣×g∣X∣T\times|{\cal A}_{1}|\times g^{|{\cal X}|} value surface under either solver; the panels differ inside the transition arrow. (a) An explicit integrator resolves the recovering branch’s relaxation at rate μjeff\mu^{\mathrm{eff}}_{j} , so each transition takes (max⁡jμjeff)Δ/c(\max_{j}\mu^{\mathrm{eff}}_{j})\Delta/c substeps. (b) Lemma 2 gives the same trajectory as a linear leg and an exponential leg joined at the closed-form crossing time t⋆t^{\star} : constant cost per transition, independent of the service-rate scale.
Figure 6 : Finite-horizon backlog-growth probability at 30 s against the dissatisfaction probability d1d_{1} (single class, Example 3.2 parameters, N(0)=7C/6N(0)=7C/6 , tier 1 from t=0t=0 ; 200 replications per point, Wilson 95% CIs). The outcome is N(30 s)>17C/6N(30\,\mathrm{s})>17C/6 . Dotted verticals mark the fluid stability threshold d⋆=(1−r/(kμ1m))/ρ=0.4175d^{\star}=(1-r/(k\mu_{1}m))/\rho=0.4175 and the fluid finite-horizon growth threshold dfluid=0.4217d^{\mathrm{fluid}}=0.4217 ; the stochastic transition is centered near dfluidd^{\mathrm{fluid}} and sharpens with CC , and no irreversibility is implied. Per- CC detail in Appendix F .
 symbolmeaning
fleetkk , mm , mkmkservers; concurrent slots per server; total capacity in jobs
NtN_{t}jobs resident in the system at time tt
γ\gamma , w0w_{0} , cmc_{m} , κ\kappa , CSLAC_{\mathrm{SLA}} , β\betaelectricity price; idle draw per server; memory cost scale and exponent; service-level penalty scale (dollars per hour) and elasticity
tiersj∈{0,…,J}j\in\{0,\dots,J\}quality tier, 00 the strongest; φ\varphi a routing mix across tiers
Δwj\Delta w_{j} , E[Sj]{\bf\sf E}[S_{j}] , μj\mu_{j}power draw per active slot; mean service time; service rate 1/E[Sj]1/{\bf\sf E}[S_{j}]
djd_{j}dissatisfaction probability: the answer fails its user, increasing in jj
Appendix
Table 3 : Notation. The symbols of the first three blocks are primitives, measurable from benchmarks, serving traces, and logged traffic; the derived quantities of the last block carry the retry feedback and are the objects every result compares.
Figure 7 : The transportation LP ( 8 ) for two classes and two tiers. Solid arrows are the dominant flows in the calibrated instance; dotted arrows are the cross-assignments the LP considers and rejects. Capacity is charged in effective slot-time S~j(x)=Mj(x) E[Sj]{\tilde{S}}_{j}(x)=M_{j}(x)\,{\bf\sf E}[S_{j}] , not posted E[Sj]{\bf\sf E}[S_{j}] : a heavy-retry class consumes capacity its service time conceals, and in the trap region the capacity constraint tightens under degradation.
Figure 8 : The boomerang of Example 3.2 , in the example’s units of thousands of jobs. Three levels anchor the vertical axis: capacity mk=3mk=3 (dotted), the reactive trigger N^fire=3.5\widehat{N}_{\mathrm{fire}}=3.5 (dash-dotted; the reactive policy of Proposition 6 instead fires at N^fire=mk\widehat{N}_{\mathrm{fire}}=mk , where the spiral already ignites), and the Little’s-law equilibrium Nˉ=∑xλx S~(x)≈2.94\bar{N}=\textstyle\sum_{x}\lambda_{x}\,{\tilde{S}}(x)\approx 2.94 of the anticipatory mix, which sits below capacity. Reactive throttling fires when NN crosses the trigger and ignites the retry spiral; the anticipatory path relaxes toward Nˉ\bar{N} and never saturates. The cost rate spikes before it saves, and the churned lifetime value accrues on a separate, monotone ledger that no cost chart displays.
Figure 9 : Demand and backlog under the policies of Table 1 , five providers, one shared diurnal profile scaled to each provider’s throughput (hour of day; the day is a finite horizon, not a cycle, so the two edges need not match). Top panels : fresh demand, with the static-optimal-mix throughput θmix\theta_{\mathrm{mix}} dashed; the morning and afternoon plateaus are the published peak windows, and the overnight spike at 03:00 is the surge. Bottom panels : the resulting backlogs (log scale), the closed-form fluid forward pass of ( 7 ) under each policy’s logged hourly actions; under the DP’s own policy the simulation’s mean hourly backlogs agree with the forward pass to a maximum relative gap of 42.5% over all providers and hours (mean 11.80%). The distilled tree is omitted: visually identical to the fluid DP on every panel. “static LP (hourly)” in the legend is the index rule of Section 5.2 .
providerperturbationDP costfeas.best baseline/DPindex rule (hourly)/DP
Anthropicbase$9.38MfeasibleNA124 ×\times
Anthropicρx ⁣↑\rho_{x}\!\uparrow (cap 1)$5.75MfeasibleNA212 ×\times
Anthropicρx×0.75\rho_{x}\times 0.75$16.47MfeasibleNA60.9 ×\times
Anthropicprℓ×0.5p_{r}\ell\times 0.5$7.16MfeasibleNA163 ×\times
Anthropicprℓ×2p_{r}\ell\times 2$13.54MfeasibleNA86.7 ×\times
AnthropicCSLA×0.5C_{\mathrm{SLA}}\times 0.5$9.38MfeasibleNA62.8 ×\times
Appendix
Table 4 : One-at-a-time sensitivity of the DES-estimated comparison (Anthropic and Google; demand profile fixed in every row except the peak/ θmix\theta_{\mathrm{mix}} provisioning rows). DP cost is the fluid rollout on the perturbed instance (mean and 95% CI over 20 forecast-noise seeds where applicable); feasibility is the DES criterion of Section 5.2 (no hour with DES-estimated \mboxP(Wattempt>300 s)>0.05\mbox{\sf P}(W_{\mathrm{attempt}}>300\,\mathrm{s})>0.05 ) evaluated on true demand; best-baseline/DP uses feasible baselines only (NA where none is feasible); index rule (hourly)/DP compares the hourly re-solved index rule against the DP. The index rule was not re-solved on noisy forecasts, so its column reads “not run” on the forecast-noise rows ( σ\sigma rows carry no archived index-rule value). The full grid is in Appendix E .
customer classes (shared across providers)
 casualspecial.agentic 
share A0.600.250.15share of fresh demand
ρx\rho_{x} A0.300.850.98probability an unsatisfied user re-asks
d0,xd_{0,x} A0.020.050.10dissatisfaction at the strongest tier
βx\beta_{x} A0.82.53.0quality sensitivity: dj(x)=min⁡{0.95, d0,x+βx(1−Ij/I0)}d_{j}(x)=\min\{0.95,\,d_{0,x}+\beta_{x}(1-I_{j}/I_{0})\}
prp_{r} A0.030.080.05churn probability upon abandonment
Appendix
Table 5 : Calibrated instances. Markers give each number’s provenance: B benchmark (Artificial Analysis), P published by the provider, A assumption anchored to public prices, C calibration choice, D derived, S full joint-action count →\to static-ledger count →\to post-screen count; the per-attempt screen of Section 5.2 removes tiers only on OpenAI (the 221221 s Sol (max), Terra (max) and the 172172 s Luna (max)), so the two stages coincide on the other providers. Symbols as in Table 3 , by class xx (Section 3.3 ).
providermeanp95occ. meanfeas. disagr.
Anthropic1.83%7.79%3.33%5.77%
OpenAI1.04%3.98%1.56%0.00%
Google8.05%39.53%11.66%1.51%
DeepSeek1.23%1.83%1.39%0.00%
Kimi0.00%0.00%0.00%0.00%
Appendix
Table 6 : Pointwise Q-regret of the fluid policy under the converged Q-function, on mutually feasible state–action pairs (uniform mean and p95, occupancy-weighted mean), with the feasibility-disagreement rate under the heuristic surrogate. Tails concentrate off-trajectory in deep-saturation grid states on the shared-pool, high- μ\mu providers; the deployment-relevant comparison is the DES-estimated policy-value gap of Table 2 .
providerpolicymean95% CIhours >0.05>0.05
Anthropicalways strongestdiv †div †24
Anthropicreactive threshold$991.88M[984.08M,984.08M, 999.69M]9
Anthropicscheduled (peak windows)div †div †19
Anthropicindex rule (hourly)$1.17B[1.16B,1.16B, 1.18B]3
Anthropicfluid DP$9.40M[9.39M,9.39M, 9.41M]0
Anthropicdistilled tree$9.44M[9.44M,9.44M, 9.45M]0
Appendix
Table 7 : DES-estimated 24-hour cost: mean and 95% CI over 30 replications per policy and provider, with the number of hours in which the hourly point estimate of \mboxP(Wattempt>300 s)\mbox{\sf P}(W_{\mathrm{attempt}}>300\,\mathrm{s}) exceeds 0.050.05 . † div: the DES-estimated cost exceeds the 1000×1000\times reporting cutoff under the policy’s own retry feedback, so no finite estimate is reported — the same convention as Table 1 . Implementation: batched discrete-event simulation at per-provider step Δt\Delta t with μmax⁡Δt≤0.1\mu_{\max}\Delta t\leq 0.1 (Anthropic 0.9960.996 s, DeepSeek 11 s, Google 0.2410.241 s, Kimi 11 s, OpenAI 0.6790.679 s), per-class Poisson fresh-arrival streams, exponential services by per-job thinning, instantaneous retries, per-class FCFS queues with wait-age buckets, and the fluid model’s proportional capacity shares (per-tier pools where published), all 30 replications simulated in parallel.
providercellDP costP0P1P2P5DPbest base/DP
Anthropicbase$9.38Mdiv †$991.88Mdiv †$1.17B$9.40MNA
Anthropicρx ⁣↑\rho_{x}\!\uparrow (cap 1)$5.75Mdiv †$1.44Bdiv †$1.22B$5.76MNA
Anthropicρx×0.75\rho_{x}\times 0.75$16.47Mdiv †$1.05Bdiv †$1.00B$16.48MNA
Anthropicprℓ×0.5p_{r}\ell\times 0.5$7.16Mdiv †$976.77Mdiv †$1.17B$7.17MNA
Anthropicprℓ×2p_{r}\ell\times 2$13.54Mdiv †$1.06Bdiv †$1.17B$13.53MNA
AnthropicCSLA×0.5C_{\mathrm{SLA}}\times 0.5$9.38Mdiv †$538.03Mdiv †$591.51M$9.42MNA
Appendix
Table 8 : Full one-at-a-time sensitivity grid (Anthropic and Google): DES-estimated 24-hour mean cost by policy, the DP’s fluid rollout cost, and the best feasible baseline/DP ratio (feasibility by the Table 7 criterion: no hour with DES-estimated \mboxP(Wattempt>300 s)>0.05\mbox{\sf P}(W_{\mathrm{attempt}}>300\,\mathrm{s})>0.05 ). The base row is taken from the same 30-replication run as Table 7 ; the deterministic perturbation rows use their own 30-replication DES sets and the two forecast-noise rows use 20 seeds, so base and perturbed cells differ by seeds as well as by the perturbation. † As in Table 7 : the cost exceeds 1000×1000\times the DP’s mean, so no finite estimate is reported. The index rule was not re-solved on noisy forecasts, so its column reads “not run” on the forecast-noise rows.
d1d_{1}prob.95% CId1d_{1}prob.95% CI
0.30000.000[0.000, 0.019]0.41750.065[0.038, 0.108]
0.32000.000[0.000, 0.019]0.41800.100[0.066, 0.149]
0.34000.000[0.000, 0.019]0.41850.100[0.066, 0.149]
0.36000.000[0.000, 0.019]0.41900.140[0.099, 0.195]
0.38000.000[0.000, 0.019]0.41950.235[0.182, 0.298]
0.39750.000[0.000, 0.019]0.42000.285[0.227, 0.351]
Appendix
Table 9 : Finite-horizon backlog-growth probability, C=mk=750C=mk=750 (200 replications per point; Wilson 95% CIs; fluid threshold d⋆=0.4175d^{\star}=0.4175 ; fluid growth threshold dfluid=0.4217d^{\mathrm{fluid}}=0.4217 ; DES step μ1Δt=0.083\mu_{1}\Delta t=0.083 ). Grid: coarse spacing 0.020.02 plus refined rows at spacing 0.00050.0005 within ±0.02\pm 0.02 of d⋆d^{\star} ; the grid is split into two side-by-side halves so that every row from 0.30000.3000 through 0.50000.5000 prints.
d1d_{1}prob.95% CId1d_{1}prob.95% CI
0.30000.000[0.000, 0.019]0.41750.000[0.000, 0.019]
0.32000.000[0.000, 0.019]0.41800.000[0.000, 0.019]
0.34000.000[0.000, 0.019]0.41850.000[0.000, 0.019]
0.36000.000[0.000, 0.019]0.41900.015[0.005, 0.043]
0.38000.000[0.000, 0.019]0.41950.025[0.011, 0.057]
0.39750.000[0.000, 0.019]0.42000.055[0.031, 0.096]
Appendix
Table 10 : Finite-horizon backlog-growth probability, C=mk=3000C=mk=3000 (200 replications per point; Wilson 95% CIs; fluid threshold d⋆=0.4175d^{\star}=0.4175 ; fluid growth threshold dfluid=0.4217d^{\mathrm{fluid}}=0.4217 ; DES step μ1Δt=0.083\mu_{1}\Delta t=0.083 ). Grid: coarse spacing 0.020.02 plus refined rows at spacing 0.00050.0005 within ±0.02\pm 0.02 of d⋆d^{\star} ; the grid is split into two side-by-side halves so that every row from 0.30000.3000 through 0.50000.5000 prints.
d1d_{1}prob.95% CId1d_{1}prob.95% CI
0.30000.000[0.000, 0.019]0.41750.000[0.000, 0.019]
0.32000.000[0.000, 0.019]0.41800.000[0.000, 0.019]
0.34000.000[0.000, 0.019]0.41850.000[0.000, 0.019]
0.36000.000[0.000, 0.019]0.41900.000[0.000, 0.019]
0.38000.000[0.000, 0.019]0.41950.000[0.000, 0.019]
0.39750.000[0.000, 0.019]0.42000.000[0.000, 0.019]
Appendix
Table 11 : Finite-horizon backlog-growth probability, C=mk=12000C=mk=12000 (200 replications per point; Wilson 95% CIs; fluid threshold d⋆=0.4175d^{\star}=0.4175 ; fluid growth threshold dfluid=0.4217d^{\mathrm{fluid}}=0.4217 ; DES step μ1Δt=0.083\mu_{1}\Delta t=0.083 ). Grid: coarse spacing 0.020.02 plus refined rows at spacing 0.00050.0005 within ±0.02\pm 0.02 of d⋆d^{\star} ; the grid is split into two side-by-side halves so that every row from 0.30000.3000 through 0.50000.5000 prints.

Explore similar work

CardsList

  1. The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs

    Jun 2, 2026Xu Wan, Speed Zhu, Jianwei Cai +4Token Budget AllocationEconomies

    1Zhejiang University · 2Tecent HY Team · 3Peking University

  2. Service-Induced Congestion in Memory-Constrained LLM Serving

    Jun 14, 2026Ruicheng Ao, Jing Dong, Gan Luo +1Large Language Model ServingLarge Language Model Memory

    Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA 02139 · Columbia Business School, Columbia University, New York, NY 10027 · School of Mathematical Sciences, Peking University, Beijing, China

  3. Learning the Cost of Reliable Inference

    Sep 23, 2026Dimitrios Rontogiannis, Ander Artola Velasco, Manuel Gomez RodriguezQuestion-Answering Benchmarks

    Max Planck Institute for Software Systems Kaiserslautern, Germany