Measuring and Charging for Retained Context in Long-Context LLM Services
Organizations: School of Statistics and Data Science, Nanjing Audit University, Nanjing 211815, Jiangsu, China
Abstract
Long-context large language model (LLM) services charge for repeated context, yet billing weights need not reflect its contribution to answer quality. We present a three-layer framework separating content relevance, observable proxies and auditable positional weights. Context-ablation experiments show that relevance dominates position and that distance-decay families do not outperform a relevance-only baseline. A measured serving stack shows that cache hits reduce repeated-prefix compute cost by over two orders of magnitude at 65,536 tokens, while cold resends constrain sustainable discounts. Cross-provider measurements separate cache value from tokenisation and list-price differences. Calibrated simulations reduce the 99th percentile of session charges by 31-42%, although revenue-matched nonlinear tariffs achieve similar reductions and median charges increase. The framework provides a bounded per-token accounting rule and measured deployment constraints, rather than a validated utility estimator.
Figures & tables
| Metric | Hard truncation | Proactive compression |
|---|---|---|
| Mean | -11.4 | -4.6 |
| Median (P50) | +58.9 | +58.9 |
| P95 | -25.2 | -21.9 |
| P99 | -42.0 | -32.8 |
| Coefficient of variation | -40.5 | -30.3 |
| Metric | Cache tariff | Context-weighted | Change (%) |
|---|---|---|---|
| Mean (USD) | 0.055 | 0.048 | -11.4 |
| Standard deviation | 0.154 | 0.081 | -47.3 |
| Median (P50) | 0.014 | 0.022 | 58.9 |
| P95 | 0.245 | 0.183 | -25.2 |
| P99 | 0.618 | 0.359 | -42.0 |
| Coefficient of variation | 2.812 | 1.673 | -40.5 |
| Schedule (cached/list) | Crossover | 200-round | Mean | P99 | CV | Grid |
|---|---|---|---|---|---|---|
| (rounds) | saving (%) | (%) | (%) | (%) | cheaper (%) | |
| DeepSeek flash (2%) | 182 | 3.3 | +10.3 | -30.9 | -42.6 | 22 |
| DeepSeek v4-pro (3.3%) | 164 | 7.4 | +7.4 | -32.8 | -43.5 | 23 |
| Qwen 3.8-flash (12.5%) | 111 | 27.8 | -11.4 | -42.0 | -40.5 | 33 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Baseline value | Robustness values or distribution |
|---|---|---|
| Interaction rounds | 200 | Fixed at 200 in the grid |
| New input per round | 250 tokens | 50, 581, 1,000 or 2,000 |
| Retained output per round | 750 tokens | 1,120, 1,200, 1,500 or 5,000 |
| Context window | 128,000 tokens | 32,000; 128,000 |
| Cache-hit rate | 0.80 | 0.50; 0.80; 0.95 |
| Utility parameter | 0.95 | 0.70; 0.80; 0.90; 0.95; 0.98; 0.99 |
| Category | DeepSeek | DeepSeek | Qwen | |
|---|---|---|---|---|
| flash (idle/peak) | v4-pro (idle/peak) | 3.8-flash | ||
| Uncached input | 1 / 2 | 4.5 / 9.0 | 0.8 | CNY / million tokens |
| Cached input | 0.02 / 0.04 | 0.15 / 0.30 | 0.1 | CNY / million tokens |
| Output | 4 / 8 | 13.5 / 27.0 | 2.7 | CNY / million tokens |
| Scenario | Parameter | Cache cost | Weighted cost | Change |
|---|---|---|---|---|
| Agent, 100 rounds | No tool/reasoning tokens | 0.289 | 0.314 | 8.6% |
| Agent, 100 rounds | 5,000 persistent tool tokens | 0.473 | 0.931 | 96.8% |
| Agent, 100 rounds | +5,000 ephemeral reasoning tokens | 0.663 | 1.121 | 69.1% |
| Router, per round | Switch probability 0.75 | 0.00905 | 0.00749 | -17.3% |
| Router, per round | Switch probability 1.00 | 0.01127 | 0.00918 | -18.5% |
| Outcome | Mean/median | Minimum | Maximum |
|---|---|---|---|
| Grid expenditure change (180 combinations) | 36.3% a | -84.1% | 300.3% |
| Seed mean-cost change (20 replications) | -8.4% | -18.8% | 0.2% |
| Seed P95 change (20 replications) | -22.0% | -33.7% | 1.5% |
| Seed P99 change (20 replications) | -40.6% | -53.8% | -19.0% |
| Seed coefficient-of-variation change | -32.5% | -46.4% | -16.4% |
| a Median; seed rows report arithmetic means. Negative values indicate savings or risk reduction. | |||
| 1,200 rounds | 9,600 rounds | ||||
|---|---|---|---|---|---|
| Window | Cache | Weighted | Cache | Weighted | |
| 32K | 0.75 | -19.8 | -10.1 | -20.1 | -10.3 |
| 32K | 0.80 | -17.2 | -7.7 | -17.4 | -7.9 |
| 32K | 0.90 | -10.1 | -1.8 | -10.3 | -1.9 |
| 32K | 1.00 | -5.9 | +1.4 | -6.0 | +1.3 |
| 128K | 0.75 | -34.2 | +12.2 | -35.4 | +12.9 |
| Stakeholder | Required condition | Main outcome |
|---|---|---|
| Model provider | Demand growth exceeds the usage-growth threshold of Section 4 of the main text and cost remains covered; the calibrated analysis requires a demand elasticity near , atypically high for a mature digital service | Sessions, rounds, retention, contribution profit |
| Application developer | Per-round historical charge stabilises and quality is preserved | Mean, CV, P95/P99, CVaR, task quality |
| End user | Portable and auditable context-age/utility metadata | Switching premium, completion rate, provider switching |
LLM Serving
Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.
See all 249 papers