You don't need long context

Peter Belcak; Saurav Muralidharan; Jindong Jiang; Yonggan Fu; Wonmin Byeon; Hanrong Ye; Ryan Angilly; Ali Taghibakhshi

cs.AINov 3, 2026

You don't need long context

Authors: Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi,

Abstract

Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.

Figures & tables

Figure 1: The cost of long context. Dollar cost of generating

1

M output tokens on an existing input KV cache as a function of the context window size, obtained by converting the measured peak decode throughput of each deployment at on-demand H100 prices (

\

3.85

per GPU-hour, Nebius, Sept. 2026). Input request lengths are drawn uniformly at random. GLM 5.2 and Kimi K2.6 are served with INT4 weights and BF16 activations, Qwen 3 Coder Next is served in BF16. Serving at a context window size of

256

K rather than

K costs

3.6\times

more per generated token for GLM,

5.6\times

more for Kimi, and

7\times$ more for Qwen.

Figure 2: The burden of long context. The token generation latency, token throughput, and memory footprint of several open-source models measured on an NVIDIA DGX Spark plotted as a function of input context length. For several models, later tokens take approximately 2x longer to generate than earlier tokens and require a larger memory footprint.

Literary works			Model context window sizes
Work	Words	Tokens	Model	Words	Tokens
The Old Man and the Sea [ 21 ]	$\sim 27$ K	$\sim 37$ K	MiniMax M2.7 [ 31 ]	$\sim 150$ K	$200$ K
The Great Gatsby [ 15 ]	$\sim 48$ K	$\sim 64$ K	Claude Opus 4.5 [ 2 ]	$\sim 150$ K	$200$ K
Qur’an (Saheeh Int’l translation) [ 38 ]	$\sim 156$ K	$\sim 207$ K	Gemma 4 31B [ 16 ]	$\sim 192$ K	$256$ K
King James New Testament [ 26 ]	$\sim 181$ K	$\sim 241$ K	Kimi K2.6 [ 32 ]	$\sim 192$ K	$256$ K
Great Expectations [ 12 ]	$\sim 184$ K	$\sim 245$ K	Qwen 3 Coder Next [ 9 ]	$\sim 197$ K	$262$ K

Table 1: Lengths of literary works against model context window sizes. Word counts use the cited English editions. Word-token conversions use approximately

1.33

tokens per English word, while actual counts vary with text and tokenizer. Models and their context window sizes are detailed in Section 2.3 .

Figure 3: Context-management regimes. The three context-management regimes considered in this work: Full-History Accumulation (FHA), End-of-Conversation Compaction (EoCC), and Single-Action Compaction (SAC).

Figure 4: Decode throughput collapses as the context window size grows. Peak sustained decode throughput (tokens/s, left axis) for open-weight coding models served under vLLM on NVIDIA H100 GPUs, as a function of the configured context window size. GLM 5.2 and Kimi K2.6 are served with INT4 experts and BF16 activations (Kimi additionally with an FP8 KV cache), Qwen 3 Coder Next in BF16. The right axis re-expresses the same curve as a speedup over the

256

K operating point. Reducing the context window size from

256

K to

16

K buys

3.6\times

(GLM 5.2),

5.6\times

(Kimi K2.6), and

7\times

(Qwen 3 Coder Next) more decoded tokens per second on identical hardware.

Figure 5: Long-horizon results as a function of total context limit. Top: accuracy changes in percentage points relative to each model’s full-history baseline; zero indicates unchanged accuracy. Bottom: baseline estimated optimized retail cost divided by the compaction configuration’s estimated optimized retail cost;

1\times

indicates unchanged cost. Orange/blue identify Claude Opus 4.5 and GPT 5.6 Sol, respectively, and stars mark the baselines. Cost estimates use the model in Appendix B .

Figure 6: Long-input results as a function of total context limit. Top: accuracy changes relative to each model’s full-input baseline. Bottom: optimized retail cost savings, using the model in Appendix B .

Figure 7: LongBench v2: conversation length versus document length. Claude Opus 4.5, compaction at

16

K /

12

K. The document-length axis is logarithmic and the conversation-length axis is linear. The dashed curve marks

y=x

Figure 8: LOONG: conversation length versus document length. Claude Opus 4.5, compaction at

48

K /

40

K (1,600 questions). The document-length axis is logarithmic and the conversation-length axis is linear. The dashed curve marks

y=x

Long-Context Reasoning in Language Models

See all 51 papers

The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management
May 21, 2026Binqi Shen, Lier Jin, Hanyu Cai +2LLM Evaluation Cost-Aware Inference
Department of Industrial Engineering and Management Sciences Northwestern University Evanston, USA · Fuqua School of Business Duke University Durham, USA · Department of Engineering Carnegie Mellon University Pittsburgh, USA +1
Trained Agentic Context Management
Oct 1, 2026Bryce SandlundLong-Context Language Modeling Long-Context Reasoning in Language Models
In-Model Context Compression with Agentic Soft Tokens for Long-Horizon LLM Agents
Oct 30, 2026Xuning He, Kai Fan, Runyu Zhang +4Context Compaction for LLM Agents LLM Context Compression
Shanghai Jiao Tong University · Nanyang Technological University · Nankai University +2

May 21, 2026J/K move · Enter open · S save

The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management

Binqi Shen, Lier Jin, Hanyu Cai, Lan Hu+1

Department of Industrial Engineering and Management Sciences Northwestern University Evanston, USA · Fuqua School of Business Duke University Durham, USA · Department of Engineering Carnegie Mellon University Pittsburgh, USA+1

LLM Evaluation Cost-Aware Inference LLM Context Management

Open paper AskPDF arXiv

Large language models (LLMs) increasingly rely on long-context processing, but expanding context windows introduces substantial computational and financial costs. Existing context reduction approaches, including retrieval and memory compression methods, are typically evaluated using performance and efficiency metrics independently, limiting systematic comparison and deployment-aware decision-making. This paper introduces The Efficiency Frontier, a unified framework for cost--performance optimization in LLM context management. The framework models context strategy selection as a deployment-aware optimization problem that jointly accounts for task performance, token cost, and preprocessing reuse through amortized cost modeling. Unlike existing evaluations that compare methods in isolation, the proposed framework enables decision-oriented analysis. It identifies when different context management strategies become preferable under varying operational conditions. Experiments on HotpotQA reveal distinct operational regimes and transition boundaries between retrieval-based and preprocessing-based strategies. Results show that deployment-aware optimization reduces effective token usage by approximately 25% at comparable performance, enabling more cost-efficient deployment of large language model systems, while amortized memory compression achieves over 50% lower token cost relative to full-context prompting in higher-performance settings. Overall, the proposed framework provides a principled and practical foundation for evaluating and deploying scalable, efficient, and sustainable LLM systems across enterprise, scientific, and public-sector applications.

Abstract

Long-Context Reasoning in Language Models

Explore similar work

The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management

Trained Agentic Context Management

In-Model Context Compression with Agentic Soft Tokens for Long-Horizon LLM Agents