Long-context language models (LCLMs) power agents that act over long trajectories, from improving a codebase to running research experiments to forecasters that condition on lengthy histories. These applications draw on three distinct long-context capabilities: recall, where the answer is a fact present in the context; state tracking, where the outcome can be computed by performing a series of transformations to a specified state; and pattern learning, where the goal is to make a prediction by sampling from a distribution inferred from many observations. Existing benchmarks primarily focus on the first capability. In this work, we introduce LEAPS (Long-context Evaluation of Adaptive Pattern learning and State tracking), a collection of 12 novel datasets, spanning agentic workflows, behavioral modeling, and forecasting of market and world events. By construction, LEAPS is resistant to data contamination, extends to millions of input tokens, and uses reliable metrics. We evaluate 30 API and open models from 8K to 512K input tokens. LEAPS distinguishes models that perform near perfectly on recall tasks, finding that such models' state tracking and pattern learning abilities can vary significantly and degrade much more dramatically with increasing input lengths--even for pattern learning, where more observations should make learning a theoretically easier task. Moreover, thinking models often underperform their base counterparts, exhausting their generation budget or oversimplifying the task. We hope LEAPS can guide the development of the next generation of LCLMs.
Figures & tables
Figure 1: In contrast to many previous long-context benchmarks that revolve around recall , we study two new capabilities in a long-context setting, state tracking and pattern learning .
State tracking
Description
Source
Cup swapping
Track positions of balls as cups are swapped.
Syn
Coins
Track the count of piles as coins are added, removed, and swapped.
Syn
Orderbook
Track buy and sell orders in a multi-actor market.
Syn
ML Dev
Follow parallel model developments and generate subsequent commands.
Syn
Commit re-ordering
Identify pairs of conflicting commits.
Real
Code tracking
Track code files across commits and changes.
Real
Table 1: Description and data source of each dataset (synthetic or real).
Figure 2: Results of API models at input length 512K in percentages. Compared with recall tasks, consisting of subsets of RULER and HELMET, Leaps distinguishes models on two new axes.
Figure 3: Results of selected open-weight models at 128K input length. Even the strongest open models struggle with both state tracking and pattern learning.
Figure 4: Performance is shown relative to each model’s performance at 8K input tokens. Models observe substantially more degradation with increasing length on state tracking and pattern learning compared to recall. Model performance drops by up to 80% from 8K to 128K on Leaps .
Figure 5: Comparison of base and post-trained models. Results shown in z-scores to standardize averaging across datasets and comparisons across input lengths.
Figure 6: Results of thinking models with generation budgets from 8K to 32K, on input lengths L={64K,128K} . Performance may improve going from 8K to 16K output but plateaus after.
Figure 7: Correlation matrix at L=96K , calculated across 23 models: API models and open-weight non-thinking models. The standard error of Spearman ρ=0 is 0.209 .
Qwen 3.5 30B Cups reasoning
GPT-OSS-120B Kaggle reasoning
The user wants to know the final positions of three balls. ⋯ I need to go through the list of “On table X, swap cups Y and Z” instructions and filter for those that apply to Table 58. 1. On table 58, swap cups 3 and 2. 2. On table 58, swap cups 3 and 4. ⋯ 816. On table 58, swap cups 5 and 2. 817. On table 58, swap cups [run out of generation budget]
maybe the responder is simply the last value of “Continuous data”? ⋯ Take first example: ⋯ responder equals last continuous data value. Check second example: answer (C) responder 0.0412. Continuous data ends with “… 0.0412”. Yes. So responder = last element of continuous data. The continuous data line ends with: “… -0.3101, 2.3888, 1.3782, 0.7434”. Yes last value is 0.7434. Thus responder = 0.7434. Thus output: (E) [Gold answer: (C)]
Table 2: Examples of unsuccessful model reasoning at 128K input tokens. Qwen 3.5 30B exhausts its generation budget while explicitly tracking swaps in a Cups sample (left), while GPT-OSS-120B assumes a spurious pattern and returns an incorrect answer on a Kaggle sample (right).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Repo
URL
License
Fabric
https://github.com/danielmiessler/Fabric
MIT
PowerShell
https://github.com/PowerShell/PowerShell
MIT
Vibe-Trading
https://github.com/HKUDS/Vibe-Trading
MIT
agency-agents
https://github.com/msitarzewski/agency-agents
MIT
astropy
https://github.com/astropy/astropy
BSD-3-Clause
biome
https://github.com/biomejs/biome
MIT/Apache-2.0
Appendix
Table 3: Repositories and licenses used for the code tracking and commit reordering datasets.
Figure 8: Difference in performance on variants of coins and orderbook. For coins, we increase the probability of swapping as opposed to adding/removing coins from 10% to 90%. For orderbook, we use reverse chronological order.
Figure 9: Results of API models at 512K input length, compared with previous synthetic benchmarks, recall (subsets of RULER and HELMET Recall), GraphWalks, MRCR, and LongBench v2. Recall is mostly saturated. Despite the variation in performance, GraphWalks, MRCR, and LongBench v2 similarly only test for recall capabilities by design. Leaps is conceptually distinct from previous benchmarks.
Recall
GraphWalks
MRCR
Avg.
Model
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
Llama-3.1-8B-Base
99.7
99.6
98.3
88.7
54.0
6.4
2.3
1.6
1.1
2.5
0.0
0.0
0.0
0.0
0.0
58.8
57.6
56.6
51.0
31.6
Llama-3.1-8B-Inst
99.5
98.8
97.5
97.3
72.5
2.5
0.5
8.1
12.8
4.7
38.3
24.5
21.1
32.0
24.6
63.0
60.1
61.0
63.8
46.3
Llama-3.1-70B-Base
100.0
99.8
99.0
95.7
49.1
30.1
21.3
12.2
5.4
2.5
0.0
0.0
0.0
0.0
0.0
65.8
63.1
60.1
56.2
28.8
Llama-3.1-70B-Inst
99.9
99.9
99.1
97.9
37.8
24.2
47.9
42.3
33.6
7.5
57.2
33.9
28.1
27.9
12.0
72.2
75.6
72.7
69.6
25.5
ProLong-8B-Base
96.8
98.5
97.7
96.6
95.1
10.5
7.7
4.4
0.5
0.1
0.0
0.0
0.0
0.0
0.0
58.3
58.5
57.1
55.3
54.4
Appendix
Table 4: Results of all models on previous benchmarks: Recall is a mix of RULER and HELMET Recall, GraphWalks, MRCR, and the average across these categories. LongBench v2 is excluded as it does not support fine granularity of input lengths.
Figure 10: Results in z-scores averaged across each category from 8K to 128K tokens. We first calculate the mean and variance for a dataset across 30 evaluated models and 6 lengths and normalize all scores into the z-score to reduce bias for any particular dataset in the category average. Absolute scores and all models are shown in Figure 11 .
Figure 11: Main results across six datasets for each category and input lengths from 8K to 128K tokens for all 30 evaluated models.
dataset
mean
std
Chess
0.118778
0.122106
Code Tracking
0.297739
0.324307
Coins
0.623230
0.301486
Coins Swap
0.543219
0.283638
Commit Reorder
0.293655
0.334649
Cups
0.257087
0.227134
Appendix
Table 5: Means and standard deviations across all models on input lengths from 8K to 128K.
Figure 12: Average absolute performance on each capability. Recall tasks are saturated, and models observe relatively limited degradation, whereas Leaps tests for new capabilities that are still not saturated and experience more degradation.
Figure 13: Results of thinking models when thinking finishes. The default generation budget is up to 16K tokens.
We study long context language models. Instead of training long context natively, or designing a long context harness, we train a model over the simplest possible harness: a tool to call itself with any specified prompt and a tool to read tokens in a range from the input context. We finetune Qwen3.6-35B-A3B on a diverse synthetic dataset using this harness. With only 8,000 tokens of context, our small model is as strong as GPT-5.4 with 1M tokens of context on the OOLONG-synth benchmark when document length exceeds 40K tokens.
Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.
We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.
Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin +10
University of Washington · Meta Superintelligence Labs · MIT +1