As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. While several carefully designed long-context evaluations have recently been released, these evaluations tend to rely on retrieval from one or more sections of the context, which allows nearly all of the context tokens to be disregarded as noise. This represents only one type of task that might be performed with long context. We introduce Oolong, a benchmark of long-context reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions. Oolong is separated into two task sets: Oolong-synth, a set of naturalistic synthetic tasks, where we can easily ablate components of the reasoning problem; and Oolong-real, a downstream setting which requires reasoning over real-world conversational data. Oolong requires models to reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations. Even frontier models struggle on Oolong, with GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text.
Figures & tables
Figure 1: Oolong poses questions that require performing a multi-step information aggregation process to determine the solution. Oolong -synth uses ICL-based tasks, which could be easily decomposed and solved iteratively, as a proxy for real-world aggregation tasks over long inputs. Oolong -real poses challenging information aggregation questions over transcripts from live-action Dungeons & Dragons shows, which can not be easily decomposed into component pieces.
Real vs synthetic data
Aggregation capabilities
Measurability
Dataset
Realistic inputs
Realistic questions
Relevant text matches distractors
Multi-step reasoning
Numeracy
Must use full input
Easy to eval
MRCR
✓
✗
✓
✓
✓
✗
✓
BABILong
✗
✓
✗
✓
✗
✗
✓
RULER
✗
✗
✗
✝
✝
✝
✓
HELMET Summ
✓
✓
✓
✓
✗
✓
✗
HELMET LongQA
✓
✓
✓
✓
✗
✝
✓
Table 1: Comparing long-context reasoning benchmarks. Oolong measures aggregation capabilities in controllable, realistic, and simple-to-evaluate settings.
Figure 2: Scores by context window length for Oolong -synth and Oolong -real. Here we show only the best model for each model family; see Appendix B for full results.
Figure 3: Does reasoning effort correlate with task performance?
Figure 4: Comparison on Oolong -synth: we provide the gold labels in the input. This leads to a consistent but small improvement.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Oolong -synth
Oolong -real
Model
Avg.
8K
16K
32K
64K
128K
Avg.
55K
118K
175K
Gemini-3-Pro
78.69
90.94
87.61
81.94
70.55
62.42
57.93
64.71
55.01
54.07
GPT-5
70.75
85.56
84.45
76.12
61.24
46.36
47.00
58.74
45.72
36.53
Gemini-2.5-Pro
55.29
88.13
69.84
56.83
36.56
25.06
52.16
60.04
49.36
47.07
o3
62.37
86.80
79.52
63.23
44.86
37.45
36.71
50.57
33.57
25.99
GPT-5-mini
63.68
85.13
77.65
64.64
50.14
40.85
34.55
49.86
29.90
23.89
Appendix
Table 2: Oolong results on a number of strong models. Nearly all models we test support at least 200K context; thus, we report Oolong scores as an average over scores on 8K-175K inputs. Oolong -synth is a more information-dense task than Oolong -real; the ranking of some models shifts slightly between the two settings. Models are sorted by the average score between the two splits. – marks a context length beyond the maximum for that model; such lengths count as 0 in the average, which we denote with † .
Figure 5: Scores by context window length for Oolong -synth and Oolong -real for closed-weight models.
Figure 6: Scores by context window length for Oolong -synth and Oolong -real for open-weight models.
Oolong -synth
Oolong -real
Model
Avg.
8K
16K
32K
64K
128K
Avg.
55K
118K
175K
Gemini-3-Pro
78.69
90.94
87.61
81.94
70.55
62.42
57.93
64.71
55.01
54.07
GPT-5
70.75
85.56
84.45
76.12
61.24
46.36
47.00
58.74
45.72
36.53
Gemini-2.5-Pro
55.29
88.13
69.84
56.83
36.56
25.06
52.16
60.04
49.36
47.07
o3
62.37
86.80
79.52
63.23
44.86
37.45
36.71
50.57
33.57
25.99
GPT-5-mini
63.68
85.13
77.65
64.64
50.14
40.85
34.55
49.86
29.90
23.89
Appendix
Table 3: Oolong results on a number of strong models. Nearly all models we test support at least 200K context; thus, we report Oolong scores as an average over scores on 8K-175K inputs. Oolong -synth is a more information-dense task than Oolong -real; the ranking of some models shifts slightly between the two settings. Models are sorted by the average score between the two splits. – marks a context length beyond the maximum for that model; such lengths count as 0 in the average, which we denote with † .
Figure 7: Comparison across reasoning levels on Oolong -real.
Dataset
Task
# Labels
Input Len
Spam ( Almeida et al., 2011 )
SMS spam classification
2
57
TREC-QC-coarse ( Li and Roth, 2002 ; Hovy et al., 2001 )
Table 4: Datasets used for Oolong -synth. † indicates cases where we combined similar labels to produce a smaller label-set for Oolong . The input length is the average per-instance length (in Llama 2 tokens), including our added date and user metadata.
Dataset
% Removed
Spam
0.635%
TREC-QC-coarse
0.048%
AGNews
0.026%
App Reviews
0.051%
Formality
0.108%
IMDB
0.042%
Appendix
Table 5: Only a small fraction of examples are removed during validation for each source dataset.
Input
Label
Passed validation?
‘Virtual Girlfriend’ Demands Gifts HONG KONG - She needs to be coddled with sweet talk and pampered with gifts, but you’ll never see her in the flesh. A Hong Kong company has developed a “virtual girlfriend” for new cell phones with video capability…
World
✗
Court Hears Case of Brain Damaged Woman (AP) AP - The Florida Supreme Court questioned lawyers Tuesday about the extent of the power handed to Gov. Jeb Bush under a law that let him order the reinsertion of a brain-damaged woman’s feeding tube.
Sci/Tech
✗
Rooney backs fiercesome threesome (AFP) AFP - Wayne Rooney believes the three-pronged attack of himself, Michael Owen and Jermain Defoe can put England on the fast track to the World Cup finals.
World
✗
From mouths of babes: What’s hot, what’s not How hot are the “Hot Dozen” toys? To find out, The Boston Globe put six toys from the 2004 Toy Wishes magazine Hot Dozen list in front of 13 kids from the Charlestown Boys and Girls Club.
Business
✗
Crematory operator to get 12 years Ray Brent Marsh, who is to enter the plea Friday, had faced up to 8,000 years in a case that shocked the nation two years ago when investigators found hundreds of corpses at his rural northwest facility.
Business
✗
Vending Machines Making Room for Healthy Products WASHINGTON (AP) – The typical vending machine fare consists of chocolate bars and potato chips, leaving few options for people seeking low-calorie or low-salt snacks. That is changing now as companies develop markets for products they expect to satisfy both nutritionists and consumers…
Sci/Tech
✓
Appendix
Table 6: AGNews examples that failed and passed validation.
Input
Label
Passed validation?
Kill Me..
positive
✗
Used to be great app. But since the last 7 updates always get - Authentication failed for: x@gmail.com
positive
✗
App force closing on changing THEME PLZ FIX!
positive
✗
You can’t uninstall & I don’t like my space being used. But am glad for the blind & deaf
positive
✗
GOOGLE TALKBACK Read carefully & then give at least a good reviews this app is made only for persons who got disability such as a blind person & if you want to get rid this simply just ROOT your device dumb head!!
negative
✗
This thing is great. I am at the bottom of the learning curve but in the few minutes I have played with it it seems like it will be easy to learn and to use. Just what I hoped for.
positive
✓
Appendix
Table 7: App Reviews examples that failed and passed validation.
Input
Label
Passed validation?
The Unit secret agent has signed on to appear in three episodes as a businessman being shown some houses - and, presumably, one bedroom in particular - by our randy realtor heroine.
formal
✗
Barely 12, with large brown eyes and stick-like arms, Fandi is 3 years older than his brother – in his eyes almost a man.
formal
✗
D-Lister Avril Lavigne appears on the cover of Z-List magazine Savvy this month.
formal
✗
ORHS is going through it now, but from all I can perceive the district is out for the quick fix, chop off the principal, instead of really trying to assist the students and dig to the bottom to find the truth.
formal
✗
Little Sally Draper (Kiernan Shipka) is a ”Patty Duke”-era girl - indeed, ”Mad Men” is currently set during the month ”The Patty Duke Show” premiered in 1963 - and on a ”Patty Duke”-like show, she’d just be a lispy Shirley Temple doll with a crush on Daddy.
formal
✗
I have tried everything possible to attract business.
formal
✓
Appendix
Table 8: Formality examples that failed and passed validation.
Input
Label
Passed validation?
Good movie, very 70s, you can not expect much from a film like this. Sirpa Lane is an actress of erotic films, a nice body but nothing exceptional. Not demand a lot from these films are light years away from the movies today, the world has changed incredibly. The plot is simple and the actors not extraordinary.
positive
✗
This film Evil Breed: The legend of samhain contains very little thought or effort. It is ridiculed with specs of ultra fast ”slasher” style death and plain disgusting acts of death. The acting was rated a D as the actors show very little ability, and the stupidity of them in the film is too questionable.
positive
✗
The movie ”Holly” may make the audience want to donate money towards organizations that improve the life for these poor youngsters, but the film’s dramatic weaknesses may reduce its chances of being seen by enough people to make a difference. Overall, I think the concept is better as a documentary and it was not as touching as a movie.
positive
✗
Although this film put Davis on the map due to her brilliantly intense performance, this film is strangely unsatisfying to me as a whole. What I cannot fathom for the life of me is just how or why Phillip would take the constant abuse this tramp constantly dishes out towards him.
positive
✗
This film is just plain horrible. John Ritter doing pratt falls, 75% of the actors delivering their lines as if they were reading them from cue cards, poor editing, horrible sound mixing, and a plot that really goes nowhere. If I could sum this film up in one word, that word would be: Suckotrocity
negative
✓
Zentropa has much in common with The Third Man, another noir-like film set among the rubble of postwar Europe. Like TTM, there is much inventive camera work. There is an innocent American who gets emotionally involved with a woman he doesn’t really understand, and whose naivety is all the more striking in contrast with the natives. But I’d have to say that The Third Man has a more well-crafted storyline. Zentropa is a bit disjointed in this respect. Perhaps this is intentional: it is presented as a dream/nightmare, and making it too coherent would spoil the effect. This movie is unrelentingly grim–“noir” in more than one sense; one never sees the sun shine. Grim, but intriguing, and frightening.
positive
✓
Appendix
Table 9: IMDB examples that failed and passed validation. Only eight examples are shown because of the example length.
Input
Label
Passed validation?
Draupadi’s eyes were diamonds. ↔ Draupadi’s eyes were beautiful.
correct
✓
The faculty meeting was an easy breeze ↔ The faculty meeting was very easy and relaxing
correct
✓
The house was a tomb. ↔ The house was big.
incorrect
✓
I had already planted the idea in her mind. ↔ I had already scared her about the idea.
incorrect
✓
It is sad to observe the fruits of ignorance. ↔ It is sad to observe the effects of ignorance.
correct
✓
Appendix
Table 10: Metaphors examples that passed validation. No examples failed validation for this dataset.
Input
Label
Passed validation?
You’re safe. → You have nothing to worry about.
neutral
✗
asks Burton, cupping his ear. → Burton wanted to ask what was going on, but he couldn’t do much of anything while bound and gagged.
contradiction
✗
I “poison my dearest Emily!” → How could you think I would poison Emily?
entailment
✗
When he finally succeeded, after a prolonged siege and heavy losses, he punished the local population by cutting off the noses and lips of all men except those who played wind instruments. → All men who played wind instruments were tasked with helping cut off people’s noses.
neutral
✗
They died slowly, their eyes bulging and faces turning blue. → They died asphyxiated.
neutral
✗
well you see that on television also → You can see that on television, as well.
entailment
✓
Appendix
Table 11: MultiNLI examples that failed and passed validation.
Input
Label
Passed validation?
No care is anything that serves as an enticement.
True
✗
No care may be anything that serves as an enticement.
True
✗
A loud utterance is an appropriate definition of produce in no context.
True
✗
Complete attention is an appropriate definition of candidate in no context.
True
✗
No stress is a message received and understood.
True
✗
Action refers to a military engagement.
True
✓
Appendix
Table 12: Negation examples that failed and passed validation.
Input
Label
Passed validation?
Go chase after her and run her over while she’s crossing the street
ham
✗
i want to grasp your pretty booty :)
ham
✗
No da if you run that it activate the full version da.
ham
✗
i am seeking a lady in the street and a freak in the sheets. Is that you?
ham
✗
Should i send you naughty pix? :)
ham
✗
WINNER!! As a valued network customer you have been selected to receive a £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.
spam
✓
Appendix
Table 13: Spam examples that failed and passed validation. The examples that failed validation were almost exclusively ham messages with sexual content; the selected messages are the least inappropriate of the examples that failed validation.
Input
Label
Passed validation?
What explosive do you get by mixing charcoal, sulfur and saltpeter?
entity
✗
What is the procedure called for drilling a hole in your skull to achieve a higher consciousness?
entity
✗
Name 11 famous martyrs.
human being
✓
What’s the Olympic motto?
description and abstract concept
✓
What is the highest waterfall in the United States?
location
✓
What does the abbreviation AIDS stand for?
abbreviation
✓
Appendix
Table 14: TREC-coarse-Q examples that failed and passed validation. Only two examples failed validation.
Input
Label
Passed validation?
Is there a God? The question to end all questions, and begin them.
Business & Finance
✗
why do we need to lie?
Health
✗
try to type the word supercalifragilisticexpialidocious. 20 times fast you can only make 10 mistakes? This is for fun!
Education & Reference
✗
do you think it is okay to tell a lie?
Education & Reference
✗
where can you purchase cesium carbonate?
Science & Mathematics
✗
What are good sources to find out about new gospel artists? Is there a site that focuses primarily on gospel?
Entertainment & Music
✓
Appendix
Table 15: Yahoo examples that failed and passed validation. One class of examples that failed validation, not demonstrated here, are sexually explicit questions labeled as seemingly random categories (likely from spam posters on the original Yahoo Answers forum).
Counting
In the above data, which of the labels is the most common? Give your final answer in the form ‘label: answer’ where answer is one of the labels: {label_list}.
In the above data, which of the labels is the least common? Give your final answer in the form ‘label: answer’ where answer is one of the labels: {label_list}.
In the above data, is label ‘{A}’ more common, less common, or the same frequency as label ‘{B}’? Give your final answer in the form ‘Answer: {A} is [X]{B}’, where [X]is ‘more common than’, ‘less common than’, or ’same frequency as’.
In the above data, how many data points should be classified as label ‘{label}’? Give your final answer in the form ‘Answer: number’.
Appendix
Table 16: Counting questions used in the Oolong -synth dataset.
User
In the above data, which user is represented most often? Give your final answer in the form ‘User: [X]’, where [X]is the user ID.
In the above data, which user is represented the second most often? Give your final answer in the form ‘User: [X]’, where [X]is the user ID.
For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user is represented most often? Give your final answer in the form ‘User: [X]’, where [X]is the user ID.
For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user is represented the second most often? Give your final answer in the form ‘User: [X]’, where [X]is the user ID.
For the following question, only consider the subset of instances that are associated with user IDs {user_names}. Among instances associated with these users, {any of the Counting questions above}
For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user has the most instances with the label {label}? Give your final answer in the form ‘User: [X]’, where [X]is the user ID.
Appendix
Table 17: User questions used in the Oolong -synth dataset.
Timeline
In the above data, which date is represented most often? Give your final answer in the form ‘Date: [X]’, where [X]is the date in the format MM/DD/YYYY.
In the above data, which date is represented most often? Give your final answer in the form ‘Date: [X]’, where [X]is the date in the format MM/DD/YYYY.
In the above data, which date is represented the second most often? Give your final answer in the form ‘Date: [X]’, where [X]is the date in the format MM/DD/YYYY.
In the above data, how many dates are represented exactly {n} times? Give your final answer in the form ‘Answer: [X]’, where [X]is the number of dates represented exactly {n} times.
In the above data, was label ‘{key}’ more common, less common, or the same frequency before {time}, as compared to after {time}? Give your final answer in the form ‘Answer: {key} is [X]before {time}’, where [X]is ‘more common’, ‘less common’, or ’the same frequency’.
In the above data, was label ‘{key}’ more common, less common, or the same frequency before {time}, as compared to after {time}? Give your final answer in the form ‘Answer: {key} is [X]before {time}’, where [X]is ‘more common’, ‘less common’, or ’the same frequency’.
Appendix
Table 18: Timeline questions used in the Oolong -synth dataset.
Rolls
Total number of rolls in this episode? (counting)
Total number of rolls by the character {character name} in this episode? (counting, character)
Total number of rolls by the player {player name} in this episode? (counting, player)
Total number of rolls of type {roll type} in this episode? (counting, roll type)
Number of rolls of natural value {roll value} in this episode? (counting, roll value)
In this episode, what percentage of rolls were of value {roll value}? round to the nearest integer. (counting, roll value)
Appendix
Table 19: Question types used in the Oolong -real dataset (single episode). For reference, we provide the question types in parentheses.
Rolls
What is the cumulative total of rolls by the end of episode {episode index}? Count the number of rolls and not the values of the rolls. (counting)
What is the cumulative total of rolls by the character {character name} at the end of episode {episode index}? Count the number of rolls and not the values of the rolls. (counting, character)
Total number of rolls across all the episodes? (counting)
Total number of rolls by the character {character name} across all episodes? (counting, character)
Total number of rolls by the player {player name} across all episodes? (counting, player)
Total number of rolls of type {roll type} across all episodes? (counting, roll type)
Appendix
Table 20: Question types used in the Oolong -real dataset (multi-episode). For reference, we provide the question types in parentheses.
Spells
What is the cumulative total of spells cast by the end of episode {episode index}? (counting, indexing)
What is the first spell cast in the episode {episode index}? (enumeration, indexing)
What is the second spell cast in the episode {episode index}? (enumeration, indexing)
What is the third spell cast in the episode {episode index}? (enumeration, indexing)
List the first spell cast in each episode? Return a comma separated list. (enumeration)
List the last spell cast in each episode? Return a comma separated list. (enumeration)
Appendix
Table 21: Question types used in the Oolong -real dataset (multi-episode). For reference, we provide the question types in parentheses.
Figure 10: The performance trend for models by type of answer and type of task on Oolong -synth.
Figure 11: Scores by context window length for various question types in Oolong -real. Some questions are only relevant for multi-episode settings.
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness. Notably absent is a systematic way to probe how models perform as we scale up the difficulty of tasks along various axes. We address this gap by proposing PredicateLongBench, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints (e.g., lexicographic ordering), drawn from a broader predicate class. The central innovation of our benchmark is the identification and systematic exploration of multiple different axes of difficulty which test multiple aspects of long context understanding. We provide two complementary generation pipelines - a fully synthetic setup using random word-like strings, and a real-world setup that samples words from natural documents while preserving their distributional properties. We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes, demonstrating the utility of our benchmark in understanding the limitations of current long-context capabilities. Furthermore, the tasks in PredicateLongBench, though challenging, are conceptually simple and do not require LLM-based generations or judges.
Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long contexts using gradient updates at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard long-context finetuning, achieving average absolute performance gains of up to 20% for Qwen-2.5 (0.5B & 7B) on synthetic and real-world long-context reasoning. PERK also maintains its advantages across model scales and families. Compared to specialized long-context LLMs, PERK matches or surpasses their performance. Finally, our analyses show PERK is more robust to reasoning complexity, length extrapolation, and the positions of relevant information in contexts. https://perk-long-context.web.app
Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.