We study a QA curation pipeline for improving long-context complex reasoning in large language models (LLMs). Our approach, π2, constructs high-quality reasoning data through rigorous QA curation: 1) extracting and expanding tables from Wikipedia, 2) from the collected tables together with relevant metadata, generating complex reasoning questions whose answers are automatically determined and validated through dual-path code execution, 3) finally, back-translating chain-of-thoughts solutions grounded in realistic context. Supervised fine-tuning with gpt-oss-20b and Qwen3-4B-Instruct-2507 on π2 yields consistent improvements across four long-context reasoning benchmarks and our alike π2-Bench, with average absolute accuracy gains of +6.25% and +3.37% respectively. Through deeper analyses, we observe that reasoning style contributes little, while faithful reasoning patterns discovered by back translation and grounded realistic long context, as π2 is designed for, are crucial for the improvement. Our code, data, and models are fully open-source at https://github.com/vtpss/pi-squared.
Figures & tables
Figure 1: Tables bridge realistic documents and complex reasoning in open-domain tasks. They represent data across sources , while being compatible with programmatic operations .
Figure 2: π2 curation pipeline . It first 1) collects tables from webpages and expands them with information retrieved from linked documents if conditions are met (indicated by dashed arrows and objects); then 2) from each collected table, generate a complex question together with an executable SQL query by an LLM and verify the SQL-derived answer with an independent LLM-generated Python solution, and finally 3) back-translates reasoning traces grounded in real free-form context from the verified answer.
Question
Answer
Evidence
Among the television roles held by Debbie Reynolds in her career, how many distinct character or guest roles did she perform specifically on shows broadcast by NBC?
3
Debbie Reynolds; Aloha Paradise; Alice (American TV series); etc.
In the 2023 Cricket World Cup, among bowlers who use a Right-arm fast style, which player took the most wickets?
Mohammed Shami
2023 Cricket World Cup; Mohammed Shami; etc.
Based on the 2022 cobalt production and reserves data reported by the USGS, which country had the highest ratio of reserves to annual production?
Australia
Cobalt
Table 1: Samples in π2 . The "Evidence" column shows the evidence Wikipedia page(s). The sample is originated from a table in the first listed page. Our samples cover diverse topics and complexity levels.
In-domain
Out-of-domain
LongSeal
OfficeQA
π2 -Bench
LongBenchV2
Oolong
Avg.
Gemini 3.1 Flash-Lite
64.96
57.14
75.29
58.14
42.38
59.58
Gemini-2.5-Pro
59.84
53.37
72.32
56.3
36.56
55.68
Qwen3.5-35B-A3B-FP8
58.5
55.74
76.97
58.22
39.76
57.84
gpt-oss-120b (low)
42.18
33.88
75
51.06
38.08
48.04
Qwen3-4B-Instruct-2507
29.14
15.12
54.68
36.09
30.51
33.11
Table 2: π2 significantly improves long-context reasoning ability of Qwen3-4B-Instruct-2507 and gpt-oss-20b . We evaluate our models and baselines on in-domain benchmarks LongSeal, OfficeQA, and our π2 -Bench; as well as out-of-domain benchmarks LongBenchV2 and Oolong. For overall comparison, we report the weight-less average performance of models across benchmarks. Bold-faced numbers indicate the best among the group. We confirm that the average improvements brought by SFT on π2 are significant at α=0.01 under t-test (i.e., with 99% confidence level).
Question: According to the WHO, in which year were the highest number of measles cases recorded worldwide? Correct Answer: 1980 Context: (Omitted due to its length. A snapshot of evidence is shown in Figure 5 )
gpt-oss-20b (low) : From the WHO data presented in the documents, the peak in worldwide measles cases occurred in 2019 , with a total of approximately 869,770 reported cases globally. This is higher than any preceding or following year reported in the sources (e.g., 2018 had 343,806 cases, and 2024 had 359,521 cases). Therefore, according to WHO statistics, 2019 was the year with the highest number of measles cases worldwide.
π2 -20b (low) : The WHO’s own global measles surveillance data (published in Document 1) lists the total number of reported measles cases worldwide by year: 1980 – 3,852,242 cases, 1990 – 1,325,074 cases, 2000 – 853,479 cases 2010 – 343,806 cases, 2019 – 869,770 cases, 2020 – 159,073 cases Among these figures, the largest reported number is the 3,852,242 cases recorded for the year 1980 . No other year shown in the official WHO data exceeds this count, and [ 31 words omitted for brevity ]. Therefore, according to the WHO’s own records, the highest number of measles cases worldwide occurred in 1980 .
Table 3: π2 equips models with stronger long-context retrieval and reasoning . We show the response of gpt-oss-20b and π2 -20b (with low reasoning effort) on a sample of LongSeal. The base model fail to reason and retrieve the correct piece of evidence, as it mistakes the phrase "climbed to 869,770 in 2019" as the signal for the correct answer, thus missing out other evidence. On the other hand, π2 -20b successfully identifies all pieces of evidence and reason thoroughly through them to derive the correct answer. That is a core capability targeted by our data curation pipeline.
Question: Who holds the all-time record at the Grammys for the most wins in the album of the year category? Correct Answer: Serban Ghenea Context: (Only show the most important evidence. Omitted remaining text for brevity.) [In Document 1: "Grammy Award for Album of the Year" Wikipedia page, as a document in the context] … Serban Ghenea (as engineer/mixer) is the most frequent winner in this category with five awards. Taylor Swift (as performer); John Hanes (as engineer/mixer); and Tom Coyne and Randy Merrill (as mastering engineers) have won the award four times …
gpt-oss-20b (low) : From the provided context ( Document 1, Document 5, Document 7, Document 9 ) we see that: - Frank Sinatra, Paul Simon, and Stevie Wonder each have three wins. - Taylor Swift has won the award four times: for Fearless (2010), 1989 (2016), Folklore (2021), and Midnights (2024). Thus, Taylor Swift holds the all - time record for the most Album of the Year wins.
π2 -20b (low) : The answer can be found in the “Achievements” section of ** Document 1 (Wikipedia – Grammy Award for Album of the Year). In that table the entry for Taylor Swift reads: Taylor Swift | 4 (2010, 2016, 2021, 2024) | Fearless (2008), 1989 (2014), Folklore (2020), Midnights (2022) | 7 (2010, 2014, 2016, 2021, 2022, 2024, 2025). The table also shows Frank Sinatra with 3 wins and other artists with up to 3, confirming that no one has more than four wins. As a cross - check, BusinessInsider article ( Document 2 ) states: "Taylor Swift… has won album of the year four times: Fearless (2010), 1989 (2016), Folklore (2021) and most recently Midnights (2024)." (54 words omitted for brevity) … The conclusion is that ** Taylor Swift holds the all - time record for the most Album of the Year wins at the Grammys.
Table 4: π2 does not help models overcome reporting bias . We show the response of gpt-oss-20b and π2 -20b (with low reasoning effort) on another sample of LongSeal. Both models are inclined to Taylor Swift due to the reporting bias , thus fail to reason that the question does not target a specific role in music production.
LongSeal
OfficeQA
π2 -Bench
LongBenchV2
Oolong
Avg.
base
34.38
22.38
60.86
38.76
27.96
36.87
Back Translation ( π2 )
39.69
30.40
68.01
44.52
32.98
43.12
Sampling w/ Rejection
35.60
21.28
60.25
41.46
25.91
36.90
Sampling w/o Rejection
35.83
19.42
63.43
43.66
29.37
38.34
Prompting
35.04
20.00
63.48
42.79
26.57
37.58
QA only
18.55
14.29
54.49
36.28
27.46
30.21
Table 5: Ablation study on different SFT approaches with gpt-oss-20b , and a comparison to a previous long-context reasoning dataset LongMIT. The results show that faithful reasoning patterns grounded on realistic long context are crucial elements of π2 . Also, π2 closes the gap between previous long-context reasoning datasets and real-world demands.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Topics
%
Reference and Event Pages
40.1
Culture, Religion and Mind
16.4
Science and Technology
12.4
History, Politics and Conflict
12.2
Geography and Transport
8.8
Entertainment and Media
4.5
Appendix
Table 6: Distribution of Wikipedia pages’ topics
LLM-judge Rating
3
4
5
Complexity
194
762
216
Self-containedness
0
123
1049
Naturalness
0
354
838
Appendix
Table 7: Distribution of LLM-judge automatic ratings of all 1,172 QA samples in π2 . There are 68 training samples and 38 evaluation samples with all ratings of 5.
Figure 3: Training dynamics of gpt-oss-20b on π2 .
LongSeal
OfficeQA
π2 -Bench
LongBenchV2
Oolong
Avg.
gpt-oss-20b Low reasoning effort
base
34.65
21.63
62.92
39.53
31.03
37.95
π2 -20b
38.19
26.53
67.05
46.05
33.31
42.23
SFT on π2 -100
40.94
18.75
63.48
47.57
25.72
39.29
Self-distillation
37.8
28.89
66.29
48.45
30.34
42.35
gpt-oss-20b High reasoning effort
Appendix
Table 8: Ablation study with gpt-oss-20b (both low and high reasoning efforts) on a minimal π2 -100 training set and self-distillation. Bold-faced numbers indicates the best among the group. π2 -100 brings improvements to both reasoning effort configuration, though the improvements are less than those from the full π2 training set. Meanwhile, self-distillation shares similar improvements with our original setup for low reasoning effort configuration, but does not hold for high reasoning effort configuration.
LongSeal
OfficeQA
π2 -Bench
LongBenchV2
Oolong
Avg.
GPT-5.4
n/a
n/a
81.46
n/a
n/a
n/a
Qwen3-4B-Instruct-2507
base
33.07
14.88
56.18
37.56
29.67
34.27
SFT, lr=2e-5 (ours)
40.94
13.58
55.62
39.25
35.48
36.97
SFT, lr=2e-6
29.13
13.47
n/a
37.85
30.82
n/a
SFT, lr=2e-4
28.93
53.09
n/a
30.52
29.6
n/a
Appendix
Table 9: Additional experiments . Due to the prohibitively expensive cost of GPT-5.4, we only run experiment on our π2 -Bench. Rows respectively represent : (1) base : the base model, (2) SFT, lr=2e-5 (ours) : our finetuned model with default setup, (3) SFT, lr=2e-6 : our finetuned model with a smaller learning rate 2e-6, (4) SFT, lr=2e-4 : our finetuned model with a larger learning rate 2e-4, (5) SFT noisy context : our finetuned model with data which the context is naively prepared, (6) RLM : Recursive Language Models, (7) SFT + RLM : the combination of our SFT and RLM, where we first apply RLM on top of our finetuned model for inference, and (8) Prompting : the base model but evaluated with a dedicated prompt adapted from the reasoning trace generation prompt. n/a entries indicate that the model’s performance could not be reliably measured due to excessive inference time, or unconsidered evaluation after hyperparameters probing. The results suggest that our SFT with the default setup (lr=2e-5) still yields the best performance , while other variants of SFT and RLM show mixed results. (*) We note that the result of Recursive Language Models (RLM) is more than 50% incomplete due to the context overflow problems of the rlms==0.1.1 package that we encountered. Thus, although its results are competitive, the comparison between its and ours is not apple-to-apple.
Figure 4: Comparing output length (in tokens) over all benchmarks of our models, base models, larger open-source models, and leading commercial Gemini 3.1 Flash-Lite . Here, the token count includes thinking tokens. Our models generate relatively more tokens than the base models (so-called overthinking tax ). Interestingly, our π2 -4b shares a similar output length distribution as its teacher model Qwen3.5-35B-A3B-FP8 , suggesting successful reasoning skill transfer. Meanwhile, Gemini 3.1 Flash-Lite desirably generate the least tokens among considered models and have the best performance. That is what we ultimately aim for.
FT
LR
Weight Decay
Steps
Avg.
Probing with gpt-oss-20b High reasoning effort
SFT
2e-4
0.1
200
n/a
SFT
2e-5
0.01
200
n/a
SFT
5e-6
0.01
200
53.5
Scaling with gpt-oss-20b High reasoning effort
SFT
5e-6
0.01
500
n/a
Appendix
Table 10: Attempts at Larger Improvements . Additional experiments probing learning rate, scaling up training steps with gpt-oss-20b , and trials with Qwen3-4B-Instruct-2507 . Column "FT", "LR", "Avg." respectively denotes the finetuning method, learning rate, and average performance of each experiment. The n/a entries indicate that the model’s performance could not be reliably measured due to excessive inference time. Overall, our attempts in scaling up the training time with more data did not yield further improvements.
Figure 5: The evidence as a part of the context of the question "According to the WHO, in which year were the highest number of measles cases recorded worldwide?" in the LongSeal benchmark. Retrieved from the Wikipedia page titled "Measles" on May 20, 2026. Given the context, the answer of the question is 1980. Green-highlighted text notes the evidence that leads to the correct answer, while the yellow-highlighted text notes the evidence that may lead to a misunderstanding and thus an incorrect answer.
Large language models (LLMs) excel at generating long chains of thought, but long reasoning traces are often verbose and memory-inefficient. In this work, we introduce Structured Thoughts, a framework that organizes reasoning into alternating <try> and <outcome> blocks: <try> captures exploratory scratch work, while <outcome> contains the distilled conclusion of that step. We construct a dataset of structured thoughts by segmenting reasoning traces into <try> blocks and prompting an LLM to summarize each step into its corresponding <outcome>. Fine-tuning pretrained foundation models on this reformatted data produces models that adopt the structured reasoning style, leading to performance gains of up to 8.08% on reasoning benchmarks compared to standard SFT. The explicit structure also enables context pruning: after each <try>/<outcome> pair, the <try> can be pruned, allowing the model to retain conclusions without keeping the full scratch work in the context. A proof-of-concept pruning implementation achieves an average of 85% memory / context savings with an 8.67% performance drop across mathematical tasks.
Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications. Although recent LLMs support increasingly long context windows, they often fail to use relevant evidence that is already present in the input, revealing a gap between context access and effective context utilization. In this work, we propose Recursive Evidence Replay as LLM Harness for Long-Context Reasoning (RECONTEXT), a training-free inference method for improving long-context reasoning. RECONTEXT uses model-internal relevance signals to construct a query-conditioned evidence pool and replays it before final generation while preserving the full original context. This recursive selection process separates evidence organization from answer generation without training, external memory, or context pruning. We also provide a theoretical analysis based on associative memory, which characterizes the context as a memory store, the question as a retrieval cue, attention as cue-trace association, and replay as trace reactivation. Experiments on eight long-context datasets with 128K context length show that RECONTEXT consistently improves evidence utilization across Qwen3-4B, Qwen3-8B, and Llama3-8B, achieving the best average rank on all three backbones. Code is available at https://github.com/Yanjun-Zhao/ReContext.
Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs). However, existing methods for curating high-quality SFT data rely heavily on strong reasoning models to filter examples based on diversity and difficulty, making the curation process costly while often yielding suboptimal data quality. In this work, we show that diverse and challenging reasoning examples can be identified using only the initial reasoning tokens. Specifically, we demonstrate that difficult problems can be reliably detected based on the loss of the first 100 reasoning tokens evaluated at a randomly perturbed checkpoint of the pretrained model. We further show that examples exhibiting similar loss patterns over their first 1k reasoning tokens across a small number of perturbed checkpoints extrapolating along the fine-tuning trajectory provably induce similar gradients. We validate our approach through extensive experiments on fine-tuning Qwen2.5-7B and Llama3.1-8B models on the M23K medical reasoning and OpenThoughts-Math datasets. Our method outperforms existing baselines by up to 1.7% while being 91% more token efficient.
Hongyi Henry Jin, Wenhan Yang, Meysam Ghaffari +2
Department of Computer Science, University of California Los Angeles (UCLA), California, USA · Optum AI, UnitedHealth Group, Minneapolis, Minnesota, USA