$π^2$: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models

Quyet V. Do; Thinh Pham; Nguyen Nguyen; Sha Li; Pratibha Zunjare; Tu Vu

cs.CLApr 6, 2026

$π^2$ : Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models

Authors: Quyet V. Do, Thinh Pham, Nguyen Nguyen, Sha Li, Pratibha Zunjare, Tu Vu

Organizations: Virginia Tech, VA 24061, USA

Abstract

We study a QA curation pipeline for improving long-context complex reasoning in large language models (LLMs). Our approach, $π^2$ , constructs high-quality reasoning data through rigorous QA curation: 1) extracting and expanding tables from Wikipedia, 2) from the collected tables together with relevant metadata, generating complex reasoning questions whose answers are automatically determined and validated through dual-path code execution, 3) finally, back-translating chain-of-thoughts solutions grounded in realistic context. Supervised fine-tuning with gpt-oss-20b and Qwen3-4B-Instruct-2507 on $π^2$ yields consistent improvements across four long-context reasoning benchmarks and our alike $π^2$ -Bench, with average absolute accuracy gains of +6.25% and +3.37% respectively. Through deeper analyses, we observe that reasoning style contributes little, while faithful reasoning patterns discovered by back translation and grounded realistic long context, as $π^2$ is designed for, are crucial for the improvement. Our code, data, and models are fully open-source at https://github.com/vtpss/pi-squared.

Figures & tables

Figure 1: Tables bridge realistic documents and complex reasoning in open-domain tasks. They represent data across sources , while being compatible with programmatic operations .

Figure 2: $\pi^{2}$ curation pipeline . It first 1) collects tables from webpages and expands them with information retrieved from linked documents if conditions are met (indicated by dashed arrows and objects); then 2) from each collected table, generate a complex question together with an executable SQL query by an LLM and verify the SQL-derived answer with an independent LLM-generated Python solution, and finally 3) back-translates reasoning traces grounded in real free-form context from the verified answer.

Question	Answer	Evidence
Among the television roles held by Debbie Reynolds in her career, how many distinct character or guest roles did she perform specifically on shows broadcast by NBC?	3	Debbie Reynolds; Aloha Paradise; Alice (American TV series); etc.
In the 2023 Cricket World Cup, among bowlers who use a Right-arm fast style, which player took the most wickets?	Mohammed Shami	2023 Cricket World Cup; Mohammed Shami; etc.
Based on the 2022 cobalt production and reserves data reported by the USGS, which country had the highest ratio of reserves to annual production?	Australia	Cobalt

Table 1: Samples in

\pi^{2}

. The "Evidence" column shows the evidence Wikipedia page(s). The sample is originated from a table in the first listed page. Our samples cover diverse topics and complexity levels.

	In-domain			Out-of-domain
	LongSeal	OfficeQA	$\pi^{2}$ -Bench	LongBenchV2	Oolong	Avg.
Gemini 3.1 Flash-Lite	64.96	57.14	75.29	58.14	42.38	59.58
Gemini-2.5-Pro	59.84	53.37	72.32	56.3	36.56	55.68
Qwen3.5-35B-A3B-FP8	58.5	55.74	76.97	58.22	39.76	57.84
gpt-oss-120b (low)	42.18	33.88	75	51.06	38.08	48.04
Qwen3-4B-Instruct-2507	29.14	15.12	54.68	36.09	30.51	33.11

Table 2:

\pi^{2}

significantly improves long-context reasoning ability of Qwen3-4B-Instruct-2507 and gpt-oss-20b . We evaluate our models and baselines on in-domain benchmarks LongSeal, OfficeQA, and our

\pi^{2}

-Bench; as well as out-of-domain benchmarks LongBenchV2 and Oolong. For overall comparison, we report the weight-less average performance of models across benchmarks. Bold-faced numbers indicate the best among the group. We confirm that the average improvements brought by SFT on

\pi^{2}

are significant at

\alpha=0.01

under t-test (i.e., with 99% confidence level).

Question: According to the WHO, in which year were the highest number of measles cases recorded worldwide? Correct Answer: 1980 Context: (Omitted due to its length. A snapshot of evidence is shown in Figure 5 )

gpt-oss-20b (low) : From the WHO data presented in the documents, the peak in worldwide measles cases occurred in 2019 , with a total of approximately 869,770 reported cases globally. This is higher than any preceding or following year reported in the sources (e.g., 2018 had 343,806 cases, and 2024 had 359,521 cases). Therefore, according to WHO statistics, 2019 was the year with the highest number of measles cases worldwide.

\pi^{2}

-20b (low) : The WHO’s own global measles surveillance data (published in Document 1) lists the total number of reported measles cases worldwide by year: 1980 – 3,852,242 cases, 1990 – 1,325,074 cases, 2000 – 853,479 cases 2010 – 343,806 cases, 2019 – 869,770 cases, 2020 – 159,073 cases Among these figures, the largest reported number is the 3,852,242 cases recorded for the year 1980 . No other year shown in the official WHO data exceeds this count, and [ 31 words omitted for brevity ]. Therefore, according to the WHO’s own records, the highest number of measles cases worldwide occurred in 1980 .

Table 3:

\pi^{2}

equips models with stronger long-context retrieval and reasoning . We show the response of gpt-oss-20b and

\pi^{2}

-20b (with low reasoning effort) on a sample of LongSeal. The base model fail to reason and retrieve the correct piece of evidence, as it mistakes the phrase "climbed to 869,770 in 2019" as the signal for the correct answer, thus missing out other evidence. On the other hand,

\pi^{2}

-20b successfully identifies all pieces of evidence and reason thoroughly through them to derive the correct answer. That is a core capability targeted by our data curation pipeline.

Question: Who holds the all-time record at the Grammys for the most wins in the album of the year category? Correct Answer: Serban Ghenea Context: (Only show the most important evidence. Omitted remaining text for brevity.) [In Document 1: "Grammy Award for Album of the Year" Wikipedia page, as a document in the context] … Serban Ghenea (as engineer/mixer) is the most frequent winner in this category with five awards. Taylor Swift (as performer); John Hanes (as engineer/mixer); and Tom Coyne and Randy Merrill (as mastering engineers) have won the award four times …

gpt-oss-20b (low) : From the provided context ( Document 1, Document 5, Document 7, Document 9 ) we see that: - Frank Sinatra, Paul Simon, and Stevie Wonder each have three wins. - Taylor Swift has won the award four times: for Fearless (2010), 1989 (2016), Folklore (2021), and Midnights (2024). Thus, Taylor Swift holds the all - time record for the most Album of the Year wins.

\pi^{2}

-20b (low) : The answer can be found in the “Achievements” section of ** Document 1 (Wikipedia – Grammy Award for Album of the Year). In that table the entry for Taylor Swift reads: Taylor Swift | 4 (2010, 2016, 2021, 2024) | Fearless (2008), 1989 (2014), Folklore (2020), Midnights (2022) | 7 (2010, 2014, 2016, 2021, 2022, 2024, 2025). The table also shows Frank Sinatra with 3 wins and other artists with up to 3, confirming that no one has more than four wins. As a cross - check, BusinessInsider article ( Document 2 ) states: "Taylor Swift… has won album of the year four times: Fearless (2010), 1989 (2016), Folklore (2021) and most recently Midnights (2024)." (54 words omitted for brevity) … The conclusion is that ** Taylor Swift holds the all - time record for the most Album of the Year wins at the Grammys.

Table 4:

\pi^{2}

does not help models overcome reporting bias . We show the response of gpt-oss-20b and

\pi^{2}

-20b (with low reasoning effort) on another sample of LongSeal. Both models are inclined to Taylor Swift due to the reporting bias , thus fail to reason that the question does not target a specific role in music production.

	LongSeal	OfficeQA	$\pi^{2}$ -Bench	LongBenchV2	Oolong	Avg.
base	34.38	22.38	60.86	38.76	27.96	36.87
Back Translation ( $\pi^{2}$ )	39.69	30.40	68.01	44.52	32.98	43.12
Sampling w/ Rejection	35.60	21.28	60.25	41.46	25.91	36.90
Sampling w/o Rejection	35.83	19.42	63.43	43.66	29.37	38.34
Prompting	35.04	20.00	63.48	42.79	26.57	37.58
QA only	18.55	14.29	54.49	36.28	27.46	30.21

Table 5: Ablation study on different SFT approaches with gpt-oss-20b , and a comparison to a previous long-context reasoning dataset LongMIT. The results show that faithful reasoning patterns grounded on realistic long context are crucial elements of

\pi^{2}

. Also,

\pi^{2}

closes the gap between previous long-context reasoning datasets and real-world demands.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Topics	%
Reference and Event Pages	40.1
Culture, Religion and Mind	16.4
Science and Technology	12.4
History, Politics and Conflict	12.2
Geography and Transport	8.8
Entertainment and Media	4.5

Appendix

Table 6: Distribution of Wikipedia pages’ topics

LLM-judge Rating	3	4	5
Complexity	194	762	216
Self-containedness	0	123	1049
Naturalness	0	354	838

Appendix

Table 7: Distribution of LLM-judge automatic ratings of all 1,172 QA samples in

\pi^{2}

. There are 68 training samples and 38 evaluation samples with all ratings of 5.

Figure 3: Training dynamics of gpt-oss-20b on $\pi^{2}$ .

	LongSeal	OfficeQA	$\pi^{2}$ -Bench	LongBenchV2	Oolong	Avg.
gpt-oss-20b Low reasoning effort
base	34.65	21.63	62.92	39.53	31.03	37.95
$\pi^{2}$ -20b	38.19	26.53	67.05	46.05	33.31	42.23
SFT on $\pi^{2}$ -100	40.94	18.75	63.48	47.57	25.72	39.29
Self-distillation	37.8	28.89	66.29	48.45	30.34	42.35
gpt-oss-20b High reasoning effort

Appendix

Table 8: Ablation study with gpt-oss-20b (both low and high reasoning efforts) on a minimal

\pi^{2}

-100 training set and self-distillation. Bold-faced numbers indicates the best among the group.

\pi^{2}

-100 brings improvements to both reasoning effort configuration, though the improvements are less than those from the full

\pi^{2}

training set. Meanwhile, self-distillation shares similar improvements with our original setup for low reasoning effort configuration, but does not hold for high reasoning effort configuration.

	LongSeal	OfficeQA	$\pi^{2}$ -Bench	LongBenchV2	Oolong	Avg.
GPT-5.4	n/a	n/a	81.46	n/a	n/a	n/a
Qwen3-4B-Instruct-2507
base	33.07	14.88	56.18	37.56	29.67	34.27
SFT, lr=2e-5 (ours)	40.94	13.58	55.62	39.25	35.48	36.97
SFT, lr=2e-6	29.13	13.47	n/a	37.85	30.82	n/a
SFT, lr=2e-4	28.93	53.09	n/a	30.52	29.6	n/a

Appendix

Table 9: Additional experiments . Due to the prohibitively expensive cost of GPT-5.4, we only run experiment on our

\pi^{2}

-Bench. Rows respectively represent : (1) base : the base model, (2) SFT, lr=2e-5 (ours) : our finetuned model with default setup, (3) SFT, lr=2e-6 : our finetuned model with a smaller learning rate 2e-6, (4) SFT, lr=2e-4 : our finetuned model with a larger learning rate 2e-4, (5) SFT noisy context : our finetuned model with data which the context is naively prepared, (6) RLM : Recursive Language Models, (7) SFT + RLM : the combination of our SFT and RLM, where we first apply RLM on top of our finetuned model for inference, and (8) Prompting : the base model but evaluated with a dedicated prompt adapted from the reasoning trace generation prompt. n/a entries indicate that the model’s performance could not be reliably measured due to excessive inference time, or unconsidered evaluation after hyperparameters probing. The results suggest that our SFT with the default setup (lr=2e-5) still yields the best performance , while other variants of SFT and RLM show mixed results. (*) We note that the result of Recursive Language Models (RLM) is more than 50% incomplete due to the context overflow problems of the rlms==0.1.1 package that we encountered. Thus, although its results are competitive, the comparison between its and ours is not apple-to-apple.

Figure 4: Comparing output length (in tokens) over all benchmarks of our models, base models, larger open-source models, and leading commercial Gemini 3.1 Flash-Lite . Here, the token count includes thinking tokens. Our models generate relatively more tokens than the base models (so-called overthinking tax ). Interestingly, our

\pi^{2}

-4b shares a similar output length distribution as its teacher model Qwen3.5-35B-A3B-FP8 , suggesting successful reasoning skill transfer. Meanwhile, Gemini 3.1 Flash-Lite desirably generate the least tokens among considered models and have the best performance. That is what we ultimately aim for.

FT	LR	Weight Decay	Steps	Avg.
Probing with gpt-oss-20b High reasoning effort
SFT	2e-4	0.1	200	n/a
SFT	2e-5	0.01	200	n/a
SFT	5e-6	0.01	200	53.5
Scaling with gpt-oss-20b High reasoning effort
SFT	5e-6	0.01	500	n/a

Appendix

Table 10: Attempts at Larger Improvements . Additional experiments probing learning rate, scaling up training steps with gpt-oss-20b , and trials with Qwen3-4B-Instruct-2507 . Column "FT", "LR", "Avg." respectively denotes the finetuning method, learning rate, and average performance of each experiment. The n/a entries indicate that the model’s performance could not be reliably measured due to excessive inference time. Overall, our attempts in scaling up the training time with more data did not yield further improvements.

Figure 5: The evidence as a part of the context of the question "According to the WHO, in which year were the highest number of measles cases recorded worldwide?" in the LongSeal benchmark. Retrieved from the Wikipedia page titled "Measles" on May 20, 2026. Given the context, the answer of the question is 1980. Green-highlighted text notes the evidence that leads to the correct answer, while the yellow-highlighted text notes the evidence that may lead to a misunderstanding and thus an incorrect answer.

Long-Context Reasoning in Language Models

See all 41 papers

Structured Thoughts For Improved Reasoning And Context Pruning
Jul 11, 2026Zain Sarwar, Supriyo Chakraborty, Berkcan Kapusuzoglu +5Efficient Language Model Reasoning Structured CoT Reasoning
University of Chicago · Capital One
ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Jul 2, 2026Yanjun Zhao, Ruizhong Qiu, Tianxin Wei +6Long-Context Inference Long-Context Reasoning in Language Models
University of Illinois Urbana-Champaign
Reasoning Quality Emerges Early: Data Curation for Reasoning Models
Jun 25, 2026Hongyi Henry Jin, Wenhan Yang, Meysam Ghaffari +2LLM Fine-Tuning Training Data Selection
Department of Computer Science, University of California Los Angeles (UCLA), California, USA · Optum AI, UnitedHealth Group, Minneapolis, Minnesota, USA

Jul 11, 2026J/K move · Enter open · S save

Structured Thoughts For Improved Reasoning And Context Pruning

Zain Sarwar, Supriyo Chakraborty, Berkcan Kapusuzoglu, Chia-Hsuan Lee+4

University of Chicago · Capital One

Efficient Language Model Reasoning Structured CoT Reasoning CoT Reasoning LLM Reasoning Structured Reasoning LLM Context Management+1

Open paper AskPDF arXiv

Large language models (LLMs) excel at generating long chains of thought, but long reasoning traces are often verbose and memory-inefficient. In this work, we introduce Structured Thoughts, a framework that organizes reasoning into alternating <try> and <outcome> blocks: <try> captures exploratory scratch work, while <outcome> contains the distilled conclusion of that step. We construct a dataset of structured thoughts by segmenting reasoning traces into <try> blocks and prompting an LLM to summarize each step into its corresponding <outcome>. Fine-tuning pretrained foundation models on this reformatted data produces models that adopt the structured reasoning style, leading to performance gains of up to 8.08% on reasoning benchmarks compared to standard SFT. The explicit structure also enables context pruning: after each <try>/<outcome> pair, the <try> can be pruned, allowing the model to retain conclusions without keeping the full scratch work in the context. A proof-of-concept pruning implementation achieves an average of 85% memory / context savings with an 8.67% performance drop across mathematical tasks.

Abstract

Long-Context Reasoning in Language Models

Explore similar work

Structured Thoughts For Improved Reasoning And Context Pruning

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

Reasoning Quality Emerges Early: Data Curation for Reasoning Models