Probing How Scalable Table Data Enhances General Long-Context Reasoning

Huaibing Xie; Guoliang Zhao; Yang Liu; Shihan Dou; Siming Huang; Yanling Xiao; Shaolei Wang; Yiting Liu

cs.CLMar 23, 2026

Probing How Scalable Table Data Enhances General Long-Context Reasoning

Authors: Huaibing Xie, Guoliang Zhao, Yang Liu, Shihan Dou, Siming Huang, Yanling Xiao, Shaolei Wang, Yiting Liu,

Organizations: Large Language Model Department, Tencent · Xi’an Jiaotong University, Xi’an, China · Fudan University, Shanghai, China

Abstract

As real-world tasks grow increasingly complex, long-context reasoning has become a core capability for Large Language Models (LLMs). However, few studies explore which data types are effective for long-context reasoning and why. We find that structured table data with periodic structures shows strong potential for long-context reasoning. Motivated by this observation, we mathematically analyze tabular dependency structures using mutual information, revealing periodic non-vanishing dependencies in table data. Furthermore, we systematically analyze the capabilities of structured table data, conduct relevant scaling experiments, and validate its underlying mechanisms for enhancing long-context reasoning, yielding several meaningful insights. Leveraging these insights, we propose a simple yet scalable pipeline(TableLong) for synthesizing high-quality, diverse, and verifiable structured table data to boost long-context reasoning via RL. Extensive experimental results demonstrate that table data significantly enhances the long-context reasoning capability of LLMs across multiple long-context benchmarks (+8.24% on average), and even improves performance on out-of-domain benchmarks (+8.06% on average). We hope that our insights provide practical guidance for effective post-training data to enhance long-context reasoning in LLMs.

Figures & tables

Figure 1: Overview of TableLong: An end-to-end table data construction pipeline for long-context reasoning.

Figure 2: The radar chart of long-context reasoning benchmarks for DS-R1-Distill-32B trained with varying length.

Reference Flagships & Baselines
Model	LB-v2	Loong	Brows	GSM-Inf	Oolong	Ruler	MRCR	Avg.
Gemini-3.0-pro	69.38	65.43	88.07	87.06	78.41	83.01	75.30	78.09
Deepeek-v3.1	52.88	50.55	56.27	47.40	51.90	42.97	46.62	49.80
Qwen-Long-L1	43.74	44.68	69.93	9.00	27.37	47.70	27.70	38.59
Main Results: Effectiveness of Our Method
Qwen3-32B	46.33	39.96	59.33	12.22	31.66	50.86	42.45	40.40

Table 1: Performance Comparison on Long-Context and Reasoning Benchmarks. We compare base models against their variants optimized via Reinforcement Learning on our dataset (denoted as + Ours ). The best results within each model family are highlighted in bold . LB-v2 : LongBench v2; Brows : BrowsCompLong; GSM-Inf : GSM-Infinite; Oolong : Oolong-Synth. For model names, Qwen2.5-32B-Inst denotes Qwen2.5-32B-Instruct, and DS-R1-Distill refers to Deepseek-R1-Distill-Qwen.

Panel of Figure 3 — Figure 3: Needle in a Haystack retrieval across document depths. Our approach significantly enhances long-context robustness, boosting the 14B model’s accuracy from 69.30% to 91.20% and the 32B model’s accuracy from 87.95% to 99.40%, achieving near-perfect performance.

Panel of Figure 4 — Figure 4: Decomposition experiments for DS-R1-Distill-32B. (a) While structure alone (“no semantics”) boosts baseline performance (+1.67%), semantics remain essential for peak results. Removing delimiters or adding noise yields negligible drops, confirming the primacy of intrinsic structure. (b) Models with “no semantics” suffer premature convergence, whereas “no visible delimiters” settings recover from low initial rewards.

Model	LB-v2	Loong	Brows	GSM-Inf	Oolong	Ruler	MRCR	Avg.
DS-R1-Distill-32B
$\|-$ 0 $\sim$ 30	44.33	42.87	71.87	18.00	40.29	69.30	37.44	46.30
$\|-$ 0 $\sim$ 100	45.33	43.46	74.31	10.00	44.10	71.82	36.46	46.50
$\|-$ 0 $\sim$ 300+	45.73	45.30	74.31	14.80	51.41	66.41	40.57	48.36
DS-R1-Distill-14B
$\|-$ 0 $\sim$ 30	36.98	35.60	72.17	6.40	41.70	82.16	27.63	43.23

Table 2: Performance comparison across varying scales of table cells. Bold values indicate the best performance within each backbone.

Model	LB-v2	Loong	Brows	GSM-Inf	Oolong	Ruler	MRCR	Avg.
DS-R1-Distill-32B
$\|-$ 1	44.93	43.65	73.39	16.80	50.78	62.97	34.10	46.66
$\|-$ 1 $\sim$ 5	45.33	44.27	73.37	15.00	49.63	64.37	37.79	47.11
$\|-$ 1 $\sim$ 30	45.73	45.30	74.31	14.80	51.41	66.41	40.57	48.36
DS-R1-Distill-14B
$\|-$ 1	34.39	37.05	71.25	6.20	42.09	77.03	31.59	42.80

Table 3: Performance comparison across varying scales of table count. Bold values indicate the best performance within each backbone.

Model	GPQA	AIME	MC	LCB
DS-R1-Distill-32B	56.06	60.00	30.28	46.71
DS-R1-Distill-32B + Ours	63.64	70.00	32.97	58.68
DS-R1-Distill-14B	55.56	43.33	23.08	43.71
DS-R1-Distill-14B + Ours	59.60	46.67	26.86	45.51

Table 4: Generalization on OOD Benchmarks. We evaluate models on GPQA-Diamond ( GPQA ), AIME 2025 ( AIME ), MultiChallenge ( MC ), and LiveCodeBench ( LCB ). Shaded rows indicate our RL-finetuned models ( + Ours ). Our method achieves significant gains, notably +11.9% on LCB for the 32B model.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Computational Setup
Hyperparameter	Value
Hardware	64 $\times$ NVIDIA H20
Parallelism Strategy	TP=8
Offload	Disabled
Training Parameters
Max Sequence Capacity	32,768

Appendix

Table 5: Detailed hyperparameters for RL training

Category	Benchmark	Subset / Task	Num. Samples	Domain Focus
Holistic	LongBench-v2 ( Bai et al., 2025 )	Full Set	503	General Real-world
Holistic	Loong ( Wang et al., 2024 )	Full Set	1,441	Spotlight QA / Retrieval
Retrieval	Browscomplong ( OpenAI, 2025a )	Standard	327	Complex Search
	MRCR ( OpenAI, 2025c )	Standard	1,437	Multi-round QA
	Ruler ( Hsieh et al., 2024 )	QA2 Task	2,495	Variable Tracking
Reasoning	GSM-Infinite ( Zhou et al., 2025 )	Standard	500	Long-range Math

Appendix

Table 6: Details of the evaluation benchmarks for long-context reasoning. ”Subset/Task” indicates the specific evaluation partition used. ”Num. Samples” denotes the number of test instances after filtering.

Category	Benchmark	Num. Samples	Domain Focus
Science	GPQA-Diamond ( Rein et al., 2024 )	198	Scientific Reasoning
Math	AIME 2025 ( Balunović et al., 2025 )	30	Mathematical Reasoning
Multi-turn Dialogue	MultiChallenge ( Deshpande et al., 2025 )	273	Real-world Multi-turn Dialogue
Code	LiveCodeBench ( Jain et al., 2024 )	167	Code Generation

Appendix

Table 7: Details of the evaluation benchmarks for other domains. ”Num. Samples” denotes the number of test instances after filtering.

Panel of Figure 5 — Figure 5: Decomposition experiments for DS-R1-Distill-14B. (a) While structure alone (“no semantics”) boosts baseline performance (+1.66%), semantics remain essential for peak results. Removing delimiters or adding noise yields negligible drops, confirming the primacy of intrinsic structure. (b) Models with “no semantics” suffer premature convergence, whereas “no visible delimiters” settings recover from low initial rewards.

Model	LB-v2	Loong	Brows	GSM-Inf	Oolong	Ruler	MRCR	Avg.
DS-R1-Distill-32B
$\|-$ 4k	45.13	43.41	74.92	11.60	47.65	68.42	37.65	46.97
$\|-$ 8k	48.11	43.89	74.01	13.20	47.87	66.05	39.04	47.45
$\|-$ 16k	45.73	45.30	74.31	14.80	51.41	66.41	40.57	48.36
DS-R1-Distill-14B
$\|-$ 4k	36.78	36.03	74.31	8.00	43.81	80.08	30.27	44.18

Appendix

Table 8: Performance comparison across varying scales of length. Bold values indicate the best performance within each backbone.

Panel of Figure 6 — Figure 6: Performance comparison on the 16k+ length range of LongBench-v2.

Model	0-4k	4k-8k	8k-16k	16k-32k	32k-64k	64-128k
DS-R1-Distill-32B	63.82	51.50	38.50	28.75	23.00	26.25
DS-R1-Distill-32B + Ours	69.49	62.75	55.50	45.25	38.50	39.00
DS-R1-Distill-14B	62.31	51.00	45.50	37.25	31.25	27.00
DS-R1-Distill-14B + Ours	66.30	52.50	47.75	40.00	33.50	34.75
Qwen3-32B	60.66	40.25	33.25	20.50	11.75	15.00
Qwen3-32B + Ours	63.69	53.75	53.00	35.75	28.25	24.25

Appendix

Table 9: Oolong-Synth performance across different length ranges

Model	0-4k	4k-8k	8k-16k	16k-32k	32k-64k	64k-128k
DS-R1-Distill-32B	-	39.43	38.52	30.41	27.46	25.62
DS-R1-Distill-32B + Ours	-	50.81	47.04	40.54	36.61	30.31
DS-R1-Distill-14B	-	37.80	38.15	32.79	26.51	14.37
DS-R1-Distill-14B + Ours	-	44.31	34.07	28.04	25.76	22.81
Qwen3-32B	-	52.44	48.15	40.88	36.27	35.94
Qwen3-32B + Ours	-	52.44	45.93	42.23	41.36	33.44

Appendix

Table 10: MRCR performance across different length ranges

Model	0-4k	4k-8k	8k-16k	16k-32k	32k-64k	64-128k
DS-R1-Distill-32B	63.82	51.50	38.50	28.75	23.00	26.25
$\|-$ 4k	70.70	61.50	52.75	39.50	31.25	31.75
$\|-$ 8k	70.01	61.50	57.25	45.75	32.00	28.25
$\|-$ 0-30	68.64	56.75	34.75	29.75	21.50	23.00
$\|-$ 0-100	72.94	60.75	47.50	32.00	25.50	25.50
$\|-$ 1	71.39	64.75	56.50	42.75	34.00	33.50

Appendix

Table 11: Oolong performance comparison across context lengths.

Group	Model	0-4k	4k-8k	8k-16k	16k-32k	32k-64k	64-128k
	DS-R1-Distill-32B	-	39.43	38.52	30.41	27.46	25.62
Length	$\|-$ 4k	-	48.37	44.81	33.45	34.92	29.69
	$\|-$ 8k	-	51.63	43.33	37.84	34.24	31.87
Cells	$\|-$ 0-30	-	48.78	42.95	36.15	33.90	27.50
	$\|-$ 0-100	-	47.97	42.95	36.94	29.15	28.44
Tables	$\|-$ 1	-	43.50	39.26	34.80	36.61	24.06

Appendix

Table 12: MRCR performance comparison across context lengths with grouping.

Panel of Figure 7 — Figure 7: The instruction prompt used for generating SQL-NLQ pairs, which constrains SQL logic while eliciting natural, task-oriented questions.

Panel of Figure 8 — Figure 8: An instance of the precise retrieval grounding task.

Panel of Figure 9 — Figure 9: An instance of the multi-hop reasoning task.

Panel of Figure 10 — Figure 10: An instance of the multi-table grounding task.

Panel of Figure 11 — Figure 11: An instance of the semantic-agnostic table task.

Panel of Figure 12 — Figure 12: An instance of the no-visible-delimiter task.

Panel of Figure 13 — Figure 13: An instance of the no-visible-delimiter task with highlighted noise.

Panel of Figure 14 — Figure 14: Comparison of reasoning chains. The baseline model (left) hallucinates column values (e.g., confusing ‘dst_ap‘ with ‘src_ap‘’s neighbor), while our model (right) correctly parses the row as a sequence.

Panel of Figure 15 — Figure 15: Comparison of reasoning strategies on the AIME 2025 grid coloring problem. The baseline (DS-R1-Distill-32B) relies on exhaustive enumeration but fails due to symmetry bias (Result: 84). In contrast, our model employs algebraic modeling and variable reduction to enforce precise constraint propagation (Result: 82).

Long-Context Reasoning in Language Models

See all 41 papers

A Decomposition Perspective to Long-context Reasoning for LLMs
Apr 9, 2026Yanling Xiao, Huaibing Xie, Guoliang Zhao +8Long-Context Reasoning in Language Models
Large Language Model Department, Tencent · Xi’an Jiaotong University, Xi’an, China · Fudan University, Shanghai, China +2
LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards
Feb 5, 2026Qize Yu, Zijun Chen, Chen Wu +6Long-Context Reasoning in Language Models RL for Language Model Reasoning
Shanghai Jiao Tong University · Tsinghua University · Beijing University of Posts and Telecommunications, Beijing, China +1
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Jun 17, 2026Xiaoyue Xu, Sikui Zhang, Xiaorong Wang +2Training Data Curation RL for Language Model Reasoning
OpenBMB · Tsinghua University

Apr 9, 2026J/K move · Enter open · S save

A Decomposition Perspective to Long-context Reasoning for LLMs

Yanling Xiao, Huaibing Xie, Guoliang Zhao, Shihan Dou+7

Large Language Model Department, Tencent · Xi’an Jiaotong University, Xi’an, China · Fudan University, Shanghai, China+2

Long-Context Reasoning in Language Models

Open paper AskPDF arXiv

Long-context reasoning is essential for complex real-world applications, yet remains a significant challenge for Large Language Models (LLMs). Despite the rapid evolution in long-context reasoning, current research often overlooks the internal complexity of the long-context reasoning task itself. In this paper, we move beyond this holistic view and decompose long-context reasoning into a set of fundamental atomic skills, and we then automatically synthesize a suite of pseudo datasets, each explicitly targeting a specific atomic skill. Our empirical analysis confirms that proficiency in these atomic skills is strongly correlated with general long-text reasoning performance. Building on this insight, we employ reinforcement learning on these pseudo datasets to sharpen the model's atomic skills, in the hope of boosting its general long-context reasoning ability. Extensive experiments across multiple benchmarks demonstrate the effectiveness of our approach: it outperforms a strong baseline by an average margin of 7.7% (improving from 46.3% to 54.0%) across Loogle, Loong, LongBench-v2, BrowscompLong, Ruler-qa2, and MRCR.

Abstract

Long-Context Reasoning in Language Models

Explore similar work

A Decomposition Perspective to Long-context Reasoning for LLMs

LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning