SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs

Jie Sun; Yu Liu; Lu Han; Qiwen Deng; Xiang Shu; Yang Xiao; Xingyu Lu; Jun Zhou

cs.CLApr 9, 2026

SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs

Authors: Jie Sun, Yu Liu, Lu Han, Qiwen Deng, Xiang Shu, Yang Xiao, Xingyu Lu, Jun Zhou,

Organizations: University of Science and Technology of China · Shanghai Innovation Institute · Nanjing University · University of Edinburgh ·

Abstract

While transformer-based Large Language Models (LLMs) theoretically support massive context windows, they suffer from severe performance degradation when processing long numerical sequences. We attribute this failure to the attention dispersion in the Softmax mechanism, which prevents the model from concentrating attention. To overcome this, we propose Separate Sequence (SepSeq), a training-free, plug-and-play framework to mitigate dispersion by strategically inserting separator tokens. Mechanistically, we demonstrate that separator tokens act as an attention sink, recalibrating attention to focus on local segments while preserving global context. Extensive evaluations on 9 widely-adopted LLMs confirm the effectiveness of our approach: SepSeq yields an average relative accuracy improvement of 35.6% across diverse domains while reducing total inference token consumption by 16.4% on average.

Figures & tables

Figure 1: (A) Average accuracy across six synthetic tasks, performance drops sharply with increasing numerical sequence length (S: 2–32, M: 33–128, L: 129–256, XL: 257–512), while SepSeq remains largely unaffected. (B) visualize the attention scores given the input: “… How many 1’s in [0 1 0 1 0 1 0 0 0] …”. (C) show the attention scores for the input with segmentation: “… How many 1’s in [0 1 0 1\n0 1 0 0\n0] …”. (B) exhibits dispersed attention across the entire sequence while (C) demonstrates segment-focused attention.

Figure 2: Visualization of the Separator Attention effect. (A-B) The separator $\tau_{sep}$ serves as a prominent attention sink, demonstrating consistently higher attention scores than $\tau_{sp}$ across both (A) model layers and (B) attention heads. (C) This concentrated attention on $\tau_{sep}$ results in a substantial reduction of cross-segment attention weights, thereby localizing the attention scope relative to the vanilla $\tau_{sp}$ .

Task	Answer Rate (%, $\uparrow$ )				Accuracy (%, $\uparrow$ )
Task	Vanilla	CoT	ICL	SepSeq (Incr.)	Vanilla	CoT	ICL	SepSeq (Incr.)
counting	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0 ( 0 +0.0%)	42.7 $\pm$ 2.7	41.2 $\pm$ 3.0	37.9 $\pm$ 3.2	76.7 $\pm$ 1.2 ( 0 +79.6% )
indexing	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0 ( 0 +0.0%)	38.8 $\pm$ 3.1	33.4 $\pm$ 3.4	34.9 $\pm$ 2.9	86.6 $\pm$ 0.8 ( +123.0% )
max-float	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0 ( 0 +0.0%)	63.9 $\pm$ 1.9	60.5 $\pm$ 2.1	63.3 $\pm$ 1.7	81.0 $\pm$ 1.1 ( 0 +26.8% )
max-int	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0 ( 0 +0.0%)	75.3 $\pm$ 1.1	68.8 $\pm$ 1.7	71.0 $\pm$ 1.4	92.4 $\pm$ 0.4 ( 0 +22.7% )
min-float	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0 ( 0 +0.0%)	63.1 $\pm$ 1.7	60.2 $\pm$ 2.2	60.4 $\pm$ 1.9	79.3 $\pm$ 1.0 ( 0 +25.7% )

Table 1: The average answer rate and accuracy for each task over 10 independent runs, reported as mean

\pm

standard deviation. Bold indicates the best average performance, while underlined values denote the best average performance excluding our method. “Incr.” indicates the percentage improvement of SepSeq over the strongest non-SepSeq baseline. Green and red indicate improvement and degradation in performance, respectively.

Model	Answer Rate (%, $\uparrow$ )				Accuracy (%, $\uparrow$ )
Model	Vanilla	CoT	ICL	SepSeq (Incr.)	Vanilla	CoT	ICL	SepSeq (Incr.)
Qwen3-8B	0 79.4 $\pm$ 1.1	0 79.5 $\pm$ 0.9	0 75.2 $\pm$ 1.3	0 87.3 $\pm$ 0.6 ( 0 +9.8% )	45.5 $\pm$ 2.8	40.3 $\pm$ 2.9	38.5 $\pm$ 3.2	69.6 $\pm$ 1.6 ( +53.0% )
Qwen3-30B-A3B	0 80.6 $\pm$ 0.9	0 80.9 $\pm$ 1.1	0 81.0 $\pm$ 0.8	0 90.2 $\pm$ 0.5 ( +11.4% )	54.3 $\pm$ 2.4	52.7 $\pm$ 2.2	52.3 $\pm$ 2.5	80.7 $\pm$ 1.1 ( +48.6% )
QwQ-32B	0 72.7 $\pm$ 1.5	0 72.2 $\pm$ 1.3	0 72.7 $\pm$ 1.2	0 75.6 $\pm$ 1.1 ( 0 +4.0% )	34.2 $\pm$ 3.4	28.9 $\pm$ 3.7	28.5 $\pm$ 3.5	57.8 $\pm$ 2.0 ( +69.0% )
DeepSeek-V3	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0 ( 0 +0.0%)	45.7 $\pm$ 2.6	44.5 $\pm$ 2.9	45.2 $\pm$ 2.7	50.9 $\pm$ 2.5 ( +11.4% )
DeepSeek-R1	0 99.9 $\pm$ 0.1	0 99.9 $\pm$ 0.0	0 99.8 $\pm$ 0.1	0 99.9 $\pm$ 0.0 ( 0 +0.0%)	61.1 $\pm$ 1.8	56.6 $\pm$ 2.3	57.9 $\pm$ 2.0	70.5 $\pm$ 1.4 ( +15.4% )

Table 2: The average answer rate and accuracy of each model across 10 tasks, computed over 10 independent runs and reported as mean

\pm

Figure 3: (A): Average Accuracy across different separate intervals. (B): Average Accuracy across different separator symbols. (C): Average Answer Rate and Accuracy across different Qwen3 model sizes: 0.6B to 14B.

Figure 4: Comparison between Vanilla and SepSeq across tasks. The left panel shows that SepSeq significantly reduces token consumption during inference (average -16.4%).

Prompting	Input Formatting
Prompting	Standard	SepSeq
Vanilla	53.25	70.35
PoT	78.90	84.45

Table 3: Average results for PoT across 10 tasks (200 samples/task).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Size	Total	Qwen3-30B-A3B-Instruct Model		Qwen3-30B-A3B-Thinking-2507
Size	Total	# Correct	Accuracy	# Correct	Accuracy
S: 2-32	50	50	100.00%	50	100.00%
M: 33-128	50	50	100.00%	36	72.00%
L: 129-256	50	50	100.00%	16	32.00%
XL: 257-512	50	7	14.00%	0	0.00%
XXL: 513-1024	50	0	0.00%	0	0.00%

Appendix

Table 4: Performance of Qwen3-30B-A3B-Instruct-2507 and Qwen3-30B-A3B-Thinking-2507 on the strict numerical sequence repetition task. The table shows the number of correct reproductions and accuracy for each sequence length range.

Task Type	Question Only (Tokens)						Full Prompt (Tokens)						#Samples
Task Type	Min	25%	50%	Mean	75%	Max	Min	25%	50%	Mean	75%	Max	#Samples
max_int	56	56	56	56.0	56	56	88	153	341	411.9	594	1070	200
min_int	56	56	56	56.0	56	56	88	149	341	410.3	594	1104	200
max_float	56	56	56	56.0	56	56	106	318	957	1252.8	1867	3613	200
min_float	56	56	56	56.0	56	56	99	334	968	1232.3	1869	3562	200
counting	24	24	24	24.0	24	24	56	117	309	392.9	565	1076	200

Appendix

Table 5: Comprehensive Token Length Statistics. Comparison between Question Only and Full Prompt across six statistical dimensions: Minimum, Quartiles (25%, 50%, 75%), Mean, and Maximum. All tasks consist of 200 samples.

Long-Context Language Model Inference

Long-context language model inference makes processing and generating over lengthy prompts faster and more memory-efficient.

See all 227 papers

AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
Apr 29, 2026Ahan Gupta, Zhihao Wang, Neel Dani +3Efficient Language Model Training LLM Training
SSAIL Lab, University of Illinois Urbana-Champaign Champaign, IL, USA · Anyscale San Francisco, CA, USA · Snowflake San Mateo, CA, USA
Training-Inference Consistent Segmented Execution for Long-Context LLMs
May 12, 2026Xianpeng Shang, Jiang Li, Zehua Duo +2Self-Attention Long-Context Language Modeling
College of Computer Science, Inner Mongolia University, Hohhot 010021, China · National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, Hohhot 010021, China · Inner Mongolia Key Laboratory of Multilingual Artificial Intelligence Technology, Hohhot 010021, China +1
Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
Jan 19, 2026Zimeng Wu, Donghao Wang, Chaozhe Jin +2LLM Inference Acceleration Long-Context LLM Inference
School of Computer Science and Engineering, Beihang University, Beijing, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China

Apr 29, 2026J/K move · Enter open · S save

AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism

Ahan Gupta, Zhihao Wang, Neel Dani, Masahiro Tanaka+2

SSAIL Lab, University of Illinois Urbana-Champaign Champaign, IL, USA · Anyscale San Francisco, CA, USA · Snowflake San Mateo, CA, USA

Efficient Language Model Training LLM Training

Open paper AskPDF arXiv

Large-language-models (LLMs) demonstrate enormous utility in long-context tasks which require processing prompts that consist of tens to hundreds of thousands of tokens. However, existing LLM training libraries do not provide easy to use abstractions to optimize for long-context training, instead focusing on optimizations for models with large parameter counts through ZeRO-3/FSDP, Tensor and Pipeline parallelism. This forces users to rewrite LLM training libraries to incorporate compositions of various complex long-context optimizations, such as sequence-parallelism, to training pipelines; a process that requires in-depth expertise, reducing developer productivity. To tackle these challenges, we introduce AutoSP: the first automated solution to automatically optimize LLM training for longer-contexts. AutoSP compiles models and applies a targeted set of optimizations: automated sequence parallelism, and long-context aware activation-checkpointing, to drastically enhance LLM trainability at negligible cost to throughput. Our evaluation demonstrates AutoSP's capability on both NVIDIA and AMD hardware, increasing training contexts by upto 2.7

\times

and 2.5

\times

respectively over competitive hand-written baseline at negligible cost to runtime performance.

Abstract

Long-Context Language Model Inference

Explore similar work

AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism

Training-Inference Consistent Segmented Execution for Long-Context LLMs

Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference