Can Large Language Models "Hyper-Thread"?

Fei Ding

cs.CLAug 23, 2026

Can Large Language Models "Hyper-Thread"?

Authors: Fei Ding

Organizations: Alibaba Group

Abstract

Large language models generate tokens sequentially, but can they execute multiple tasks concurrently while forming each token? Broader attention allocation may provide a mechanism for such task concurrency. Existing approaches to scaling inference primarily rely on longer generations, more samples, or additional verification stages, while attention dispersion is often treated as a signal of interference or error. Task concurrency within serial generation therefore remains underexplored. We propose the Model Hyper-Threading Hypothesis and evaluate its predictions using multiple coordinated tasks that share state within the same problem. We design three conditions (Baseline, Serial Functional Scheduling, and Concurrent Functional Loading) and evaluate their benefits and costs using accuracy, output-token distributions, and attention metrics. On an AIME 2025 development set, Concurrent Functional Loading achieves the highest accuracy. Relative to Serial Functional Scheduling, its typical output length is similar and it is shorter on most problems, while exhibiting greater attention dispersion and higher task-relevant coverage, albeit with a heavier output-length tail. Within-step concurrency and its causal mechanism still require direct tests. Our results show that more dispersed attention can coexist with higher accuracy, providing preliminary behavioral and correlational evidence for the hyper-threading hypothesis. These findings motivate a shift in perspective on inference scaling from "generating more tokens" toward "having each generation step carry more tasks," pointing to a new avenue for improving reasoning performance.

Figures & tables

Condition	Functional organization	Additional functional requirements
Baseline	No additional scheduling	Only requires a Chinese response and a standardized answer format.
Serial Functional Scheduling	Serial staged scheduling	Applies the shared verification objectives through grouped checks after each paragraph.
Concurrent Functional Loading	Concurrent joint loading	Maintains the same objectives throughout generation, interrupting and repairing immediately when an error is detected.

Table 1: The three conditions in the within-problem functional experiment.

Condition	Accuracy	Mean tokens	Median tokens	P90	P95	Maximum	$H_{\mathrm{LB}}$	Top-1 $p$
Baseline	16/30 (53.33%)	2,932.73	1,229.0	7,591	10,461	24,258	0.6750	84.4059%
Serial Functional Scheduling	17/30 (56.67%)	2,689.83	1,564.5	6,018	7,424	10,397	0.6955	83.9987%
Concurrent Functional Loading	23/30 (76.67%)	4,117.47	1,531.0	7,655	23,418	24,744	0.7110	83.7262%

Table 2: Exploratory main results on all 30 AIME 2025 problems. P90/P95 use the empirical nearest-rank definition.

Comparison	Accuracy difference	95% bootstrap interval	Incorrect $\rightarrow$ correct	Correct $\rightarrow$ incorrect	McNemar $p$
Concurrent Functional Loading $-$ Baseline	+23.33 pp	$[+6.67,+40.00]$ pp	8	1	0.0391
Concurrent Functional Loading $-$ Serial Functional Scheduling	+20.00 pp	$[+3.33,+36.67]$ pp	7	1	0.0703
Serial Functional Scheduling $-$ Baseline	+3.33 pp	$[-10.00,+16.67]$ pp	3	2	1.0000

Table 3: Paired accuracy changes on the same problems. Each direction is the former condition relative to the latter.

Comparison	Mean difference	95% bootstrap interval	Median difference	Problems where former is shorter
Concurrent Functional Loading $-$ Baseline	$+1{,}184.73$	$[+256.6,+2387.7]$	$+113.5$	11/30
Concurrent Functional Loading $-$ Serial Functional Scheduling	$+1{,}427.63$	$[-420.2,+3683.0]$	$-165.5$	17/30

Table 4: Paired output-length changes under Concurrent Functional Loading relative to the controls. Differences are the former minus the latter.

Problem	Baseline	Concurrent Functional Loading	Serial Functional Scheduling
I_13	$181\times$ (9,269)	$204\checkmark$ (15,164)	$146\times$ (1,858)
I_14	$53\times$ (10,461)	$56\times$ (24,744)	$23\times$ (2,727)
II_10	$907\checkmark$ (1,008)	$907\checkmark$ (7,655)	$907\checkmark$ (896)
II_13	$216\times$ (24,258)	$48\times$ (23,418)	$375\times$ (7,424)

Table 5: Four post hoc identified long-tail cases. Parentheses contain output tokens;

\checkmark/\times

denote correct/incorrect.

Condition or comparison	$H_{\mathrm{norm}}$	$R_{\mathrm{eff}}$	$D_{\tau}$	$C_{\mathrm{rel}}$
Baseline	$0.530$	$0.043$	$0.035$	$0.540$
Serial Functional Scheduling	$0.550$	$0.045$	$0.039$	$0.580$
Concurrent Functional Loading	$0.580$	$0.051$	$0.048$	$0.640$
Concurrent Functional Loading $-$ Baseline	$+0.050$	$+0.009$	$+0.012$	$+0.100$
Concurrent Functional Loading $-$ Serial Functional Scheduling	$+0.030$	$+0.007$	$+0.008$	$+0.060$

Table 6: Evidence from direct attention measurements;

D_{\tau}

uses

\tau=2

. Entries are group means rounded to three decimal places; differences are computed from unrounded values.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Accuracy	Total output tokens	Mean tokens	Median tokens
Baseline	15/26 (57.69%)	42,986	1,653.31	1,159.5
Serial Functional Scheduling	16/26 (61.54%)	67,790	2,607.31	1,554.0
Concurrent Functional Loading	21/26 (80.77%)	52,543	2,020.88	1,348.0

Appendix

Table 7: Sensitivity analysis on 26 problems after removing four post hoc identified long-tail cases; this does not replace the main result on all 30 problems.

Currently supported	Currently unsupported
Concurrent Functional Loading has a higher point estimate of accuracy in a single development-set run.	Concurrent execution of multiple functions within one token has been established.
The advantage of Concurrent Functional Loading does not accompany universal problem-wise lengthening.	Concurrent Functional Loading uses fewer tokens on average in the full sample or is resource-equivalent.
Mean-length differences are materially affected by a small number of search tails.	The result after removing four problems post hoc is the primary result.
The results are consistent with the behavioral predictions of the hyper-threading hypothesis.	Output predictive entropy is self-attention dispersion or mechanistic evidence.
Measured attention dispersion and task-relevant coverage coexist with the accuracy advantage.	Attention dispersion has been established as the cause of higher accuracy or within-step functional concurrency.

Appendix

Table 8: Claims supported and unsupported by the current evidence.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 280 papers

ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models
Nov 24, 2025Long Lian, Sida Wang, Felix Juefei-Xu +7Parallel Reasoning in Language Models RL for Language Model Reasoning
Meta Superintelligence Labs (MSL), Menlo Park, CA, United States · UC Berkeley, Berkeley, CA, United States · UCSF, San Francisco, CA, United States
Message Passing Enables Efficient Reasoning
Jul 1, 2026Xuecheng Liu, Daman Arora, Gokul Swamy +1Parallel Reasoning in Language Models Inference-Time Scaling
Carnegie Mellon University
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinAutoregressive Decoding Efficient Language Model Reasoning
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY

Nov 24, 2025J/K move · Enter open · S save

ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models

Long Lian, Sida Wang, Felix Juefei-Xu, Tsu-Jui Fu+6

Meta Superintelligence Labs (MSL), Menlo Park, CA, United States · UC Berkeley, Berkeley, CA, United States · UCSF, San Francisco, CA, United States

Parallel Reasoning in Language Models RL for Language Model Reasoning Inference-Time Scaling Efficient Language Model Reasoning LLM Inference Acceleration Large Reasoning Models

Open paper AskPDF arXiv

Scaling inference-time computation has enabled Large Language Models (LLMs) to achieve strong reasoning performance, but their inherently sequential decoding incurs substantial latency, motivating parallelization of the generation process. However, existing parallel reasoning approaches suffer from performance degradation compared to their sequential counterparts, and often rely on specialized inference engines. We introduce ThreadWeaver, a framework for adaptive parallel reasoning that matches the accuracy of comparably sized sequential reasoning models while significantly reducing inference latency via three key innovations: 1) a two-stage parallel trajectory generator that produces high-quality parallel chain-of-thought data for supervised fine-tuning; 2) a trie-based rollout design that enables parallel reasoning on any off-the-shelf autoregressive inference engine; and 3) a parallelization-aware reinforcement learning framework that trains the model to balance reasoning accuracy with effective parallelization. Across six challenging math reasoning benchmarks, ThreadWeaver trained on top of Qwen3-8B achieves performance on par with cutting-edge sequential reasoning models (79.9% on AIME24 and 71.9% on average) while delivering up to 1.53x speedup in token latency, establishing a new Pareto frontier between accuracy and efficiency.

Abstract

LLM Inference

Explore similar work

ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models

Message Passing Enables Efficient Reasoning

Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference