Can Large Language Models "Hyper-Thread"?

Fei Ding

cs.CLAug 23, 2026

Can Large Language Models "Hyper-Thread"?

Authors: Fei Ding

Organizations: Alibaba Group

Abstract

Large language models generate tokens sequentially, but can they execute multiple tasks concurrently while forming each token? Broader attention allocation may provide a mechanism for such task concurrency. Existing approaches to scaling inference primarily rely on longer generations, more samples, or additional verification stages, while attention dispersion is often treated as a signal of interference or error. Task concurrency within serial generation therefore remains underexplored. We propose the Model Hyper-Threading Hypothesis and evaluate its predictions using multiple coordinated tasks that share state within the same problem. We design three conditions (Baseline, Serial Functional Scheduling, and Concurrent Functional Loading) and evaluate their benefits and costs using accuracy, output-token distributions, and attention metrics. On an AIME 2025 development set, Concurrent Functional Loading achieves the highest accuracy. Relative to Serial Functional Scheduling, its typical output length is similar and it is shorter on most problems, while exhibiting greater attention dispersion and higher task-relevant coverage, albeit with a heavier output-length tail. Within-step concurrency and its causal mechanism still require direct tests. Our results show that more dispersed attention can coexist with higher accuracy, providing preliminary behavioral and correlational evidence for the hyper-threading hypothesis. These findings motivate a shift in perspective on inference scaling from "generating more tokens" toward "having each generation step carry more tasks," pointing to a new avenue for improving reasoning performance.

Figures & tables

Condition	Functional organization	Additional functional requirements
Baseline	No additional scheduling	Only requires a Chinese response and a standardized answer format.
Serial Functional Scheduling	Serial staged scheduling	Applies the shared verification objectives through grouped checks after each paragraph.
Concurrent Functional Loading	Concurrent joint loading	Maintains the same objectives throughout generation, interrupting and repairing immediately when an error is detected.

Table 1: The three conditions in the within-problem functional experiment.

Condition	Accuracy	Mean tokens	Median tokens	P90	P95	Maximum	$H_{\mathrm{LB}}$	Top-1 $p$
Baseline	16/30 (53.33%)	2,932.73	1,229.0	7,591	10,461	24,258	0.6750	84.4059%
Serial Functional Scheduling	17/30 (56.67%)	2,689.83	1,564.5	6,018	7,424	10,397	0.6955	83.9987%
Concurrent Functional Loading	23/30 (76.67%)	4,117.47	1,531.0	7,655	23,418	24,744	0.7110	83.7262%

Table 2: Exploratory main results on all 30 AIME 2025 problems. P90/P95 use the empirical nearest-rank definition.

Comparison	Accuracy difference	95% bootstrap interval	Incorrect $\rightarrow$ correct	Correct $\rightarrow$ incorrect	McNemar $p$
Concurrent Functional Loading $-$ Baseline	+23.33 pp	$[+6.67,+40.00]$ pp	8	1	0.0391
Concurrent Functional Loading $-$ Serial Functional Scheduling	+20.00 pp	$[+3.33,+36.67]$ pp	7	1	0.0703
Serial Functional Scheduling $-$ Baseline	+3.33 pp	$[-10.00,+16.67]$ pp	3	2	1.0000

Table 3: Paired accuracy changes on the same problems. Each direction is the former condition relative to the latter.

Comparison	Mean difference	95% bootstrap interval	Median difference	Problems where former is shorter
Concurrent Functional Loading $-$ Baseline	$+1{,}184.73$	$[+256.6,+2387.7]$	$+113.5$	11/30
Concurrent Functional Loading $-$ Serial Functional Scheduling	$+1{,}427.63$	$[-420.2,+3683.0]$	$-165.5$	17/30

Table 4: Paired output-length changes under Concurrent Functional Loading relative to the controls. Differences are the former minus the latter.

Problem	Baseline	Concurrent Functional Loading	Serial Functional Scheduling
I_13	$181\times$ (9,269)	$204\checkmark$ (15,164)	$146\times$ (1,858)
I_14	$53\times$ (10,461)	$56\times$ (24,744)	$23\times$ (2,727)
II_10	$907\checkmark$ (1,008)	$907\checkmark$ (7,655)	$907\checkmark$ (896)
II_13	$216\times$ (24,258)	$48\times$ (23,418)	$375\times$ (7,424)

Table 5: Four post hoc identified long-tail cases. Parentheses contain output tokens;

\checkmark/\times

denote correct/incorrect.

Condition or comparison	$H_{\mathrm{norm}}$	$R_{\mathrm{eff}}$	$D_{\tau}$	$C_{\mathrm{rel}}$
Baseline	$0.530$	$0.043$	$0.035$	$0.540$
Serial Functional Scheduling	$0.550$	$0.045$	$0.039$	$0.580$
Concurrent Functional Loading	$0.580$	$0.051$	$0.048$	$0.640$
Concurrent Functional Loading $-$ Baseline	$+0.050$	$+0.009$	$+0.012$	$+0.100$
Concurrent Functional Loading $-$ Serial Functional Scheduling	$+0.030$	$+0.007$	$+0.008$	$+0.060$

Table 6: Evidence from direct attention measurements;

D_{\tau}

uses

\tau=2

. Entries are group means rounded to three decimal places; differences are computed from unrounded values.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Accuracy	Total output tokens	Mean tokens	Median tokens
Baseline	15/26 (57.69%)	42,986	1,653.31	1,159.5
Serial Functional Scheduling	16/26 (61.54%)	67,790	2,607.31	1,554.0
Concurrent Functional Loading	21/26 (80.77%)	52,543	2,020.88	1,348.0

Appendix

Table 7: Sensitivity analysis on 26 problems after removing four post hoc identified long-tail cases; this does not replace the main result on all 30 problems.

Currently supported	Currently unsupported
Concurrent Functional Loading has a higher point estimate of accuracy in a single development-set run.	Concurrent execution of multiple functions within one token has been established.
The advantage of Concurrent Functional Loading does not accompany universal problem-wise lengthening.	Concurrent Functional Loading uses fewer tokens on average in the full sample or is resource-equivalent.
Mean-length differences are materially affected by a small number of search tails.	The result after removing four problems post hoc is the primary result.
The results are consistent with the behavioral predictions of the hyper-threading hypothesis.	Output predictive entropy is self-attention dispersion or mechanistic evidence.
Measured attention dispersion and task-relevant coverage coexist with the accuracy advantage.	Attention dispersion has been established as the cause of higher accuracy or within-step functional concurrency.

Appendix

Table 8: Claims supported and unsupported by the current evidence.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 276 papers

Nov 24, 2025

ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models

Scaling inference-time computation has enabled Large Language Models (LLMs) to achieve strong reasoning performance, but their inherently sequential decoding incurs substantial latency, motivating parallelization of the generation process. However, existing parallel reasoning approaches suffer from performance degradation compared to their sequential counterparts, and often rely on specialized inference engines. We introduce ThreadWeaver, a framework for adaptive parallel reasoning that matches the accuracy of comparably sized sequential reasoning models while significantly reducing inference latency via three key innovations: 1) a two-stage parallel trajectory generator that produces high-quality parallel chain-of-thought data for supervised fine-tuning; 2) a trie-based rollout design that enables parallel reasoning on any off-the-shelf autoregressive inference engine; and 3) a parallelization-aware reinforcement learning framework that trains the model to balance reasoning accuracy with effective parallelization. Across six challenging math reasoning benchmarks, ThreadWeaver trained on top of Qwen3-8B achieves performance on par with cutting-edge sequential reasoning models (79.9% on AIME24 and 71.9% on average) while delivering up to 1.53x speedup in token latency, establishing a new Pareto frontier between accuracy and efficiency.

Long Lian, Sida Wang, Felix Juefei-Xu +7

Meta Superintelligence Labs (MSL), Menlo Park, CA, United States · UC Berkeley, Berkeley, CA, United States · UCSF, San Francisco, CA, United States

Parallel Reasoning in Language Models RL for Language Model Reasoning Inference-Time Scaling Efficient Language Model Reasoning LLM Inference Acceleration Large Reasoning Models

Ask PDF

Jul 1, 2026

Message Passing Enables Efficient Reasoning

While inference-time scaling has improved the reasoning abilities of large language models (LLMs), the need to generate long chains-of-thought (CoTs) is a computational bottleneck. Thus, in contrast to sequential scaling methods like CoT, recent parallel scaling techniques instead use fork and join (FJ) primitives to divide work across multiple LLM threads. However, in the fork-join paradigm, threads are typically transient and do not communicate pointwise with one another which limits scalability. To tackle this, we introduce Message Passing Language Models (MPLMs), a framework for LLM reasoning in which threads communicate directly via lightweight send and receive primitives. MPLMs enable efficient scaling through two key mechanisms: (1) reduced communication costs, achieved by avoiding redundant context sharing, and (2) preemption, which allows threads to terminate early based on partial information from their peers. We demonstrate the promise of MPLMs on 3 classes of tasks. First, on Sudoku puzzles, we show that MPLMs require an asymptotically smaller context than both serial CoT and parallel FJ. We then fine-tune a single model to solve 25 x 25 puzzles that remain challenging for standard CoT and FJ approaches, as well as frontier reasoning models without tools. Second, on 3-SAT puzzles, the capability of preemption allows termination of unpromising branches, which results in improved efficiency. Finally, we show that appropriately prompted large pre-trained models follow the MPLM protocol, achieving competitive results on long-context question answering relative to popular fork-join approaches.

Xuecheng Liu, Daman Arora, Gokul Swamy +1

Carnegie Mellon University

Parallel Reasoning in Language Models Inference-Time Scaling Efficient Language Model Reasoning LLM Inference Acceleration Language Modeling Large Reasoning Models

Ask PDF

Sep 13, 2026

Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference

Large language models (LLMs) demonstrate impressive capabilities, but their deployment presents significant efficiency challenges. Autoregressive decoding imposes substantial inference latency and under-utilizes hardware accelerators in low batch size regimes. Discrete diffusion models can generate in parallel but struggle to match autoregressive quality without many diffusion denoising steps. Long-context reasoning creates memory bottlenecks that strain even state-of-the-art accelerators. My thesis is that language models can direct their own inference execution strategy by annotating semantic dependence -- which tokens depend on which others -- in their generation. I call such models self-orchestrating language models. For each system, I design a runtime that acts on these annotations to parallelize autoregressive decoding, evict intermediate context, or derive denoising orders, achieving Pareto-optimal quality-efficiency trade-offs. I demonstrate this approach through three self-orchestrating systems. First, PASTA uses semantic dependence to parallelize autoregressive decoding, training the model to annotate which output chunks can generate independently. Second, TIP uses semantic dependence to evict intermediate reasoning steps from the KV cache, reducing memory consumption while preserving accuracy. Third, Planned Diffusion uses semantic dependence to derive a denoising order for discrete diffusion, autoregressively generating a plan that specifies which chunks to denoise in parallel.

Tian Jin

Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY

Autoregressive Decoding Efficient Language Model Reasoning Diffusion Language Model Inference Long-Context Language Model Inference LLM Inference Acceleration Language Modeling+1

Ask PDF

Abstract

LLM Inference

Explore similar work

ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models

Message Passing Enables Efficient Reasoning

Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference