LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling

Zijian Jia; Peixuan Zhang; Genying Li; Di Wu; Yujie Fu; Si Li; Boxin Shi; Shuchen Weng

cs.AIOct 23, 2026

LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling

Authors: Zijian Jia, Peixuan Zhang, Genying Li, Di Wu, Yujie Fu, Si Li, Boxin Shi, Shuchen Weng

Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Yuanpei College, Peking University · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance · State Key Laboratory of Multimedia Information Processing, Peking University ·

Abstract

Continuous diffusion models have emerged as a promising non-autoregressive paradigm for text generation over continuous embedding spaces. Along the continuous trajectory, the residual token identity directly shapes the denoising objective. However, the geometric trajectory misaligns with temporal progression, triggering severe temporal compression. In this paper, we show that embedding norm decay causes Euclidean distance to yield spurious overtaking, indicating that token competition must be evaluated directionally. In this space, a candidate overtakes the source token only when the effective noise magnitude surpasses their semantic separation. We formalize these pairwise dynamics as the Competition Threshold Theory (CTT). Crucially, high-dimensional overtaking requires extreme noise levels, causing standard trajectories to concentrate identity decay into a narrow window near the noisy endpoint. To resolve this, we propose Semantic Retention Scheduling (SRS) to align time allocation with the progression of identity decay, and demonstrate that existing token identity metrics are fundamentally statistical aggregations of these pairwise crossings. Building on these theoretical insights, we introduce LLaCO, a generative framework powered by competition-aware semantic scheduling. LLaCO adapts a pretrained backbone to continuous representations via an embedding directional adapter and timestep-conditioned modulation. To operate within the directional space, LLaCO employs a spherical noising process formulated as a geodesic interpolation toward a mask-biased endpoint, eliminating norm decay while preserving accurate token competition. Guided by SRS, LLaCO reallocates the denoising trajectory proportionally to the logarithmic rank decay. Instantiated as an 8B model, LLaCO outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.

Figures & tables

Figure 1: Identity decay dynamics along continuous noising trajectories. (a) Top-1 source-token retention across pretrained embeddings along a standard linear trajectory, comparing directional cosine ranking against Euclidean decoding. (b) Endpoint-normalized retention statistics of unit token directions evaluated along the linear noising trajectory. (c) Endpoint-normalized retention statistics of unit token directions evaluated along the spherical geodesic trajectory. (d) Comparison between the preservation scale guaranteed by our full-spectrum theoretical bound and the empirically measured location where identity decay reaches tolerance

\delta

, evaluated across varying tolerance thresholds

\delta

with markers highlighting

\delta=0.1

. See Appendices 8.4 and 8.5 for details.

Figure 2: Predicted and measured competition statistics on pretrained embeddings against the effective noise magnitude

\kappa

: (a) expected rank

\mathbb{E}[r]

, (b) top-1 retention

P

, (c) logarithmic retention

Q_{\log}

, and (d) linear retention

Q

. Retention statistics are endpoint-normalized. Lower panels show prediction errors; shading denotes 95% bootstrap intervals.

Figure 3: Overview of the LLaCO framework. (a) The model architecture features an embedding directional adapter and timestep-conditioned modulation to produce token probabilities for cross-entropy training and clean-direction estimation. (b) Spherical noising proceeds along a geodesic trajectory toward a mask-biased endpoint. (c) Semantic Retention Scheduling leverages logarithmic rank retention to reallocate the optimization budget during both training and inference. Here,

t

denotes

t_{\mathrm{sem}}

for simplicity.

Discrete diffusion
Model / configuration	ARC-E	ARC-C	MMLU	GPQA	BBH	GSM8K	MATH	HumanEval	MBPP
LLaDA-8B	75.08	53.67	63.97	26.79	53.16	78.70	43.82	45.12	40.20
LLaDA-1.5-8B	79.46	55.20	64.19	27.01	52.72	79.38	44.26	46.34	43.20
Dream-7B	83.50	55.29	69.74	30.13	54.68	80.14	51.20	54.27	57.80
LLaCO-8B	76.68	54.78	69.89	31.25	51.30	80.74	62.12	48.17	46.20
Continuous diffusion

Table 1: Overall performance comparison and ablations. Models are grouped by generation paradigm, with LLaCO-8B listed in both groups for reference. Indented rows under LLaCO-8B alter a single factor of the default configuration and are retrained. Bold values denote the best result within each primary group.

Varying block size ( $N=B$ )					Varying steps ( $B=8$ )
$B$	GSM8K	MATH	HumanEval	MBPP	$N$	GSM8K	MATH	HumanEval	MBPP
4	80.74	62.12	48.17	46.20	4	67.32	45.80	28.66	32.60
8	70.96	47.70	31.71	33.40	8	70.96	47.70	31.71	33.40
16	48.07	24.28	15.24	20.20	16	67.48	44.76	34.15	31.40

Table 2: Effect of inference block size

B

and steps per block

N

with the default LLaCO-8B checkpoint. Bold marks the default setting, which is best on all tasks; the gray row repeats

(8,8)

for reference.

	Pairwise cosine		Mean max.	Near-orthogonal pairs (%)		Mean candidate count			Maximum cosine quantiles
Model	Mean	Std.	cosine	$\|a\|\leq 0.1$	$\|a\|\leq 0.2$	$a>0.3$	$a>0.5$	$a>0.7$	P50	P90	P99
LLaDA-8B-Instruct	0.0300	0.0415	0.4309	94.838	99.940	5.0420	0.4033	0.0024	0.4203	0.5854	0.6756
LLaDA2.0-mini	0.0197	0.0335	0.4687	98.299	99.960	8.0793	0.9089	0.0481	0.4674	0.6544	0.7469
Qwen3-8B	0.0172	0.0212	0.2316	99.912	99.998	0.3860	0.0132	0.0002	0.2117	0.3496	0.5074
Llama-3.1-8B-Instruct	0.0161	0.0196	0.3361	99.879	99.993	1.8340	0.2087	0.0073	0.3177	0.5397	0.6822
Dream-v0-Instruct-7B	0.0172	0.0336	0.3161	98.937	99.990	1.6938	0.0496	0.0002	0.3024	0.4480	0.5793

Table 3: Directional geometry of pretrained input token embeddings. Each model uses 4,096 uniformly sampled source tokens and its full eligible candidate vocabulary, excluding self-pairs. Coverage uses absolute cosine; candidate counts use signed cosine. Maximum-cosine statistics first take the maximum over candidates for each source, then compute its mean and quantiles across sources. Counts below one are averages over sources.

Half-retention effective noise magnitude ( $\kappa_{0.5}$ )
Estimate	$P$	$Q_{\log}$	$Q$
Near-orthogonal (pred.)	14.419	23.998	91.002
Spectrum (pred.)	13.941	23.261	88.605
Gaussian (meas.)	14.169	23.670	89.688
Sphere (meas.)	14.165	23.665	89.671
90%–10% transition width

Table 4: Half-retention locations and transition widths in effective-noise coordinates for the common 4,096-source, 128-noise experiment. Pred. denotes theoretical predictions; meas. denotes measurements with Gaussian or paired spherical endpoints.

$\delta$	Mean-max	Full spectrum	Measured
0.010	6.961	8.479	9.146
0.050	7.390	9.007	10.225
0.100	7.601	9.266	10.898
0.200	7.829	9.546	11.839

Table 5: Preservation scales in the separate bound-validation sample. Mean-max denotes the heuristic substitution; Full spectrum solves

B_{\mathrm{full}}(\kappa)=\delta

; Measured denotes the first-overtaking

\delta

quantile.

Family	Weighted RMSE
Endpoint-normalized Logistic	0.004
Stretched exponential	0.005
Beta survival	0.006
Generalized power	0.007

Table 6: Reported fitting errors for endpoint-constrained families relative to the isotonic

Q_{\log}

curve, weighted by bin sample counts.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 471 papers

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
Mar 3, 2026Junzhe Shen, Jieru Zhao, Ziwei He +1Continuous Diffusion Language Models Diffusion Language Model Decoding
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University · Shanghai Innovation Institute +1
Continuous Latent Diffusion Language Model
May 7, 2026Hongcan Guo, Qinyu Zhao, Yian Zhao +8Non-Autoregressive Text Generation Diffusion Transformer
ByteDance Seed · The Australian National University · Peking University +2
Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models
Oct 1, 2026Yair Schiff, Omer Belhasin, Roy Uziel +7Diffusion Language Model Inference Language Modeling
NVIDIA · Cornell University

Mar 3, 2026J/K move · Enter open · S save

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

Junzhe Shen, Jieru Zhao, Ziwei He, Zhouhan Lin

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University · Shanghai Innovation Institute+1

Continuous Diffusion Language Models Diffusion Language Model Decoding

Open paper AskPDF arXiv

We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.

Abstract

Diffusion Language Models

Explore similar work

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

Continuous Latent Diffusion Language Model

Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models