LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling
Authors: Zijian Jia, Peixuan Zhang, Genying Li, Di Wu, Yujie Fu, Si Li, Boxin Shi, Shuchen Weng
Organizations: School of Artificial Intelligence, Beijing University of Posts and Telecommunications · Yuanpei College, Peking University · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance · State Key Laboratory of Multimedia Information Processing, Peking University · National Engineering Research Center of Visual Technology, Peking University · Beijing Academy of Artificial Intelligence
Continuous diffusion models have emerged as a promising non-autoregressive paradigm for text generation over continuous embedding spaces. Along the continuous trajectory, the residual token identity directly shapes the denoising objective. However, the geometric trajectory misaligns with temporal progression, triggering severe temporal compression. In this paper, we show that embedding norm decay causes Euclidean distance to yield spurious overtaking, indicating that token competition must be evaluated directionally. In this space, a candidate overtakes the source token only when the effective noise magnitude surpasses their semantic separation. We formalize these pairwise dynamics as the Competition Threshold Theory (CTT). Crucially, high-dimensional overtaking requires extreme noise levels, causing standard trajectories to concentrate identity decay into a narrow window near the noisy endpoint. To resolve this, we propose Semantic Retention Scheduling (SRS) to align time allocation with the progression of identity decay, and demonstrate that existing token identity metrics are fundamentally statistical aggregations of these pairwise crossings. Building on these theoretical insights, we introduce LLaCO, a generative framework powered by competition-aware semantic scheduling. LLaCO adapts a pretrained backbone to continuous representations via an embedding directional adapter and timestep-conditioned modulation. To operate within the directional space, LLaCO employs a spherical noising process formulated as a geodesic interpolation toward a mask-biased endpoint, eliminating norm decay while preserving accurate token competition. Guided by SRS, LLaCO reallocates the denoising trajectory proportionally to the logarithmic rank decay. Instantiated as an 8B model, LLaCO outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.
Figures & tables
Figure 1: Identity decay dynamics along continuous noising trajectories. (a) Top-1 source-token retention across pretrained embeddings along a standard linear trajectory, comparing directional cosine ranking against Euclidean decoding. (b) Endpoint-normalized retention statistics of unit token directions evaluated along the linear noising trajectory. (c) Endpoint-normalized retention statistics of unit token directions evaluated along the spherical geodesic trajectory. (d) Comparison between the preservation scale guaranteed by our full-spectrum theoretical bound and the empirically measured location where identity decay reaches tolerance δ , evaluated across varying tolerance thresholds δ with markers highlighting δ=0.1 . See Appendices 8.4 and 8.5 for details.
Figure 2: Predicted and measured competition statistics on pretrained embeddings against the effective noise magnitude κ : (a) expected rank E[r] , (b) top-1 retention P , (c) logarithmic retention Qlog , and (d) linear retention Q . Retention statistics are endpoint-normalized. Lower panels show prediction errors; shading denotes 95% bootstrap intervals.
Figure 3: Overview of the LLaCO framework. (a) The model architecture features an embedding directional adapter and timestep-conditioned modulation to produce token probabilities for cross-entropy training and clean-direction estimation. (b) Spherical noising proceeds along a geodesic trajectory toward a mask-biased endpoint. (c) Semantic Retention Scheduling leverages logarithmic rank retention to reallocate the optimization budget during both training and inference. Here, t denotes tsem for simplicity.
Model / configuration
ARC-E
ARC-C
MMLU
GPQA
BBH
GSM8K
MATH
HumanEval
MBPP
Discrete diffusion
LLaDA-8B
75.08
53.67
63.97
26.79
53.16
78.70
43.82
45.12
40.20
LLaDA-1.5-8B
79.46
55.20
64.19
27.01
52.72
79.38
44.26
46.34
43.20
Dream-7B
83.50
55.29
69.74
30.13
54.68
80.14
51.20
54.27
57.80
LLaCO-8B
76.68
54.78
69.89
31.25
51.30
80.74
62.12
48.17
46.20
Continuous diffusion
Table 1: Overall performance comparison and ablations. Models are grouped by generation paradigm, with LLaCO-8B listed in both groups for reference. Indented rows under LLaCO-8B alter a single factor of the default configuration and are retrained. Bold values denote the best result within each primary group.
Varying block size ( N=B )
Varying steps ( B=8 )
B
GSM8K
MATH
HumanEval
MBPP
N
GSM8K
MATH
HumanEval
MBPP
4
80.74
62.12
48.17
46.20
4
67.32
45.80
28.66
32.60
8
70.96
47.70
31.71
33.40
8
70.96
47.70
31.71
33.40
16
48.07
24.28
15.24
20.20
16
67.48
44.76
34.15
31.40
Table 2: Effect of inference block size B and steps per block N with the default LLaCO-8B checkpoint. Bold marks the default setting, which is best on all tasks; the gray row repeats (8,8) for reference.
Pairwise cosine
Mean max.
Near-orthogonal pairs (%)
Mean candidate count
Maximum cosine quantiles
Model
Mean
Std.
cosine
∣a∣≤0.1
∣a∣≤0.2
a>0.3
a>0.5
a>0.7
P50
P90
P99
LLaDA-8B-Instruct
0.0300
0.0415
0.4309
94.838
99.940
5.0420
0.4033
0.0024
0.4203
0.5854
0.6756
LLaDA2.0-mini
0.0197
0.0335
0.4687
98.299
99.960
8.0793
0.9089
0.0481
0.4674
0.6544
0.7469
Qwen3-8B
0.0172
0.0212
0.2316
99.912
99.998
0.3860
0.0132
0.0002
0.2117
0.3496
0.5074
Llama-3.1-8B-Instruct
0.0161
0.0196
0.3361
99.879
99.993
1.8340
0.2087
0.0073
0.3177
0.5397
0.6822
Dream-v0-Instruct-7B
0.0172
0.0336
0.3161
98.937
99.990
1.6938
0.0496
0.0002
0.3024
0.4480
0.5793
Table 3: Directional geometry of pretrained input token embeddings. Each model uses 4,096 uniformly sampled source tokens and its full eligible candidate vocabulary, excluding self-pairs. Coverage uses absolute cosine; candidate counts use signed cosine. Maximum-cosine statistics first take the maximum over candidates for each source, then compute its mean and quantiles across sources. Counts below one are averages over sources.
Table 4: Half-retention locations and transition widths in effective-noise coordinates for the common 4,096-source, 128-noise experiment. Pred. denotes theoretical predictions; meas. denotes measurements with Gaussian or paired spherical endpoints.
δ
Mean-max
Full spectrum
Measured
0.010
6.961
8.479
9.146
0.050
7.390
9.007
10.225
0.100
7.601
9.266
10.898
0.200
7.829
9.546
11.839
Table 5: Preservation scales in the separate bound-validation sample. Mean-max denotes the heuristic substitution; Full spectrum solves Bfull(κ)=δ ; Measured denotes the first-overtaking δ quantile.
Family
Weighted RMSE
Endpoint-normalized Logistic
0.004
Stretched exponential
0.005
Beta survival
0.006
Generalized power
0.007
Table 6: Reported fitting errors for endpoint-constrained families relative to the isotonic Qlog curve, weighted by bin sample counts.
We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.
Junzhe Shen, Jieru Zhao, Ziwei He +1
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University · Shanghai Innovation Institute +1
Large language models have achieved remarkable success under the autoregressive paradigm, yet high-quality text generation need not be tied to a fixed left-to-right order. Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling. We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition. Cola DLM first learns a stable text-to-latent mapping with a Text VAE, then models a global semantic prior in continuous latent space with a block-causal DiT, and finally generates text through conditional decoding. From a unified Markov-path perspective, its diffusion process performs latent prior transport rather than token-level observation recovery, thereby separating global semantic organization from local textual realization. This design yields a more flexible non-autoregressive inductive bias, supports semantic compression and prior fitting in continuous space, and naturally extends to other continuous modalities. Through experiments spanning 4 research questions, 8 benchmarks, strictly matched ~2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs, we identify an effective overall configuration of Cola DLM and verify its strong scaling behavior for text generation. Taken together, the results establish hierarchical continuous latent prior modeling as a principled alternative to strictly token-level language modeling, where generation quality and scaling behavior may better reflect model capability than likelihood, while also suggesting a concrete path toward unified modeling across discrete text and continuous modalities.
Hongcan Guo, Qinyu Zhao, Yian Zhao +8
ByteDance Seed · The Australian National University · Peking University +2
Recent works on continuous diffusion for discrete data have demonstrated performance on par with comparable discrete diffusion models. However, these continuous counterparts lack key features that are essential to practical use as language models, namely variable-length generation and support for a key-value cache, and they still lag behind the frontier of autoregressive and discrete diffusion quality. In this work, we address these limitations. We do so by introducing a model parameterization that uses position-dependent noise schedules to define semi-autoregressive (SAR) continuous diffusion language models (DLMs). Together with efficient training and sampling algorithms, we call this framework Clock Diffusion, and we present two special cases of our method: block and sliding window generation. We then define ClockDLMs, a family of Gaussian DLMs based on sliding window Clock Diffusion that attain state-of-the-art diffusion likelihood bounds on OpenWebText, even beating the performant block SAR discrete diffusion models. ClockDLMs trained on TinyGSM also substantially outperform continuous baselines on the GSM8K benchmark and match and exceed comparable SAR discrete diffusion models. Finally, building on our parameterization, we propose more efficient samplers that we dub Cache Grab, which adapt techniques from accelerated inference in discrete diffusion, such as committing tokens whose probabilities exceed a confidence threshold and self-speculative decoding, further improving our models' quality and efficiency.