Diffusion models promise efficient parallel text generation but rely on bidirectional attention, creating a structural mismatch with pre-trained Autoregressive (AR) models. This incompatibility precludes reusing robust AR priors, necessitating prohibitive pre-training from scratch. To bridge this gap, we propose FLUID, a framework that efficiently adapts AR backbones to the diffusion paradigm. By enforcing Strictly Causal Alignment, FLUID enables seamless initialization from standard GPT-style checkpoints, circumventing the need for massive pre-training. Furthermore, we introduce Elastic Horizons, an entropy-driven mechanism that dynamically modulates denoising strides based on local information density rather than fixed schedules. Experiments demonstrate that FLUID achieves state-of-the-art performance while reducing training costs by orders of magnitude, effectively reconciling established AR foundations with efficient parallel generation. Our code is available at https://github.com/Oli-lab-nun/FLUID/tree/main.
Figures & tables
Figure 1: Causal mismatch in fixed-size block diffusion. First-pass accuracy decays significantly faster for high-entropy data (e.g., complex reasoning steps in MATH500) than low-entropy text (e.g., GSM8k) as lookahead increases.
Figure 2: From Autoregressive to FLUID: Evolution of four language model generation paradigms. The diagram highlights the pivotal transition to FLUID (d), which integrates Unidirectional Attention to maintain strictly causal alignment with AR priors, and Dynamic Mask Blocks to adaptively modulate generation horizons based on sequence entropy, resolving the structural and dynamic mismatches inherent in prior paradigms.
Figure 3: The Schematic Framework of FLUID. The Macro View (Top) depicts the Adaptive Diffusion Progression, where the model dynamically adjusts the lookahead horizon Kt . The Micro View (Bottom) details the mechanism at step t : the Elastic Horizon Modeling derives the optimal window size from hidden states to guide the Elastic Horizon Diffusion Process, which iteratively refines masked tokens under strict causal constraints.
Figure 4: Illustration of Training Objective. The original sequence is transformed via dynamic expansion. Block x1 ( k=4 ) exhibits a mixed state of stochastically restored ground-truth ( x2 ) and injected noise ( x∗ ), governed by a strictly causal attention matrix.
General
Math & Science
Code
Model
Type
Tokens
MMLU
IFEVAL
GSM8K
MATH500
HEval
MBPP
LLaMA-3-8B †
AR
15T
68.4 (5)
49.7 (0)
78.3 (4)
27.4 (0)
59.8 (0)
57.6 (4)
Qwen-2.5-7B †
AR
18T
76.6 (5)
74.7 (0)
91.6 (0)
72.8 (0)
84.8 (0)
79.2 (4)
Gemma2 9B †
AR
8T
–
–
76.7 (0)
–
68.9 (0)
74.9 (0)
Deepseek 7B ¶
AR
2T
49.4 (0)
–
63.0 (0)
–
48.2 (-)
35.2 (-)
LLaDA-8B
Diff
2.0T
65.5 (5)
59.9 (0)
78.6 (4)
36.2 (0)
47.6 (0)
34.2 (4)
Table 1: Main Comparison on Standard Benchmarks. We compare FLUID against state-of-the-art Diffusion and AR baselines under Instruct settings. “Tokens” denotes the amount of data used for pre-training (for AR) or adaptation (for Diffusion). Note that FLUID achieves comparable performance to strong baselines using orders of magnitude less training data (2.7B vs. Trillions). Results indicated by † and ¶ are sourced from Chu et al. (2024) , Yang et al. (2024) , and Bi et al. (2024) . Best results in diffusion methods are bolded , and our results are highlighted in aaa .
Figure 5: Semantic Quality Evaluation via Skywork-Reward-V2.
Figure 6: Efficiency and Accuracy Comparison. (a) All models are evaluated without KV cache or FlashAttention to ensure a fair architectural comparison. (b) Verification of K-Head Accuracy.
Components
Benchmarks
Method
Causal
Elastic
GSM8K
MATH500
HEval
Baseline
✗
✗
82.0
51.2
42.2
Baseline
✗
✓
82.5
53.6
42.8
Baseline
✓
✗
90.6
59.2
54.9
FLUID (Ours)
✓
✓
91.9
61.8
60.4
Table 2: Ablation study on adaptation strategies. Impact of strictly causal masking ( Causal ) and entropy-aware dynamic horizons ( Elastic ) relative to a bidirectional fixed-block baseline.
Noise Ratio
GSM8K
MATH500
HEval
0%
91.0
60.8
59.8
5%
91.3
61.1
59.8
10% (Ours)
91.9
61.8
60.4
15%
91.1
61.5
60.0
Table 3: Ablation on the stochastic restoration noise ratio in Stage I training. Impact of varying proportions of injected noise within the masked span.
Threshold τ
GSM8K
MATH500
HEval
2.6
90.9
60.4
59.8
2.7
91.1
61.5
59.9
2.8 (Ours)
91.9
61.8
60.4
2.9
90.5
61.5
59.5
3.0
90.4
60.2
59.3
Table 4: Ablation study on the competence boundary τ .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Decoding Trajectories of Bidirectional Diffusion. (a) Incremental decoding ( K=1 ) triggers an unintended sequential path, revealing a structural redundancy in bidirectional attention. (b) Block-wise decoding ( K=4 ) exhibits bi-terminal convergence, where non-linear filling from sequence boundaries leads to causal mismatch. These behaviors substantiate FLUID’s design of strictly causal alignment and adaptive horizon modulation.
Figure 8: Training loss convergence of FLUID during Stage I. The rapid stabilization post-initialization highlights the efficiency of adapting pre-trained AR models to the diffusion paradigm via strictly causal alignment.
Figure 9: Visualization of FLUID’s decoding trajectory, token entropy, and adaptive horizon on an arithmetic reasoning prompt.
Stage
Trained Module
Hardware
Est. Time
Stage I
LoRA Adapter
4 × 64GB GPUs
∼ 80 hours
Stage II
K-Head
4 × 64GB GPUs
negligible
Total
–
4 × 64GB GPUs
∼ 320 GPU-hours
Appendix
Table 5: Computational overhead of FLUID adaptation. The total cost is approximately 320 GPU-hours, with Stage I accounting for nearly all of the training time.