We introduce Clover-35B-A3B-Preview (Clover), a mixture-of-experts diffusion language model with 35 billion total parameters and 3 billion activated parameters, supporting thinking-mode generation. Clover adapts Gated Delta Networks to block diffusion by combining recurrent memory of previously generated context with bidirectional softmax attention inside the current block. Historical memory readout competes with current-block tokens in a shared softmax, with learnable block-size calibration and a separate, zero-initialized right-context convolution branch. Clover leads the seven evaluated agent metrics, including 73.6 on SWE-bench Verified and 51.7 on Terminal-Bench 2.1. On internal agent workloads, it reaches a reported generation speed of 680 tokens/s using one NVIDIA GB300 GPU at BF16 precision and batch size one. We organize the evaluation around model quality, drafting and verification costs, and deployment under constrained memory. We also describe a decision-model variant that generates answers to related questions in one denoising pass. On an internal set of 500 groups, joint prediction raises question accuracy from 90.37% to 93.88% and all-correct group accuracy from 65.4% to 77.0%.
Figures & tables
Figure 1: Clover architecture. (a) Three BD-GDN–dMoE layers and one full-attention–dMoE layer form a group repeated ten times. Backbone normalizations are RMSNorm. (b) Pink boxes are linear projections; the Q , K , V , and Z paths carry query, key, value, and output-gate features. Stacked boxes indicate multiple heads: 16 query heads, 16 key heads, and 32 value heads; each query/key head serves two value heads. The historical readout from Sb and block-local information are combined by the shared softmax. Output RMSNorm operates per value head, and gated head outputs are concatenated before the output projection. (c) dMoE combines token-level top-eight routing within a block coreset with a shared expert. Solid lines carry features, dashed arrows show routing, and gray experts are inactive. For one token, pi is the normalized weight of routed expert i , and gs is the shared expert’s sigmoid gate.
Figure 2: The two block-diffusion adaptations in BD-GDN. (a) The query-specific history readout rj from ( 9 ) competes with Nb current-block values in one softmax. Its logit includes learnable block-size calibration. (b) Independent causal and right-context depthwise kernels are summed before SiLU. The right taps start at zero and read only valid successors in the current block; clean prefill and commit use the causal kernel.
Benchmark
\ours Clover 35B-A3B-Preview
LLaDA2.2 flash 104B-A7.4B
DiffusionGemma 26B-A4B
Mercury 2.5
Qwen3.6 35B-A3B
Qwen3.5 35B-A3B
Knowledge
GPQA Diamond
\first 83.8
47.5
68.2
76.3
\second 83.3
78.3
HLE text-only
12.8
5.3
10.9
8.9
\second 21.2
\first 21.6
MMLU-Pro
\second 84.8
71.0
80.9
84.1
84.7
\first 85.2
Math
MATH-500
\second 96.2
93.2
94.4
93.8
\second 96.2
\first 97.2
Table 1: Main benchmark results. All models are evaluated in-house under the protocol described in the text. Higher is better; \first red bold marks the best value and \second blue bold the second-best distinct value within each row, including ties. Shading identifies Clover. GDPval-AA v2 uses Elo, so its differences are not percentage points.
Workload
Model
A (tokens/round)
Decode TPS
Clover-35B-A3B-Preview
6.8743
465.2
MATH-500
Qwen3.6-35B-A3B + DFlash 2
6.5897
604.3
Clover-35B-A3B-Preview
8.2096
556.6
Agent replay
Qwen3.6-35B-A3B + DFlash 2
4.1548
396.7
Clover-35B-A3B-Preview
4.9970
356.7
Daily chat
Qwen3.6-35B-A3B + DFlash 2
2.9500
286.5
Table 2: H200 performance on three workloads. A is committed tokens per verification round; Decode TPS is defined in Eq. 18 .
Figure 3: Daily chat: answer length, body-only decode throughput, and the fraction of fully accepted draft rounds. Filled circles show each metric; in the first panel, they indicate mean body length, while open diamonds indicate the minimum. Connecting segments span minimum to mean. Thinking is excluded from body-only TPS.
Figure 4: Full agent replay: end-to-end throughput and committed tokens per verification round. Solid/hatched bars use block size or verification budget 8/16. Clover is faster without penalties, but can sometimes produce repetitive outputs.
Benchmark
\ours Clover 35B-A3B-Preview
MiniCPM-V 4.7 35B-A3B
LFM2.5 24B-A2B
Nemotron 3.5 Lightning (30B-A3B)
Gemma 4 26B-A4B
IFBench
72.3
47.7
74.7
71.9
\first 77.2 *
GPQA Diamond
\first 83.8
81.3
74.4
75.4
82.3
MMLU-Pro
\first 84.8
83.5
—
81.9
82.6
HLE text-only
12.8
9.4
7.4
11.7
\first 17.2
LiveCodeBench v6
72.6
56.6
—
—
\first 77.1
BFCL-v4 1
\first 67.9
—
65.1
53.8
57.1
Table 3: Comparison with efficient and on-device models. Higher is better; \first red bold marks the best reported value in each row, shading identifies Clover, and dashes denote unavailable results. Sources and harnesses differ as detailed below. These scores are not asserted to come from Clover’s INT4 deployment.
Model / placement
Expert weights
GPU / CPU experts 1
CPU backend
Accuracy (%)
Decode (tokens/s)
MiniCPM-V 4.7
INT4 g128
128 / 128
AVX-VNNI-256
97.4
49.51
Qwen3.6-35B-A3B
INT4 g128
128 / 128
AVX-VNNI-256
93.8
47.04
Clover-35B-A3B-Preview
Original placement, top-8
INT4 g32
96 / 160
AVX-512
96.8
66.64
Optimized placement, top-8
INT4 g32
96 / 160
AVX-512
97.6
97.52
\highlightrow Optimized placement, top-6
INT4 g32
96 / 160
AVX-512
96.0
106.25
Table 4: Full MATH-500 evaluation on one RTX 5070 Ti (16 GB), using SGLang with KTransformers. Accuracy is manually reviewed; decode throughput excludes prefill and includes thinking tokens.
Task
Mode
Question / cell accuracy
Complete-set accuracy
Multiple500
Independent
90.37
65.4
\highlightrow Multiple500
Joint
93.88
77.0
Sudoku
Independent
78.30
22.0
\highlightrow Sudoku
Joint
95.60
76.0
Table 5: Joint versus independent prediction with the same Clover decision model. All values are percentages; higher is better. Complete-set accuracy requires every question in a group or every missing cell in a board to be correct.
Model
JevBench accuracy (%)
Decision Index 0.2.1
Jev v1.13
86.58
57.93
\highlightrow Clover-Decision
88.31
58.23
Table 6: External decision benchmarks. JevBench reports accuracy over 231 questions; Decision Index reports the full suite’s composite score. Both metrics are higher-is-better. Jev v1.13 is a frozen API evaluation reference.
Figure 5: Group latency as the number of questions increases. Measurements use the same GPU and checkpoint, with one outer task at a time and a single denoising pass per call. Each point averages 16 measurements: four repetitions on each of four fixed cases. We vary group size by selecting the first 1, 2, 4, or 8 questions from each case.
Diffusion language models offer a promising path to efficient agents through parallel token generation. However, efficient interaction requires adapting both where computation is spent and how much text is generated. We introduce a metacognitive controller that guides diffusion generation according to the needs of the current response and the agent's interaction with the environment. It learns when to finalize tokens and how far generation should extend, reducing overall computation within responses. Our controller, a lightweight plug-in for the diffusion backbone, is trained on agent trajectories to balance correctness and generation cost. This approach connects the agent's generation process to the interaction the agent serves. Experiments across five benchmarks demonstrate that our approach reduces inference FLOPs by 20.6% on average compared with the most efficient baseline, while maintaining competitive task accuracy. Ablation studies further validate the effectiveness of our approach.
We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.
DiffusionGemma Team, Adrien Ali Taïga, James Assiene +41
Diffusion language models have seen exciting recent progress, offering far more flexibility in generative trajectories than autoregressive models. This flexibility has motivated a growing body of research into new approaches to diffusion language modeling, which typically begins at the scale of GPT-2 small (150 million parameters). However, these advances introduce new issues with evaluation methodology. In this technical note, we discuss the limitations of current methodology and propose principled augmentations to ensure reliable comparisons. We first discuss why OpenWebText has become the standard benchmark, and why alternatives such as LM1B are inherently less meaningful. We then discuss the limitations of likelihood evaluations for diffusion models, and explain why relying on generative perplexity alone as a metric can lead to uninformative results. To address this, we show that generative perplexity and entropy are two components of the KL divergence to a reference distribution. This decomposition explains generative perplexity's sensitivity to entropy, and naturally suggests generative frontiers as a principled method for evaluating model generative quality. We conclude with empirical observations on model quality at this scale. We include a blog post with interactive content to illustrate the argument at https://patrickpynadath1.github.io/blog/eval_methodology/.
Patrick Pynadath, Jiaxin Shi, Ruqi Zhang
Department of Computer Science, Purdue University West Lafayette, Indiana, 47906, USA