cs.CLAug 24, 2026

XTC: Head-Aware Sampling by Excluding Top Choices

Authors: Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder, Sanjay Basu, Ravid Shwartz-Ziv

Organizations: Independent Researcher · Thoughtworks · New York University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Standard decoding rules for autoregressive language models promote diversity by rescaling the full next-token distribution or truncating its low-probability tail. These strategies overlook a common regime of open-ended generation in which several continuations are plausible but too much probability mass remains concentrated on the most generic choice. We introduce XTC (Exclude Top Choices), a lightweight head-aware decoding operator that targets this regime directly. XTC identifies tokens whose probabilities exceed an absolute plausibility threshold ττ: when at least two qualify, it removes the dominant eligible choices with probability ρρ and retains only the weakest plausible alternative before renormalization. Across 60 experiments on Gemma 3 27B Q4, Gemma 3 12B Q6, and DeepSeek R1 14B Q6, with scaling validation on Llama 3.3 70B Q4, XTC improves the diversity-repetition Pareto frontier. On creative generation, Distinct-2 increases by 11--15% and repeat trigrams decrease by 27--47% across the four models. Combined with temperature scaling, gains reach 38% in Distinct-2 and 71% in repeat-trigram reduction over baseline. A blinded Amazon Mechanical Turk study with 150 Master raters yields a 62.3% creativity preference for XTC (p<10−4p<10^{-4}) without reduced fluency, while a GPT-4o control judge reproduces the Anthropic-judge direction on every measure. On IFEval with Llama 3.3 70B Q4, XTC preserves prompt-level strict accuracy within 1.7 percentage points of baseline while recovering most of the diversity gain; a temperature setting matched on Distinct-2 reduces IFEval by 8.8 points. The effect is additive with temperature and repetition penalties, robust across quantization levels and model families, and consistent across twelve prompt genres. XTC has been adopted by llama.cpp, ExLlamaV2, and text-generation-webui.

Figures & tables

Figure 1: The XTC operator at one decoding step. (a) Given the next-token distribution ptp_{t} and an absolute plausibility floor τ\tau , the eligible set Et={v:pt(v)≥τ}E_{t}=\{v:p_{t}(v)\geq\tau\} is highlighted in blue. The toy distribution shows the head-ambiguity regime: three tokens are individually plausible, with substantial mass concentrated on the most generic continuation. (b) The two dominant eligible tokens are removed and the weakest plausible alternative is kept. The non-eligible tail is untouched. (c) Renormalization redistributes the removed mass over the surviving support, yielding the transformed distribution qtq_{t} . The retained underdog (green) now carries the largest probability among the surviving options.
Figure 2: Main creative evaluation across six diversity and repetition metrics (24 prompts, 8 samples each, Gemma 3 27B q4). Error bars show 95% bootstrap CIs. XTC ( ρ=1.0\rho{=}1.0 , τ=0.1\tau{=}0.1 ) achieves the best score on every metric, outperforming the baseline as well as temperature, top- pp , and typical- pp on all diversity measures while simultaneously reducing repetition. Top- pp and typical- pp at 0.950.95 degrade diversity relative to baseline on Distinct-2, confirming that tail truncation can worsen head-dominance problems. Distinct-2, Embedding Cosine Distance, and Semantic Cluster Entropy: higher is better. Self-BLEU-4, Repeat Trigram Rate, Pairwise ROUGE-L: lower is better.
Figure 3: Pareto frontiers on two diversity-vs-repetition planes (24 creative prompts, Gemma 3 27B q4). Left: Embedding cosine distance vs. repeat trigram rate. Right: Distinct-2 vs. Self-BLEU-4. Stars mark Pareto-optimal conditions. XTC Medium and Strong dominate the frontier on both planes, while top- pp and typical- pp are Pareto-dominated by the baseline.
Figure 4: Claude Opus 4.7 and an OpenAI gpt-4o cross-vendor control on XTC vs. baseline (Gemma 3 27B q4, n=24n{=}24 paired samples per condition). Bars are paired Δ\Delta on a 1–5 scale with 95% bootstrap CIs. ⋆\star marks intervals that exclude zero. Opus and GPT-4o agree that XTC significantly improves repetition. GPT-4o additionally resolves significant gains on creativity and lexical diversity. Overall quality is null on both judges, supporting the no-degradation hypothesis. The cross-vendor agreement neutralizes same-family-bias concerns.
Figure 5: Llama 3.3 70B q4 scaling validation: Claude Opus 4.7 LLM judge on XTC vs. baseline ( n=24n{=}24 paired samples per condition, 95% bootstrap CIs). Opus resolves significant positive XTC effects on creativity and lexical diversity ( ⋆\star ). The repetition delta is positive but its CI grazes zero. Overall quality is null. The cross-vendor GPT-4o control was not run on this scaling configuration. The cross-vendor agreement is established on the 27B q4 panel (Figure 4 ).
Figure 6: Blinded AMT human evaluation on Gemma 3 27B q4 ( n=100n{=}100 prompts, 3 annotators per pair, 150 Master-qualified workers). (a) Stacked preference rates: XTC wins on creativity in 62.3% of head-to-head comparisons against the baseline (vs. 21.0% baseline wins, 16.7% ties), and ties or beats baseline in 84.4% of quality comparisons. (b) Net preference ( XTC -minus-baseline) and Fleiss’ κ\kappa inter-annotator agreement: the creativity gain is significant at p<10−4p<10^{-4} (two-sided binomial) with moderate agreement ( κ=0.58\kappa{=}0.58 ), and the quality gain is significant at p<10−3p<10^{-3} with κ=0.44\kappa{=}0.44 .
Figure 7: Instruction-following preservation on Llama 3.3 70B q4. (a) IFEval prompt-level strict accuracy across four conditions. XTC Light ( ρ=0.25\rho{=}0.25 , τ=0.10\tau{=}0.10 ) shifts accuracy by only −0.4-0.4 pp, and XTC Medium ( ρ=0.50\rho{=}0.50 , τ=0.10\tau{=}0.10 ) by −1.7-1.7 pp. A temperature setting (T=1.15) chosen to deliver a comparable Distinct-2 gain collapses IFEval by −8.8-8.8 pp. (b) Diversity gain vs. instruction-following cost: XTC Medium delivers ∼\sim 5 ×\times more Distinct-2 per IFEval-point lost than temperature scaling, locating XTC in the upper region of the diversity-vs-fidelity Pareto plane.
Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

 Distinct-2 ↑\uparrow Repeat trigram rate ↓\downarrow
Model familyBaseXTCΔ%\Delta\% [95% CI] BaseXTCΔ\Delta [95% CI]
Gemma 3 12B (q6) †0.7040.784+11.4+11.4 [ +7.4,+17.6+7.4,+17.6 ] 0.0470.025−0.022-0.022 [ −0.042,−0.006-0.042,-0.006 ]
DeepSeek R1 14B (q6) ‡0.6150.751+13.6+13.6 [ +10.2,+16.5+10.2,+16.5 ] 0.0380.021−0.017-0.017 [ −0.024,−0.009-0.024,-0.009 ]
Gemma 3 27B (q4)0.6090.689+13.1+13.1 [ +11.0,+18.1+11.0,+18.1 ] 0.0820.049−0.033-0.033 [ −0.047,−0.021-0.047,-0.021 ]
Llama 3.3 70B (q4)0.5680.653+15.1+15.1 [ +12.4,+19.8+12.4,+19.8 ] 0.0400.029−0.011-0.011 [ −0.027,−0.000-0.027,-0.000 ]
Average0.6240.719+13.3+13.3 0.0520.031−0.021-0.021
Appendix
Table 1: Cross-model generalization across four model families spanning three architectures (Gemma 3, Qwen-derived DeepSeek R1, Llama 3.3) and parameter counts from 12B to 70B. Means and 95% paired-bootstrap CIs are computed over per-prompt deltas (24 prompts for the 27B and 70B runs, 10 for the 12B/14B cross-model runs). Distinct-2 is reported as paired percent change. Repeat trigram rate is reported as the absolute paired difference, since the small baseline values make percent change unstable. Every model shows a Distinct-2 CI that excludes zero. Repeat trigram improvements are also significant on three of the four families. The Llama 70B repeat-trigram CI just touches zero, consistent with that model’s already-low baseline repetition. The XTC strongest condition reported per family sits within the paper’s operating region: ρ∈{0.75,1.0}\rho\in\{0.75,1.0\} , τ∈{0.05,0.1}\tau\in\{0.05,0.1\} . † Cross-model run uses 10 prompts per condition (vs. 24 for the primary 27B/70B runs). ‡ DeepSeek R1’s chat template injects a native reasoning-trace prefix on every response. We re-ran this configuration with max_tokens =8192 and stripped the <think>...</think> prefix before metric computation, so Distinct-2 and repeat trigram rate here reflect the post-reasoning answer text.
Figure 8: Cross-model generalization: baseline vs. XTC on the three primary model families (Gemma 3 27B q4, Gemma 3 12B q6, DeepSeek R1 14B q6). The Llama 3.3 70B q4 scaling validation is shown separately in Figure 9 .
Figure 9: XTC scaling from 12B to 70B parameters on the strongest XTC condition available in each run. The strongest-XTC Distinct-2 delta is monotone increasing in parameter count on the three q4/q6 points plotted: Gemma 3 12B q6 ( ρ=0.75\rho{=}0.75 , τ=0.05\tau{=}0.05 ) +11.4%+11.4\% , Gemma 3 27B q4 xtc_strong ( ρ=1.0\rho{=}1.0 , τ=0.1\tau{=}0.1 ) +13.1%+13.1\% , Llama 3.3 70B q4 xtc_strong ( ρ=1.0\rho{=}1.0 , τ=0.1\tau{=}0.1 ) +15.1%+15.1\% . The dashed trend line connects the three strongest-XTC points.
Figure 10: Quantization robustness: Gemma 3 27B at q4 vs. q6. No qualitative divergence at any operating point.
Figure 11: Diversity vs. repetition tradeoff on long-form prompts for q4 and q6.
Figure 12: DeepSeek R1 14B sampler composition on creative prompts.
Figure 13: DeepSeek R1 14B repetition composition.
Figure 14: Design ablation on DeepSeek R1 14B. Keep Least achieves the highest diversity across all four metrics.
Figure 15: Dose-response on DeepSeek R1 14B.
Figure 16: Gemma 3 12B q6 sampler composition.
Figure 17: Gemma 3 12B q6 repetition composition.
Figure 18: Dose-response on Gemma 3 12B q6.
Figure 19: Design ablation comparing four head-exclusion strategies under matched parameters (Gemma 3 27B q4, ρ=1.0\rho{=}1.0 , τ=0.1\tau{=}0.1 ). Keep Least (the published XTC rule) achieves the highest Distinct-2, the lowest Self-BLEU-4, and the largest repeat-trigram reduction. Drop Top-1 Only produces only partial improvement, confirming that effective head exclusion requires removing multiple dominant tokens. The XTC default bar is highlighted with a thicker outline in each panel.
Figure 20: Design ablation on Gemma 3 12B q6. Keep Least achieves the highest diversity.
Figure 21: Sampler composition: each sampler alone (gray) vs. paired with XTC Medium (green). Adding XTC improves Distinct-2 and reduces Self-BLEU-4 in every pairing, with no instances of degradation.
 BaselineTemp 1.3XTC MedXTC ρ=.75\rho{=}.75T1.3 + XTC
Distinct-2 ↑\uparrow0.6090.7420.6580.7700.841
Self-BLEU-4 ↓\downarrow0.2730.1910.2250.1520.113
Repeat trigram ↓\downarrow0.0940.0540.0710.0380.027
Embed. distance ↑\uparrow0.2910.3560.2930.3520.448
Compress. gzip ↑\uparrow2.492.692.582.762.93
Compress. xz ↑\uparrow2.953.233.063.293.47
Appendix
Table 2: Extended metric comparison (Gemma 3 27B q4). Arrows indicate preferred direction. The composition T=1.3 with XTC ( ρ=0.75\rho{=}0.75 , τ=0.05\tau{=}0.05 ) achieves the best score on every metric . XTC alone at ρ=0.75\rho{=}0.75 outperforms T=1.3 alone on five of ten metrics.
Figure 22: Temperature ×\times XTC factorial interaction across all three model families (Gemma 3 27B q4, Gemma 3 12B q6, DeepSeek R1 14B q6, 24 prompts, 5 seeds each). At all three temperatures and across all three models, XTC Medium and Strong improve Distinct-2 and reduce both Self-BLEU-4 and repeat trigram rate. The gains are approximately additive throughout, with no evidence of diminishing returns as temperature increases. DeepSeek panels are on the reasoning-trace basis (Table 1 caption).
Figure 23: Repetition penalty ×\times XTC factorial interaction across all three model families. Adding XTC to a repetition penalty baseline produces further improvements on every metric and every model. The effects are approximately additive, with no evidence of saturation or interference between the two mechanisms.
Figure 24: Interaction heatmaps: Distinct-3 surface over the temperature ×\times XTC grid and the repetition penalty ×\times XTC grid for each of the three model families. Diagonal structure (improvement increasing along both axes independently) is consistent with additive composition.
Figure 25: Synergy analysis: measured joint improvement vs. the sum of individual effects. Bars near zero indicate additive composition. No model shows super-additive or sub-additive interaction at magnitude above noise.
Figure 26: Forest plot of paired differences vs. baseline with 95% bootstrap confidence intervals and permutation test pp -values. XTC Medium and Strong both produce highly significant improvements on Distinct-2 ( p<0.001p{<}0.001 each) and significant improvements on Self-BLEU-4. Repeat trigram rate reductions are significant at every XTC strength ( p≤0.002p{\leq}0.002 ).
Figure 27: Per-prompt win/tie/loss counts vs. baseline across 24 prompts on Gemma 3 27B q4. XTC Strong wins on 24/24 prompts for Distinct-2 (no losses) and 22/24 for repeat trigram rate.
ConditionDistinct-2 ↑\uparrowRepeat trigram ↓\downarrowOpus judge ↑\uparrowGPT-4o judge ↑\uparrow
Top- pp 0.950.95 (reference)0.5420.0817.127.18
Eta sampling ( η=3×10−4\eta{=}3{\times}10^{-4} )0.5810.0607.447.39
Min- pp ( pmin⁡=0.10p_{\min}{=}0.10 )0.5980.0527.657.70
XTC ( ρ=1.0\rho{=}1.0 , τ=0.1\tau{=}0.1 )0.6730.0488.158.08
Min- pp + XTC0.6950.0358.328.25
Appendix
Table 3: Strong tail-shaping baselines on the 24-prompt creative pool (Gemma 3 27B q4). XTC alone outperforms top- pp , eta sampling, and min- pp on all four metrics, and the min- pp + XTC composition is best on every column. Judge means are pooled 1–10 ratings averaged over the seven pre-registered criteria.
ConditionIFEval (%) ↑\uparrowΔ\Delta vs. baselineDistinct-2 gain (%)IF-cost / Distinct-2
Baseline ( T=1.0T{=}1.0 )81.2n/an/an/a
XTC Light ( ρ=0.25\rho{=}0.25 )80.8−0.4-0.4+8.0+8.00.05×0.05\times
XTC Medium ( ρ=0.50\rho{=}0.50 )79.5−1.7-1.7+13.5+13.50.13×0.13\times
Temperature ( T=1.15T{=}1.15 )72.4−8.8-8.8+14.0+14.00.63×0.63\times
Appendix
Table 4: IFEval prompt-level strict accuracy on Llama 3.3 70B q4. The final column is IFEval-points lost per percentage point of Distinct-2 gained: XTC Medium pays 0.130.13 IFEval points per Distinct-2 percentage gained, vs. 0.630.63 for matched-Distinct-2 temperature scaling, a ∼\sim 5 ×\times ratio in favor of XTC . Per-prompt 95% bootstrap CIs on the four conditions are pending re-import of the per-prompt IFEval result CSVs from the GPU machine where the run was executed. The camera-ready will replace the point estimates here with p^±CI95\widehat{p}\pm\mathrm{CI}_{95} entries.
Figure 28: Head-to-head against deliberately strong comparators on Gemma 3 27B q4 (24 prompts, 8 samples each). Temperature 1.3 alone substantially improves diversity, but the composition T=1.3 + XTC ( ρ=0.75\rho{=}0.75 , τ=0.05\tau{=}0.05 ) achieves the highest score on all four metrics. Referenced from Section 4.2 .
Figure 29: Combined Pareto frontier overlaying conditions from three experiment configurations. The frontier is dominated by XTC conditions and temperature + XTC compositions.
Figure 30: Diversity-vs-repetition scatter for all strong baseline conditions. XTC conditions cluster in the desirable upper-left region.
Figure 31: Diversity-vs-Self-BLEU tradeoff for temperature, XTC , and compositions. The composition of T=1.3 with XTC occupies the far upper-left corner.
Figure 32: XTC parameter space: change from baseline ( Δ\Delta ) across a 4×44\times 4 grid of intervention probability ρ\rho and eligibility threshold τ\tau . Green indicates improvement. The strongest effects occur at high probability and low threshold.
Figure 33: Fine-grained operating region sweep at τ=0.05\tau=0.05 . The response is approximately linear in ρ\rho , with diminishing returns above ρ=0.75\rho=0.75 .
Figure 34: Structured quality retention on JSON extraction and constrained rewriting (12 prompts). XTC Light and Medium maintain quality at or near baseline.
Figure 35: Code exactness safety sweep: eval pass rate, test pass rate, and code parse validity as a function of XTC probability.
Figure 36: Token-level mechanism diagnostics across 64 generation positions. Activation rate ( ≈\approx 40%), top-1 changed rate, eligible token count ( ≈\approx 1.5–1.6), and removed mass ratio ( ≈\approx 25%).
Figure 37: Aggregated mechanism diagnostics across three XTC operating points.
Figure 38: Diversity gain stratified by prompt-level activation rate. High-activation prompts show larger gains, supporting the conditional hypothesis.
Figure 39: Prompt-openness stratification: diversity gain increases with prompt openness, consistent with the hypothesis that XTC benefits are largest when head multiplicity is naturally present.
Figure 40: Δ\Delta Distinct-2 by prompt genre (12 genres) relative to baseline. XTC produces positive gains across all 12 genres.
Figure 41: Absolute repeat trigram rate by prompt genre. XTC Strong achieves the lowest repetition across all 12 genres.
Figure 42: Extended diversity metrics: compression ratios, homogenization, template rate, and self-repetition.
Figure 43: Embedding geometry metrics: chamfer distance, remote clique, nn -gram diversity, and cosine distance.
Figure 44: Normalized multi-metric radar profile. XTC Strong and Medium expand the profile envelope across all axes.
Figure 45: Repetition evaluation on long-form generation prompts.
Figure 46: Repetition benchmark: XTC alone, repetition penalty + XTC , and compositions on long-form generation.
Figure 47: Extended repetition metrics: self-repetition, homogenization BLEU, and template rate all decrease under XTC .
JudgeConditionOverallCreativityRepetitionLexical
Claude Opus 4.7XTC Medium−0.05-0.05 [−0.35,+0.25][-0.35,+0.25]+0.15+0.15 [−0.05,+0.35][-0.05,+0.35]+0.05+0.05 [−0.35,+0.40][-0.35,+0.40]−0.10-0.10 [−0.40,+0.15][-0.40,+0.15]
XTC Strong−0.09-0.09 [−0.41,+0.23][-0.41,+0.23]+0.09+0.09 [−0.18,+0.36][-0.18,+0.36]+0.45\mathbf{+0.45} [+0.14,+0.77][+0.14,+0.77]+0.23+0.23 [+0.00,+0.45][+0.00,+0.45]
top- pp 0.950.95−0.11-0.11 [−0.37,+0.16][-0.37,+0.16]−0.11-0.11 [−0.26,+0.00][-0.26,+0.00]−0.21-0.21 [−0.58,+0.16][-0.58,+0.16]−0.21-0.21 [−0.42,+0.00][-0.42,+0.00]
GPT-4o (control)XTC Strong+0.05+0.05 [−0.15,+0.28][-0.15,+0.28]+0.39\mathbf{+0.39} [+0.11,+0.65][+0.11,+0.65]+0.51\mathbf{+0.51} [+0.22,+0.78][+0.22,+0.78]+0.28\mathbf{+0.28} [+0.04,+0.50][+0.04,+0.50]
top- pp 0.950.95−0.18-0.18 [−0.39,+0.02][-0.39,+0.02]−0.10-0.10 [−0.27,+0.05][-0.27,+0.05]−0.30-0.30 [−0.61,+0.00][-0.61,+0.00]−0.18-0.18 [−0.36,+0.00][-0.36,+0.00]
Appendix
Table 5: Per-judge paired deltas vs. baseline on Gemma 3 27B q4 ( n=24n{=}24 paired samples per condition, 95% bootstrap CIs). Bold entries have a 95% interval that excludes zero. Both judges agree on direction across every criterion. Both resolve a significant repetition improvement at XTC Strong, and GPT-4o additionally resolves creativity and lexical-diversity improvements that Opus shows as directionally positive but inside its CI. Neither judge resolves a significant negative XTC effect on overall quality.
JudgeConditionOverallCreativityRepetitionFluency
Claude Opus 4.7XTC (p=0.75, t=0.05)−0.08-0.08 [−0.25,+0.08][-0.25,+0.08]−0.04-0.04 [−0.21,+0.12][-0.21,+0.12]−0.04-0.04 [−0.38,+0.29][-0.38,+0.29]−0.21-0.21 [−0.42,+0.00][-0.42,+0.00]
temp 1.151.15 + XTC+0.04+0.04 [−0.17,+0.21][-0.17,+0.21]+0.00+0.00 [−0.25,+0.21][-0.25,+0.21]+0.00+0.00 [−0.29,+0.29][-0.29,+0.29]−0.29-0.29 [−0.54,−0.04][-0.54,-0.04]
temp 1.301.30 + XTC−0.04-0.04 [−0.33,+0.25][-0.33,+0.25]+0.04+0.04 [−0.21,+0.29][-0.21,+0.29]−0.04-0.04 [−0.33,+0.25][-0.33,+0.25]−0.67-0.67 [−1.08,−0.25][-1.08,-0.25]
Appendix
Table 6: Opus paired deltas vs. baseline on the Gemma 3 12B q6 cross-model creative-eval pool ( n=24n{=}24 paired samples per condition). Baseline Opus repetition is 4.334.33 , near the ceiling, leaving no absolute headroom for further improvement. Composing temp 1.301.30 with XTC degrades fluency, consistent with the standard temperature-over-warming pattern. Automatic Distinct-2 on this run is +11.4%+11.4\% (Table 1 ).
More in this topic

Language Model Decoding

Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.

See all 161 papers

Explore similar work

CardsList
Oct 1, 2026

Gacha Decoding: Eliciting Diverse Generations Through Instruction Following

We introduce Gacha Decoding, an inference-time method for eliciting diverse language model generations that scales with model capability. Across open-ended domains (in-the-wild chat, creative writing, planning for image generation, and protein design), Gacha Decoding significantly outperforms existing generation diversity approaches at equal quality (up to 2.4x Vendi over the next-best prior approach), reaching the same number of high-quality modes with over an order of magnitude fewer samples (11.0x) and discovering novel modes that no other approach surfaces. Our key insight is to treat diversity as an instruction-following problem: rather than relying on the LM's token entropy, we combine its instruction-following capability with randomness from an external RNG tool to scalably identify and realize distinct modes of the response space. This approach of "planning with dice" enables Gacha to invert the long-observed tension between diversity and model capability. As the underlying LM becomes a better instruction follower, diversity under Gacha Decoding consistently improves--even as its token entropy and diversity under prior approaches decline. Together, our results highlight that instruction following, rather than token entropy alone, can drive generation diversity.

Scott Geng, Yufei Zhang, Joseph Lee +3

University of Washington · Meta Superintelligence Labs

Language Model DecodingDiverse Text GenerationOutput Diversity in Language Models
AskPDF
Sep 27, 2026

Improving the Diversity of LLM Outputs without a Trade-off

We propose DAST (Diversifying Arithmetic Sampling with TokenTour), a method that increases the diversity of LLM outputs without any change to the marginal distribution and with negligible generation-time overhead (a few microseconds). We observe that token IDs are often arranged in a meaningless order and reassign them so that tokens with similar meanings appear consecutively. This can be done in advance in a few hundred seconds per model, and the resulting order can be reused for all subsequent generations. By combining this order with arithmetic sampling (or quasi-Monte Carlo methods), we make similar tokens less likely to be generated across runs while preserving the distribution. Our method not only produces qualitatively good ideas but also significantly improves performance on the downstream task of ProtoQA.

Ryoma Sato

National Institute of Informatics

Output Diversity in Language ModelsLanguage Model DecodingDiversity-Aware Sampling
AskPDF
Sep 28, 2026

Low-Confidence Remasking Traps Flexibility: Realizing Arbitrary-Order Potential for Diverse Rollouts in Diffusion LLMs

Masked diffusion language models support arbitrary-order generation, suggesting a natural way to produce diverse outputs. However, recent work argues that this flexibility reduces diversity by delaying high-uncertainty tokens that can lead to different generation paths. We trace this diversity loss not to arbitrary-order generation itself, but largely to low-confidence remasking (LCR), a widely used decoding rule. At each step, LCR samples a token at every masked position but commits only the sampled token with the highest probability, filtering out the rest. We show that this mechanism can exponentially suppress lower-probability tokens as more positions compete, and observe the same suppression in LLaDA. In contrast, top-probability position selection (TPP), which has often been conflated with LCR under the shared label confidence-based decoding, avoids this diversity loss. TPP first selects the position whose most likely token has the highest probability, then samples directly from that position's distribution. Replacing LCR with TPP restores diversity and yields Pass@kk comparable to left-to-right decoding, suggesting that the reported diversity loss stems largely from LCR's filtering rather than from generating high-confidence positions first. To further exploit order flexibility, we introduce Entropy-Guided Initialization (EGI), which samples the first token at the highest-entropy position and then follows TPP. This simple modification further improves rollout diversity and solution coverage beyond left-to-right decoding, with gains extending to downstream policy optimization, highlighting the potential of arbitrary-order generation for diverse rollouts.

Moongyu Jeon, Dongjae Jeon, Bumjun Kim +2

Yonsei University · KRAFTON AI · Kookmin University

Masked Diffusion ModelsDiffusion Language Model DecodingDiffusion Language Models
AskPDF