Deterministic inference is a comforting ideal in classical software: the same program on the same input should always produce the same output. As large language models move into real-world deployment, this ideal has been imported wholesale into inference stacks. Recent work from the Thinking Machines Lab has presented a detailed analysis of nondeterminism in LLM inference, showing how batch-invariant kernels and deterministic attention can enforce bitwise-identical outputs, positioning deterministic inference as a prerequisite for reproducibility and enterprise reliability. In this paper, we take the opposite stance. We argue that, for LLMs, deterministic inference kills. It kills the ability to model uncertainty, suppresses emergent abilities, collapses reasoning into a single brittle path, and weakens safety alignment by hiding tail risks. LLMs implement conditional distributions over outputs, not fixed functions. Collapsing these distributions to a single canonical completion may appear reassuring, but it systematically conceals properties central to artificial cognition. We instead advocate Stochastic CHAOS, treating distributional variability as a signal to be measured and controlled. Empirically, we show that deterministic inference is systematically misleading. Single-sample deterministic evaluation underestimates both capability and fragility, masking failure probability under paraphrases and noise. Phase-like transitions associated with emergent abilities disappear under greedy decoding. Multi-path reasoning degrades when forced onto deterministic backbones, reducing accuracy and diagnostic insight. Finally, deterministic evaluation underestimates safety risk by hiding rare but dangerous behaviors that appear only under multi-sample evaluation.
Figures & tables
Figure 1
Figure 1: GLUE Robustness Heatmap under Deterministic vs. Stochastic Decoding. Each cell shows the robustness ratio Rt(d)(m) (higher is better) for task t∈{MNLI,QQP,QNLI,SST-2} , decoding mode d∈{Stoch,Det} , and model m (columns, same zoo as in the FRACTURE analysis). Darker green indicates that paraphrased, perturbed, and adversarial variants preserve most of the model’s original GLUE accuracy; purple indicates severe degradation. Across tasks and models, Stochastic rows are consistently greener than their Deterministic counterparts , showing that bitwise-deterministic greedy decoding systematically underestimates the distributional generalization capacity of the underlying model . In other words, deterministic evaluation replays the GLUE mistake : it optimizes for one canonical completion per prompt, while stochastic, distributional evaluation reveals that the model’s competence is broader—and its brittleness more severe— than the single trace suggests.
Figure 2: Robustness ratios for Claude across GLUE tasks. Under stochastic decoding (teal), Claude attains robustness ratios between 0.85 and 0.91 across MNLI, QQP, QNLI, and SST-2, whereas deterministic decoding (orange) stays in the lower 0.79–0.82 band. This yields absolute stochastic–deterministic gaps in the range of 0.05–0.12 . The tight stochastic violins on QNLI and SST-2 indicate low variance across perturbation types , while the slightly wider shapes on MNLI and QQP reveal task-dependent sensitivity . Overall, Claude is consistently more robust when decoded stochastically , and the gains are not marginal but numerically substantial.
Figure 3: Robustness ratios for DeepSeek across GLUE tasks. Stochastic decoding places DeepSeek in a high-robustness regime, with ratios spanning 0.85–0.93 across tasks, while deterministic decoding lags behind at 0.76–0.81 . The stochastic–deterministic gap ranges from about 0.05 up to 0.16 absolute points, making DeepSeek one of the models with the largest decoding-induced robustness gains . QNLI and SST-2 show the highest stochastic robustness, whereas MNLI and QQP display broader violins, reflecting increased variability under perturbations . These numbers highlight that DeepSeek’s strong robustness is tightly coupled to stochastic inference ; deterministic decoding leaves significant robustness “on the table.”
Figure 4: Robustness ratios for Gemma-2 9B across GLUE tasks. With stochastic decoding, Gemma-2 9B achieves robustness ratios between 0.82 and 0.91 , while deterministic decoding stays in the 0.77–0.82 range. The task-wise stochastic–deterministic differences vary from essentially 0.00 (one task where deterministic is on par) up to about 0.09 absolute points. QQP and SST-2 show the highest stochastic robustness , while MNLI and QNLI are slightly lower and more spread out. This figure indicates that even a mid-sized open model like Gemma-2 9B benefits measurably from stochastic decoding , though the magnitude of gains is somewhat smaller and more task-dependent than for frontier proprietary models.
Figure 5: Robustness ratios for Gemma-2 27B across GLUE tasks. Scaling to 27B pushes the stochastic robustness band to 0.86–0.90 , while deterministic decoding lies in the slightly lower interval 0.79–0.84 . Stochastic–deterministic gaps span roughly 0.02–0.10 across tasks, smaller than for some proprietary models but still systematically positive . MNLI and QQP show clear upward shifts compared to Gemma-2 9B, and SST-2 reaches the top of the model’s robustness range with narrow, high violins . The combination of higher means and reduced spread suggests that Gemma-2 27B is both more robust and more stable , yet still meaningfully boosted by stochastic decoding.
Figure 6: Robustness ratios for GPT-4o mini across GLUE tasks. Under stochastic decoding, GPT-4o mini attains robustness ratios in the 0.84–0.89 range, whereas deterministic decoding falls between 0.76 and 0.84 . The resulting gaps are on the order of 0.05–0.09 absolute points depending on the task. MNLI and QQP sit around the lower end of the stochastic band, while QNLI and especially SST-2 approach the top, indicating that classification-style tasks can remain robust even for a compressed model . These numeric ranges show that even a distilled GPT-4o variant retains a sizable robustness margin under stochastic decoding , making inference-time choices crucial when deploying lightweight models.
Figure 7: Robustness ratios for GPT-4o across GLUE tasks. GPT-4o shows one of the strongest robustness profiles : stochastic ratios consistently lie between 0.87 and 0.93 , while deterministic decoding drops to 0.75–0.84 . Task-wise stochastic–deterministic gaps range from about 0.04 up to 0.16 absolute points, with the largest differences on QNLI and SST-2. The tight, high violins for stochastic decoding indicate high robustness and low variance , whereas deterministic violins are wider and noticeably shifted down. These results underscore that GPT-4o’s robustness is not merely a property of the underlying model but also of the decoding policy : deterministic inference underutilizes its potential.
Figure 8: Robustness ratios for GPT-3.5 across GLUE tasks. For stochastic decoding, robustness ratios span 0.84–0.91 , situating GPT-3.5 below GPT-4o but still in a relatively strong band. Deterministic decoding compresses the model into the 0.78–0.83 range, with per-task gaps of roughly 0.06–0.11 absolute points. QQP and MNLI exhibit the largest downward shifts and broader violins under deterministic decoding, signaling heightened vulnerability to adversarial paraphrases in these settings. Taken together, the figure positions GPT-3.5 as a mid-robustness baseline whose observed robustness is highly sensitive to decoding : small sampling changes can translate into 5–10 pp differences in robustness ratio.
Figure 9: Robustness ratios for LLaMA-2 7B across GLUE tasks. Stochastic decoding yields robustness ratios between 0.81 and 0.89 , while deterministic decoding ranges more widely from 0.72 up to 0.86 . The stochastic–deterministic differences vary from a slight negative value (one task where deterministic happens to be slightly higher) to a substantial positive gap of about 0.15 absolute points. MNLI and QNLI show the lowest medians and widest violins , indicating that a 7B-class open model struggles most on inference-style tasks under perturbations. Numerically, this figure illustrates that LLaMA-2 7B sits at the lower end of the robustness spectrum and is highly decoding-sensitive , making it an informative but fragile baseline.
Figure 10: Robustness ratios for LLaMA-2 13B across GLUE tasks. After scaling to 13B, stochastic robustness climbs to the 0.84–0.94 range, while deterministic decoding stays in a narrower but lower interval of 0.79–0.82 . The resulting stochastic–deterministic gaps fall between 0.03 and 0.14 absolute points, with the largest gains again on MNLI and QNLI. Compared to LLaMA-2 7B, both decoding modes shift upward and the stochastic violins become tighter , especially on QQP and SST-2. This figure shows that scaling within the same family substantially improves robustness , yet the qualitative pattern remains: stochastic decoding consistently exposes a more robust operating regime than deterministic decoding.
Figure 11: Robustness ratios for LLaMA-3 7B across GLUE tasks. Despite having the same parameter count as LLaMA-2 7B, LLaMA-3 7B achieves higher stochastic robustness, with ratios in the 0.83–0.90 range. Deterministic decoding occupies 0.77–0.84 , and stochastic–deterministic gaps are more modest but still positive at roughly 0.06–0.08 absolute points. QQP and QNLI show the highest robustness and the tightest violins , while MNLI remains the most challenging task. Quantitatively, this figure suggests that architectural and data improvements from LLaMA-2 to LLaMA-3 shift the entire robustness band upward , even though the fundamental advantage of stochastic decoding persists.
Figure 12: Robustness ratios for LLaMA-3 8B across GLUE tasks. Under stochastic decoding , LLaMA-3 8B attains robustness ratios in the 0.89–0.96 band (roughly 0.91 on MNLI, 0.80 on QQP, 0.86 on QNLI, and 0.95 on SST-2), whereas deterministic decoding falls to the 0.74–0.79 band across the same tasks. The stochastic–deterministic gaps range from about 0.06 (QQP, QNLI) up to nearly 0.18 (SST-2), showing large decoding-induced robustness gains . The high, tight stochastic violin on SST-2 in particular indicates that LLaMA-3 8B becomes extremely robust when decoded stochastically , while deterministic decoding systematically underestimates its robustness.
Figure 13: Robustness ratios for LLaMA-3 70B across GLUE tasks. Stochastic decoding places LLaMA-3 70B in a strong robustness band of 0.84–0.96 : around 0.85 on MNLI, 0.88 on QQP, 0.88 on QNLI, and near 0.96 on SST-2. In contrast, deterministic decoding compresses robustness into the lower 0.74–0.83 interval. The resulting stochastic–deterministic differences span roughly 0.04–0.13 absolute points, with the largest margins on SST-2 and QQP . Compared with LLaMA-3 7B, these numbers show that scaling to 70B significantly strengthens robustness while preserving the same qualitative advantage of stochastic decoding .
Figure 14: Robustness ratios for Mistral-7B across GLUE tasks. With stochastic decoding, Mistral-7B achieves robustness ratios between 0.84 and 0.90 on MNLI, QQP, and QNLI, and around 0.78–0.82 on SST-2. Deterministic decoding yields slightly lower values on most tasks, in the 0.79–0.84 range for MNLI/QQP/QNLI and around 0.77–0.82 on SST-2. Stochastic–deterministic gaps are moderate ( 0.02–0.06 absolute), except for SST-2 where deterministic decoding is marginally higher, illustrating that the decoding advantage can flip on specific tasks . Overall, the figure highlights that Mistral-7B is reasonably robust but exhibits nuanced, task-specific trade-offs between stochastic and deterministic decoding .
Figure 15: Robustness ratios for Mixtral-8 × 7B across GLUE tasks. Stochastic decoding places the mixture-of-experts model in a high band of 0.83–0.95 : about 0.92 on MNLI, 0.86 on QQP, 0.83 on QNLI, and 0.89 on SST-2. Deterministic decoding yields 0.77–0.84 across tasks, often trailing stochastic decoding by 0.05–0.10 absolute points. The largest gaps appear on MNLI and SST-2, where violins are clearly separated, while QQP shows a smaller but still positive advantage for stochastic decoding. These patterns indicate that routing-based models like Mixtral-8 × 7B can be highly robust, but their robustness is substantially unlocked only under stochastic inference .
Figure 16: Robustness ratios for Mixtral-8 × 22B across GLUE tasks. Scaling Mixtral to 8 × 22B yields stochastic robustness ratios in the 0.84–0.95 band: about 0.84 on MNLI, 0.85 on QQP, 0.93 on QNLI, and 0.90 on SST-2. Deterministic decoding remains in a lower 0.78–0.82 band across all tasks. The stochastic–deterministic margins are modest ( 0.03–0.06 ) on MNLI/QQP/SST-2 but become very large on QNLI ( ≈0.10–0.15 ). The very tall, narrow stochastic violin for QNLI emphasizes high and stable robustness , whereas deterministic decoding exhibits both lower means and larger spread. Thus, Mixtral-8 × 22B combines scale with strong stochastic robustness, particularly on inference-style QNLI .
Figure 17: Robustness ratios for Phi-2 across GLUE tasks. Despite being a small model, stochastic decoding propels Phi-2 to surprisingly high robustness ratios: around 0.86 on MNLI, 0.93–0.95 on QQP, 0.96–0.98 on QNLI, and 0.89–0.93 on SST-2. In contrast, deterministic decoding stays in the 0.74–0.80 band across tasks. This yields very large stochastic–deterministic gaps of roughly 0.10–0.17 absolute points, some of the largest differences in the entire model suite . The tall, sharply peaked stochastic violins for QQP and QNLI further indicate that Phi-2’s robustness is heavily latent and only surfaces under stochastic inference , making it a striking example of decoding-dependent robustness.
Figure 18: Robustness ratios for Vicuna-7B across GLUE tasks. With stochastic decoding , Vicuna-7B reaches robustness ratios of roughly 0.88 on MNLI, 0.87 on QQP, 0.90 on QNLI, and 0.82–0.84 on SST-2. Deterministic decoding lies around 0.83 on MNLI, 0.78 on QQP, 0.79 on QNLI, and 0.85–0.87 on SST-2. This produces positive stochastic–deterministic gaps of 0.05–0.11 on MNLI/QQP/QNLI, but a negative gap on SST-2 where deterministic decoding is ≈0.03–0.04 higher. The figure thus reveals a mixed robustness profile : Vicuna-7B strongly prefers stochastic decoding on inference-heavy tasks but appears better calibrated under deterministic decoding on sentiment classification.
Figure 20: Few-shot ICL accuracy and exploration gains across models on BESSTIE tasks. Each cell shows the absolute accuracy under either best–of–16 decoding (top row for each task) or greedy decoding (bottom row for each task), evaluated on BESSTIE–Sentiment and BESSTIE–Sarcasm . For the greedy rows we additionally print the accuracy gap ( “ ↓d% ” ) relative to best–of–16, where d=EGt,mICL(16)×100 . The warm vs. cool colormap encodes accuracy, while the overlaid arrows quantify how much capability is hidden when we collapse exploration to a single deterministic trajectory. Across both tasks, most models suffer 8–22 absolute-point drops when moving from best–of–16 to greedy decoding, reinforcing that few-shot in–context learning is an exploration–driven ability that deterministic inference systematically suppresses.
Figure 21: Exploration–ICL landscapes for LLaMA-2 13B on BESSTIE. Left: Sentiment (empirical best-of-16 gain ≈15 pp) shows a broad ridge of exploration benefit concentrated around temperatures T∈[0.65,0.80] and sample counts k∈[8,32] (i.e., log2k∈[3,5] ), with gains tapering smoothly toward both very low and very high exploration. Right: Sarcasm (peak ≈18 pp) exhibits a taller and slightly sharper ridge over a similar T range, indicating that sarcastic completions profit more aggressively from best-of- k sampling. In both panels, the x-axis spans temperature T∈[0.05,1.0] , the y-axis covers log2k∈[0,6] (i.e., k∈[1,64] ), and the color scale encodes exploration gain ΔAccICL in the numeric range [0,0.25] (corresponding to [0,25] percentage points).
Figure 22: Exploration–ICL landscapes for LLaMA-3 8B . Left: Sentiment has a relatively low best-of-16 gain of only ≈6 pp, with a shallow ridge centred near T≈0.7 and small-to-moderate k ( k∈[4,16] ), indicating limited upside from exploration on this task. Right: Sarcasm (peak ≈13 pp) shows a visibly stronger and more extended plateau, with useful gains persisting for T∈[0.65,0.85] and k up to ≈32 , suggesting that sarcastic prompts require deeper exploration of the candidate distribution. Across both plots, the numeric ranges are fixed to T∈[0.05,1.0] , log2k∈[0,6] and ΔAccICL∈[0,0.25] , making cross-model comparison in later figures scale-consistent .
Figure 23: Exploration–ICL landscapes for LLaMA-3 70B . Left: Sentiment (peak gain ≈5 pp) is characterized by a very flat surface with only a low-amplitude bump at T≈0.7 and k≈8 – 16 , indicating that the strong base model already solves most cases under greedy decoding. Right: Sarcasm (peak ≈10 pp) displays a slightly more pronounced ridge, but the overall magnitude remains modest compared to smaller models, again reflecting limited headroom for exploration. Formally, the figure keeps T in [0.05,1.0] , log2k in [0,6] , and ΔAccICL clipped to [0,0.25] , so the visually compressed ridges here are a real signal of reduced exploration benefit rather than an artefact of scaling.
Figure 24: Exploration–ICL landscapes for Gemma-2 9B . Left: Sentiment shows a substantial ridge with peak gain ≈14 pp, spanning T∈[0.65,0.8] and k∈[8,32] , and quickly flattening for very low k and overly hot temperatures. Right: Sarcasm is even more exploration-sensitive, achieving a peak of ≈17 pp and maintaining high gains over a wide band T∈[0.65,0.85] and k∈[8,48] , where the surface height stays above roughly 0.10 (i.e., 10 pp). The color scale is again fixed to [0,0.25] , so the taller, warmer ridge for sarcasm versus sentiment visually encodes a true difference in exploration headroom for the same backbone.
Figure 25: Exploration–ICL landscapes for Gemma-2 27B . Left: Sentiment exhibits one of the strongest ridges in our study, with peak gain ≈17 pp and a high plateau for T∈[0.65,0.8] and k∈[8,48] , where ΔAccICL remains in the [0.10,0.20] (10–20 pp) band. Right: Sarcasm (peak ≈10 pp) has a noticeably shorter and narrower ridge, concentrated near T≈0.7 and k∈[8,24] , suggesting that this larger Gemma variant is more exploration-hungry on sentiment than on sarcasm. Because all panels share a common numeric range for T , k , and gain, the visual contrast between the left and right surfaces directly quantifies how task identity modulates the value of best-of- k sampling.
Figure 26: Exploration–ICL landscapes for Mistral-7B . Left: Sentiment (peak gain ≈10 pp) has a clean, single ridge around T≈0.7 and k∈[8,24] ; below k=4 or above k=32 the surface rapidly collapses toward 0 . Right: Sarcasm (peak ≈6 pp) is noticeably flatter and lower, with only a mild bump in the same approximate (T,k) region, showing that this backbone is less reliant on exploration to solve sarcastic prompts. Within the global numeric ranges T∈[0.05,1.0] , log2k∈[0,6] , and ΔAccICL∈[0,0.25] , Mistral-7B thus appears as a model where exploration is useful but not critical , especially relative to Gemma-2.
Figure 27: Exploration–ICL landscapes for Mixtral-8x7B . Left: Sentiment and Right: Sarcasm both peak at roughly ≈7 pp, with gently sloping ridges around T∈[0.65,0.8] and k∈[8,24] . The similarity of the two surfaces—both staying mostly within the [0.03,0.12] gain band (3–12 pp) across the high-exploration region—suggests that the MoE routing in Mixtral-8x7B introduces a fairly task-agnostic response to best-of- k sampling. Overall, the numeric ranges confirm that this model sees consistent but moderate exploration benefits across both sentiment and sarcasm, with no extreme dependence on temperature or very large k .
Figure 28: Exploration–ICL landscapes for Mixtral-8x22B . Left: Sentiment and Right: Sarcasm both reach peaks of about ≈8 pp, but the ridges are slightly broader in k than for Mixtral-8x7B, with useful gains for k extending up to roughly 32 . Within T∈[0.65,0.8] and k∈[8,32] , ΔAccICL often stays above 0.05 (5 pp), while quickly dropping outside this band. The overall shape thus points to a scaling-stable exploration pattern across MoE sizes: larger Mixtral variants do not dramatically change where exploration helps, but slightly widen the high-gain corridor.
Figure 29: Exploration–ICL landscapes for Vicuna-7B . Left: Sentiment reaches a moderate peak of ≈9 pp, with a compact ridge around T≈0.7 and k∈[8,24] , and limited gain outside this region. Right: Sarcasm is dramatically different: the surface climbs up to ≈17 pp, with a tall ridge covering T∈[0.65,0.85] and k∈[8,48] , where gains stay well above 0.10 (10 pp). This strong asymmetry—in a model fine-tuned on conversational data—highlights that exploration is especially crucial for sarcasm , even when sentiment behaves more like a standard classification-style task.
Figure 30: Exploration–ICL landscapes for Phi-2 . Left: Sentiment shows a surprisingly strong ridge for a small model, with peak gain ≈11 pp and a concentrated band of high values around T∈[0.65,0.8] and k∈[8,24] ; here, gains in the [0.06,0.12] (6–12 pp) range are common. Right: Sarcasm is much flatter, with peak ≈5 pp and only a small bump near T≈0.7 and k∈[8,16] , quickly collapsing towards zero for larger k or temperatures too far from the sweet spot. Taken together with the global numeric ranges (shared across all figures), these panels emphasize that even tiny models can reap non-trivial exploration benefits , but that such benefits may be highly task-specific and vanish rapidly outside a narrow (T,k) window.
Figure 31: Entropy–exploration relationship in BESSTIE few-shot ICL. Each marker is a task–model pair (t,m) from open LLMs in Figure 20 , for either BESSTIE–Sentiment (circles) or BESSTIE–Sarcasm (triangles). The x –axis shows the normalized label entropy Ei[H~i,t,m]∈[0,1] , where for each example i we estimate a label distribution p^ℓ,i,t,m from temperature–scaled stochastic samples and compute Hi,t,m=−∑ℓp^ℓ,i,t,mlogp^ℓ,i,t,m . We then normalize by the task arity, H~i,t,m=Hi,t,m/logCt , and average over i . The y –axis plots the ICL exploration gain EGt,mICL(k=16)=Acct,mICL(k=16)−Acct,mgreedy , i.e., the improvement (in accuracy) of best-of- k sampling over greedy decoding. Solid (Sentiment) and dashed (Sarcasm) curves show quadratic fits f(h)≈ah2+bh+c to the points in each task. We observe a clear inverted-U relationship: both low-entropy regimes ( Ei[H~i,t,m]≲0.2 , nearly deterministic labels) and very high-entropy regimes ( ≳0.8 , almost uniform confusion) yield negligible exploration gains ( EGICL≲0.05 ), while intermediate entropies ( ≈0.3 – 0.7 ) produce the largest gains ( EGICL≈0.10 – 0.20 ). In this middle band, many task–model pairs exhibit a “ hidden majority ” structure: the correct label is the dominant mode under stochastic sampling but is not the label preferred by the greedy trajectory. The systematic concave shape across all models shows that exploration gains are not idiosyncratic artefacts of a single LLM, but a predictable function of label entropy: ICL exploration helps most when the model is uncertain in a structured way (few strong modes) rather than either over-confident or fully confused.
Model
Policy
Mean scores (0–1)
Psuccstyle
ssem
clen
cinc
cavoid
cstyle
LLaMA–2
Greedy
0.76
0.58
0.71
0.83
0.52
0.34
Multi ( k=8 )
0.78
0.64
0.76
0.86
0.61
0.40
LLaMA–3
Greedy
0.82
0.63
0.78
0.87
0.58
0.48
Multi ( k=8 )
0.84
0.72
0.83
0.90
0.69
0.64
Gemma–2
Greedy
0.80
0.61
0.76
0.85
0.56
0.45
Table 1: Where do gains come from? Per–dimension effects of stochastic search on InstruSum. For each model m we report mean semantic adequacy ssem and mean constraint scores clen,cinc,cavoid,cstyle under greedy and multi–sample decoding with budget k=8 , together with the full success rate Psuccstyle(e,m,k) .
Figure 32: Style exploration gain as a function of search budget. For each model m , we plot the style exploration gain EGmstyle(k) on InstruSum as the search budget k increases from 1 (greedy decoding) to 32 samples. Gains rise sharply between k=2 and k=8 and mostly saturate by k⋆=8 . Larger, recent models such as LLaMA–3 and Mixtral–8 × 22B reach higher plateaus ( EGmstyle(k⋆)≈0.10 – 0.13 ), whereas smaller models like Phi–2 show fast saturation with low gains , indicating limited headroom for improving instruction adherence via search. Overall, the figure illustrates that distributional exploration systematically improves style/constraint satisfaction, but the attainable gains are strongly model–dependent .
Figure 33: Linking ICL exploration gains to style exploration gains. Each point is a model m , with the x–axis showing its ICL exploration gain EGmICL on BESSTIE and the y–axis showing its style exploration gain EGmstyle(k⋆) on InstruSum at k⋆=8 . The regression line and correlation ( ρ≈0.80 ) indicate that models with larger ICL gains typically also gain more in style/constraint satisfaction. Outliers such as Phi–2 ( ICL–strong, style–weak ) and Mixtral–8 × 22B ( strong on both ) show that this relationship is architecture–dependent . Together with Figure 32 , this scatter plot supports our claim that distributional exploration surfaces latent abilities in both ICL and style–constrained generation, while the degree to which they are buried under greedy decoding is highly model–specific .
Semantic–constraint density landscapes, part I. Panels (a) – (d) show four representative models, illustrating how probability mass can be concentrated on an under–constrained ridge even when higher–quality, constraint–respecting summaries are present in the tails of the distribution.
Figure 35: Semantic–constraint density landscapes across models on InstruSum. Taken together, panels (a) – (h) reveal a consistent pattern: many instruction–tuned LLMs place most of their probability mass on an under–constrained ridge of semantically strong but stylistically misaligned summaries, while only a smaller portion of the distribution occupies the high semantics & high constraints region near the Pareto frontier. As models become larger and more recent (e.g., LLaMA–3 , Mixtral–8 × 22B ), this high–quality corner receives increasingly more mass, yet greedy decoding remains anchored to the ridge. The figure therefore supports our central takeaway: apparent failures to follow detailed style and format instructions are often a search artifact of the decoding policy rather than a hard limitation of the underlying conditional distribution.
Figure 36: Reasoning graph for a multi–step ticket–sales word problem. The problem (shown below the tree) asks how much money a school donates after selling adult and child tickets over two days, with different prices and a fixed per–ticket expense. Internal nodes represent partial chains of thought: an initial parsing stage, a split into Strategy A ( group by day ) versus Strategy B ( group by ticket type ), and finer subplans such as “Monday first” vs. “Tuesday first” or “Adults first” vs. “Children first”. Leaves are classified as Correct , Near–miss , or Failure , with outcome types indicated by the colored markers in the legend box. The thick red path denotes the greedy chain of thought , which ends in a near–miss leaf, whereas the dashed green path shows an alternative, fully correct strategy that the model also assigns nontrivial probability to. Other sampled branches are drawn in light gray. This illustrates how multi–sample decoding exposes a rich, multi–path landscape of reasoning, while deterministic inference collapses it into a single brittle trace.
Figure 37: Aggregate reasoning-path analysis. Left: models with greater sampled path diversity tend to realize larger accuracy gains from stochastic exploration. Right: sampled reasoning graphs separate collapsed failures, brittle successes, and robust successes, exposing behaviors that a single greedy chain cannot reveal.
Model
Accgreedy
Accmulti(16)
ΔAcc(16)
Hm
Sm
% Collapsed / % Brittle
DeepSeek–R1–Distill
0.67
0.82
+0.15
1.68
0.38
31% / 29%
LLaMA–3.1–8B–Instruct
0.62
0.73
+0.11
1.49
0.45
26% / 33%
Mixtral–8 × 7B–Instruct
0.64
0.72
+0.08
1.37
0.52
21% / 37%
Table 2: Reasoning–path diversity and brittleness across modern reasoning LLMs. For each model m (macro–averaged over GSM8K, SVAMP, and StrategyQA), we report greedy vs. multi–sample CoT accuracy, the resulting exploration gain ΔAccm(16) at budget k=16 , the average reasoning–path entropy Hm , the greedy path support Sm (fraction of sampled traces that follow the greedy prefix at each step), and the fraction of collapsed failures (greedy wrong, some sampled path correct) and brittle successes (greedy correct, but <30% of samples correct). The simulated values reflect a consistent trend: models with higher Hm enjoy larger accuracy gains under stochastic decoding but also exhibit lower Sm and more collapsed failures, indicating that deterministic inference increasingly under–represents their internal multi–path uncertainty.
Figure 38: DeepSeek–R1–Distill: stochastic exploration exposes rich, high-gain reasoning modes across tasks. This 3D plot shows, for DeepSeek–R1–Distill , the joint distribution of average reasoning–path entropy Hm,d (x–axis) and accuracy gain ΔAccm,d from multi–sample over greedy decoding (y–axis), with dataset layers d∈{GSM8K,SVAMP,StrategyQA} separated along the z–axis. Grey point clouds represent deterministic behavior under greedy decoding: they cluster tightly near ΔAccm,d≈0 with slightly lower entropies, indicating a narrow set of chains of thought that the model actually emits when forced to be deterministic. Colored point clouds correspond to stochastic multi–sample decoding and span Hm,d≈1.45 – 1.90 and ΔAccm,d≈0.05 – 0.22 , revealing a much richer, higher–diversity regime of reasoning that greedy decoding never visits. Arrows from greedy centers (black stars) to multi–sample centroids (colored circles) show a consistent shift toward higher path entropy and substantially higher accuracy on all three datasets, with the largest displacement on GSM8K . Taken together, this figure illustrates that, for DeepSeek, imposing deterministic decoding collapses a broad landscape of competent reasoning strategies into a single, brittle trace that systematically underestimates the model’s true multi–path capabilities.
Figure 39: LLaMA–3.1–8B–Instruct: heterogeneous gains across arithmetic and verbal reasoning. For LLaMA–3.1–8B , we again plot average path entropy Hm,d vs. accuracy gain ΔAccm,d , layered over GSM8K , SVAMP , and StrategyQA . The deterministic clouds (grey) remain tightly packed near ΔAccm,d≈0 , reflecting low apparent diversity when the model is evaluated with greedy decoding. In contrast, the stochastic clouds concentrate around Hm,d≈1.35 – 1.70 and ΔAccm,d≈0.03 – 0.12 , clearly separated from the deterministic regime and revealing nontrivial gains from distributional exploration. The greedy → multi–sample displacement (arrows from black stars to colored circles) is largest on StrategyQA , where verbal multi–hop reasoning admits many distinct but valid chains of thought; arithmetic datasets exhibit smaller but consistent gains in both diversity and accuracy. Overall, this figure shows that even a strong instruction–tuned model like LLaMA–3.1 hides a substantial fraction of its reasoning flexibility when run deterministically, and that the benefits of exploration are strongest precisely on tasks with rich, verbal reasoning structure .
Figure 40: Mixtral–8 × 7B–Instruct: moderate but consistent diversity and accuracy gains. This figure presents the same 3D landscape for Mixtral–8 × 7B–Instruct . Here, the stochastic point clouds occupy an intermediate band, with Hm,d≈1.30 – 1.55 and ΔAccm,d≈0.02 – 0.10 across GSM8K , SVAMP , and StrategyQA . The deterministic clouds again lie tightly near ΔAccm,d≈0 , but the gap to the stochastic regime is smaller than for DeepSeek or LLaMA–3.1. Arrows from greedy centers to multi–sample centroids consistently move toward higher reasoning diversity and higher accuracy , yet the magnitude of these shifts is shallower : Mixtral already allocates noticeable probability mass to high–quality chains of thought that greedy decoding sometimes captures. This intermediate pattern suggests that Mixtral’s internal reasoning landscape is less severely collapsed by deterministic inference than DeepSeek’s, but still benefits from exploration—especially on GSM8K and SVAMP , where multi–sample decoding uncovers additional correct, diverse solution paths. In combination with Figures 38 and 39 , this figure highlights that the extent of reasoning-path collapse under greedy decoding is strongly architecture-dependent , even when all models are evaluated on the same three reasoning benchmarks.
Policy
Parameters
Deployment analogue
Effect on risk visibility
Greedy πgreedy
T=0 , single sample
Deterministic API / eval scripts
Hides all non–argmax harmful modes
Temp. sampling πT,p
T∈{0.7,1.0} , top– p=0.9
Interactive chat, creative generation
Exposes some harmful tails as T increases
Multi–sample ( k )
k∈{8,16,32} samples, best–of– k or full set
Regenerate button, best–of– k reranking
Makes rare harmful modes increasingly likely to appear
Adv. search (optional)
Beam / attacker–guided sampling
Strong adaptive adversary
Upper bound on reachable harmful behavior
Table 3: Decoding policies used in safety evaluation. For each model m we evaluate a spectrum of decoders, from strictly deterministic greedy decoding—which is commonly used in academic benchmarks—to stochastic and multi–sample policies that better approximate real deployments. The last column highlights the main way in which each policy can hide or reveal harmful but low–probability behaviors.
Figure 41: Decoder–level risk as a function of harmful tail mass and search budget. We plot the theoretical decoder–level risk r~k(q)=1−(1−q)k as a function of the per–prompt harmful tail mass q (x–axis, log–scaled) and the number of independent samples k (y–axis), with a cool–to–warm colormap indicating low to high risk. The thick black curve on the front edge shows the k=1 slice (deterministic greedy evaluation): for small tails, r~1(q)≈q , so prompts with q≈10−3 and q≈0.02 look similarly safe (risks around 0.001 vs. 0.02 ). The colored k=8,16,32 curves on the surface reveal that, under realistic multi–sample budgets, the same tail masses yield much higher risk: at q=10−3 , risk rises from ≈0.001 (greedy) to ≈0.03 for k=32 , while at q=0.02 it jumps from ≈0.02 to well above 0.45 . The warm plateau near the back shows that even moderately sized harmful tails become near-certain failures once k is large. The stark visual gap between the flat–looking k=1 boundary and the lifted multi–sample surface makes explicit the illusion of robustness created by deterministic–only safety evaluation.
Model
ASRmgreedy
ASRmstoch(8)
ΔASRm(8)
(ρmrobust,ρmconcealed,ρmdet-fail)
Im(8)
Phi–2
0.60
0.75
+0.15
(0.25,0.15,0.60)
0.20
Vicuna–7B–v1.5
0.52
0.70
+0.18
(0.30,0.18,0.52)
0.26
Mistral–7B–Instruct
0.30
0.55
+0.25
(0.45,0.25,0.30)
0.45
Gemma–2–9B–IT
0.19
0.45
+0.26
(0.55,0.26,0.19)
0.58
Mixtral–8 × 7B–Instruct
0.13
0.40
+0.27
(0.60,0.27,0.13)
0.68
LLaMA–3.1–8B–Instruct
0.06
0.32
+0.26
(0.68,0.26,0.06)
0.81
Table 4: Concealed risk and deterministic illusion across models. For each model m we report deterministic and stochastic attack success rates (ASR) on our jailbreak suites, the stochastic risk gap ΔASRm(8) at budget k=8 , the prompt–level fractions of robustly safe, concealed–risk, and deterministic–failure prompts (ρmrobust,ρmconcealed,ρmdet-fail) , and the deterministic illusion index Im(8)=ΔASRm(8)/ASRmstoch(8) , i.e., the fraction of total stochastic risk that is invisible under greedy evaluation. In our simulations, stronger models display lower ASRmgreedy but much higher ΔASRm(8) and Im(8) , indicating that most of their risk is concealed under deterministic evaluation.
Figure 42: Decoder–level risk landscape for LLaMA–2 7B . The surface shows r~k(q)=1−(1−q)k over prompt percentile (x–axis; 0 to 1 , sorted by harmful tail mass q ), samples k (y–axis; 1≤k≤32 ), and risk (z–axis and color; 0 to 1 ). The black, green, and orange curves trace slices at k=1 (greedy), k=8 , and k=16 . For LLaMA–2 7B, roughly the lowest 40–50% of prompts stay near r~k(q)≈0 even as k grows, but the upper 20–30% form a steep ridge where moving from k=1 to k∈{8,16} drives risk into the 0.8−1.0 band, revealing a substantial high–risk tail that greedy testing largely misses.
Figure 43: Decoder–level risk landscape for LLaMA–2 13B . Axes and color scale match Fig. 42 . Compared to 7B, the greedy slice is lower over much of the distribution: the bottom 50–60% of prompts remain very close to r~1(q)≈0 , and even the k=8 curve stays below ≈0.2 for most prompts. However, the top 15–25% of prompts still exhibit a sharp transition where r~k(q) under k∈{8,16} jumps to 0.7−1.0 , indicating that capability scaling reduces the mass of dangerous prompts but leaves a concentrated band where multi–sample risk remains extreme.
Figure 44: Decoder–level risk landscape for LLaMA–3 8B . As before, we plot r~k(q) over prompt percentile, samples k , and risk. LLaMA–3 8B shows a broader low–risk plateau than LLaMA–2: approximately the lowest 55–65% of prompts remain below r~k(q)≈0.1 even at k=8 , and the greedy curve is tightly bound to zero in this region. Beyond the ≈65 th percentile, the k=8 and k=16 slices bend sharply upward, with the top 15–20% of prompts reaching 0.8−1.0 risk. This reflects a model whose overall safety improves, but whose tail prompts still become almost surely harmful under multi–sample decoding.
Figure 45: Decoder–level risk landscape for LLaMA–3 70B . For the 70B variant, the low–risk region widens further: roughly the lowest 70% of prompts remain at r~k(q)≲0.05 for k=1 and stay below ≈0.15 even at k=8 . Yet the remaining 10–15% of prompts still show a dominant ridge where the k=8 and k=16 curves rise into the 0.8−1.0 range. Thus, large–scale alignment compresses the dangerous set into a smaller fraction of prompts, but does not fully remove the near–certain failure zone that appears under stochastic sampling.
Figure 46: Decoder–level risk landscape for Gemma–2 2B . Gemma–2 2B exhibits a moderate plateau of low risk: around the lowest 45–55% of prompts stay at r~1(q)≈0 and remain below ≈0.15 for k=8 . In the upper half of the distribution, however, the k=8 and k=16 slices rise quickly, with the top 20% of prompts approaching r~k(q)≈0.8−1.0 . The contrast between the nearly flat greedy curve in the bulk and the steep rise in the tail again illustrates concealed risk : deterministic ASR underestimates how often multi–sample decoding will find harmful completions.
Figure 47: Decoder–level risk landscape for Gemma–2 27B . Scaling to 27B flattens the low–percentile region further: roughly the lowest 60–70% of prompts stay at r~1(q)≈0 and below about 0.1 even for k=8 . Yet, as with other stronger models, a compact top 10–15% still hosts a steep ridge where k=8 and k=16 yield r~k(q)≈0.9−1.0 . Gemma–2 27B therefore combines excellent average safety with a persistent, narrowly concentrated high–risk tail that only becomes evident when exploring the decoder stochastically.
Figure 48: Decoder–level risk landscape for Mistral 7B . The surface is slightly flatter than for LLaMA–2 7B in the low–percentile region: roughly the lowest 55–65% of prompts remain close to r~k(q)≈0 even at k=8 , and the greedy slice stays near zero over a broad plateau. However, the top 20% of prompts exhibit a steep ridge: going from k=1 to k∈{8,16} increases risk from ≈0.05−0.1 to nearly 0.9−1.0 . This highlights how decoder stochasticity exposes rare but extremely high–risk modes that remain invisible under purely deterministic testing.
Figure 49: Decoder–level risk landscape for Mixtral 8 × 7B . Mixtral 8 × 7B displays a pronounced two–regime structure. For approximately the lowest 60–70% of prompts, all three slices k=1,8,16 remain below r~k(q)≈0.2 , forming a wide, cool (blue) plateau where even multi–sample decoding yields low risk. Beyond the ≈70 th percentile, however, the k=8 and k=16 curves bend sharply upward, with the top 10–15% of prompts saturating to r~k(q)≈1 . This is a canonical concealed–risk pattern : strong average safety coexists with a compact high–tail region where stochastic sampling makes harmful completions almost inevitable.
Figure 50: Decoder–level risk landscape for Mixtral 8 × 22B . The larger Mixtral variant further suppresses greedy risk: the black k=1 curve hugs zero for roughly the lowest 70% of prompts, and even at k=8 the bulk of prompts stay below r~k(q)≈0.1−0.2 . Nevertheless, the upper 10–15% of prompts still form a bright ridge where increasing the search budget from k=1 to k=16 lifts risk into the 0.8−1.0 band. Scaling the model therefore reduces the measure of dangerous prompts but does not fully eliminate the high–risk tail exposed by stochastic decoding.
Figure 51: Decoder–level risk landscape for DeepSeek–R1 Distill . DeepSeek–R1 Distill exhibits an even more compressed tail: the greedy slice remains almost zero for the lowest 75–80% of prompts, and even k=8 keeps most of this mass below r~k(q)≈0.1 . However, a narrow top 10% of prompts still shows a sharp escalation where k=8 and k=16 rapidly push risk to 0.9−1.0 . This regime exemplifies the central theme of our analysis: highly capable, well–aligned models can have excellent deterministic safety profiles yet still hide a small set of prompts where multi–sample decoding yields near–certain failure.
Figure 52: Decoder–level risk landscape for Phi–2 . The surface shows r~k(q)=1−(1−q)k over prompt percentile (x–axis; 0 to 1 , sorted by harmful tail mass q ), samples k (y–axis; 1≤k≤32 ), and risk (z–axis and color; 0 to 1 ). For Phi–2, the greedy slice k=1 is relatively high across a broad band: roughly the middle 30–50% of prompts already reach r~1(q)≈0.1−0.3 , and the upper tail climbs to ≈0.4−0.5 . Increasing the budget to k∈{8,16} pushes much of the upper half into r~k(q)≈0.6−1.0 , yielding a large red region where both deterministic and stochastic ASR are high.
Figure 53: Decoder–level risk landscape for Vicuna–7B . Axes and color scale match Fig. 52 . Relative to Phi–2, the greedy slice is noticeably lower: about the lowest 50–60% of prompts stay near r~1(q)≈0 , and even up to the ≈70 th percentile the greedy risk rarely exceeds 0.15 . However, for the top 20–25% of prompts the k=8 and k=16 curves bend sharply upward, with the worst 10–15% saturating at r~k(q)≈0.8−1.0 . Vicuna–7B therefore has lower average greedy ASR than Phi–2 but still retains a compact high–risk tail where stochastic decoding uncovers failures that deterministic evaluation largely misses.
Figure 54: Attack success vs. sampling budget for different models. Each panel shows the stochastic attack success rate ASRmstoch(k) as a function of sampling budget k∈{1,2,4,8,16,32} for a single model m on our combined jailbreak suites. The horizontal dotted line marks the deterministic ASR ASRmgreedy (i.e., k=1 under greedy decoding), which is often close to zero for stronger models. As k increases, many models exhibit a sharp rise from ASRmstoch(1)≈0 to ASRmstoch(16) or ASRmstoch(32) in the 10 – 30% range, revealing substantial hidden risk that is entirely invisible under standard greedy evaluation. The gap between the dotted baselines and the curves at k=8 or k=16 corresponds directly to the stochastic risk gap ΔASRm(k) and feeds into the illusion index Im(k) reported in Table 4 .
Figure 56: More capable models exhibit larger deterministic illusions. Each point corresponds to a model m , with the x–axis showing a capability proxy Cm (average normalized score across our reasoning benchmarks) and the y–axis showing the illusion index Im(8) at budget k=8 . The fitted trend line reveals a striking pattern: models with higher capability scores tend to have larger Im(8) , meaning that a growing fraction of their true stochastic risk is concealed under greedy evaluation. In our data, smaller models cluster in the lower–left region (moderate capability, modest illusion), while frontier–scale models lie in the upper–right, combining strong performance with substantial hidden harmful tails . This formalizes the concern that “safer under greedy” does not necessarily mean “safer under realistic stochastic use.”
Figure 57: Decomposing safety into robustly safe prompts, concealed risk, and deterministic failures. For each model m , we partition attack prompts into three disjoint categories: robustly safe (the greedy completion is safe and all k=8 stochastic samples are safe), concealed risk (the greedy completion is safe, but at least one of the k stochastic samples is harmful), and deterministic failure (the greedy completion itself is already harmful). The horizontal stacked bars show, for each model, the empirical fraction of prompts assigned to each category, while the y–axis labels additionally report in parentheses the fraction of total risk that is hidden, hidden_sharem=concealedm+det_failmconcealedm , among all risky prompts for that model. Models are ordered from top to bottom by their total stochastic attack success rate ASRmstoch(k)=concealedm+det_failm , so that overall risk monotonically increases down the figure. Stronger, more safety-tuned models in the lower rows exhibit very small deterministic-failure mass but extremely large concealed-risk bands and correspondingly high hidden shares, indicating that most of their stochastic risk arises from prompts that would be certified safe by any evaluator that only inspects the greedy output. By contrast, weaker or poorly aligned models near the top allocate more mass to deterministic failures and less to concealed risk, meaning that a larger portion of their risk remains directly visible under standard greedy evaluation. This compositional view makes the illusion of robustness visually explicit : a seemingly safe region under deterministic decoding can in fact contain a substantial reservoir of hidden stochastic risk.
Figure 58: Case study I: a prompt that appears safe under greedy evaluation but hides a harmful stochastic tail. (a) For a single jailbreak prompt x⋆ , we draw k=16 completions {Yj(x⋆)}j=116 from the model and display them as a 4×4 grid. Each cell is annotated with its sample index j (with j=1 marked as Greedy ) and colored by the human harm label: green for safe , orange for borderline / evasive , and red for harmful . Even though the greedy completion Ygreedy(x⋆)=Y1(x⋆) is labeled safe, we observe multiple harmful samples among the remaining Yj(x⋆) when we enable sampling. (b) The right panel shows the full prompt x⋆ and a one-line summary of each of the 16 sampled completions, again color–coded by label. In this particular case, we obtain ∑j=116Zj(x⋆)=3 harmful completions and an empirical harmful mass qθ(x⋆)≈0.19 , which yields a tail–risk estimate r16(x⋆)=1 , i.e., a batch of 16 samples almost surely contains at least one harmful continuation. Deterministic (greedy) evaluation would thus certify x⋆ as safe , even though realistic multi–sample usage reveals a substantial harmful tail. This single–prompt case study makes the illusion of robustness visually explicit: what looks green under greedy decoding still hides significant red mass once we examine the full stochastic behavior.
In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents. Agentic systems are frequently required to sample from distributions, often inferred from observed data, a process which needs to be emulated by the LLM. This leads to a distinct failure point: while standard RL agents rely on external sampling mechanisms, LLMs fail to map their internal probability estimates to their stochastic outputs. Through rigorous empirical analysis across multiple model families, model sizes, prompting styles, and distributions, we demonstrate the extent of this failure. Crucially, we show that while powerful frontier models can convert provided random seeds to target distributions, their ability to sample directly from specific distributions is fundamentally flawed.
Xiangming Gu, Soham De, Michalis Titsias +3
Google DeepMind · National University of Singapore
Diffusion language models (DLMs) have emerged as a promising paradigm for large language models (LLMs), yet the non-deterministic behavior of DLMs remains poorly understood. The existing non-determinism evaluations for LLMs predominantly rely on dataset-level metrics under fixed inference configurations, providing limited insight into how model behavior varies across runs and evaluation conditions. In this work, we show that dataset-level metrics systematically attenuate non-determinism in diffusion language models by aggregating sample-level prediction quality across different runs. As a result, configurations with similar aggregate performance can exhibit substantially different behaviors on individual inputs, leaving fine-grained instability and distinct error patterns uncharacterized. To address this limitation, we conduct a fine-grained evaluation of non-determinism based on sample-level prediction differences across a range of model-related factors-including guidance scale, diffusion steps, and Monte Carlo sampling-as well as system-related factors such as batch size, hardware, and numerical precision. Our analysis reveals that non-determinism in DLMs is pervasive and structured, with code generation exhibiting markedly higher sensitivity to factor-level choices than question answering. To attribute sources of non-determinism evaluation, we introduce Factor Variance Attribution (FVA), a cross-factor analysis metric that decomposes observed non-determinism into variance attributable to different evaluation factor settings. Our findings highlight the need for fine-grained, factor-aware evaluation to enable reliable non-determinism assessment of diffusion language models.
Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen +5
Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, USA · Department of Computer Science & Engineering, Texas A&M University, College Station, USA · Department of Biochemistry, Case Western Reserve University, Cleveland, USA +2
This discussion argues that sequential statistical inference can naturally contribute to LLM trustworthiness. In deployment, LLM systems are queried repeatedly, conditioned on evolving contexts, and incorporate user or tool feedback, and may exhibit behavioral shifts after model updates or distribution changes. The discussion is organized around three tasks: representation, modeling LLM interactions as dependent stochastic processes rather than isolated prompt--response pairs; validity, developing uncertainty guarantees that remain meaningful under dependence, repeated use, and adaptation; and monitoring, using sequential alarms and change-point detection to identify shifts in calibration, hallucination rates, refusal behavior, fairness, or other task-relevant properties. This perspective complements recent surveys by viewing trustworthy LLM deployment as a problem of statistical process control.
Yao Xie
H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology