The Illusion of Stochasticity in LLMs

Xiangming Gu; Soham De; Michalis Titsias; Larisa Markeeva; Petar Veličković; Razvan Pascanu

cs.CLApr 8, 2026

The Illusion of Stochasticity in LLMs

Authors: Xiangming Gu, Soham De, Michalis Titsias, Larisa Markeeva, Petar Veličković, Razvan Pascanu

Organizations: Google DeepMind · National University of Singapore

Abstract

In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents. Agentic systems are frequently required to sample from distributions, often inferred from observed data, a process which needs to be emulated by the LLM. This leads to a distinct failure point: while standard RL agents rely on external sampling mechanisms, LLMs fail to map their internal probability estimates to their stochastic outputs. Through rigorous empirical analysis across multiple model families, model sizes, prompting styles, and distributions, we demonstrate the extent of this failure. Crucially, we show that while powerful frontier models can convert provided random seeds to target distributions, their ability to sample directly from specific distributions is fundamentally flawed.

Figures & tables

Figure 1: LLMs are heavily biased towards “C” rather than sampling uniformly when being prompted to generate a multi-choice question.

Model	A	B	C	D	Best-response attacker success
Gemini-3 Flash	0.034	0.521	0.373	0.071	0.966
Gemini-3.1 Pro	0.002	0.297	0.700	0.001	0.999
Gemini-3.5 Flash	0.001	0.408	0.571	0.020	0.999

Table 1: Server-selection distributions and best-response attacker success rates.

Panel of Figure 2 — Figure 2: Empirical distribution estimates, for various target distributions, using 1024 independent samples drawn from LLMs. ( Left ) Qwen3-8B and ( Right ) Gemini-2.5-Pro. The x-axis corresponds to the sample space while the y-axis shows the estimated empirical frequencies. It is demonstrated that LLMs cannot reliably sample from uniform and Gaussian distributions through independent sampling.

	Qwen3-8B	Qwen3-32B	Gemini-2.5-Pro	Gemini-3.0-Pro
Uniform $\{0,1,...,9\}$	$<$ 1e-300	1.37e-235	$<$ 1e-300	$<$ 1e-300
Uniform $[0,1]$	8.06e-113	2.69e-111	3.86e-243	3.87e-103
Gaussian $\mathcal{N}(0,1)$	9.53e-122	2.02e-104	2.62e-199	1.81e-42

Table 2: Goodness-of-fit tests using samples generated by LLMs for three representative distributions. We report p-values for all tests, which are almost zero (smaller than 0.05), which indicates LLMs cannot reliably sample from the target distribution.

Panel of Figure 3 — Figure 3: Empirical distribution estimates, for different orders of random set in the prompt, ( Left ) {left, right, up, down, fire} ( Right ) {up, down, left, right, fire} .

Model	0	1	2	3	4	5	6	7	8	9
Qwen3-8B	$<0.01$	$<0.01$	0.03	1.56	66.16	31.25	0.94	0.06	$<0.01$	$<0.01$
Qwen3-8B-Base	8.21	10.54	13.53	13.53	10.54	13.53	8.21	10.54	6.39	4.98
OLMo3-7B-Think	0.56	0.02	1.05	69.39	4.44	8.29	0.07	15.48	0.47	0.22
OLMo3-7B-Base	10.28	5.86	8.01	13.20	13.62	12.40	9.96	11.65	9.36	5.68

Table 3: Output logit distributions (%) over numbers 0–9 for reasoning and base models.

Panel of Figure 4 — Figure 4: Empirical distribution estimates, using ( Left ) Qwen3-8B to sample from a uniform discrete distribution with temperature value 0.2 and 2.0; ( Right ) Qwen3-14B to sample from a Gaussian distribution with temperature value 0.6 and 2.0. Across different temperatures, LLMs tend to have similar failure mode, e.g., the preference to 7 and 5, or ranges around $\pm\sigma/2$ instead of $\mu$ .

Panel of Figure 5 — Figure 5: From left to right , we showcase the auto-correlation of the sample sequence, against different time lags, for four sampling approaches when sampling from a uniform distribution between 0 and 9 with Gemini-2.5-Pro: independent sampling, sequential sampling with-all-history, sequential sampling with-last-history, batched sampling.

Panel of Figure 6 — Figure 6: Empirical transition counts between pairs of states when sampling from a uniform distribution between 0 and 9 with Gemini-2.5-Pro. ( Left ) sequential sampling with-all-history ( Right ) sequential sampling with-last-history.

Panel of Figure 7 — Figure 7: Empirical distribution estimates, for a uniform discrete distribution between 0 and 9, using the Gemini family under the setup of ( Left ) tool-use ( Right ) tool-use, prompting to use current time as random seed.

Panel of Figure 8 — Figure 8: Truncated responses when prompting LLMs (Qwen3-32B) to use a random number in [0, 1] to sample from ( Left ) uniform discrete distribution on colors and ( Right ) Gaussian distribution $\mathcal{N}(-23,1)$ . More examples are shown in Appendix E .

	Qwen3-1.7B	Qwen3-4B	Qwen3-8B	Gemini-2.5-Pro
Uniform $\{0,1,...,9\}$	21.3%	92.6%	97.5%	94.3%
Uniform $[0,1]$	19.9%	96.2%	97.8%	92.3%

Table 4: Accuracies of LLMs simulating PRNG algorithms for uniform distributions. For discrete distribution, we consider a sample correct when it is exactly the same as PRNG output. While for continuous one, we consider their difference is smaller than 0.01.

Panel of Figure 9 — Figure 9: LLMs can reliably convert a uniform distribution in [0, 1] to various distributions. From left to right , we use Qwen3-30B-A3B to sample from a uniform discrete distribution between 0 and 9; Qwen3-32B to sample from a non-uniform distribution on directions; Qwen3-14B to sample from a Gaussian; Gemini-3.0-Pro to sample from a Gaussian mixture model.

Qwen3 model size	0.6B	1.7B	4B	8B	14B	32B
Uniform {0,1,…,9}	0.00	2.33e-4	0.23	0.23	0.23	0.23
Gaussian $\mathcal{N}(0,1)$	1.99e-232	9.50e-186	1.61e-46	0.29	0.21	0.78

Table 5: Goodness-of-fit tests when prompting Qwen3 family to convert

\mathcal{U}[0,1]

to uniform {0, 1, 2, …, 9} and Gaussian

\mathcal{N}(0,1)

. We bold the scenarios that achieve

p>0.05

, which show the correlation with model size.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: Example responses of ( Left ) Gemini-2.5-Pro and ( Right ) Gemini-3.0-Pro when prompting to generate a multi-choice question.

Panel of Figure 11 — Figure 11: Empirical distribution estimates, for various distributions, using 1024 independent samples drawn from ( Top ) OLMO-3-7B-Think ( Bottom ) OLMO-3-32B-Think. The x-axis corresponds to the sample space while y-axis shows the estimated empirical frequencies.

Panel of Figure 12 — Figure 12: Example of thinking trace of Gemini-2.5-Flash when sampling from a uniform discrete distribution between 0 and 9.

Panel of Figure 13 — Figure 13: Example of thinking trace of Gemini-2.5-Flash when sampling from a uniform continuous distribution within [0, 1].

Panel of Figure 14 — Figure 14: Example of thinking trace of Gemini-2.5-Pro when sampling from a uniform discrete distribution between 0 and 9.

Panel of Figure 15 — Figure 15: Example of thinking trace of Gemini-2.5-Pro when sampling from a Gaussian distribution with mean 0 and standard deviation 1.

Panel of Figure 16 — Figure 16: Example of response of Qwen3-8B when sampling from a uniform discrete distribution between 1 and 100.

Panel of Figure 17 — Figure 17: Example of response of Qwen3-14B when sampling from a uniform discrete distribution on letters.

Panel of Figure 18 — Figure 18: Example of response of Qwen3-32B when sampling from a uniform continuous distribution within [0, 1].

Panel of Figure 19 — Figure 19: Example of response of Qwen3-30B-A3B when sampling from a Gaussian distribution with mean 0 and standard deviation 1.

Panel of Figure 20 — Figure 20: Examples of response of ( Left ) Gemini-2.5-Pro and ( Right ) Qwen3-8B without chain-of-though reasoning.

Panel of Figure 21 — Figure 21: Empirical distribution estimates, using ( Left ) Qwen3-8B when sampling from a uniform discrete distribution from 1 to 100 with (left) and without (right) chain-of-thought reasoning. ( Right ) Gemini-2.5-Flash when sampling from uniform discrete distribution from 0 to 9 with (left) and without (right) chain-of-thought reasoning.

Panel of Figure 22 — Figure 22: From left to right , we compare the empirical distribution estimates, for four sampling approaches when sampling from a uniform discrete distribution between 0 and 9 with Gemini-2.5-Pro: independent sampling, sequential sampling with all history, sequential sampling with last history, batched sampling.

Panel of Figure 23 — Figure 23: From left to right , we compare four sampling approaches by adopting Gemini-2.5-Pro to sample from a Gaussian distribution with mean 0 and standard deviation 1: independent sampling, sequential sampling with all history, sequential sampling with last history, batched sampling.

Panel of Figure 24 — Figure 24: ( Left ) Empirical distribution estimates and transition counts of Qwen3-32B when sequential sampling from uniform discrete distribution between 0 and 9 ( Left ) with all history ( Right ) with last history.

Panel of Figure 25 — Figure 25: Example of thinking trace of Gemini-2.5-Pro when using sequential sampling with all history from a uniform discrete distribution between 0 and 9.

Panel of Figure 26 — Figure 26: Example of thinking trace of Gemini-2.5-Pro when using sequential sampling with all history from a uniform discrete distribution between 0 and 9.

Panel of Figure 27 — Figure 27: Example of thinking trace of Gemini-2.5-Pro when using sequential sampling with last history from a uniform discrete distribution between 0 and 9.

Panel of Figure 28 — Figure 28: Example of thinking trace of Gemini-2.5-Pro when using sequential sampling with last history from a uniform discrete distribution between 0 and 9.

Panel of Figure 29 — Figure 29: Empirical distribution estimates for the number of generated numbers when using Gemini-2.5-Pro to batched sample from ( Left ) a uniform discrete distribution between 0 and 9 ( Right ) a Gaussian distribution with mean 0 and standard deviation 1.

Panel of Figure 30 — Figure 30: Empirical distribution estimates of first and second sample across independent batched sampling: left uniform discrete distribution between 0 and 9; right Gaussian distribution with mean 0 and standard deviation 1.

Panel of Figure 31 — Figure 31: Example of thinking trace of Gemini-2.5-Pro when batched sampling from a uniform discrete distribution between 0 and 9.

Panel of Figure 32 — Figure 32: Example of thinking trace of Gemini-2.5-Pro when batched sampling from a uniform discrete distribution between 0 and 9.

Panel of Figure 33 — Figure 33: Example of thinking trace of Qwen3-32B when batched sampling from a uniform discrete distribution between 0 and 9.

Panel of Figure 34 — Figure 34: Example code generation and execution results of ( Left ) Gemini-2.5-Flash and ( Right ) Gemini-2.5-Pro with tool-use.

Panel of Figure 35 — Figure 35: Example code generation and execution results of ( Left ) Gemini-2.5-Flash and ( Right ) Gemini-3.0-Pro with tool-use.

Panel of Figure 36 — Figure 36: Prompt design for using LLMs to sample from a uniform discrete distribution with a PRNG algorithm and a random seed without tool calling (empty lines and code comments are truncated).

Panel of Figure 37 — Figure 37: Prompt design for using LLMs to sample from a uniform continuous distribution with a PRNG algorithm (empty lines and code comments are truncated).

Panel of Figure 38 — Figure 38: Prompt design for using LLMs to sample from a Gaussian distribution with a PRNG algorithm (empty lines and code comments are truncated).

Model	Uniform $\{0,1,...,9\}$	Uniform $[0,1]$	Gaussian $\mathcal{N}(0,1)$
Qwen3-1.7B	5.27e-310	2.57e-30	1.98e-78
Qwen3-4B	0.02	1.00	1.42e-84
Qwen3-8B	1.00	1.00	8.68e-83
Qwen3-14B	0.98	1.00	5.93e-137
Gemini-2.5-Flash	0.22	1.30e-35	2.33e-138
Gemini-2.5-Pro	1.00	1.00	2.09e-15

Appendix

Table 6: Goodness-of-fit tests when prompting Qwen3 and Gemini-2.5 families to simulate PRNG algorithms for three representative distributions. We bold the scenarios that achieve

p>0.05

, which shows the tendency of better sampling results w.r.t. model capabilities.

Panel of Figure 39 — Figure 39: From left to right , we visualize the failures of LLMs simulating PRNG algorithms for Gaussian distributions: Gemini-2.5-Flash, Gemini-2.5-Pro, Qwen3-8B, Qwen3-32B.

Panel of Figure 40 — Figure 40: Truncated responses when prompting ( Left ) Gemini-2.5-Pro and ( Right ) Qwen3-8B to simulate a PRNG algorithm for uniform discrete distribution between 0 and 9. The thinking traces and some special tokens are also truncated as they are too verbose.

Panel of Figure 41 — Figure 41: Truncated responses when prompting ( Left ) Gemini-2.5-Pro and ( Right ) Qwen3-14B to simulate a PRNG algorithm for uniform continuous distribution [0, 1]. The thinking traces and some special tokens are also truncated as they are too verbose.

Panel of Figure 42 — Figure 42: Truncated responses when prompting ( Left ) Gemini-2.5-Pro and ( Right ) Qwen3-32B to simulate a PRNG algorithm for Gaussian distribution $\mathcal{N}(0,1)$ . The thinking traces and some special tokens are also truncated as they are too verbose. We use bold red to mark the first mistake LLM made. Typically LLMs struggle with large number multiplications.

Panel of Figure 43 — Figure 43: Responses when prompting ( Left ) Gemini-2.5-Pro ( Right ) Qwen3-30B-A3B to use a random number in [0, 1] to sample from a uniform discrete distribution between 0 and 9.

Panel of Figure 44 — Figure 44: Responses when prompting ( Left ) Gemini-2.5-Pro ( Right ) Qwen3-14B to use a random number in [0, 1] to sample from a Gaussian distribution with mean 0 and standard deviation 1.

Panel of Figure 45 — Figure 45: Responses when prompting ( Left ) Gemini-3.0-Pro to sample from a Gaussian Mixture Model (GMM); ( Right ) Qwen3-32B to sample from a non-uniform discrete distribution on directions usi a random number in [0, 1] as input.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 759 papers

Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition
Jan 12, 2026Tanmay Joshi, Shourya Aggarwal, Anusa Saha +7Emergent Abilities in Language Models LLM Reliability
Pragya Lab, BITS Pilani Goa, India · Raapid Lab, USA · Apple, USA +1
UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs
Jun 4, 2026Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo +4LLM Evaluation Language Model Generation Evaluation
University of British Columbia · 2Independent Researcher
Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Models
Jun 4, 2026Huiyuan Zheng, Houtao Zhang, Boyang Wang +2Multimodal Large Language Models Language Model Bias Evaluation
Fudan University · Beihang University

Jan 12, 2026J/K move · Enter open · S save

Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition

Tanmay Joshi, Shourya Aggarwal, Anusa Saha, Aadi Pandey+6

Pragya Lab, BITS Pilani Goa, India · Raapid Lab, USA · Apple, USA+1

Emergent Abilities in Language Models LLM Reliability LLM Evaluation

Open paper AskPDF arXiv

Deterministic inference is a comforting ideal in classical software: the same program on the same input should always produce the same output. As large language models move into real-world deployment, this ideal has been imported wholesale into inference stacks. Recent work from the Thinking Machines Lab has presented a detailed analysis of nondeterminism in LLM inference, showing how batch-invariant kernels and deterministic attention can enforce bitwise-identical outputs, positioning deterministic inference as a prerequisite for reproducibility and enterprise reliability. In this paper, we take the opposite stance. We argue that, for LLMs, deterministic inference kills. It kills the ability to model uncertainty, suppresses emergent abilities, collapses reasoning into a single brittle path, and weakens safety alignment by hiding tail risks. LLMs implement conditional distributions over outputs, not fixed functions. Collapsing these distributions to a single canonical completion may appear reassuring, but it systematically conceals properties central to artificial cognition. We instead advocate Stochastic CHAOS, treating distributional variability as a signal to be measured and controlled. Empirically, we show that deterministic inference is systematically misleading. Single-sample deterministic evaluation underestimates both capability and fragility, masking failure probability under paraphrases and noise. Phase-like transitions associated with emergent abilities disappear under greedy decoding. Multi-path reasoning degrades when forced onto deterministic backbones, reducing accuracy and diagnostic insight. Finally, deterministic evaluation underestimates safety risk by hiding rare but dangerous behaviors that appear only under multi-sample evaluation.

Abstract

LLM Reliability

Explore similar work

Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Models