Inference-time sampling is a central mechanism for improving large language models, with temperature controlling the tradeoff between reliability and diversity. While recent work suggests that varying temperature across samples can improve multi-sample inference, simple fixed-temperature schemes remain the standard in practice. We investigate this discrepancy in the canonical pass@K setting, asking when temperature diversity is useful, and when a single tuned temperature suffices. To this end, we analyze per-question success probabilities across multiple models and datasets, tracking how they vary with sampling temperature. We identify several consistent properties of these profiles, including that harder questions are less likely to share efficient sampling temperatures, although this trend is only logarithmic in hardness. We then derive an efficient method for evaluating the theoretically optimal multi-temperature allocation scheme, and show that although such a scheme improves performance, its gains over single-temperature baselines are typically modest when temperature is tuned as a function of the sampling budget K. We explain this phenomenon through mathematical mechanisms that we call dulling effects, and show how they ensure the robustness of temperature choices across different question distributions. Our results clarify when multi-temperature sampling is useful, and how temperatures should ideally be chosen.
Figures & tables
Figure 1: Temperature profiles for 51 Qwen3-8B questions. The green curve, which attains its maximum at τ=0 , illustrates the first canonical form, whereas the red and blue curves, which peak at positive temperatures with corresponding probabilities PE1 and PE2 , illustrate the second.
Figure 2: Three hardness-based views on Qwen3-8B on LiveCodeBench: peak temperature (left), efficient range size (middle), and maximal overlap ratio of efficient ranges (right) vs. hardness (log 10 scale). Left: blue/red curves show sliding averages over all instances, with/without small ideal-temperature instances. Middle: linear fit over instances. Right: maximal coverage in log-uniform hardness bins. Temperatures vary with hardness, while range size and overlap shrink logarithmically.
Figure 3: Pass@ K on log 10 scale for Mistral 7B on LiveCodeBench under fixed-temperature strategies (blueish) vs. the optimal allocation (red), normalized by the uniform-temperature baseline (dotted). While the optimal allocation consistently outperforms fixed strategies for some K , for each K at least one achieves comparable performance.
Figure 4: Sensitivity of the test pass@ K score to the question hardness distribution used for temperature selection (Mistral 7B Instruct, LiveCodeBench). Left: Goldilocks protection. Questions whose hardness is within a factor 10D of K keep their weight, a 10% loss requires reweighting the others by β∗ , which grows exponentially with D . Right: KL divergence. Even far from the test distribution in KL divergence, most random training distributions W cost under 10% .
Appendix figures & tables37 assets
Supplementary material from the paper’s appendix.
Appendix
Constraint
Function-level meaning
Bernstein-coefficient meaning
Non-negativity
Pqi(τ)≥0 for all τ
All coefficients cj≥0
Unimodality
Pqi(τ) has a single peak
Sequence {cj} is unimodal
Appendix
Table 1: Shape constraints translated from the function level to the Bernstein-coefficient level. The {cj} are the Bernstein-basis weights of the fitted profile.
Figure 5: Per-question peak temperature vs. hardness on LiveCodeBench, with sliding averages over all questions (blue) and over the large ideal temperature subset (red), across all four models. n is the number of included questions.
Figure 6: Per-question peak temperature vs. hardness on MATH-500, drawn as in Figure 5 . n is the number of included questions.
Figure 8: Per-question efficient range size vs. hardness on LiveCodeBench, with the OLS fit on log10H (slope and R2 in each legend), across all four models. n is the number of included questions.
Figure 10: Per-question efficient range size vs. hardness on Hi-ToM, drawn as in Figure 9 . n is the number of included questions.
Figure 12: Empirical RQ(H) on MATH-500, drawn as in Figure 11 . n is the number of included questions.
Figure 14: Ratio to uniform pass@ K , rtu(K)=pass@K(strategy)/pass@K(uniform) , on LiveCodeBench for all four models, read as described in Appendix F.1 . The red trace is the Section 4.1 optimum, evaluated leave-one-out (LOO) at each K of the colored-curve grid. n is the number of included questions.
Figure 16: Ratio to uniform pass@ K on Hi-ToM for all four models, defined as in Figure 15 ; n is the number of included questions. Solid curves show the temperatures that are the dominant choice in the optimal allocation at some K , namely every grid temperature from 1.125 to 1.575 for GPT OSS 20B, from 1.425 to 1.875 for Mistral 7B Instruct, 2.5 and 3.0 for Qwen3-8B, and from 1.05 to 1.575 for Qwen3-0.6B. Dashed curves show the other temperatures of a reference set drawn in every panel, τ∈{0.3,0.6,0.9,1.2,1.5,1.8,2.5,3.0} . A temperature has the same color in every panel.
K
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
1
0.344
0.361
0.372
0.381
0.389
0.396
0.402
0.407
2
0.424
0.466
0.491
0.508
0.521
0.532
0.542
0.549
3
0.467
0.522
0.555
0.575
0.590
0.602
0.612
0.619
4
0.495
0.560
0.595
0.617
0.633
0.645
0.654
0.660
6
0.531
0.607
0.646
0.669
0.685
0.696
0.704
0.709
8
0.556
0.638
0.678
0.701
0.716
0.727
0.734
0.738
Appendix
Table 2: Pass@ K for GPT OSS 20B. Low temperatures τ∈[0.1,0.8] above, high temperatures τ∈[0.9,2.0] together with the row-aggregate columns below.
K
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
1
0.178
0.184
0.186
0.185
0.181
0.177
0.171
0.164
2
0.198
0.216
0.227
0.234
0.238
0.238
0.236
0.231
3
0.210
0.236
0.254
0.265
0.273
0.277
0.277
0.274
4
0.219
0.252
0.274
0.289
0.299
0.305
0.308
0.306
6
0.233
0.275
0.303
0.323
0.338
0.347
0.352
0.353
8
0.243
0.292
0.325
0.349
0.366
0.378
0.386
0.388
Appendix
Table 3: Pass@ K for Mistral 7B Instruct. Low temperatures τ∈[0.1,0.8] above, high temperatures τ∈[0.9,2.0] together with the row-aggregate columns below.
K
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1
0.585
0.585
0.583
0.582
0.580
0.578
0.576
0.575
0.574
0.573
2
0.615
0.626
0.632
0.637
0.639
0.641
0.642
0.643
0.643
0.643
3
0.628
0.645
0.656
0.663
0.668
0.672
0.675
0.676
0.678
0.679
4
0.636
0.658
0.671
0.681
0.688
0.693
0.696
0.699
0.701
0.702
6
0.647
0.675
0.693
0.705
0.713
0.720
0.725
0.729
0.731
0.733
8
0.655
0.688
0.707
0.720
0.730
0.738
0.744
0.748
0.751
0.753
Appendix
Table 4: Pass@ K for Qwen3-8B, low-temperature half ( τ∈[0.1,1.9] ). τ∈[0.1,1.0] above, τ∈[1.1,1.9] below. Continued in Table 5 ; the row-aggregate columns are computed across the full 38-temperature sweep and appear there.
K
2.0
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
1
0.501
0.475
0.444
0.409
0.369
0.326
0.281
0.236
0.191
0.150
2
0.601
0.584
0.563
0.536
0.502
0.462
0.415
0.363
0.307
0.250
3
0.643
0.630
0.613
0.592
0.564
0.530
0.488
0.438
0.381
0.319
4
0.669
0.657
0.642
0.624
0.600
0.570
0.532
0.486
0.431
0.369
6
0.699
0.689
0.676
0.660
0.640
0.615
0.583
0.543
0.493
0.434
8
0.719
0.709
0.696
0.682
0.663
0.640
0.612
0.575
0.530
0.475
Appendix
Table 5: Pass@ K for Qwen3-8B, high-temperature half ( τ∈[2.0,4.2] ); continuation of Table 4 . τ∈[2.0,2.9] above, τ∈[3.0,4.2] together with the row-aggregate columns below. Aggregates are computed across the full 38-temperature sweep.
K
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1
0.011
0.022
0.032
0.041
0.047
0.051
0.051
0.048
0.043
0.036
2
0.020
0.041
0.061
0.077
0.087
0.093
0.094
0.089
0.079
0.066
3
0.029
0.059
0.086
0.107
0.122
0.130
0.130
0.124
0.111
0.093
4
0.036
0.075
0.108
0.134
0.152
0.161
0.162
0.154
0.139
0.117
6
0.050
0.102
0.146
0.179
0.201
0.213
0.214
0.204
0.185
0.158
8
0.061
0.125
0.177
0.215
0.240
0.254
0.255
0.244
0.222
0.191
Appendix
Table 6: Pass@ K for Qwen3-0.6B. Low temperatures τ∈[0.1,1.0] above, high temperatures τ∈[1.1,2.0] together with the row-aggregate columns below.
Figure 17: Distribution of scoreB/scoreA as a function of KL(W∥uniform) on the Mistral 7B dataset ( 359 included questions, peak pass-rate ≥0.1% ), with W drawn from a Dirichlet smooth-base sampler over the questions. Each panel corresponds to one value of K∈{4,32,128,512} ; the same 10,000 random W ’s are used in every panel, so differences reflect only the change of K . Each panel is a 6×6 column-normalized heatmap (KL bands on x , ratio bands on y ); the cell label reports the percent of W ’s in that band, and the fill is on a log-scaled viridis from ≤0.01% to 100% . The cell label 100 is reserved for columns in which every W lands in that ratio band; otherwise the label is the two-decimal percentage. At K=4 the argmax is brittle (top-row occupancy falls to 69.71% at the highest KL band); at K∈{32,128} the rightmost band still keeps ∼84% of W ’s within 10% of scoreA ; at K=512 the rightmost band keeps 87.87% , marginally the most robust regime.
Figure 18: Distribution of scoreB/scoreA as a function of KL(W∥uniform) for GPT OSS 20B ( 766 included questions, entropy ceiling ln(766)≈6.64 nats), constructed as in Figure 17 .
Figure 19: Distribution of scoreB/scoreA as a function of KL(W∥uniform) for Qwen3-8B ( 647 included questions, entropy ceiling ln(647)≈6.47 nats, later data release, see text), constructed as in Figure 17 .
Figure 20: Distribution of scoreB/scoreA as a function of KL(W∥uniform) for Qwen3-0.6B ( 85 included questions, entropy ceiling ln(85)≈4.44 nats), constructed as in Figure 17 .
Figure 21: Mistral 7B Instruct on MATH-500 ( 455 included questions), constructed as in Figure 17 .
Figure 23: Qwen3-8B on MATH-500 ( 496 included questions), constructed as in Figure 17 .
Figure 25: Mistral 7B Instruct on Hi-ToM ( 615 included questions), constructed as in Figure 17 .
Figure 27: Qwen3-8B on Hi-ToM ( 466 included questions), constructed as in Figure 17 .
Figure 29: β∗ against D on LiveCodeBench, at a 10% loss (left in each panel) and a 1% loss (right), with one curve per K . n is the number of included questions.
Figure 31: β∗ against D on Hi-ToM, drawn as in Figure 30 . n is the number of included questions. For GPT OSS 20B at a 10% loss, with pass@ K computed exactly from the fitted pass rates, the curve of K=512 needs β>1012 at every D and lies on the top row.
Figure 32: The three simulated hardness shapes, peaked (top), reversed (middle), and flat (bottom).
Figure 33: β∗ against D for the three simulated shapes, at a 10% loss (left in each panel) and a 1% loss (right), with one curve per K , drawn as in Figure 30 . Each shape has 1000 simulated questions, with hardness up to a cutoff of 500,000 and temperature profiles from Mistral 7B Instruct.
Figure 34: Ratio to uniform pass@ K , rtu(K) , of Qwen3-0.6B on Hi-ToM at τ=1.5 , defined as in Figure 15 with min_p in place of the temperature. Colored curves are single- min_p allocations, each with its paired-bootstrap ±1σ band. Solid curves show the values that are the dominant choice in the optimal allocation at some K , namely 0.001 , 0.003 , and 0.01 , and dashed curves show the other values. The red trace is the LOO optimum. The population is the 648 kept questions, with ν=500 , and the vertical axis is compressed below 1 .
Figure 35: Ratio of pass@ K to that of τ=1.5 with min_p=0.003 , for Qwen3-0.6B on Hi-ToM. Colored curves are the 5 values at τ=2.0 , and the dark line at 1 is the reference. Faint curves are the other τ=1.5 values. Those of min_p=0 and 0.2 are cut at the lower edge, and those of 0.3 , 0.5 , and 0.7 lie entirely below the plotted range, so 6 of the 9 are visible. Points are ratios of two means over the 648 questions, with 400 second-stage samples and no error bars.
Figure 36: Ratio of pass@ K with min_p at τ=2.0 to that of the best plain temperature of Appendix F.2 , for Mistral 7B Instruct on Hi-ToM. Colored curves are the 5 values, and the dark line at 1 is the best plain temperature at each K , picked on the same questions. Points are ratios of two means over the 617 questions, with ν=1500 and no error bars.
Figure 37: Ratio to uniform pass@ K , rtu(K) , on Hi-ToM at τ=2.0 , defined as in Figure 34 and computed from measured pass rates. n is the number of included questions. Colored curves are single- min_p allocations, each with its paired-bootstrap ±1σ band. Solid curves show the values that are the dominant choice in the optimal allocation at some K , dashed curves the others, and the red trace is the LOO optimum. K runs to 384 for Qwen3-0.6B and to 1024 for Mistral 7B Instruct. For Mistral 7B Instruct, 19 questions have a best pass rate below the 0.1% inclusion floor of the paper’s analysis, 8 of them with no correct sample, and the curve of 0.0056 lies under the red trace. The vertical axis is compressed below 1 .
Figure 38: min_p profiles of 51 Hi-ToM questions in each study, the counterpart of Figure 1 . Lines join measured pass rates, with no fit, and grid values are spaced equally on the horizontal axis. In each third of the grid, with 4 , 3 , and 3 values at τ=1.5 and 2 , 2 , and 1 at τ=2.0 , we highlight, among the questions that peak there, the one with the most correct samples at its peak, with the peak at the smallest min_p among tied values and ties between questions broken by id. Bars are exact Clopper-Pearson 1σ intervals, and the vertical axis stops at 60% . The 48 grey questions are drawn at random, with a fixed seed, from the other questions of the study.
Figure 39: Per-question peak min_p against hardness, with sliding averages over all questions (blue) and over the questions whose peak lies above the smallest grid value (red), the counterpart of Figure 5 . The peak is the min_p of the largest measured pass rate, the smallest among tied values. Grid values are spaced equally on the vertical axis, so the averages are means of grid position. The window is 30 questions, and the averages are drawn only where the window is full. No further low-hardness points are dropped, since Hq takes discrete values here.
Figure 40: Per-question efficient range size against hardness, with the OLS fit on log10Hq , the counterpart of Figure 9 . Here ∣Eq∣ is the number of grid values in the efficient range, since the 10 and 5 values of these grids do not support the interpolated tR−tL .
Figure 41: Maximal coverage against hardness, the counterpart of Figure 11 . Questions are placed in 17 log-uniform hardness bins, and the bar height is the largest share of the in-bin questions whose efficient ranges Eq hold one grid value. Empty bins have no bar, and the bins wholly above H=100 hold 9 , 2 , and 68 questions in the three studies.
Figure 42: Distribution of scoreB/scoreA as a function of KL(W∥uniform) for Qwen3-0.6B on Hi-ToM at τ=1.5 ( 648 included questions, ν=500 , K∈{4,32,128,256} , entropy ceiling ln(648)≈6.47 nats), constructed as in Figure 17 with min_p in place of the temperature.
Figure 43: Distribution of scoreB/scoreA as a function of KL(W∥uniform) for Qwen3-0.6B on Hi-ToM at τ=2.0 ( 648 included questions, ν=400 , K∈{4,32,128,256} , entropy ceiling ln(648)≈6.47 nats), constructed as in Figure 17 with min_p in place of the temperature.
Figure 44: Distribution of scoreB/scoreA as a function of KL(W∥uniform) for Mistral 7B Instruct on Hi-ToM at τ=2.0 ( 598 included questions, ν=1500 , K∈{4,32,128,512} , entropy ceiling ln(598)≈6.39 nats), constructed as in Figure 17 with min_p in place of the temperature.
Figure 45: β∗ against D on Hi-ToM with min_p in place of the temperature, drawn as in Figure 30 , for K∈{4,32,128,256} for Qwen3-0.6B and K∈{4,32,128,512} for Mistral 7B Instruct. Each panel shows D up to the last grid value before any of its curves exceeds 106 : 0.8 , 1.0 , and 0.6 in (a), (b), and (c). Hardness is measured at each question’s best value, and pass@ K is the unbiased estimator of the sample counts. For Qwen3-0.6B at τ=2.0 and a 10% loss, no value loses 10% at K=256 , so C is empty and the panel has no curve for it. Every other curve has a reachable loss at D=0 .
The quality of text generated by large language models depends critically on the decoding sampling strategy. While mainstream methods such as Top-k, Top-p, and Min-p achieve a balance between diversity and accuracy through probability-space truncation, they share an inherent limitation: extreme sensitivity to the temperature parameter. Recent logit-space approaches like Top-nσ achieve temperature invariance but rely on global statistics that are susceptible to long-tail noise, failing to capture fine-grained confidence structures among top candidates. We propose \textbf{Min-k Sampling}, a novel dynamic truncation strategy that analyzes the local shape of the sorted logit distribution to identify "semantic cliffs": sharp transitions from high-confidence core tokens to uncertain long-tail tokens. By computing a position-weighted relative decay rate, Min-k dynamically determines truncation boundaries at each generation step. We formally prove that Min-k achieves strict temperature invariance and empirically demonstrate its low sensitivity to hyperparameter choices. Experiments on multiple reasoning benchmarks, creative writing tasks, and human evaluation show that Min-k consistently improves text quality, maintaining robust performance even under extreme temperature settings where probability-based methods collapse. We make our code, models, and analysis tools publicly available.
Large language models generate text by sampling each token from a predicted distribution, and a temperature parameter sets how far the draw strays from the most probable tokens. Truncation samplers such as top-p and min-p discard the least probable tokens before the draw, so that sampling at high temperature stays coherent. Their reported accuracy gains come from temperatures of 1.5 to 3, while the defaults of deployed systems cluster between 0.6 and 1.0. Whether they change accuracy at those defaults, and for which models, has not been measured. We test thirteen open-weight models on GSM8K and MMLU-Pro at temperatures 0.7, 1.0, and 1.3 in one controlled pipeline, ten of them under eight decoding configurations. Six of the thirteen models lose 17 to 38 accuracy points on MMLU-Pro between 0.7 and 1.3, and the other seven lose at most 10. The lost accuracy comes from generations that run to the token limit or never state an answer. These results suggest that truncation samplers improve accuracy primarily when higher temperatures substantially degrade model performance. Where accuracy remains stable across temperatures, none of the tested truncation samplers improves on plain temperature sampling.
Repetitive, degenerate generation is a persistent failure of LLM decoding, and the standard remedies (temperature, nucleus, and top-k sampling) are static heuristics. By Kac's lemma and the AEP, a stationary source takes time exponential in its entropy to repeat a block. We therefore maximize the entropy of the sampling distributions under one cross-entropy budget that keeps the output plausible under the model. Offline, the solution is temperature sampling, with the inverse temperature as the budget's Lagrange multiplier. Our contribution is the online problem: the conditionals p1,p2,… are revealed one at a time, and the sampling distribution qt must be chosen causally from p1,…,pt while the budget couples the entire horizon. The optimal temperature is thus a global shadow price that must be estimated from a prefix. We propose Entropic Online Sampling (EOS), which re-solves for the temperature on each prefix. We prove it loses O(Rlogn) entropy against the offline optimum while overshooting the budget by O(Rlogn), with R an instance-dependent drift parameter. A budget-tightened variant is exactly feasible at the same rate. Conversely, every exactly feasible online algorithm must lose Ω(R), so EOS is optimal up to a logn factor. When the conditionals are drawn from a finite mixture, R=O~(n). Experiments confirm that EOS tracks the offline temperature and beats fixed-temperature sampling in entropy.