Organizations: School of Computer and Information Engineering, Henan University · Department of Statistics, LMU Munich · Munich Center for Machine Learning (MCML)
The quality of text generated by large language models depends critically on the decoding sampling strategy. While mainstream methods such as Top-k, Top-p, and Min-p achieve a balance between diversity and accuracy through probability-space truncation, they share an inherent limitation: extreme sensitivity to the temperature parameter. Recent logit-space approaches like Top-nσ achieve temperature invariance but rely on global statistics that are susceptible to long-tail noise, failing to capture fine-grained confidence structures among top candidates. We propose \textbf{Min-k Sampling}, a novel dynamic truncation strategy that analyzes the local shape of the sorted logit distribution to identify "semantic cliffs": sharp transitions from high-confidence core tokens to uncertain long-tail tokens. By computing a position-weighted relative decay rate, Min-k dynamically determines truncation boundaries at each generation step. We formally prove that Min-k achieves strict temperature invariance and empirically demonstrate its low sensitivity to hyperparameter choices. Experiments on multiple reasoning benchmarks, creative writing tasks, and human evaluation show that Min-k consistently improves text quality, maintaining robust performance even under extreme temperature settings where probability-based methods collapse. We make our code, models, and analysis tools publicly available.
Figures & tables
Figure 1: Comparison of decoding methods under different temperature and confidence conditions. See Appendix A for detailed token information. Top: High-confidence case, where Min- k accurately selects correct tokens and remains temperature-invariant. Bottom: Low-confidence case, where Min- k identifies semantic boundaries while reducing noise tokens.
i∗←argmaxii⋅(max(l)−min(l))li−li+1
Algorithm 1 Min- k Sampling
Dataset
Model
LLaMA3-8B-Instruct
Qwen3-4B-Instruct
T
1.0
2.0
3.0
4.0
5.0
10.0
1.0
2.0
3.0
4.0
5.0
10.0
AQuA
Top- k
46.85
40.94
17.72
8.66
6.69
0.00
75.20
77.56
71.65
62.20
44.09
0.00
Top- p
49.61
31.10
12.99
3.54
1.18
0.00
77.95
77.17
70.87
52.76
19.29
0.00
Min- p
50.00
32.68
24.02
7.48
7.08
0.00
77.56
76.77
74.41
69.69
55.12
0.00
Top- nσ
48.03
47.24
46.06
46.06
44.49
49.61
78.35
77.17
77.17
78.35
77.17
77.17
Min- k
50.00
48.03
44.49
52.76
50.39
46.06
79.13
79.13
79.13
79.92
78.35
79.13
Table 1: Exact Match (%) for different temperatures (1.0–10.0) and sampling strategies on AQuA, GPQA-main, GSM8K and MATH500 using the LLaMA-3-8B-Instruct and Qwen3-4B-Instruct models. The detailed results for LLaMA-3-70B-Instruct and Qwen3-30B-Instruct are provided in Appendix G . Best results in bold .
Method
LLaMA3-8B-Instruct
Qwen3-4B-Instruct
T=1.0
T=3.0
T=1.0
T=3.0
Top- k
50.80
1.40
50.00
13.80
Top- p
49.60
0.00
54.60
0.40
Mirostat
48.20
1.40
48.40
18.00
η -sampling
52.60
0.00
49.80
0.20
Min- p
53.40
51.60
49.40
46.60
Table 2: Win rates (%) on creative writing for both models. Best results in bold .
Method
LLaMA3-8B-Instruct
Qwen3-4B-Instruct
Total (%)
Top- nσ
33
34
67 (33.5)
Tie
26
32
58 (29.0)
Min- k
41
34
75 (37.5)
Total
100
100
200 (100)
Table 3: Human evaluation outcomes for Min- k and Top- nσ , stratified by model. Best results in bold .
Figure 2: Sensitivity analysis of Min- k with respect to τ on GSM8K. The heatmap shows accuracy (%) under different combinations of τ and temperature. The method maintains highly stable performance ( ∼ 74-79%) across the entire tested parameter space, demonstrating strong robustness without a clear degradation threshold even at very high temperatures.
Figure 3: Semantic Noise Rate Comparison (Log Scale). We evaluate the ratio of incoherent tokens generated by different methods across temperatures ranging from 1.0 to 10.0 on GSM8K (128 samples). While traditional methods collapse (noise rate → 1.0) as temperature rises, Min- k remains remarkably stable, consistently filtering out long-tail noise without manual tuning.
Method
Tokens/s
Latency (ms)
Slowdown
Greedy
39.33
25.43
0.00%
Top- k
38.28
26.13
2.67%
Top- nσ
38.22
26.16
2.81%
Min- p
38.10
26.24
3.11%
Top- p
37.97
26.34
3.46%
Min- k
37.86
26.41
3.73%
Table 4: Latency benchmark on LLaMA-3-8B-Instruct (single A100). Relative slowdown is computed against greedy decoding.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Detailed probabilities and logits of the model’s top-13 candidate tokens under the high-confidence prompt, at temperatures T=1.0 and T=5.0 .
Figure 5: Detailed probabilities and logits of the model’s top-13 candidate tokens under the low-confidence prompt, at temperatures T=1.0 and T=5.0 .
Figure 6: Visualization of candidate set size k across decoding steps under different temperature settings. Left: T=1.0 . Right: T=5.0 . The distributions are nearly identical, indicating strong temperature invariance.
Analysis of Component Contributions
Full-Min- k
76.15
Complete methodical, and balances all factors.
w/o Weight (1/ i )
64.14
Validate the necessity of Head-bias. Removing it increases k and introduces more noise.
w/o Range Norm ( Rl )
41.02
Validate Temperature Invariance. Performance drops significantly at high T when removed.
w/o Fallback
76.15
Validate Robustness. Prevents degradation under extremely flat distributions caused by k=1 .
Appendix
Table 5: Ablation study of Min- k design components on the GSM8K dataset ( T=4.0 ). We compare the proposed Full Min- k against variants without position weighting ( 1/i ), range normalization ( Rl ), and the fallback mechanism. The results demonstrate that range normalization is fundamental for temperature invariance (as seen in the collapse of the "w/o Range Norm" variant), while position weighting effectively filters tail noise.
Decay Function
GSM8K Accuracy (%)
Creative Writing Win Rate (%)
T=1.0
T=4.0
T=1.0
T=4.0
1/i0 (No Weighting)
75.97
64.14
52.80
11.00
1/i0.5 (Sqrt)
77.41
71.57
49.00
49.20
1/i1 (Linear, Ours)
77.39
76.15
58.60
53.20
1/i2 (Square)
78.54
76.04
50.20
50.40
log(i)
74.07
72.71
51.60
48.20
Appendix
Table 6: Impact of different decay functions on reasoning (GSM8K) and creative writing (AlpacaEval) tasks. While aggressive decay ( 1/i2 ) slightly favors strict reasoning by filtering tail noise, it harms generation quality. The proposed linear decay ( 1/i ) achieves the best overall generalization across domains and temperatures.
Dataset
Model
LLaMA3-70B-Instruct
Qwen3-30B-Instruct
T
1.0
2.0
3.0
4.0
5.0
10.0
1.0
2.0
3.0
4.0
5.0
10.0
AQuA
Top- k
73.23
70.87
54.33
27.56
15.75
3.54
54.55
50.26
42.62
19.23
4.23
1.24
Top- p
74.02
68.90
35.83
10.24
4.33
0.00
57.02
55.26
37.95
5.83
1.79
0.00
Min- p
74.41
70.87
65.35
42.52
16.54
0.79
59.67
56.84
49.57
29.63
10.35
0.68
Top- nσ
74.80
74.80
70.47
71.65
72.83
72.83
61.26
60.83
59.87
59.23
60.22
60.12
Min- k
73.23
73.62
72.83
72.83
72.05
73.62
59.46
59.06
60.24
60.63
60.79
59.84
Appendix
Table 7: Exact Match (%) for different temperatures (1.0–10.0) and sampling strategies on AQuA, GPQA-main, GSM8K and MATH500 using the LLaMA-70B-Instruct and Qwen3-30B-Instruct. Best results in bold .