Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)
Organizations: Thoughtworks · Apart Research · New York University & Meta · Columbia University ·
Abstract
Open-weight language models are widely observed to get worse the longer they generate. Often, attempts to fix this involve more training. At high temperature, The sampler, i.e. the rule that picks the next token out of the model's probabilities, decides whether long and agentic generation survives and/or thrives. We test 29 open-weight models from 1B to 122B parameters on ten established benchmarks plus needle retrieval out to one million tokens, covering SWE-bench Verified coding agents, long chain-of-thought reasoning, long-form writing, long structured output and long-context reading. Samplers that keep their pool of candidate tokens small and steady as temperature rises don't lose accuracy. Samplers that let in low-probability tokens fall apart. At temperature 2, coding agents running top-p, top-k or Top-H resolve 0 of 100 GitHub issues, while the same agents running top-nsigma resolve 46 to 62 and win every issue the two samplers disagree on, on three Qwen3.5 models. Over all benchmarks top-nsigma keeps a median 100% of its temperature-1 score where top-p keeps 26%. Counting how many tokens each sampler lets through explains why. In one Llama-3.1-8B story, going from temperature 1 to 2 grows top-p's pool from 4 tokens to 51,845 while top-nsigma stays at 3. Across 133 combinations of sampler setting and temperature, the typical pool size predicts judged quality (Spearman correlation of -0.77), and when we pick the pool at temperature 1 and only apply temperature afterwards, Top-H, min-p and p-less stop collapsing. At the default temperature of 1 most samplers perform about the same, We release all code and data to the public.
Figures & tables
| Median tokens kept per step | |||
|---|---|---|---|
| Sampler | Keeps the tokens that… | temp. 1 | temp. 2 |
| Pure sampling | (all tokens) | all | all |
| Top- 50 | are among the 50 most likely | 51 | 51 |
| Top- 0.9 | together cover 90% of the probability | 2 | 90,000 |
| Top-H 0.4 | keep the set’s entropy under 40% of the full entropy | 1 | 54 |
| Min- 0.1 | have at least 10% of the top token’s probability | 1 | 14 |
| Sampler | Resolved | W/L vs top- | Resolved | W/L vs top- |
|---|---|---|---|---|
| Pure sampling | 46% | 6/6 | 0% | 0/0 |
| Top- 0.9 | 46% | base | 0% | base |
| Min- 0.1 | 46% | 7/7 | 48% | 24/0 ∗ |
| Top-H 0.4 | 56% | 9/4 | 18% | 9/0 ∗ |
| P-less | 48% | 6/5 | 58% | 29/0 ∗ |
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Sampler | Resolved | W/L vs top- | Resolved | W/L vs top- | Resolved | W/L vs top- |
|---|---|---|---|---|---|---|
| Pure sampling | 34% | 7/9 | 0% | 0/1 | 0% | 0/0 |
| Top- 50 | 37% | 10/9 | 3% | 3/1 | 0% | 0/0 |
| Top- 0.9 | 36% | base | 1% | base | 0% | base |
| Min- 0.1 | 35% | 12/13 | 50% | 49/0 ∗ | 16% | 16/0 ∗ |
| Top-H 0.4 | 33% | 10/13 | 25% | 24/0 ∗ | 0% | 0/0 |
| Sampler | Resolved | W/L vs top- | Resolved | W/L vs top- | Resolved | W/L vs top- |
|---|---|---|---|---|---|---|
| Pure sampling | 72% | 6/3 | 0% | 0/15 ∗ | 0% | 0/0 |
| Top- 50 | 60% | 1/4 | 14% | 0/8 ∗ | 0% | 0/0 |
| Top- 0.9 | 66% | base | 30% | base | 0% | base |
| Min- 0.1 | 70% | 5/3 | 68% | 20/1 ∗ | 50% | 25/0 ∗ |
| Top-H 0.4 | 68% | 5/4 | 56% | 15/2 ∗ | 0% | 0/0 |
| Sampler | Resolved | W/L vs top- | Resolved | W/L vs top- |
|---|---|---|---|---|
| Top- 0.9 | 47% | base | 0% | base |
| Min- 0.1 | 44% | 13/16 | 18% | 18/0 ∗ |
| Top-H 0.4 | 48% | 9/8 | 0% | 0/0 |
| Top- 1 | 46% | 10/11 | 46% | 46/0 ∗ |
| Sampler | Resolved | W/L vs top- | Resolved | W/L vs top- | Resolved | W/L vs top- |
|---|---|---|---|---|---|---|
| Pure sampling | 30% | 5/9 | 0% | 0/1 | – | – |
| Top- 50 | 30% | 3/7 | 2% | 1/1 | – | – |
| Top- 0.9 | 38% | base | 2% | base | 0% | base |
| Min- 0.1 | 36% | 5/6 | 42% | 20/0 ∗ | – | – |
| Top-H 0.4 | 36% | 3/4 | 24% | 12/1 ∗ | – | – |
| Sampler | Score | W/L | Score | W/L | Score | W/L |
|---|---|---|---|---|---|---|
| Top- 0.95 | 42.4 | 37.5 | 27.4 | |||
| Top-H 0.4 | 33.2 | 50–186 ∗ | 38.6 | 127–111 | 40.8 | 228–30 ∗ |
| Min- 0.1 | 41.7 | 102–113 | 41.8 | 137–74 ∗ | 36.0 | 157–30 ∗ |
| P-less | 31.2 | 59–225 ∗ | 33.1 | 101–166 ∗ | 37.5 | 216–67 ∗ |
| Top- 1 | 41.2 | 100–118 | 39.6 | 118–87 ∗ | 40.2 | 219–29 ∗ |
| Sampler | Score | W/L | Score | W/L | Score | W/L |
|---|---|---|---|---|---|---|
| Top- 0.95 | 57.2 | 57.1 | 39.2 | |||
| Top-H 0.4 | 44.6 | 71–267 ∗ | 51.8 | 89–171 ∗ | 55.3 | 324–73 ∗ |
| Min- 0.1 | 55.3 | 110–139 | 57.7 | 113–104 | 54.2 | 287–54 ∗ |
| P-less | 43.6 | 63–274 ∗ | 46.4 | 78–244 ∗ | 51.2 | 286–100 ∗ |
| Top- 1 | 57.4 | 125–122 | 58.2 | 116–99 | 57.9 | 326–36 ∗ |
| Sampler | Score | W/L | Score | W/L | Score | W/L |
|---|---|---|---|---|---|---|
| Top- 0.9 | 31.1 | 29.1 | 13.4 | |||
| Top- 50 | 28.7 | 202–277 ∗ | 24.3 | 186–331 ∗ | 7.6 | 99–233 ∗ |
| Top-H 0.4 | 31.0 | 211–200 | 31.0 | 268–205 ∗ | 14.7 | 209–169 ∗ |
| Min- 0.1 | 30.3 | 222–253 | 28.5 | 246–240 | 23.9 | 399–100 ∗ |
| P-less | 31.0 | 215–203 | 31.2 | 274–193 ∗ | 30.8 | 532–73 ∗ |
| Sampler | Score | W/L | Score | W/L | Score | W/L |
|---|---|---|---|---|---|---|
| Top- 0.9 | 38.8 | 36.6 | 25.6 | |||
| Top- 50 | 38.3 | 298–314 | 36.9 | 336–329 | 29.6 | 442–322 ∗ |
| Top-H 0.4 | 39.2 | 295–284 | 39.1 | 346–272 ∗ | 35.4 | 557–261 ∗ |
| Min- 0.1 | 38.7 | 287–292 | 39.2 | 365–288 ∗ | 36.0 | 579–265 ∗ |
| P-less | 39.9 | 312–280 | 39.9 | 366–268 ∗ | 39.8 | 667–239 ∗ |
| Sampler | Score | W/L | Score | W/L | Score | W/L |
|---|---|---|---|---|---|---|
| Top- 0.9 | 92.9 | 91.4 | 82.3 | |||
| Top- 50 | 92.4 | 43–61 | 91.8 | 88–76 | 85.2 | 290–186 ∗ |
| Top-H 0.4 | 93.3 | 40–25 | 92.5 | 78–41 ∗ | 84.6 | 248–166 ∗ |
| Min- 0.1 | 92.9 | 38–39 | 92.1 | 75–50 ∗ | 91.1 | 385–70 ∗ |
| P-less | 93.6 | 47–23 ∗ | 93.5 | 90–15 ∗ | 92.6 | 402–35 ∗ |
| Sampler | Loop | First | Last | Loop | First | Last | Loop | First | Last |
|---|---|---|---|---|---|---|---|---|---|
| Pure sampling | 0.41 | 4.0 | 2.5 | 0.03 | 1.1 | 1.0 | 0.00 | 1.0 | 1.0 |
| Top- 50 | 0.56 | 5.2 | 2.7 | 0.00 | 3.2 | 1.5 | 0.00 | 2.0 | 1.2 |
| Top- 0.9 | 0.95 | 5.2 | 2.5 | 0.05 | 1.5 | 1.2 | 0.00 | 1.0 | 1.0 |
| Min- 0.1 | 0.97 | 5.2 | 2.5 | 0.64 | 5.3 | 2.4 | 0.09 | 3.8 | 1.6 |
| Top-H 0.4 | 0.97 | 4.8 | 2.6 | 0.20 | 4.5 | 2.2 | 0.00 | 1.8 | 1.1 |
| Sampler | pass@1 | W/L | pass@1 | W/L | pass@1 | W/L |
|---|---|---|---|---|---|---|
| Top- 0.95 | 62.0 | base | 60.9 | base | 50.6 | base |
| Top- 1 | 63.4 | 14/9 | 63.7 | 22/12 | 63.4 | 53/8 ∗ |
| Min- 0.1 | 62.6 | 14/12 | 62.9 | 21/14 | 63.1 | 50/6 ∗ |
| P-less | 62.0 | 14/14 | 62.6 | 22/16 | 62.6 | 49/7 ∗ |
| Top-H 0.4 | 61.7 | 15/16 | 62.6 | 22/16 | 62.9 | 50/7 ∗ |
| Sampler | pass@1 | W/L | pass@1 | W/L | pass@1 | W/L |
|---|---|---|---|---|---|---|
| Top- 0.95 | 52.0 | base | 49.1 | base | 46.3 | base |
| Top- 1 | 52.9 | 20/17 | 51.4 | 18/10 | 50.3 | 24/10 ∗ |
| Min- 0.1 | 49.4 | 16/25 | 51.1 | 15/8 | 48.6 | 21/13 |
| P-less | 46.3 | 14/34 ∗ | 45.4 | 12/25 ∗ | 49.7 | 24/12 |
| Top-H 0.4 | 46.9 | 16/34 ∗ | 48.6 | 22/24 | 49.7 | 26/14 |
| Sampler | Singleton at | |||
|---|---|---|---|---|
| Pure sampling | all | all | all | 0.00 |
| Top- 50 | 51 | 51 | 51 | 0.00 |
| Top- 0.9 | 2 | 41,508 | 89,581 | 0.46 |
| Min- 0.1 | 1 | 3 | 14 | 0.59 |
| Top-H 0.4 | 1 | 10 | 54 | 0.62 |
| P-less | 1 | 1 | 21 | 0.93 |
| Temperature first | Candidate set fixed at | ||||||
| Sampler | |||||||
| Top- 0.9 | 4.7 | 1.0 | 1.0 | 1.0 | 3.4 | 2.7 | 2.6 |
| Top- 50 | 4.8 | 1.9 | 1.4 | 1.3 | 2.0 | 1.4 | 1.2 |
| Top-H 0.4 | 4.2 | 2.5 | 1.3 | 1.2 | 4.2 | 4.5 | 4.5 |
| Min- 0.1 | 4.6 | 4.4 | 2.6 | 1.1 | 4.5 | 4.6 | 4.6 |
| P-less | 4.2 | 4.6 | 2.9 | 1.1 | 4.2 | 4.2 | 4.1 |
| Arena-Write healthy | Arena-Write last window | LongBench-Write | WritingBench | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sampler | ||||||||||||
| Pure sampling | 0.78 | 0.14 | 0.00 | 6.4 | 2.0 | 1.0 | 78.6 | 32.1 | 15.0 | 6.85 | 3.71 | 1.26 |
| Top- 50 | 0.93 | 0.39 | 0.15 | 6.8 | 4.5 | 2.7 | 79.8 | 67.4 | 43.1 | 6.94 | 6.48 | 5.00 |
| Top- 0.9 | 0.92 | 0.23 | 0.01 | 6.8 | 2.6 | 1.2 | 80.8 | 42.4 | 21.1 | 6.89 | 4.37 | 2.20 |
| Min- 0.1 | 0.92 | 0.95 | 0.69 | 6.8 | 6.9 | 5.8 | 80.3 | 81.3 | 74.9 | 6.88 | 6.97 | 6.67 |
| Top-H 0.4 | 0.90 | 0.72 | 0.10 | 6.6 | 5.9 | 2.3 | 78.8 | 76.6 | 37.8 | 6.84 | 6.81 | 4.35 |
| Sampler | Loop | First | Last | Loop | First | Last | Loop | First | Last |
|---|---|---|---|---|---|---|---|---|---|
| Pure sampling | 0.46 | 3.9 | 2.4 | 0.00 | 1.1 | 1.1 | 0.00 | 1.0 | 1.0 |
| Top- 50 | 0.55 | 5.3 | 2.5 | 0.01 | 3.2 | 1.6 | 0.00 | 1.9 | 1.1 |
| Top- 0.9 | 0.96 | 5.2 | 2.5 | 0.04 | 1.5 | 1.1 | 0.00 | 1.0 | 1.0 |
| Min- 0.1 | 0.97 | 5.1 | 2.5 | 0.66 | 5.3 | 2.6 | 0.10 | 3.9 | 1.5 |
| Top-H 0.4 | 1.00 | 5.0 | 2.4 | 0.21 | 4.4 | 2.0 | 0.00 | 1.8 | 1.1 |
| Sampler | Loop | First | Last | Loop | First | Last | Loop | First | Last |
|---|---|---|---|---|---|---|---|---|---|
| Pure sampling | 0.62 | 7.1 | 4.1 | 0.00 | 2.1 | 1.1 | 0.00 | 1.1 | 1.0 |
| Top- 50 | 0.50 | 7.0 | 4.1 | 0.10 | 6.7 | 2.7 | 0.00 | 3.4 | 1.5 |
| Top- 0.9 | 0.69 | 7.0 | 4.2 | 0.17 | 2.6 | 1.1 | 0.00 | 1.7 | 1.0 |
| Min- 0.1 | 0.56 | 7.3 | 4.3 | 0.54 | 7.4 | 4.2 | 0.21 | 7.2 | 4.0 |
| Top-H 0.4 | 0.81 | 6.7 | 3.7 | 0.40 | 7.3 | 4.2 | 0.00 | 2.8 | 1.5 |
| Rule | Stack | Loop | Quality | Loop | Quality | Loop | Quality |
|---|---|---|---|---|---|---|---|
| Top- 0.9 | none | 0.95 | 3.2 | 0.06 | 1.3 | 0.00 | 1.0 |
| + DRY | 0.38 | 3.4 | 0.00 | 1.2 | 0.00 | 1.0 | |
| + top-2 floor | 0.88 | 3.3 | 0.05 | 1.4 | 0.00 | 1.0 | |
| Min- 0.1 | none | 0.97 | 3.1 | 0.64 | 3.3 | 0.11 | 2.4 |
| + DRY | 0.34 | 3.2 | 0.22 | 3.5 | 0.05 | 2.4 | |
| Context depth | 2K | 16K | 64K | 128K |
| Quality gap, | -0.03 | +0.08 | -0.37 | -0.08 |
| Quality gap, | +1.21 | +1.43 | +1.21 | +1.41 |
| Quality gap, | +1.50 | +1.51 | +1.58 | +1.48 |
| Entropy at , gemma4-12b | 0.12 | 0.10 | 0.19 | – |
| Entropy at , llama3.1-8b | 2.91 | 2.42 | 1.66 | – |
| Entropy at , nemotron3-30b-a3b | 1.18 | 1.11 | 1.08 | 1.32 |
| Sampler | Loop | First | Last | Loop | First | Last | Loop | First | Last |
|---|---|---|---|---|---|---|---|---|---|
| Pure sampling | 0.04 | 4.2 | 4.4 | 0.03 | 1.0 | 1.2 | 0.00 | 1.0 | 1.0 |
| Top- 50 | 0.21 | 5.7 | 4.5 | 0.00 | 3.6 | 2.7 | 0.00 | 2.1 | 1.6 |
| Top- 0.9 | 0.50 | 5.6 | 4.2 | 0.01 | 1.5 | 1.4 | 0.00 | 1.0 | 1.0 |
| Min- 0.1 | 0.64 | 5.6 | 4.0 | 0.14 | 5.8 | 4.5 | 0.00 | 4.1 | 3.9 |
| Top-H 0.4 | 0.75 | 5.5 | 3.8 | 0.11 | 4.8 | 4.0 | 0.01 | 1.7 | 1.6 |
| Sampler | Loop | First | Last | Loop | First | Last |
|---|---|---|---|---|---|---|
| Pure sampling | 0.56 | 5.4 | 3.4 | 0.00 | 2.5 | 1.1 |
| Top- 50 | 0.69 | 6.5 | 3.3 | 0.06 | 4.4 | 1.4 |
| Top- 0.9 | 0.83 | 6.3 | 3.3 | 0.21 | 3.1 | 1.4 |
| Min- 0.1 | 0.88 | 6.3 | 3.6 | 0.69 | 6.5 | 3.4 |
| Top-H 0.4 | 0.90 | 6.3 | 3.4 | 0.31 | 5.6 | 2.6 |
Language Model Decoding
Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.
See all 182 papers