Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes

Deepon Halder; Raj Dabre

cs.CLMar 15, 2026

Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes

Authors: Deepon Halder, Raj Dabre

Organizations: AI4Bharat · IIEST Shibpur · IIT Madras

Abstract

Probabilistic language generators are theoretically modeled as discrete stochastic processes, yet standard decoding strategies (Top-k, Top-p) impose static truncation rules that fail to accommodate the dynamic information density of natural language. This misalignment often forces a suboptimal trade-off: static bounds are either too restrictive for high-entropy creative generation or too permissive for low-entropy logical reasoning. In this work, we formalize the generation process as a trajectory through a relative probability manifold. We introduce Top-b (Adaptive Relative Band Sampling), a decoding strategy that regulates the candidate set via a dynamic bandwidth coefficient coupled strictly to the instantaneous Shannon entropy of the model's distribution. We provide a theoretical framework demonstrating that Top-b acts as a variance-minimizing operator on the tail distribution. Empirical validation on GPQA and GSM8K benchmarks indicates that Top-b significantly reduces generation entropy and inter-decoding variance while maintaining competitive reasoning accuracy, effectively approximating a self-regulating control system for autoregressive generation.

Figures & tables

Figure 1: Structural comparison of static cumulative truncation (Top- $p$ ) versus entropy-regulated relative thresholding (Top- $b$ ). (a) In low-entropy reasoning regimes, Top- $p$ admits a long tail of low-probability distractor tokens, increasing the risk of logical incoherence. (b) In high-entropy creative regimes, Top- $p$ ’s static cumulative threshold arbitrarily truncates viable tokens, artificially restricting diversity. In contrast, Top- $b$ establishes a dynamic probability band anchored to the distribution mode ( $p_{\max}$ ). (c) Under low entropy, the Top- $b$ bandwidth strictly contracts to prune the distractor tail and enforce deterministic reasoning. (d) Under high entropy, the bandwidth expands to safely retain linguistic diversity. This illustrates how Top- $b$ continuously adapts its sampling support to the local information density of the language process.

Method	Accuracy	Avg_Entropy
Top-b (Ours)	0.2285	0.1579
Eta	0.2099	0.1592
Min-p	0.2199	0.1714
Epsilon	0.2389	0.1712
Top-p	0.2132	0.1764
Temperature	0.2121	0.1774

Table 1: GPQA accuracy and average entropy. Top-b minimizes average entropy, indicating a reduction in generation stochasticity and confusion.

Figure 2: Accuracy variance across random seeds on GPQA. Top-b exhibits the lowest variance, indicating higher deterministic stability compared to Top-p and stochastic sampling methods.

Figure 3: Entropy trajectory over generation steps for Top-b and Top-p sampling. Top-b induces a monotonic reduction in entropy, while Top-p maintains higher entropy in later stages due to broader candidate sets.

Figure 4: Interaction between bandwidth parameter $b$ and temperature $T$ on GPQA performance. Mid-range $b$ values regularize high-temperature sampling, improving accuracy without inducing mode collapse.

Method	Average Accuracy	Variance
Top- $b$ (Ours)	0.2680	$1.136\times 10^{-4}$
Top- $p$	0.2613	$1.1207\times 10^{-3}$
Top- $k$	0.2093	$3.797\times 10^{-4}$
Min- $p$	0.2427	$2.380\times 10^{-4}$

Table 2: Preliminary results on GSM8K demonstrating Top-b’s superior accuracy and variance reduction.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Schematic of entropy-induced branching. Top-b acts as a pruning operator, collapsing diffuse branches into a single high-likelihood continuation.

Prompt Context	Top Tokens	Orig. Prob (%)	Top- $b$	Top- $p$	Top- $k$	Min- $p$
Prompt: ”2+2=”
Support Size ( $\|\mathcal{S}\|$ )			1 token	14 tokens	5 tokens	9 tokens
	4	39.75	100.0	44.2	54.8	46.6
	5	15.50	-	17.2	21.5	18.3
	\n	7.81	-	8.7	10.8	9.2
Prompt: ”A rainbow is an optically brilliant meteorological event resulting from refraction, reflection, and dispersion of”

Appendix

Table 3: Comparison of selected support sizes (

|\mathcal{S}|

) and renormalized probabilities across different contexts. Top-

b

consistently isolates the high-probability manifold, severely restricting the distractor tail compared to Top-

p

Autoregressive Decoding

Autoregressive decoding generates outputs step by step, conditioning each token on earlier ones while balancing quality, control, and inference cost.

See all 55 papers

Entropy-informed Decoding: Adaptive Information-Driven Branching
May 10, 2026Benjamin Patrick Evans, Sumitra Ganesh, Leo ArdonLanguage Model Decoding LLM Inference Acceleration
JP Morgan AI Research, London, UK · JP Morgan AI Research, New York, USA
XTC: Head-Aware Sampling by Excluding Top Choices
Aug 24, 2026Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder +2Language Model Decoding Diversity-Aware Sampling
Independent Researcher · Thoughtworks · New York University
Min- $k$ Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics
Apr 13, 2026Yuanhao Ding, Meimingwei Li, Esteban Garces Arias +3Language Model Decoding Adaptive Inference
School of Computer and Information Engineering, Henan University · Department of Statistics, LMU Munich · Munich Center for Machine Learning (MCML)

May 10, 2026J/K move · Enter open · S save

Entropy-informed Decoding: Adaptive Information-Driven Branching

Benjamin Patrick Evans, Sumitra Ganesh, Leo Ardon

JP Morgan AI Research, London, UK · JP Morgan AI Research, New York, USA

Language Model Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Large language models (LLMs) achieve remarkable generative performance, yet their output quality is dependent on the decoding strategy. While sampling-based methods (e.g., top-k, nucleus) and search-and-select based methods (e.g., beam search, best-of-n, majority voting) can improve upon greedy decoding, both approaches suffer from limitations: sampling generally commits to a single path, while search often expends excessive computation regardless of task complexity. To address these, we introduce Entropy-informed decoding (EDEN), a plug-and-play, model-agnostic decoding framework that adaptively allocates computation based on the model's own uncertainty, approximating higher-width beam search with fewer expansions. At each generation step, EDEN estimates the entropy of the output token distribution and adjusts the branching factor monotonically with the entropy, expanding more candidates in high-entropy regions and following a greedier path in low-entropy regions, improving token efficiency. Experiments across complex tasks, including mathematical reasoning, code generation, and scientific questions, demonstrate that EDEN consistently improves output quality over existing decoding strategies, achieving better accuracy-expansion trade-offs than fixed-width beam search. By treating next-token selection as a noisy maximisation problem, we prove that branching factors monotone in entropy are guaranteed to find better (i.e. more probable) continuations than any fixed branching factor within the same total expansion budget, and derive explicit regret rates characterising the benefit of the adaptive allocation.

Abstract

Autoregressive Decoding

Explore similar work

Entropy-informed Decoding: Adaptive Information-Driven Branching

XTC: Head-Aware Sampling by Excluding Top Choices

Min-kkk Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics

Min- $k$ Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics