cs.AIAug 2, 2026

Perspectives on Tsallis Statistics for Artificial Intelligence

Authors: Kleyton da Costa, Bernardo Modenesi

Organizations: University College London, London, United Kingdom · Division of Biostatistics & School of Computing, University of Utah, United States

Abstract

Tsallis statistics generalizes Boltzmann-Gibbs statistical mechanics through a single real parameter qq that controls the weight assigned to rare and frequent events. Originally proposed to describe physical systems with long-range correlations, multifractal geometry, and heavy-tailed fluctuations, the framework has become a recurring ingredient in modern artificial intelligence (AI): it underlies sparse attention mechanisms (\textsc{sparsemax} and αα-\textsc{entmax}), maximum-entropy reinforcement learning with controllable exploration, robust and heavy-tailed probabilistic models, and a family of generalized loss functions and regularizers. This paper offers a structured perspective on where Tsallis statistics meets AI. We first review the mathematical core: qq-entropy and its variational (maximum-entropy) foundation, the qq-exponential and qq-logarithm, the qq-central limit theorem, qq-Gaussian distributions, and their dynamical origin in superstatistics, emphasizing the properties that matter for machine learning. We then survey applications across softmax generalization, reinforcement learning, sequential and graph neural models, generative and probabilistic modeling, loss design, and optimization, extracting the recurring design pattern in each case: a tunable interpolation between dense/uniform and sparse/peaked behavior governed by qq. We further argue that the heavy-tailed weight spectra and gradient-noise statistics empirically observed in deep networks are themselves nonextensive signatures, placing modern learning dynamics within the scope of qq-statistics. Finally, we discuss methodological pitfalls, the relationship to information geometry and qq-exponential families, and open directions, arguing that qq should be treated as a learnable inductive bias rather than a fixed hyperparameter.

Explore similar work

CardsList
Jun 18, 2026stat.ML

Statistical Properties of Training & Generalization

Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.
Itay Lavie, Noam Levi, Yonatan Kahn
Scaling LawsArchitectural Inductive BiasesPhysics-Informed Deep LearningInductive BiasStatisticsScaling
AskPDF
Apr 28, 2026cs.LG

How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

SFT-then-RLVR is widely used for post-training reasoning models, but why this specific ordering, and why RLVR-only stalls at cold start, have lacked a unifying theoretical account. We provide that account under a unified loss family JQJ_Q using the Tsallis qq-logarithm. JQJ_Q is a single-parameter family that interpolates between RLVR (at q=0q{=}0, the \textit{exploitation pole}) and the log-marginal-likelihood over latent trajectories (at q=1q{=}1, the \textit{density-estimation pole}), under which the standard pipeline corresponds to a stepwise q=1→0q{=}1 \to 0 schedule. All members share the same per-example gradient direction, differing only by a per-instance amplification Pθ−qP_θ^{-q} that reweights each instance independently of the learning rate. Under gradient flow analysis, we show that the exploitation pole requires Ω(1p0)Ω(\frac{1}{p_0}) time to escape cold start but is robust to label noise, while the density-estimation pole escapes in Θ(log⁡(1p0))Θ\big(\log(\frac{1}{p_0})\big) but memorizes label noise. This separation explains how SFT (q=1q{=}1) first moves the model out of the cold-start regime, followed by the more robust RLVR (q=0q{=}0), under the SFT-then-RLVR paradigm. We further derive two Monte Carlo estimators that directly optimize fixed-qq on the JQJ_Q continuum, without annotated rationales: Gradient-Amplified RL (GARL) and Posterior-Attenuated Fine-Tuning (PAFT), with shared bias O(qMPθq)O\big(\frac{q}{M P_θ^q}\big) but different variance and stability properties. On FinQA, HotPotQA, and MuSiQue, GARL at sufficiently high qq substantially mitigates cold-start stalling, escaping cold start where GRPO fails entirely. In warm start, GARL at low qq dominates FinQA where training is stable; on HotPotQA and MuSiQue, GARL destabilizes and PAFT at q=0.75q{=}0.75 remains stable, reaching 47.947.9 \texttt{m@16} on HotPotQA (+13.9+13.9 over GRPO).
Chu-Cheng Lin, Eugene Ie
Supervised Fine-TuningLarge Reasoning ModelsCold-StartLearning RatesRandomized Global-Local Density EstimationCommits
AskPDF
Aug 2, 2026cs.LG

Statistical Mechanics of Learning on Product Wasserstein Manifolds

Normally the statistical mechanics of learning treats constraints on weight distributions as restrictions that shrink the space of possible solutions. Therefore, it reduces model capacity. In this paper we would like to take a contrary approach, which, however, is based on the earlier work on distribution-constrained perceptrons. Rather than treating a prescribed weight distribution as a mere restriction, we propose that it defines the intrinsic geometry upon which learning naturally unfolds. We formulate both deep neural networks and variational quantum circuits as gradient flows on a product of Wasserstein manifolds -- one classical Wasserstein space for each layer and one quantum Wasserstein space for the circuit parameters. Within this geometry, the capacity reduction, which was previously associated with distributional constraints, appears as the metric structure of the constraint manifold itself. We develop a hierarchical mean-field description for deep networks, extend the framework to the quantum setting using the quantum Wasserstein distance of order 1, and introduce two such practical algorithms, Hierarchical DisCo-SGD and Quantum DisCo, that follow approximate geodesics on the manifold of the product itself. Experiments on teacher-student problems, standard image classification tasks, and small variational quantum classifiers show that respecting these distributional geometries improves generalization, stabilizes training, and reduces the severity of barren plateaus compared with unconstrained and purely norm-based baselines. This approach firstly reframes structural constraints as geometric priors and suggests a route for incorporating biological, spectral, or hardware-derived distributional information into both learning systems, viz., classical and quantum learning.
Srinivasa Rao P Vangmayi P Reddy
Quantum LearningWasserstein DistanceDistributional LearningMean-Field LimitMultilayer PerceptronsProducts
AskPDF