Language Model Perplexity

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 30

All topics

Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2
Oct 1, 2026Yohan Chatelain, Pablo de Oliveira CastroFull-Precision Performance Language Model Perplexity
Krembil Centre for Neuroinformatics, CAMH, Toronto, Canada · Universite Paris-Saclay, UVSQ, LI-PaRAD, Versailles, France
Lasting Effects of Abstract Pretraining Beyond Perplexity
Sep 30, 2026Zachary Shinnick, Hemanth Saratchandran, Damien Teney +1Large Language Model Pretraining Pretraining
Australian Institute for Machine Learning (AIML), Adelaide University · Idiap Research Institute · Metacognition AI
JARQ: Joint Alternating Refinement for Quantization
Sep 29, 2026Xinyu Wang, Sicheng Lyu, Xiao-Wen ChangPost-Training Quantization Language Model Perplexity
McGill University
GroupMask: Layer-Adaptive Group-wise Sparsity for Semi-Structured LLM Pruning
Sep 27, 2026Zhengao Li, Shuoqiu Li, Xiaofang Zhang +7Unstructured Pruning Large Language Model Compression
Florida State University · University of Connecticut · College of William & Mary +1
Temporal Taxation Compounds Under Post-Training Compression of Whisper Models
Sep 23, 2026Srishti Ginjala, Eric Fosler-Lussier, Christopher W. Myers +1Whisper-Large-V3 Character Error Rate
The Ohio State University, Columbus, OH, USA · Air Force Research Laboratory, USA
Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression
Sep 14, 2026Huicheng Zhang, Xiyao Feng, Ze-Tong Li +6Large Language Model Compression Singular Value Decomposition
Hong Kong University of Science and Technology (Guangzhou), Guangdong 511453, China · QudeLeap Research, Shanghai 200030, China
Psychosis involves a deficit of information compression in connected speech
Sep 14, 2026Samuele Vallisa, Claudio Palominos, Rui He +6Linguistics Surprisal
Grammar and Cognition Lab, Department of Translation & Language Sciences, Universitat Pompeu Fabra, Barcelona, Spain · Department of Neurosciences, Health Sciences Institute, Dokuz Eylul University, Izmir, Turkey. · Department of Psychiatry, Dokuz Eylul University, Izmir, Turkey. +4
Parallelism Strategy Chaining for Fast Training Convergence
Sep 7, 2026Minchul Kang, Changyong Shin, Younghun Go +4Large Language Model Training Language Model Perplexity
Department of Computer Science and Engineering, Korea University, Seoul, South Korea
Language-Conditional Dequantization: Recovering What Quantization Steals from Non-English Languages
Aug 12, 2026Nirmal ThomasLarge Language Model Quantization Quantized
Prathama International
Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents
Aug 10, 2026Bingzhen Liu, Xiaomeng Fan, Yuwei Wu +4Self-Evolution Large Language Model Agents
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University · Alaya Lab
CurveFP: Co-Designing Numerical Representation and Product Arithmetic for Language Models
Aug 8, 2026Ye QiaoFull-Precision Performance Task Arithmetic
Department of Electrical Engineering and Computer Science University of California, Irvine Irvine, CA, USA
Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination
Aug 7, 2026Ruijie Hou, Yueyang Jiao, Zhao Wang +1Mitigation Language Model Perplexity
Zhejiang University
When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruning
Aug 3, 2026Andrew ZhangData Compression Methods Language Model Perplexity
KTH Royal Institute of Technology
Surprisal Theory is Tautological (without Rational Grounding)
Jul 23, 2026Ryan CotterellSurprisal Linguistics
ETH Zürich
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Jul 23, 2026Ji Ho BaeRotary Position Embedding Positional Encoding
JRTI, Seoul, 05855, Republic of Korea
Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation
Jul 20, 2026Claudia Grosser, Maike Heuer, Denis Krompass +1Pretraining Heterogeneous Federated Learning
Technical University Munich · Siemens AG
Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls
Jul 9, 2026Peter HollowsLogit Lens Repetition
PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning
Jul 8, 2026Yazdan Jamshidi, Alexey ShvetsUnstructured Pruning Large Language Model Compression
Palo Alto Networks
Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition
Jul 6, 2026Mohammad Zeineldeen, Albert Zeyer, Haoran Zhang +3Automatic Speech Recognition Language Model Perplexity
AppTek.ai GmbH, Aachen, Germany · Machine Learning and Human Language Technology Group, Faculty of Computer Science, RWTH Aachen University Aachen, Germany
Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion LMs
Jul 1, 2026Shuai Zhang, Zijie Chen, Hongliang He +2Diffusion Language Models Language Model Perplexity
Zhejiang University · Westlake University · Ant Group
When transformers learn "impossible" languages, what do they learn?
Jun 29, 2026Ram Janarthan, Coleman Haley, Sharon GoldwaterLanguage Modeling Large Language Models Fail
University of Edinburgh
Why Do Accumulated Transformations Extrapolate?
Jun 23, 2026Mahesh GodavartiRotary Position Embedding Linear Attention
A Carrot, Inc.
Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment
Jun 17, 2026Miloš Nikolić, Ali Hadi Zadeh, Enrique Torres Sanchez +1Large Language Model Quantization Kullback-Leibler Divergence
ByteShape · University of Toronto · Vector Institute for Artificial Intelligence
Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics
Jun 7, 2026Antonio Franca, Alexander TongLanguage Model Perplexity Single-Token Output Distributions
AITHYRA Institute, Vienna, Austria.
Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference
Jun 1, 2026Chi-Wei Huang, Chia-Chi TsaiPost-Training Quantization Model Weights
National Cheng Kung University
The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling
May 22, 2026Al KariLanguage Modeling Language Model Perplexity
Manceps Inc.
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training
May 13, 2026Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1Low-Rank Structure Language Model Perplexity
University of Massachusetts Lowell
Most Current Model Organisms Are Leaky: Perplexity Differencing Often Reveals Finetuning Objectives
May 1, 2026Mohammed Abu Baker, Luca Baroni, Dan WilhelmLarge Language Model Safety Model Fine-Tuning
Meridian Visiting Researcher Programme.
Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning
Apr 27, 2026Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim +3Language Model Perplexity Layer-Wise
Neural Superintelligence Lab, MODULABS, Republic of Korea · University of Southern California, United States · Boston University, United States +2
Neural Continuous-Time Markov Chain: Discrete Diffusion via Decoupled Jump Timing and Direction
Apr 17, 2026Jingyuan Li, Xiaoyi Jiang, Fukang Wen +5Diffusion Models Language Model Perplexity
2Beijing Institute of Mathematical Sciences and Applications · 3Wuhan University · 4MathonAI +1

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2

Yohan Chatelain, Pablo de Oliveira Castro

Krembil Centre for Neuroinformatics, CAMH, Toronto, Canada · Universite Paris-Saclay, UVSQ, LI-PaRAD, Versailles, France

Full-Precision Performance Language Model Perplexity Transformer Architectures Multilayer Perceptrons Cross-Entropy Losses Emulators

Open paper AskPDF arXiv

Should low-precision transformer inference use stochastic rounding (SR) or round-to-nearest (RN)? The answer depends on where in the network you look. We isolate this effect by holding the numerical format fixed and varying only the rounding rule at individual operation sites. To enable experiments at freely chosen precisions, we extend the PRISM vectorized rounding library to arbitrary virtual precision via a variable-precision stochastic rounding (VPSR) algorithm, proving that the rounding decision is evaluated exactly in hardware floating point. We develop two analyses providing complementary insight into this site-level trade-off. First, a probabilistic forward-error bound for linear projections shows that SR's error envelope grows as

O(\sqrt{n} u)

in reduction length

n

, versus

O(n u)

for RN, a gap that widens rapidly at low precision and is most pronounced in the long multilayer perceptron (MLP) down-projection. Second, a second-order decomposition of expected cross-entropy loss change at the output softmax into signed drift, drift curvature, and a Fisher-weighted variance penalty reveals why the two sites behave oppositely: MLP noise is predominantly a uniform logit shift to which softmax is invariant, so SR's variance is largely discounted; head noise is non-uniform across the vocabulary and is not. On DistilGPT-2 at

t=6

significand bits, observations match theory: SR in the MLP raises perplexity to 1.15x the full-precision reference, versus 2.21x for RN. At the language-model head, the ordering reverses because SR introduces non-uniform variance, whereas deterministic RN carries none. In a mixed-precision configuration (MLP output at

t=6

), assigning SR to the MLP and RN to the head brings perplexity within 1.10x of the full-precision reference, a 28% reduction over matched-bit RN.

Momentum

Related topics

Latest papers 30