Language Modeling

Follow this topic Ask about this topic

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

525 papers73 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

Sampling from Flow Language Models via Marginal-Conditioned Bridges
May 13, 2026Iskander Azangulov, Leo ZhangFlow Matching Language Model Decoding
Department of Statistics, University of Oxford
Effective Context in Transformers: An Analysis of Fragmentation and Tokenization
May 13, 2026Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan TchamkertenLanguage Modeling Subword Tokenization
Department of Communications and Electronics Institut Polytechnique de Paris
N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation
May 13, 2026Aleksander Lorenc, Frédéric Berdoz, Joël Mathys +1Efficient Transformer Inference LLM Inference Acceleration
ETH Zurich
The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next
May 13, 2026Adil AminLLM Evaluation Forecasting Benchmarks
ZEHEN Labs
The Efficiency Gap in Byte Modeling
May 13, 2026Celine Lee, Jing Nathan Yan, Chen Liang +9Language Model Scaling Laws Autoregressive Language Modeling
1Google DeepMind · 2Work done while at Google DeepMind · Department of Computer Science, Cornell University.
In-Context Learning Operates as Concept Subspace Learning
May 12, 2026Wei Tang, Xinyan Jiang, Fakhri Karray +1Representation Learning Mechanistic Interpretability
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Shanghai Advanced Research Institute, Chinese Academy of Science
WriteSAE: Sparse Autoencoders for Recurrent State
May 12, 2026Jack YoungRecurrent Language Models Language Model Steering
Indiana University
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
May 12, 2026Rian Touchent, Eric de la ClergerieLanguage Model Pretraining Masked Language Modeling
Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris
A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuning
May 12, 2026Haibo Chen, Xin Wang, Jiaheng Chao +2LLM Alignment Graph Representation Learning
Tsinghua University
To Whom Do Language Models Align? Measuring Principal Hierarchies Under High-Stakes Competing Demands
May 12, 2026Fangyi Yu, Nabeel Seedat, Jonathan Richard Schwarz +1LLM Alignment Language Model Safety Evaluation
Thomson Reuters Foundational Research, 5 Canada Square, London · Imperial College London · University of Oxford
Do Language Models Encode Knowledge of Linguistic Constraint Violations?
May 12, 2026Hardy, Sebastian PadóLLM Interpretability Mechanistic Interpretability
IMS, University of Stuttgart, Stuttgart, Germany · Universitas Mikroskil, Medan, Indonesia.
GRAFT: Graph-Tokenized LLMs for Tool Planning
May 12, 2026Xinyi Gao, Xinyu Ren, Junliang Yu +3Tool-Use Planning Language Modeling
The University of Queensland · Griffith University
A Theory of Time-Sensitive Language Generation: Sparse Hallucination Beats Mode Collapse
May 11, 2026Atul Ganju, Travis McVoy, Shaddin Dughmi +1Hallucination in Language Models Language Modeling
University of Southern California
HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model
May 11, 2026Noam Kayzer, Dan Revital, Ori Bar Joseph +10Mixture-of-Experts Language Models Open-Weight Language Models
PwC Next
Language Modeling with Hyperspherical Flows
May 11, 2026Justin Deschenaux, Caglar GulcehreNormalizing Flows Continuous Normalizing Flows
EPFL, Lausanne, Switzerland
A Single-Layer Model Can Do Language Modeling
May 11, 2026Zanmin WangLanguage Model Pretraining Recurrent Language Models
The Truth Lies Somewhere in the Middle (of the Generated Tokens)
May 11, 2026Sophie L. Wang, Phillip Isola, Brian CheungLLM Interpretability Autoregressive Language Modeling
1MIT.
From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models
May 11, 2026Zehao Li, Yasuhiro Yoshikai, Shumpei Nemoto +2Chemical Language Models Transformer Interpretability
Laboratory of Molecular Pharmacokinetics, Graduate School of Pharmaceutical Sciences, The University of Tokyo, 7-3-1 Hongo, Bunkyo, Tokyo, Japan · The Institute of Statistical Mathematics (ISM), Research Organization of Information and Systems, 190-8562 Tachikawa, Tokyo, Japan
Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes
May 10, 2026A. BochkovLanguage Model Pretraining Language Modeling
Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
May 10, 2026Lin Zheng, Vasilisa Bashlovkina, Timothy Dozat +3Byte-Level Language Model Language Modeling
1Google DeepMind · 2The University of Hong Kong
Position: Avoid Overstretching LLMs for every Enterprise Task
May 10, 2026Kuldeep Singh, Anson Bastos, Isaiah Onando Mulang'Language Modeling Neuro-Symbolic AI
Microsoft, India · SAP, Germany
Understanding, Predicting, and Steering LLM Behavior via Representational Effective Theory
May 10, 2026Muhammed Ustaomeroglu, Guannan QuLLM Interpretability Mechanistic Interpretability
Carnegie Mellon University
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Mixture-of-Experts Language Models Recurrent Transformers
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
Fast Byte Latent Transformer
May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Speculative Decoding LLM Inference Acceleration
FAIR at Meta · Stanford University · University of Washington
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
May 8, 2026Andrea Sassella, Andrea Chizzola, Tommaso Bianchi +2LLM Evaluation Mixture-of-Experts Language Models
2DEIB, Politecnico di Milano · 1AIRIC, Politecnico di Milano
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3KV-Cache Management Memory-Efficient Inference
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models with Stream-Specialized Adapters
May 8, 2026Abdulvahap Mutlu, Şengül Doğan, Türker TuncerState Space Language Models Language Modeling
Department of Digital Forensics Engineering, Technology Faculty, Firat University, Elazig, Turkey
Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions
May 8, 2026Boyu Shi, Chang Liu, ChuanBao Gao +2LLM Pruning LLM Interpretability
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
LoopQ: Quantization for Looped Language Models
May 8, 2026Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng +1Recurrent Transformers Language Modeling
National Taiwan University
Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators
May 7, 2026Anupama Sridhar, Alexander JohansenState Space Language Models Koopman Operator Learning
California, USA · Stanford University Computer Science California, USA

May 13, 2026J/K move · Enter open · S save

Sampling from Flow Language Models via Marginal-Conditioned Bridges

Iskander Azangulov, Leo Zhang

Department of Statistics, University of Oxford

Flow Matching Language Model Decoding Language Modeling Flow-Based Generative Modeling

Open paper AskPDF arXiv

Flow Language Models (FLMs) are a recently introduced class of language models which adapt continuous flow matching for one-hot encoded token sequences. Their denoisers have a special structure absent from generic continuous diffusion models: each block of the denoising mean is a posterior marginal distribution over the clean token at that position. Standard DDPM-style samplers collapse these marginals to a single conditional-mean endpoint and bridge toward this simplex-valued point, which is generally not a valid one-hot sequence. We argue that the natural sampler for an FLM is instead posterior-predictive. At each reverse step, we sample a clean one-hot endpoint from the factorized posterior defined by the FLM token marginals, and then sample the next continuous state from the analytic Ornstein--Uhlenbeck bridge conditioned on that endpoint. The method is training-free, uses the same model evaluations as standard sampling, and gives a principled interface for token-level decoding controls such as temperature scaling and nucleus truncation. We show that, under exact posterior marginals, the endpoint approximation error is exactly the conditional multi-information among token positions. The induced one-step bridge kernel preserves all token-wise posterior-predictive marginals and loses only the residual cross-position dependence. Finally, we prove a Girsanov path-space comparison showing that the marginal-conditioned bridge has a no-larger denoising-error term than the frozen conditional-mean bridge, with strict improvement whenever intermediate coordinate-wise bridge observations reveal additional information about the clean token. Experiments with FLMs show that the sampler improves the quality--diversity tradeoff. Code is available at: github.com/imbirik/mcb.