Language Modeling

Follow this topic Ask about this topic

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

525 papers73 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

A Continuous-Time Markov Chain Framework for Insertion Language Models
Jun 8, 2026Dhruvesh Patel, Benjamin Rozonoyer, Soumitra Das +3Markov Models Markov Chains
UMass Amherst · IBM Research · University of Toronto & Vijil
Causally Evaluating the Learnability of Formal Language Tasks
Jun 8, 2026Vésteinn Snæbjarnarson, Anej Svete, Josef Valvoda +3Language Modeling Causal Interventions in Language Models
1ETH Zürich · University of Copenhagen · *Now at Google
End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12LLM Compression Long-Context Language Model Inference
New York University · Modal Labs · Harvard University +5
Chiaroscuro Attention: Spending Compute in the Dark
Jun 6, 2026Prateek Kumar SikdarTransformer Transformer Attention
Accenture, India
A Geometric Account of Activation Steering through Angle-Norm Decomposition
Jun 4, 2026Georgii Aparin, Tatiana GaintsevaLanguage Model Steering Language Modeling
Huawei Noah’s Ark Lab · Queen Mary University of London
HAARES Half-Split Residual Basis Routing for Deep Transformers
Jun 4, 2026Kehan WangTransformer Residual Learning
Paper under double-blind review
YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition
Jun 4, 2026PSBC LLM Team, Huawei LLM Team, Ruihan Long +56Financial Language Model Evaluation LLM Compression
LLMCodec: Adapting Video Codecs for Efficient Weight Compression of Large Language Models
Jun 4, 2026Rui Wang, Yan Zhao, Li Song +1Model Compression LLM Quantization
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University. Shanghai, China
What's in a Name? Morphological Shortcuts by LLMs in Pharmacology
Jun 4, 2026Kaijie Mo, Thomas Yang, Chantal Shaib +6Shortcut Learning in Language Models Hallucination in Language Models
The University of Texas at Austin · Northeastern University · MD Anderson Cancer Center
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
Jun 3, 2026Wenkai Chen, Tianshu Li, Wenyong Huang +3Mixture-of-Experts Language Models Language Modeling
Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd.
Dynamic Short Convolutions Improve Transformers
Jun 2, 2026Oliver Sieberling, Bharat Runwal, Rameswar Panda +1Dynamic Convolution Transformer
Massachusetts Institute of Technology · MIT-IBM Watson AI Lab
When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models
Jun 2, 2026Ding Zhang, Runtao Zhou, Wenqing Zheng +3LLM Interpretability Mechanistic Interpretability
University of Virginia · Capital One
Causal Evidence of Stack Representations in Modeling Counter Languages Using Transformers
Jun 2, 2026Nishit SinghTransformer Interpretability Transformer
Birla Institute of Technology and Science, Pilani
Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs
Jun 2, 2026Ting LiuLLM Quantization On-Device Language Model Inference
SymbolicLight Research Foshan, Guangdong, China
Pretraining Language Models on Historical Text
Jun 2, 2026Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber +5Language Model Pretraining LLM Evaluation
University of Waterloo · Vector Institute · AIML, Adelaide University +3
Linguistic Productivity in Large Language Models: Models Coerce, but do not Preempt
Jun 1, 2026Claire Bonial, Claire Benet Post, Laura Michaelis +1LLM Evaluation Language Modeling
Georgetown University · University of Colorado Boulder · University of Bath
Large Byte Model: Teaching Language Models About Compiled Code
Jun 1, 2026Florian Störtz, Catalin-Andrei Stan, Alexandru Dinu +4Malware Classification Byte-Level Language Model
CrowdStrike · CrowdStrike U.K. · CrowdStrike Romania +1
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
Jun 1, 2026Muyu He, Yuchen Liu, Qingya Huang +1Transformer Attention Language Modeling
Independent · Drexel University
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
Jun 1, 2026Vera Neplenbroek, Gabriele Sarti, Arianna Bisazza +1Social Bias in Language Models Language Modeling
Institute for Logic, Language and Computation, University of Amsterdam · Khoury College of Computer Sciences, Northeastern University · Center for Language and Cognition, University of Groningen
On the Persistent Effects of Lexicality in Large Language Models
Jun 1, 2026Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani +3LLM Interpretability Language Modeling
Dalhousie University Halifax, NS · University of Kentucky Lexington, KY · Carnegie Mellon University Pittsburgh, PA
On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
Jun 1, 2026Mind Lab, :, Vin Bo +64LLM Fine-Tuning Adapter Tuning
PortBERT: Navigating the Depths of Portuguese Language Models
Jun 1, 2026Raphael Scheible-Schmitt, Henry He, Armando B. MendesLanguage Model Pretraining Portuguese Language Models
School of Computation, Information and Technology, Technical University of Munich, Munich, Germany · IS2E - Intelligent Systems, Science and Engineering, LIACC polo on Azores University, Ponta Delgada, Portugal
On the Generalization Gap in Self-Evolving Language Model Reasoning
May 31, 2026Zhenting Qi, Susanna Maria Baby, Stefanie Anna Baby +5LLM Self-Refinement Language Modeling
Google Research · Harvard University · Google +1
Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading
May 31, 2026Lin YaoLanguage Model Decoding Confidence Estimation in Language Models
School of Computer Science, Shanghai Jiao Tong University, Shanghai, 200240, China · Zhongguancun Academy, Beijing, 100097, China
SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition
May 30, 2026Jayanta Dey, Shikhar Srivastava, Itamar Lerner +2Memory-Augmented Neural Networks Continual Learning
University of Texas at San Antonio, USA · University of Rochester, USA
Language Models Can Resolve Reference Compositionally, But It's Not Their Native Strength: The Case of the Personal Relation Task
May 29, 2026Bart Evelo, Meaghan Fowlie, Denis PapernoLLM Evaluation LLM Grounding
Fine-Tuning Improves Information Conveyance in Language Models
May 29, 2026Yuwei Cheng, Weiyi Tian, Haifeng XuLanguage Model Generation Evaluation LLM Fine-Tuning
Department of Data Science University of Chicago Chicago, IL 60637 · Department of Computer Science University of Chicago Chicago, IL 60637
Speculative Decoding Across Languages
May 28, 2026Nirajan Paudel, Michael Ginn, Luc De Nardi +1Multilingual Language Models Speculative Decoding
University of Colorado
Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model
May 28, 2026Thang Dang, Akira Nakagawa, Kenichi Kobayashi +1Document Chunking Byte-Level Language Model
Fujitsu Research of America · Pittsburgh, PA, USA · Fujitsu Limited +1
EvoMD-LLM: Learning the Language of Species Evolution in Reactive Molecular Dynamics
May 28, 2026Zhichen Tang, Zhengzheng Dang, Yulin Chen +3Autoregressive Language Modeling Language Modeling
Global College, Shanghai Jiao Tong University, Shanghai, China · Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai, China

Jun 8, 2026J/K move · Enter open · S save

A Continuous-Time Markov Chain Framework for Insertion Language Models

Dhruvesh Patel, Benjamin Rozonoyer, Soumitra Das, Tahira Naseem+2

UMass Amherst · IBM Research · University of Toronto & Vijil

Markov Models Markov Chains Language Modeling Diffusion Language Models

Open paper AskPDF arXiv

Insertion Language Models (ILMs) offer several advantages over left-to-right generation and mask-based generation. However, existing formulations of insertion-based generation have largely been ad-hoc. In this paper, we derive a diffusion-style denoising objective for ILMs from first principles by formulating the noising process as a continuous-time Markov chain on the space of variable-length sequences. We show that previous formulations of ILMs can be viewed as special cases of this denoising framework. Through empirical evaluation on a synthetic planning task, we show that the proposed approach retains the benefits of insertion-based generation over left-to-right generation and masked diffusion models. In language modeling, our diffusion-based approach is competitive with left-to-right generation and masked diffusion models, while offering additional flexibility in sampling compared to existing insertion language models.