Language Modeling

Follow this topic Ask about this topic

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

525 papers73 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

M $^2$ RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
Mar 15, 2026Mayank Mishra, Shawn Tan, Ion Stoica +2Recurrent Neural Networks Language Modeling
UC Berkeley · Learning Machine · MIT-IBM Watson Lab +2
Drift and selection in LLM text ecosystems
Mar 15, 2026Søren RiisModel Collapse LLM Training
Queen Mary University of London, London, United Kingdom
An Alternative Trajectory for Generative AI
Mar 14, 2026Margarita Belova, Yuval Kansal, Yihao Liang +2Language Modeling Neuro-Symbolic AI
Princeton University, USA
BERTology of Molecular Property Prediction
Mar 13, 2026Mohammad Mostafanejad, Paul Saxe, T. Daniel CrawfordChemical Language Models Molecular Property Prediction
Department of Chemistry, Virginia Tech, Blacksburg, Virginia 24061, USA · Molecular Sciences Software Institute, Blacksburg, Virginia 24060, USA
Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study
Mar 13, 2026Liel Binyamin, Elior SulemLanguage Model Pretraining Language Modeling
Faculty of Computer and Information Science, Institute for Applied AI Research Data Science Research Center Ben-Gurion University of the Negev
RetroReasoner: A Reasoning LLM for Strategic Retrosynthesis Prediction
Mar 13, 2026Hanbum Ko, Chanhui Lee, Ye Rin Kim +4AI-Assisted Scientific Research Language Modeling
Department of Artificial Intelligence, Korea University · Materials Intelligence Lab, LG AI Research · Department of Statistics, Korea University
Interpreting Negation in GPT-2: Layer- and Head-Level Causal Analysis
Mar 12, 2026Abdullah Al Mofael, Lisa M. Kuhn, Ghassan Alkadi +1Language Modeling Circuits in Language Models
∗Department of Computer Science, Southeastern Louisiana University, Hammond, LA, USA · †Department of Mathematics, Southeastern Louisiana University, Hammond, LA, USA
Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantization
Mar 12, 2026Takato YasunoLanguage Modeling Small Language Models
Long-Context Encoder Models for Polish Language Understanding
Mar 12, 2026Sławomir Dadas, Rafał Poświata, Marek Kozłowski +4Long-Context Language Modeling Long-Context Modeling
National Information Processing Institute, Warsaw, Poland · PKO Bank Polski, Warsaw, Poland
Learning Transferable Sensor Models via Language-Informed Pretraining
Mar 12, 2026Yuliang Chen, Arvind Pillai, Yu Yvonne Wu +6Multimodal Pretraining Language Modeling
Dartmouth College
Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Language
Mar 12, 2026Remigiusz Kinas, Paweł Kiszczak, Sergio P. Perez +4LLM Compression Language Modeling
Bielik.AI, Ingenix.ai · Bielik.AI, Vstorm · NVIDIA +3
Making Bielik LLM Reason (Better): A Field Report
Mar 11, 2026Adam Trybus, Bartosz Bartnicki, Remigiusz KinasLLM Reasoning Language Modeling
Institute of Philosophy, Jagiellonian University, 52 Grodzka St., Kraków 31-044, Poland · Bielik.ai (Speakleash Foundation), 33/4 Niemczańska St., Wrocław 50-561, Poland
HoloByte: Continuous Hyperspherical Distillation for Tokenizer-Free Modeling
Mar 10, 2026Vladimer KhasiaByte-Level Language Model Language Modeling
Independent Researcher
Sabiá-4 Technical Report
Mar 10, 2026Thiago Laitz, Thales Sales Almeida, Hugo Abonizio +6Portuguese Language Models Language Modeling
Maritaca AI · Jusbrasil
N-gram-like Language Models Predict Naturalistic Reading Time Best
Mar 10, 2026James A. Michaelov, Roger P. LevyLanguage Modeling Autoregressive Language Modeling
Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology · MIT Libraries CREOS, Massachusetts Institute of Technology
Build, Borrow, or Just Fine-Tune? A Political Scientist's Guide to Choosing NLP Models
Mar 10, 2026Shreyas MeherLLM Fine-Tuning Language Modeling
School of Social and Behavioural Sciences, Erasmus University Rotterdam
Exclusive Self Attention
Mar 10, 2026Shuangfei ZhaiSelf-Attention Transformer
Apple
Quantifying Memorization and Privacy Risks in Genomic Language Models
Mar 9, 2026Alexander Nemecek, Wenbiao Li, Xiaoqian Jiang +2Memorization in Language Models Privacy Leakage in Language Models
Case Western Reserve University Cleveland, Ohio, USA · The University of Texas Health Science Center at Houston Houston, Texas, USA · Rutgers University Newark, New Jersey, USA
Eyla: Toward an Identity-Anchored LLM Architecture with Integrated Biological Priors -- Vision, Implementation Attempt, and Lessons from AI-Assisted Development
Mar 9, 2026Arif AdittoLanguage Modeling AI-Assisted Software Development
Independent Researcher, Dhaka, Bangladesh
Towards a more efficient bias detection in financial language models
Mar 9, 2026Firas Hadj Kacem, Ahmed Khanfir, Mike PapadakisFinancial Language Model Evaluation Intersectional Bias in Language Models
SnT, University of Luxembourg, Luxembourg · RIADI, ENSI, University of Manouba, Tunisia
The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling
Mar 8, 2026J. Clayton Kerce, Alexis FoxTransformer Transformer Interpretability
Georgia Tech Research Institute
Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training
Mar 7, 2026Chuxue Cao, Honglin Lin, Zhanping Zhong +5Financial Language Model Evaluation LLM Training
Shanghai Artificial Intelligence Laboratory, OpenDataLab, OpenDataArena · Hong Kong University of Science and Technology
Experiences Build Characters: The Linguistic Origins and Functional Impact of LLM Personality
Mar 6, 2026Xi Wang, Mengdie Zhuang, Jiqun LiuPersonality Modeling in Language Models Language Modeling
University of Sheffield, Sheffield, UK · University of Oklahoma, Oklahoma, USA
Addressing the Ecological Fallacy in Larger LMs with Human Context
Mar 6, 2026Nikita Soni, Dhruv Vijay Kunjadiya, Pratham Piyush Shah +3Language Modeling LLM Training
Department of Computer Science, Stony Brook University · College of Connected Computing, Vanderbilt University, USA
FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications
Mar 5, 2026Yunfan Zhang, Yijie Bei, Jetashree Ravi +1LLM Evaluation Instruction Following
Columbia University · Fireworks AI
Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model
Mar 4, 2026Jakub PrejznerQuantization LLM Quantization
BitSharp, Independent Researcher, Rzeszów, Poland
Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes
Mar 4, 2026Fangyu Ding, Ding Ding, Sijin Chen +8Diffusion Language Models Language Modeling
HKUST · Huawei Foundation Model Dept · CUHK
N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognition
Mar 4, 2026Florent Meyer, Laurent Guichard, Yann Soullard +3Inference-Time Adaptation of Language Models Language Modeling
ANTAI, Rennes, France · Univ Rennes, CNRS, IRISA - UMR 6074, Rennes, France
ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Mar 3, 2026Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3Language Modeling Byte-Level Language Model
Rice University · Amazon Science
Tucano 2 Cool: Better Open Source LLMs for Portuguese
Mar 3, 2026Nicholas Kluge Corrêa, Aniket Sen, Shiza Fatimah +4Portuguese Language Models Language Modeling
Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab · Lamarr Institute for Machine Learning and Artificial Intelligence · Center for Science and Thought +2

Mar 15, 2026J/K move · Enter open · S save

M $^2$ RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling

Mayank Mishra, Shawn Tan, Ion Stoica, Joseph E. Gonzalez+1

UC Berkeley · Learning Machine · MIT-IBM Watson Lab+2

Recurrent Neural Networks Language Modeling Autoregressive Language Modeling Long-Context Language Modeling

Open paper AskPDF arXiv

Transformers are highly parallel but are limited to computations in the TC

^0

complexity class, excluding tasks such as entity tracking and code execution that provably require greater expressive power. Motivated by this limitation, we revisit non-linear Recurrent Neural Networks (RNNs) for language modeling and introduce Matrix-to-Matrix RNN (M

^2

RNN): an architecture with matrix-valued hidden states and expressive non-linear state transitions. We demonstrate that the language modeling performance of non-linear RNNs is limited by their state size, and show how the state size expansion mechanism enables efficient use of tensor cores. Empirically, M

^2

RNN achieves perfect state tracking generalization at sequence lengths not seen during training. These benefits also translate to large-scale language modeling. In hybrid settings that interleave recurrent layers with attention, Hybrid M

^2

RNN outperforms equivalent Gated DeltaNet hybrids by

0.4

0.5

perplexity points on a 7B MoE model, while using

3\times

smaller state sizes for the recurrent layers. Notably, replacing even a single recurrent layer with M

^2

RNN in an existing hybrid architecture yields accuracy gains comparable to Hybrid M

^2

RNN with minimal impact on training throughput. Further, the Hybrid Gated DeltaNet models with a single M

^2

RNN layer also achieve superior long-context generalization, outperforming state-of-the-art hybrid linear attention architectures by up to

8

points on LongBench. Together, these results establish non-linear RNN layers as a compelling building block for efficient and scalable language models.