Language Modeling

Follow this topic Ask about this topic

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

525 papers73 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

Message Passing Enables Efficient Reasoning
Jul 1, 2026Xuecheng Liu, Daman Arora, Gokul Swamy +1Parallel Reasoning in Language Models Inference-Time Scaling
Carnegie Mellon University
Prototype Language Models
Jul 1, 2026Dan Ley, Giang Nguyen, Himabindu Lakkaraju +1LLM Interpretability Language Modeling
1Harvard University · 2Guide Labs Inc. *Work initiated during an internship at Guide Labs. · 2Guide Labs Inc.
Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors
Jun 30, 2026Mark OskinTransformer Interpretability Transformer FFNs
School of Computer Science and Engineering University of Washington
DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks
Jun 29, 2026Romain Karpinsky, Julien Mozziconacci, Mickaël DelceyLanguage Model Pretraining Fine-Tuning
CNRS, LORIA · Nancy, 54000, France · CNRS +1
BERTomelo: Your Portuguese Encoder Best Friend
Jun 27, 2026Rennê Ruan Alves Oliveira, Gustavo Cordeiro Galvão Van Erven, Luís Paulo Faina GarciaLanguage Model Pretraining Portuguese Language Models
Department of Computer Science University of Brasília Brasília - DF, Brazil
Scaling limit of the Random Language Model
Jun 26, 2026Eric De GiuliLanguage Model Scaling Laws Language Modeling
Department of Physics, Toronto Metropolitan University, M5B 2K3, Toronto, Canada
Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment
Jun 26, 2026Zhuo Zuo, Li Yue, Wenhao Zheng +2Numerical Reasoning in Language Models Maximum Mean Discrepancy
College of Computer Science, Sichuan University, Chengdu, China · Dongfang Electric (Chengdu) Academy of Science and Technology Co., Ltd., Chengdu, China · Institute of Medical Information & Library, CAMS & PUMC, Beijing, China
Masked Language Flow Models
Jun 26, 2026Iskander Azangulov, Kianoosh Ashouritaklimi, Leo Zhang +2Masked Diffusion Models Instruction Following
Department of Statistics, University of Oxford
Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation
Jun 24, 2026Han Jeon, Shiv Medler, Joseph Voyles +1LLM Safety Benchmarks LLM-as-a-Judge
PricewaterhouseCoopers, U.S.A.
Riazi-8B: An Urdu Large Language Model for Mathematical Reasoning
Jun 24, 2026Azher Ali, Ibtsam Haider, Raja Khurram Shahzad +2Numerical Reasoning in Language Models LLM Fine-Tuning
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan · Department of Communication, Quality Management and Information Systems, Mid Sweden University, Ostersund, Sweden
LLMs Prompted for Legal Context Object More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context
Jun 23, 2026Anastasiia Kucherenko, François Brouchoud, Dimitri Percia David +1LLM Prompting Small Language Models
IEM, HEG, HES-SO Valais-Wallis Sierre, Switzerland
What Does a Chemical Language Model Know About Molecules?
Jun 22, 2026Christian Kenneth, Etowah Adams, Liam Bai +1Chemical Language Models Mechanistic Interpretability
Independent · Department of Systems Biology, Columbia University, New York · Generate:Biomedicines, Massachusetts +1
moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT
Jun 21, 2026Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos +1Language Model Pretraining Portuguese Language Models
UNICAMP, Campinas, Brazil · Tropic AI · Maritaca AI
Atomistic Language Models Understand and Generate Materials
Jun 19, 2026Sathya Edamadaka, Krithik Ramesh, Ju Li +1Crystal Structure Prediction Language Modeling
Massachusetts Institute of Technology · Lyra Labs · Lila Sciences
FiLM-Coordinated Dual-Branch Transformer for Global-Local Dependency Modeling in Language Modeling
Jun 19, 2026Zhiqiang Zhou, Xu Ling, Junliang DaiTransformer Language Modeling
IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources
Jun 18, 2026Arash Ghafouri, Mahdi Firouzmandi, Hossein Saberi +1Language Model Pretraining Language Modeling
Department of Artificial Intelligence and Cognitive Science, Imam Hossein Comprehensive University, Tehran, Iran.
The Stanford EDGAR Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data
Jun 16, 2026Nick Bettencourt, Xiaowei Ding, Kay GieseckeDocument Understanding Financial Language Model Evaluation
Department of Applied Statistics and Data Science University of California, Los Angeles · School of Information Management Nanjing University · Advanced Financial Technologies Lab Stanford University
MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
Jun 15, 2026Xianxuan Long, Zhikai Chen, Shenglai Zeng +3Temporal Reasoning in Language Models LLM Agent Memory
Michigan State University · Case Western Reserve University
Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammars
Jun 15, 2026Zongru Liu, Pengyu Ji, Pengcheng Wang +1Language Modeling
School of Information Science and Technology, ShanghaiTech University
The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance
Jun 15, 2026Csaba Kiss, Roland Molontay, Gabriele PergolaPharmacovigilance Language Modeling
Department of Stochastics, Institute of Mathematics, Budapest University of Technology and Economics, Műegyetem rkp. 3., Budapest, H-1111, Hungary · Institute of Biostatistics and Network Science, Semmelweis University, Üllői út 26., Budapest, H-1085, Hungary · Department of Computer Science, University of Warwick, Coventry, CV4 7AL, UK
PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino
Jun 13, 2026Jann Railey Montalan, David Demitri Africa, Jimson Paulo Layacan +3LLM Evaluation Language Modeling
AI Singapore · Nanyang Technological University · UK AI Security Institute +2
Simplifying the Modeling of Arbitrary Conditionals in Natural Language
Jun 12, 2026Yinhan Lu, Eric Elmoznino, Léo Gagnon +3Transformer Autoregressive Language Modeling
Mila — Quebec AI Institute · McGill University · Université de Montréal
Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models
Jun 12, 2026Rituraj Sharma, Tu VuRecurrent Neural Networks Recurrent Transformers
Virginia Tech
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Jun 12, 2026Sangwhan Moon, Daisuke Oba, Youmi Ma +2Byte-Level Language Model LLM Reliability
Google LLC / Mountain View, CA, USA · Institute of Science Tokyo / Tokyo, Japan · Mohamed bin Zayed University of Artificial Intelligence / Abu Dhabi, United Arab Emirates
MÖVE: A Holistic LLM Benchmark for the German Public Sector
Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Multilingual Language Model Evaluation LLM Evaluation
Innovations Department, Bundesdruckerei GmbH, Berlin, Germany.
Augmenting Molecular Language Models with Local $n$ -gram Memory
Jun 10, 2026Xinni Zhang, Zijing Liu, He Cao +2Memory-Augmented Language Models Language Modeling
1The Chinese University of Hong Kong · 2International Digital Economy Academy
Beyond representational alignment with brain-guided language models for robust reasoning
Jun 10, 2026Mingqing Xiao, Kai Du, Zhouchen LinLogical Reasoning Language Modeling
1State Key Lab of General AI, School of Intelligence Science and Technology, Peking University. · 3Microsoft Research Asia. · Department of Psychological and Cognitive Sciences, Tsinghua University.
Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs
Jun 9, 2026Ruixuan Huang, Jinyuan Shi, Hantao Huang +5Continual Learning for LLMs LLM Compression
1HKUST · 2Moffett AI · 3Independent Contributor
Large Language Models as Modal Models in Linguistics
Jun 9, 2026Haruto Suzuki, Saku SugawaraLanguage Modeling
1Keio University · 2National3UniversityInstituteofofTokyo
Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation
Jun 9, 2026Xuanchen Li, Tianrui Wang, Yuheng Lu +11Speech Translation Automatic Speech Recognition
1Tianjin University · 2Shanghai Jiao Tong University · 3Zhipu AI

Jul 1, 2026J/K move · Enter open · S save

Message Passing Enables Efficient Reasoning

Xuecheng Liu, Daman Arora, Gokul Swamy, Andrea Zanette

Carnegie Mellon University

Parallel Reasoning in Language Models Inference-Time Scaling Efficient Language Model Reasoning LLM Inference Acceleration Language Modeling Large Reasoning Models

Open paper AskPDF arXiv

While inference-time scaling has improved the reasoning abilities of large language models (LLMs), the need to generate long chains-of-thought (CoTs) is a computational bottleneck. Thus, in contrast to sequential scaling methods like CoT, recent parallel scaling techniques instead use fork and join (FJ) primitives to divide work across multiple LLM threads. However, in the fork-join paradigm, threads are typically transient and do not communicate pointwise with one another which limits scalability. To tackle this, we introduce Message Passing Language Models (MPLMs), a framework for LLM reasoning in which threads communicate directly via lightweight send and receive primitives. MPLMs enable efficient scaling through two key mechanisms: (1) reduced communication costs, achieved by avoiding redundant context sharing, and (2) preemption, which allows threads to terminate early based on partial information from their peers. We demonstrate the promise of MPLMs on 3 classes of tasks. First, on Sudoku puzzles, we show that MPLMs require an asymptotically smaller context than both serial CoT and parallel FJ. We then fine-tune a single model to solve 25 x 25 puzzles that remain challenging for standard CoT and FJ approaches, as well as frontier reasoning models without tools. Second, on 3-SAT puzzles, the capability of preemption allows termination of unpromising branches, which results in improved efficiency. Finally, we show that appropriately prompted large pre-trained models follow the MPLM protocol, achieving competitive results on long-context question answering relative to popular fork-join approaches.