Language Modeling

Follow this topic Ask about this topic

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

525 papers73 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

When and How Long? The Readout-Mediator Angle in Temporal Reasoning
May 27, 2026Shreyas Fadnavis, Praitayini Kanakaraj, Felix WyssTemporal Reasoning in Language Models Temporal Reasoning
Bioscope AI
Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay
May 27, 2026Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen +2Multilingual Language Model Evaluation Pragmatic Reasoning in Language Models
Nanyang Technological University · University of Mississippi · Indiana University
Sense Representations Are Inducible Interfaces
May 27, 2026Jan Christian Blaise Cruz, Alham Fikri AjiLanguage Model Steering Language Modeling
MBZUAI · SEACrowd
When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs
May 27, 2026Marcell Fekete, Johannes Bjerva, Tamás KáldiVLM Evaluation Visual Question Answering
Department of Computer Science, Aalborg University, Copenhagen, Denmark · Department of Psycholinguistics and Neurolinguistics, ELTE Research Centre for Linguistics, Budapest, Hungary · ELTE Bárczi Gusztáv Faculty of Special Needs Education, Budapest, Hungary
Periodic RoPE for Infinite Context LLMs
May 27, 2026Simin HuoRotary Positional Embeddings Long-Context Language Modeling
Shanghai Jiao Tong University
DunbaaBERT: From Sacrifice to Semantics
May 26, 2026Iffat Maab, Waleed Jamil, Raphael SchmittLanguage Model Pretraining Language Modeling
Research and Development Center for Large Language Models (LLMC), National Institute of Informatics, Tokyo · Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany +1
Innovation: An Almost Characterization of Hallucination
May 26, 2026Nishant P. Das, Piyush SrivastavaHallucination in Language Models Language Model Calibration
School of Technology and Computer Science, Tata Institute of Fundamental Research, Mumbai, Maharashtra - 400 005, India.
Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
May 26, 2026Xiao-Wen Yang, Ziyu Han, Xi-Hua Zhang +4Latent Dynamics Modeling Test-Time Scaling
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Nanjing, China
Is Inference Mediated by Distinct Semantic Structures in LLMs? A Mechanistic Interpretation
May 25, 2026Nura Aljaafari, Marco Valentino, André FreitasMechanistic Interpretability Language Modeling
University of Manchester, United Kingdom · University of Sheffield, United Kingdom · Idiap Research Institute, Switzerland +1
Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning
May 24, 2026Chenyou Guo, Zongqi Liu, Yizhou Zhang +2Scientific Reasoning LLM Fine-Tuning
Ocean University of China · Peking University · Monash University
The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling
May 22, 2026Al KariDecoder-Only Language Models Message Passing Neural Networks
Manceps Inc.
Skill Weaving: Efficient LLM Improvement via Modular Skillpacks
May 21, 2026Zhuo Li, Guodong Du, Zesheng Shi +5LLM Compression LLM Inference Acceleration
Harbin Institute of Technology, Shenzhen, China · The Hong Kong Polytechnic University · Huawei Technologies Co., Ltd. +1
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Encoder Spike Sparsity
May 20, 2026Ting LiuLanguage Model Pretraining Spiking Neural Networks
SymbolicLight Research Foshan, Guangdong, China
LoCar: Localization-Aware Evaluation of In-Vehicle Assistants through Fine-Grained Sociolinguistic Control
May 20, 2026Seogyeong Jeong, Kiwoong Park, Seyoung Song +4LLM Evaluation Language Modeling
KAIST · BMW Group
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Efficient Transformer Inference Recurrent Transformers
1Rice University · 2Apple · 3UC Santa Cruz +1
Self-Training Doesn't Flatten Language -- It Restructures It: Surface Markers Amplify While Deep Syntax Dies
May 20, 2026Ming LiuSelf-Training for Language Models Syntactic Generalization in Language Models
Amazon
Readers make targeted regressions to plausible errors in reanalysis of "noisy-channel garden-path" sentences
May 18, 2026Thomas Hikaru Clark, Roger Levy, Edward GibsonLanguage Modeling Computational Psycholinguistics
MIT Brain and Cognitive Sciences 43 Vassar Street, Cambridge MA, USA
Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models
May 17, 2026Ethan TangLLM Evaluation Language Modeling
School of Computing and Augmented Intelligence · Arizona State University · Tempe, AZ 85281
MiniGPT: Rebuilding GPT from First Principles
May 17, 2026Jibin JosephLanguage Model Pretraining Decoder-Only Language Models
Department of Computer Science The University of Texas at Austin Austin, TX 78712
Parallel Recursive LSTM
May 16, 2026Tristan Gaudreault, Yongyi MaoParallel Training of Recurrent Networks Recurrent Neural Networks
School of Electrical Engineering and Computer Science University of Ottawa Ottawa, ON
Language Acquisition Device in Large Language Models
May 16, 2026Masato Mita, Taiga Someya, Ryo Yoshida +1Language Model Pretraining Synthetic Data Pretraining
The University of Tokyo
Recall Isn't Enough: Bounding Commitments in Personalized Language Systems
May 15, 2026Rui Tang, Yichi Zhang, Xi Chen +4Memory-Augmented Language Models Controllable Text Generation
OpenAsk · Stern School of Business, New York University · Bank of Hebei +4
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
May 15, 2026Tomasz SteiferGated Attention Recurrent Transformers
Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland
GiLT: Augmenting Transformer Language Models with Dependency Graphs
May 15, 2026Tianyu Huang, Yida Zhao, Chuyan Zhou +1Transformer Language Modeling
School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging
Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis
May 14, 2026William Timkey, Brian Dillon, Tal LinzenSurprisal Theory Language Modeling
Department of Linguistics, New York University, 10 Washington Pl, New York, NY 10003, USA · Department of Linguistics, University of Massachusetts Amherst, 650 N Pleasant St, Amherst, MA 01002, USA · Center for Data Science, New York University, 60 5th Ave, New York, NY 10011, USA
Emotion-Attended Stateful Memory (EASM):The Architecture for Hyper-Personalization at Scale
May 14, 2026Vineet Kotecha, Vansh GuptaMemory-Augmented Language Models Personalized Language Models
divAIne Research
Teaching Large Language Models When Not to Know: Learning Temporal Critique for Ex-Ante Reasoning
May 14, 2026Chenlu Ding, Jiancan Wu, Yanchen Luo +3LLM Prompting LLM Fine-Tuning
University of Science and Technology of China · The Hong Kong Polytechnic University · University of Notre Dame
VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use
May 13, 2026Juan S. SantillanaSmall Language Models LLMs for Cybersecurity
Globant
A Hierarchical Language Model with Predictable Scaling Laws and Provable Benefits of Reasoning
May 13, 2026Jason Gaitonde, Frederic Koehler, Elchanan Mossel +2Long-Context Language Modeling Language Model Scaling Laws
Duke University · University of Chicago · Massachusetts Institute of Technology +1

May 27, 2026J/K move · Enter open · S save

When and How Long? The Readout-Mediator Angle in Temporal Reasoning

Shreyas Fadnavis, Praitayini Kanakaraj, Felix Wyss

Bioscope AI

Temporal Reasoning in Language Models Temporal Reasoning Language Modeling Causal Interpretability Hidden-State Probing

Open paper AskPDF arXiv

A linear probe can decode a representation almost perfectly and yet be completely irrelevant to how the model uses it. On calendar-date duration reasoning in language models, a

\sin

\cos

probe recovers day-of-year from a layer's activations, yet ablating its direction has no effect on the model's answers -- while ablating a four-dimensional subspace found by Distributed Alignment Search (DAS) at the same layer collapses performance entirely. We measure the angle between these two subspaces -- the readout-mediator angle -- and find it indistinguishable from the angle between two random subspaces (the Haar-uniform null), meaning the probe has learned a direction orthogonal to the model's actual computation. Reverse-engineering the circuit reveals why: attention heads route month-grained context through learned QK offsets at

{\pm}30

and

{\pm}61

days, and MLPs then convert when (absolute date) into how long (duration) -- all downstream of the causal subspace the probe never touches. Sparse-autoencoder decomposition confirms the split: probe-aligned and DAS-aligned features encode semantically disjoint concepts with negligible causal overlap. The dissociation replicates across four scales (

1.5

9\,

B) and two model families, with preliminary evidence on two further domains (spatial displacement, symbolic arithmetic), suggesting that readout-mediator orthogonality is a general failure mode of probe-based interpretability. This directly undermines proposals to deploy probes as runtime safety monitors: the probe can report high confidence on a direction the model has silently abandoned.