Language Modeling

Follow this topic Ask about this topic

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

525 papers73 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

Instruction-Tuned Models Locally Produce More Syntactic Overlap -- But Not More Conditional Reuse -- Than Humans Do
Jul 28, 2026Zandi EberstadtLanguage Modeling Instruction Tuning
Department of Computer Science, University of Oxford
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
Jul 27, 2026Darina Gold, Alexander Schwirjow, Viktor Haag +4On-Device Language Model Inference Training Data Curation
IIS Fraunhofer
MS-GPT: Rethinking MS/MS De Novo Structure Elucidation as Spectrum-Induced Posterior Querying of a Molecule-Language Model
Jul 26, 2026Xin Zhao, Yumin Liu, Zhuo Li +5Language Modeling Molecular Generation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science Shanghai Jiao Tong University Shanghai, China · Instrumental Analysis Center Shanghai Jiao Tong University Shanghai, China · ByteDance Inc. Beijing, China +1
Spectral Dynamics of Semantic Drift in Clinical Multi-Agent Language Model Networks
Jul 23, 2026Amritesh BanerjeeSemantic Drift in Language Models Language Modeling
University of Massachusetts Amherst, USA
From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models
Jul 23, 2026Lin Tian, Marian-Andrei RizoiuMultilingual Language Model Evaluation Text Classification
Behavioral Data Science University of Technology Sydney Sydney, Australia
Exposure is Optional: Learning Unlike Coordination in Language Models
Jul 22, 2026Jiamu Luo, Shane Steinert-ThrelkeldSyntactic Generalization in Language Models Language Modeling
Department of Linguistics, University of Washington, Seattle, WA 98195, USA
surprisal is Not a Theory
Jul 22, 2026Andrés Buxó-Lugo, Aniello De Santo, Morgan Grobol +2Surprisal Theory Language Modeling
aUniversity at Buffalo, Buffalo, NY, USA · bUniversity of Utah, Salt Lake City, UT, USA · cUniversité Paris Nanterre, Paris, France +1
Multi-Head Attention Residuals
Jul 22, 2026Cheng Luo, Zefan Cai, Junjie HuTransformer Attention Residual Connections
Independent Researcher · University of Wisconsin–Madison
From Dependency to Compositionality: A Neurosymbolic Lifting of LLM Outputs via Combinatory Categorial Grammar
Jul 21, 2026Remo PareschiLLM Grounding Language Modeling
STAKE Lab, University of Molise, Campobasso, Italy
L1 Augmented Attention as an Improved Vector Similarity Metric
Jul 20, 2026Kurt GoddenTransformer Attention Language Modeling
Walsh College, Troy Michigan, USA
Arithmetic and its remedies select the evidence on learned geometry
Jul 20, 2026Kwan Soo ShinLanguage Modeling Language Model Post-Training
PolymathMinds Lab, Asan, Republic of Korea.
Explaining and Tuning Transformer-based LLMs in Arithmetic Tasks with Human Strategies
Jul 19, 2026Luyu Qiu, Jianing Li, Hwanhee Kim +4Transformer Interpretability Language Modeling
1The Hong Kong University of Science and Technology · 2The Hong Kong Polytechnic University · University of California, Berkeley
Linear representations of grammaticality in neural language models
Jul 16, 2026Jane Li, Najoung KimSyntactic Generalization in Language Models Language Modeling
Department of Cognitive Science Johns Hopkins University Baltimore, MD · Department of Linguistics Boston University Boston, MA
Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns
Jul 13, 2026Yong Yang, Xiang Guan, Sophie Arheix-Parras +7Multimodal Large Language Models Language Modeling
Department of Computer Science and Engineering, University of South Carolina · Department of Psychology, University of South Carolina · Department of Communication Sciences and Disorders, University of South Carolina +2
Dzongkha Next Word Prediction System
Jul 13, 2026Prerna Chhetri, Tenzin Yoezer, Phuntsho Wangmo +1Gated Recurrent Unit Next-Token Prediction
Information Technology Department, College of Science and Technology, Royal University of Bhutan
The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese
Jul 12, 2026Siyuan Song, Zhiheng Qian, Yunhao Zhang +11Language Model Pretraining LLM Evaluation
Princeton University · Shanghai Jiao Tong University · Chinese Academy of Sciences +5
Energy-guided Recursive Model
Jul 11, 2026Yifei Zhao, Ying TangLanguage Modeling Recursive Language Models
Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Physics, University of Electronic Science and Technology of China, Chengdu 611731, China
A Sovereign, Open-Source Foundation Model for German and English
Jul 10, 2026The Soofi-Team, :, Benedikt Droste +29Language Model Pretraining Multilingual Language Models
ellamind · Fraunhofer IAIS · Lamarr +8
Creativity, honesty and designed forgetting emerge in small hyperbolic language models
Jul 10, 2026Kwan Soo Shin, In Seok Kang, Yunkyung MinLLM Sycophancy Memory-Augmented Language Models
PolymathMinds Lab, Seoul, Republic of Korea. · Department of Chemical Engineering, POSTECH, Pohang, and aSSIST University, Seoul, Republic of Korea. · Korean Educational Development Institute, Jincheon, Republic of Korea.
Co-LMLM: Continuous-Query Limited Memory Language Models
Jul 8, 2026Yair Feldman, Linxi Zhao, Nathan Godey +5Language Model Pretraining Memory-Augmented Language Models
Department of Computer Science Cornell University
DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation
Jul 8, 2026Jordan Painter, Dipankar Srirag, Adarsh Kappiyath +3Language Modeling LLM Alignment
Institute for People-Centred AI, University of Surrey, Surrey, United Kingdom · University of New South Wales, Sydney, Australia
Semantic Drift and the Stability of Operator Control in Reasoning-Class Decision Support Systems
Jul 8, 2026M. L. Kaluzhsky, V. A. EfirovLanguage Model Steering Semantic Drift in Language Models
Expert Department, Regional Strategy Development Fund, Omsk, Russia · Omsk State Technical University, Omsk, Russia · Expert Council, Interregional Public Fund “Regional Strategy Development Fund”, Omsk, Russia
FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention
Jul 8, 2026Athanasios ZerisFrequency-Domain Feature Learning Transformer Attention
Independent Researcher, Athens, Greece.
BlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching Speech
Jul 7, 2026Ho Lam Chung, Bo-Xuan Zheng, Cheng-Chieh Huang +8TTS Synthesis Language Modeling
National Taiwan University
DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modeling
Jul 6, 2026Wataru Nakata, Yuki Saito, Hiroshi SaruwatariSpeech Processing Language Modeling
The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology, Japan
Full Glyph Images Beat Token Embeddings: A Controlled Study for Transformers
Jul 4, 2026Shuyang Xiang, Hao GuanLanguage Modeling Tokenization in Language Models
Independent Researcher. · Institute of Software, Chinese Academy of Sciences.
Induction Heads Interpolate N-Grams
Jul 2, 2026Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman +1Transformer Interpretability In-Context Learning
1TML Lab, EPFL, Switzerland. · 2IIT Madras, India. · †Work done during an internship at the TML Lab, EPFL.
BamiBERT: A New BERT-based Language Model for Vietnamese
Jul 2, 2026Dat Quoc Nguyen, Thinh Pham, Chi Tran +1Language Model Pretraining Transformer
Qualcomm AI Research · Virginia Tech
Probing Chemical Language Models: Effects of Pre-training and Fine-tuning
Jul 2, 2026Anna Karnysheva, Dietrich Klakow, Ji-Ung LeeLanguage Model Pretraining Chemical Language Models
1RTG Neuroexplicit Models · 2Spoken Language Systems · Saarland University +1
Parameter Golf: What Really Works?
Jul 1, 2026Prashanna Mani Paudel, Shivanand Venkanna SheshappanavarLLM Evaluation LLM Compression
Geometric Intelligence Research Lab., Department of Electrical Engineering and Computer Science University of Wyoming

Jul 28, 2026J/K move · Enter open · S save

Instruction-Tuned Models Locally Produce More Syntactic Overlap -- But Not More Conditional Reuse -- Than Humans Do

Zandi Eberstadt

Department of Computer Science, University of Oxford

Language Modeling Instruction Tuning

Open paper AskPDF arXiv

Syntactic convergence - the tendency of speakers to adapt in language towards the grammatical profiles of their interlocutors - is a well-documented, largely implicit feature of human dialogue. Whether large language models exhibit analogous syntactic convergence towards human users relative to human baselines (including across a broad range of syntactic constructions) remains an open question. Using substitution-paradigm data from Blevins et al. [2026] in which model generations replace one speaker's turns in pre-existing human dialogues, this study measures turn-adjacent reuse of context-free grammar (CFG) rules across sixteen open-weight Llama and Gemma models (1B-70B, pretrained and instruction-tuned) at 1,901 matched positions per model. Every model showed greater CFG-rule overlap with the preceding human turn than with a sampled unrelated human prime (a context-associated effect that does not isolate syntactic priming from topical continuity), and this actual-versus-random difference was larger for lower-frequency rules. Each instruction-tuned model also showed greater natural-output overlap with the actual prime than the human response it replaced, and all eight matched architecture pairs exhibited greater actual-prime overlap after instruction tuning. However, relative to pretrained variants, instruction-tuned outputs overlapped more with unrelated primes, showed a smaller actual-versus-random increment, and had lower conditional rule-reuse odds once target rule-set size was held constant. In exploratory analyses, each model exhibited greater mean lexical and semantic similarity to the preceding turn than matched human responses did. Instruction-tuned models additionally produced responses with greater mean semantic similarity than their pretrained counterparts in all eight architecture pairs, whereas the lexical similarity results were more heterogeneous.