Language Modeling

Ask about this topic

Overview

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

CardsList

523 papers84 papers in the last four weeks, up 33% on the four weeks before. 0.5% of all new papers.

  1. Asynchronous Flow Map Language Models

    Oct 30, 2026Chanhyuk Lee, Wonjung Kim, Patrick Pynadath +5Language ModelingOne-Step Generative Modeling

    KAIST · Purdue University · The University of Tokyo

  2. Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese

    Oct 29, 2026Pedro Carvalho BromLanguage ModelingSmall Language Models

    Department of Mathematics Federal Institute of Brasília Brasília, Brazil

  3. Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention

    Oct 28, 2026Yixiao Qian, Yaxin Zhang, Song Chen +3Linear AttentionDelta-Rule Linear Attention

    College of Control Science and Engineering, Zhejiang University · Polytechnic Institute, Zhejiang University · Department of Mathematics, National University of Singapore +1

  4. StateLoop: Persistent State for Looped Transformers

    Oct 26, 2026Feichen Song, Boyi Zeng, He Li +7Persistent Memory for Language ModelsLanguage Modeling

    Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute

  5. Synthetic Child-Adult Interactions Are Not Developmentally Plausible

    Oct 26, 2026Bastian Bunzeck, Francesca PadovaniLanguage ModelingSynthetic Data Evaluation

    Computational Linguistics, Department of Linguistics, Bielefeld University · Center for Language and Cognition (CLCG), University of Groningen

  6. Wave Is All You Need: A Quantum Language Model Based on Fully Connected Quantum Boltzmann Machine

    Oct 25, 2026Jun LiuLanguage ModelingQuantum NLP

    School of Economics, Hunan University of Finance and Economics, Changsha 410205, China

  7. CoLink: Global Compression and Knowledge Linking for Multi-Turn Dialogue Modeling

    Oct 25, 2026Lian Yan, Zhenbo Xia, Haotian Wang +4Language ModelingLLM Context Compression

    Research Center of Language Technology, Faculty of Computing, Harbin Institute of Technology, Harbin, Heilongjiang 150001, China · Institute for Advanced Algorithms Research, Shanghai, 201306, China · Research Center of Artificial Intelligence of Things, Faculty of Computing, Harbin Institute of Technology, Harbin, Heilongjiang 150001, China, and also with the National Key Laboratory of Smart Farm Technologies and Systems, Harbin, Heilongjiang 150001, China

  8. Rectified Language Flow

    Oct 24, 2026Baiyu Su, Runlong Liao, Lizhang Chen +1Language ModelingNon-Autoregressive Text Generation

    The University of Texas at Austin

  9. Kolibri: A Sovereign European Model on the Pareto Frontier

    Oct 22, 2026Aleph Alpha, :, Moritz Althaus +65Language ModelingLanguage Model Pretraining

  10. Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation

    Oct 22, 2026Hongcan Guo, Yian Zhao, Feng Wang +4Language ModelingDiffusion Language Model Decoding

    The University of Hong Kong · ByteDance Seed · Peking University +1

  11. Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

    Oct 22, 2026Aman Behera, Namit Solanki, Mehul AnandState Space Language ModelsLong-Context Language Modeling

    IIT Roorkee · AISSMS IOIT, Pune · Independent Researcher

  12. LAMB: A Modern Contextual Language Model for Latin

    Oct 22, 2026Aidan McCarthyLanguage ModelingLanguage Model Pretraining

  13. ULME: Predictive Coupling for Mutually Reinforcing Embedding and Language Modeling

    Oct 22, 2026Yang Liu, Zhiqiang Yuan, Shu-Dong Huang +5Text EmbeddingsText Embedding Models

    Pattern Recognition Center, WeChat AI, Tencent Inc. · College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education

  14. GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation

    Oct 21, 2026Quang Anh Kiet Tran, Nam Anh Tran, Manh Hung Trinh +3Small Language ModelsLanguage Modeling

    UIT, VNU-HCM · TwoHearts Technologies Co., Ltd., Ho Chi Minh City, Vietnam · IU, VNU-HCM +3

  15. Large Flow Language Models

    Oct 20, 2026Arnav Chavan, Chanhyuk Lee, Udbhav Bamba +4Language ModelingFlow-Based Generative Modeling

    UC Berkeley · Amazon · KAIST +1

  16. Can Language Models Get "Aphasia" From Neuronal Ablation?

    Oct 20, 2026Pratik Worah, Randall BatemanCausal Interventions in Language ModelsDisease Progression Modeling

  17. BCT: Bayes-Consistent Tempering for Flow Language Models

    Oct 20, 2026Daisuke Oba, Hiroki Furuta, Naoaki OkazakiLanguage ModelingFlow-Based Generative Modeling

    Institute of Science Tokyo · AIST · NII LLMC

  18. BERT Is Already a Denoiser: Cheaply Unlocking Iterative Generation in Masked Language Models

    Oct 19, 2026Julio C. Amador Diaz LopezLanguage ModelingConditional Generation

    *SiftyML, LTD

  19. Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

    Oct 19, 2026Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker +24Language ModelingLanguage Model Pretraining

    NVIDIA · AI4Bharat

  20. Modal Sense Disambiguation in Language Models: The Role of Context, Training Domain, and Input Frequency

    Oct 19, 2026Ezgi Başar, Yevgen Matusevych, Annemarie van Dooren +1Word Sense DisambiguationLanguage Modeling

    Center for Language and Cognition (CLCG), University of Groningen

  21. Weighted State Souping for Efficient Long-Context Retrieval

    Oct 19, 2026William L. Tong, Aryo Lotfi, Emmanuel Abbe +3State Space Language ModelsLong-Context Retrieval

    Harvard University · Apple

  22. Aligning Cultural Behaviors in Arabic Language Models via Preferences

    Oct 18, 2026Junhong Liang, Daniil Orel, Saeed Almheiri +11Arabic Language ModelsCultural Alignment

    Mohamed bin Zayed University of Artificial Intelligence · IBM Research AI

  23. Prosodic Structure as a Critical Component in Phonemic Language Modelling at BabyLM Scale

    Oct 16, 2026Bill K. S. Ku, Janet B. PierrehumbertLanguage ModelingMasked Language Modeling

    Department of Engineering Science, University of Oxford

  24. Chinese Decision: A Family of Language Models for Single-Pass Probabilistic Decision-Making

    Oct 16, 2026Qiang YanLanguage Modeling

    Institute of Computing Technology, Chinese Academy of Sciences

  25. FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

    Oct 16, 2026Ali Sarabadani, Massimo Piccardi, Mehran Abolhasan +3Byte-Level Language ModelMultimodal Federated Learning

    Cyber Digital Centre, University of Technology Sydney

  26. Pearl-Chat: Engineering a native Luganda language model from first principles in JAX and Flax NNX

    Oct 15, 2026Wesley Kambale, Roland SankaraLanguage ModelingLow-Resource Language Processing

    AI for Uganda Initiative · ISBAT University

  27. From Parallels to Divergence: Language Models' Learning of Action Nominal Constructions

    Oct 15, 2026Iskar Deng, Shane Steinert-ThrelkeldSyntactic Generalization in Language ModelsLanguage Modeling

    University of Washington

  28. Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling

    Oct 15, 2026Tong Zhu, Zhe Xu, Zefeng He +7LLM Decision-MakingEfficient Multimodal Inference

    Shanghai AI Lab · Shanghai Jiao Tong University · Fudan University +1

  29. The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction

    Oct 15, 2026Hyunsik KimTokenization in Language ModelsByte-Level Language Model

    Samsung Research

  30. Wolof-Adapted Industry-Ready Open-Weight Language Models

    Oct 14, 2026Gwénolé Lecorvé, Delphine Charlet, Pierre Adam +15Language ModelingOpen-Weight Language Models

    Orange Research, France