Language Modeling

Ask about this topic

Overview

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

CardsList

525 papers73 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

  1. Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

    Mar 3, 2026Shiza Fatimah, Aniket Sen, Sophia Falk +3Language ModelingLanguage Model Pretraining

    Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab · Lamarr Institute for Machine Learning and Artificial Intelligence · Helmholtz-Institut für Strahlen- und Kernphysik +1

  2. Recursive Models for Long-Horizon Reasoning

    Mar 2, 2026Chenxiao Yang, Nathan Srebro, Zhiyuan LiRecursive Language ModelsLanguage Modeling

    Toyota Technological Institute at Chicago

  3. AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth

    Mar 2, 2026Shixiang Song, He Li, Zitong Wang +6Recurrent Language ModelsInference-Time Adaptation of Language Models

    LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Innovation Institute · Sun Yat-sen University +2

  4. D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation

    Mar 2, 2026Zhao Yang, Hengchang Liu, Chuan Cao +1Molecular Foundation ModelsGenomic Foundation Models

    Gaoling School of Artificial Intelligence, Renmin University of China · Zhongguancun Academy

  5. Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain

    Mar 2, 2026Yuma Okochi, Fabio Milentiansen Sim, Tomoyasu OkadaFinancial Language Model EvaluationFinancial Reasoning in Language Models

    Nomura Research Institute, Ltd. · NRI Indonesia

  6. A Typologically Grounded Evaluation Framework for Word Order and Morphology Sensitivity in Multilingual Masked LMs

    Feb 28, 2026Anna Feldman, Libby Barak, Jing PengMultilingual Language Model EvaluationLanguage Modeling

    Montclair State University New Jersey, USA

  7. How Large Language Models Get Stuck: Early structure with persistent errors

    Feb 27, 2026Alokesh Manna, William Snyder, Whitney TaborLanguage ModelingLLM Training

    Department of Statistics University of Connecticut, Storrs, USA · Department of Linguistics University of Connecticut, Storrs, USA · Department of Psychological Sciences University of Connecticut, Storrs, USA

  8. Monotropic Artificial Intelligence: Toward a Cognitive Taxonomy of Domain-Specialized Language Models

    Feb 27, 2026Antonio de Sousa Leitão Filho, Allan Kardec Duailibe Barros Filho, Fabrício Saul Lima +2Structural EngineeringLanguage Modeling

    Aia Context, São Luís, MA, Brazil · Federal University of Maranhão, São Luís, MA, Brazil

  9. The GRADIEND Python Package: An End-to-End System for Gradient-Based Feature Learning

    Feb 27, 2026Jonathan Drechsel, Steffen HerboldLLM InterpretabilityLanguage Modeling

    Faculty of Computer Science and Mathematics, University of Passau

  10. From Tokens To Agents: A Researcher's Guide To Understanding Large Language Models

    Feb 27, 2026Daniele BaroloLanguage ModelingLLM Alignment

  11. SleepLM: Natural-Language Intelligence for Human Sleep

    Feb 27, 2026Zongzhe Xu, Zitao Shuai, Eideen Mozaffari +3Multimodal PretrainingCross-Modal Retrieval

    University of California, Los Angeles

  12. A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT

    Feb 25, 2026Louis Estève, Christophe Servan, Thomas Lavergne +1Language Model PretrainingDiversity-Aware Sampling

    Université Paris-Saclay, CNRS, LISN

  13. Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization

    Feb 25, 2026Suyash Maniyar, Deepali Singh, Rohith ReddyLLM GroundingLanguage Modeling

  14. Deep Sequence Modeling with Quantum Dynamics: Language as a Wave Function

    Feb 24, 2026Ahmed Nebli, Hadi Saadatdoorabi, Kevin YamSequence ModelingQuantum-Inspired ML

    cAI Technology GmbH

  15. Subgroups of U(d)U(d) Induce Natural RNN and Transformer Architectures

    Feb 20, 2026Joshua NunleyRecurrent Neural NetworksTransformer

    Cognitive Science Program & Department of Informatics, Indiana University

  16. On the scaling relationship between cloze probabilities and language model next-token prediction

    Feb 19, 2026Cassandra L. Jacobs, Morgan GrobolLanguage ModelingLanguage Model Scaling

    Department of Linguistics, University at Buffalo, Buffalo, NY, USA · MoDyCo, Université Paris Nanterre, Paris, France

  17. ArXiv-to-Model: A Practical Study of Scientific LM Training

    Feb 19, 2026Anuj GuptaLanguage Model PretrainingLanguage Modeling

    Independent Researcher India

  18. ALPS: A Diagnostic Challenge Set for Arabic Linguistic & Pragmatic Reasoning

    Feb 19, 2026Hussein S. Al-Olimat, Ahmad AlshareefArabic NLPArabic Language Models

    Independent Researchers

  19. Dialects of Translationese Shape Language Model Learning

    Feb 18, 2026Jenny KunzLanguage Modeling

    Department of Computer and Information Science Linköping University

  20. Avey-B

    Feb 17, 2026Devang Acharya, Mohammad HammoudLanguage ModelingTransformer

    Avey AI

  21. RF-GPT: Teaching AI to See the Wireless World

    Feb 16, 2026Hang Zou, Yu Tian, Bohao Wang +4Wireless CommunicationsLanguage Modeling

    Research Institute for Digital Future, Khalifa University, 127788 Abu Dhabi, UAE · College of Information Science and Electronic Engineering, Zhejiang University, 310027, Hangzhou, China · Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France

  22. Competition for attention predicts good-to-bad tipping in AI

    Feb 16, 2026Neil F. Johnson, Frank Y. HuoLLM SafetyLanguage Modeling

    Physics Department and Dynamic Online Networks Laboratory, George Washington University, Washington D.C. 20052, U.S.A.

  23. Left-right asymmetry in predicting brain activity from LLMs' representations emerges with their formal linguistic competence

    Feb 13, 2026Laurent Bonnasse-Gahot, Christophe PallierLanguage Modeling

    Centre d’Analyse et de Mathématique Sociales CNRS, EHESS, Paris, France · Cognitive Neuroimaging Unit CNRS, INSERM, CEA, Neurospin Center, 91191 Gif-sur-Yvette, France

  24. Compressed Sensing for Capability Localization in Large Language Models

    Feb 11, 2026Anna Bair, Yixuan Even Xu, Mingjie Sun +1Attention Head AnalysisLLM Interpretability

    Carnegie Mellon University

  25. SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

    Feb 3, 2026Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh +2LLM Safety BenchmarksMultilingual Language Model Evaluation

    Compumacy for Artificial Intelligence Solutions, Cairo, Egypt · Computer Science Department, College of Computer Science and Engineering, Taibah University, Yanbu 46522, KSA · Electrical and Computer Engineering Department, George Mason University, VA, USA +2

  26. From Directions to Regions: Decomposing Activations in Language Models via Local Geometry

    Feb 2, 2026Or Shafran, Shaked Ronen, Omri Fahn +3Language Model SteeringLLM Interpretability

    Blavatnik School of Computer Science and AI, Tel Aviv University, Israel · New York University, New York, NY, USA · Goodfire.

  27. Distilling Token-Trained Models into Byte-Level Models

    Feb 1, 2026Zishuo Bao, Jiaqi Leng, Junxiong Wang +2LLM Fine-TuningByte-Level Language Model

    Fuzhou University · NYU Shanghai · Fudan University +2

  28. Understanding LLM Failures: A Multi-Tape Turing Machine Analysis of Systematic Errors in Language Model Reasoning

    Jan 27, 2026Magnus BomanTokenization in Language ModelsLLM Interpretability

    Karolinska Institutet, Stockholm, Sweden

  29. Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

    Jan 21, 2026Injin Kong, Hyoungjoon Lee, Yohan JoMasked Diffusion ModelsAutoregressive Diffusion

    Graduate School of Data Science, Seoul National University · Department of Biosystems & Biomaterials Science and Engineering, Seoul National University

  30. Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

    Jan 12, 2026Xin Cheng, Rui Tian, Wangding Zeng +18Memory-Augmented Language ModelsLanguage Model Scaling Laws

    Peking University · DeepSeek-AI