Overview

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

CardsList

549 papers41 papers in the last four weeks, down 49% on the four weeks before. 0.3% of all new papers.

  1. SinLlama -- A Large Language Model for Sinhala

    Aug 12, 2025H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake +3Language Model PretrainingMultilingual Language Models

  2. RooseBERT: A New Deal For Political Language Modelling

    Aug 5, 2025Deborah Dore, Elena Cabrio, Serena VillataLanguage Model PretrainingArgument Mining

  3. Understanding the Interplay between Memorization and Learning in Large Language Models

    Jul 20, 2025Bishwamittra Ghosh, Soumi Das, Qinyuan Wu +4Memorization in Language ModelsLLM Training

  4. ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining

    May 26, 2025Melis Ilayda Bal, Volkan Cevher, Michael MuehlebachLanguage Model PretrainingEfficient Language Model Training

  5. GLaMoR: Consistency Checking of OWL Ontologies using Graph Language Models

    Apr 26, 2025Justin Mücke, Ansgar ScherpLogical ReasoningLanguage Modeling

  6. Kongzi: A Historical Large Language Model with Fact Enhancement

    Apr 13, 2025Jiashu Yang, Ningning Wang, Yian Zhao +5Language ModelingRL for Language Model Reasoning

  7. LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

    Feb 20, 2025Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev +4Transformer InterpretabilityLong-Context Language Modeling

  8. Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

    May 22, 2024Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov +2Language Model PretrainingMultilingual Language Models