Language Modeling
Overview
Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.
Latest papers
549 papers41 papers in the last four weeks, down 49% on the four weeks before. 0.3% of all new papers.
RooseBERT: A New Deal For Political Language Modelling
Understanding the Interplay between Memorization and Learning in Large Language Models
Lilith: Developmental Modular LLMs with Chemical Signaling
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
GLaMoR: Consistency Checking of OWL Ontologies using Graph Language Models
Kongzi: A Historical Large Language Model with Fact Enhancement
LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers
Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian
Vikhr'' refers to the name of the Mistral LLM series and means a strong gust of wind.'' Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model's performance but also significantly improves its computational and contextual efficiency. We also expanded the instruction datasets and corpora for continued pre-training. The model weights, instruction sets, and code are publicly available.