Language Modeling

Follow this topic Ask about this topic

Language modeling studies how systems learn from sequences, most often by predicting the next token or reconstructing masked tokens, enabling text generation and language understanding. Common approaches include autoregressive Transformers, masked language models, and alternatives such as recurrent or state-space networks, mixture-of-experts models, and diffusion methods. Researchers improve models through data and tokenizer design, continued pretraining, instruction tuning, retrieval, distillation, and compression, evaluating them with perplexity, linguistic tests, and task benchmarks. A central tension is how to make models more efficient and reliable while ensuring benchmark success reflects robust reasoning, generalization, and competence across languages and domains.

Latest papers

525 papers73 papers in the last four weeks, up 22% on the four weeks before. 0.5% of all new papers.

Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning
Apr 7, 2026Navan Preet Singh, Xiaokun Wang, Anurag Garikipati +3LLM Fine-Tuning Language Modeling
Forta, Houston, TX · East China Normal University, Shanghai, China · Incept Labs, Houston, TX +1
YoNER: A New Yorùbá Multi-domain Named Entity Recognition Dataset
Apr 7, 2026Peace Busola Falola, Jesujoba O. Alabi, Solomon O. Akinola +3Named Entity Recognition Low-Resource Language Processing
University of Ibadan, Nigeria · Saarland University, Germany · Atiba University, Nigeria +1
HUKUKBERT: Domain-Specific Language Model for Turkish Law
Apr 6, 2026Mehmet Utku Öztürk, Tansu Türkoğlu, Buse Buz-YalugLegal NLP Language Modeling
Kalitte Inc., Türkiye · Aibrite Inc., Türkiye · University of Eastern Finland, Finland
FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation
Apr 6, 2026Runzhe Zhang, Letian Chen, Wenpeng Zhang +2Language Modeling Flow-Based Generative Modeling
Shanghai Jiao Tong University · ByteDance
Olmo Hybrid: From Theory to Practice and Back
Apr 3, 2026William Merrill, Yanhong Li, Tyler Romero +19Hybrid Language Models Hybrid Transformer-State Space Model Architectures
Allen Institute for AI · ETH Zürich · Lambda ML Team +2
JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency
Apr 3, 2026Aichen Cai, Anmeng Zhang, Anyu Li +66Language Modeling Mixture-of-Experts Language Models
GaelEval: Benchmarking LLM Performance for Scottish Gaelic
Apr 2, 2026Peter Devine, William Lamb, Beatrice Alex +5Multilingual Language Model Evaluation LLM Evaluation
University of Edinburgh · Lancaster University · University of Cardiff +2
BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
Apr 2, 2026Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef +2Language Modeling Multimodal Learning
Diabolocom · MICS, CentraleSupélec, Université Paris-Saclay · Artefact Research Center +1
The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook
Apr 2, 2026Xinlei Yu, Zhangquan Chen, Yongbo He +36Continuous Latent Reasoning Language Modeling
National University of Singapore · Fudan University · Tsinghua University +16
$k$ NNProxy: Efficient Training-Free Proxy Alignment for Black-Box Zero-Shot LLM-Generated Text Detection
Apr 2, 2026Kahim Wong, Kemou Li, Haiwei Wu +1AI-Generated Text Detection Robustness of Machine-Generated Text Detection
State Key Laboratory of Internet of Things for Smart City and the Department of Computer and Information Science, Faculty of Science and Technology, University of Macau, Macao 999078, China · School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Phase transition on a context-sensitive random language model with short range interactions
Apr 1, 2026Yuma Toji, Jun Takahashi, Vwani Roychowdhury +1Language Modeling Statistical Physics of Learning
Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Hokkaido 060-0814, Japan · Institute for Solid State Physics, The University of Tokyo, 5-1-5 Kashiwanoha, Kashiwa, Chiba 277-8581, Japan · Henry Samueli School of Engineering and Applied Science, University of California, Los Angeles, California 90095
Metriplector: From Field Theory to Neural Architecture
Mar 31, 2026Dan Oprisa, Peter TothNeural Algorithmic Reasoning Language Modeling
Spheroid Labs∗
Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models
Mar 31, 2026Christine Zhang, Dan Jurafsky, Chen ShaniLanguage Modeling LLM Alignment
Stanford University
Generative Chemical Language Models for Energetic Materials Discovery
Mar 30, 2026Andrew Salij, R. Seaton Ullberg, Megan C. Davis +5Molecular Generation Materials Science
Theoretical Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA · Weapon Stockpile Modernization Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA · Computing and Artificial Intelligence Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
Categorical Perception in Large Language Model Hidden States: Structural Warping at Digit-Count Boundaries
Mar 30, 2026Jon-Paul CacioliTokenization in Language Models Representation Geometry in Language Models
Independent Researcher, Melbourne, Australia
The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions
Mar 29, 2026Jarosław HryszkoState Space Language Models Language Modeling
Institute of Computer Science, Faculty of Mathematics and Computer Science, Jagiellonian University, Krak´ow, Poland
Conversational Agents and the Understanding of Human Language: Reflections on AI, LLMs, and Cognitive Science
Mar 29, 2026Andrei Popescu-BelisConversational Agents Language Modeling
HEIG-VD / HES-SO, 1401 Yverdon-les-Bains, Switzerland · EPFL, 1015 Lausanne, Switzerland
Weight Tying Biases Token Embeddings Towards the Output Space
Mar 27, 2026Antonio Lopardo, Avyukth Harish, Catherine Arnett +1Language Modeling Representation Geometry in Language Models
EleutherAI · University of California Berkeley
Demystifying When Pruning Works via Representation Hierarchies
Mar 25, 2026Shwai He, Guoheng Sun, Haichao Zhang +2LLM Pruning Language Modeling
University of Maryland, College Park, USA · Northeastern University, USA
Is AI Catching Up to Human Expression? Exploring Emotion, Personality, Authorship, and Linguistic Style in English and Arabic with Six Large Language Models
Mar 24, 2026Nasser A AlsadhanEmotion Representation in Language Models AI-Generated Text Detection
College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia
Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications
Mar 23, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóHybrid Language Models Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Cross-Family Speculative Decoding for Polish Language Models on Apple~~Silicon: An Empirical Evaluation of Bielik~~11B with UAG-Extended MLX-LM
Mar 22, 2026Krzysztof FonalSpeculative Decoding LLM Inference Acceleration
Wrocław University of Science and Technology, Poland
A chemical language model for reticular materials design
Mar 20, 2026Dhruv Menon, Vivek Singh, Xu Chen +7Chemical Language Models Materials Science
Department of Chemical Engineering & Biotechnology, University of Cambridge, Cambridge, UK · Department of Chemistry, University of California – Berkeley, Berkeley, CA, USA · Bakar Institute of Digital Materials for the Planet, Berkeley, CA, USA +2
The production of meaning in the processing of natural language
Mar 20, 2026Christopher J. Agostino, Quan Le Thien, Nayan D'Souza +1LLM Interpretability Language Modeling
NPC Worldwide, Bloomington, IN, USA · Department of Physics, Indiana University, Bloomington, IN, USA · Department of Linguistics, Indiana University, Bloomington, IN, USA +1
What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?
Mar 19, 2026Gagan Bhatia, Ahmad Muhammad Isa, Maxime Peyrard +1Tokenization in Language Models Temporal Reasoning in Language Models
University of Aberdeen · Université Grenoble Alpes & CNRS
IQuest-Coder-V1 Technical Report
Mar 17, 2026Jian Yang, Wei Zhang, Shawn Guo +35Recurrent Language Models Code Language Models
EngGPT2: Sovereign, Efficient and Open Intelligence
Mar 17, 2026G. Ciarfaglia, A. Rosanova, S. Cipolla +17Language Modeling Open-Weight Language Models
NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling
Mar 17, 2026Zhengzheng TangSpiking Language Models Spiking Neural Networks
Boston University
A Family of LLMs Liberated from Static Vocabularies
Mar 16, 2026Aleph Alpha, :, Adnen Abdessaied +35Language Modeling Byte-Level Language Model
Aleph Alpha Research
Pretraining and Benchmarking Modern Encoders for Latvian
Mar 16, 2026Arturs ZnotinsLanguage Model Pretraining Low-Resource Language Processing
Institute of Mathematics and Computer Science, University of Latvia, 29 Raina bulv., Riga, LV-1459, Latvia

Apr 7, 2026J/K move · Enter open · S save

Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning

Navan Preet Singh, Xiaokun Wang, Anurag Garikipati, Madalina Ciobanu+2

Forta, Houston, TX · East China Normal University, Shanghai, China · Incept Labs, Houston, TX+1

LLM Fine-Tuning Language Modeling RL Fine-Tuning

Open paper AskPDF arXiv

We present an innovative multi-stage optimization strategy combining reinforcement learning (RL) and supervised fine-tuning (SFT) to enhance the pedagogical knowledge of large language models (LLMs), as illustrated by EduQwen 32B-RL1, EduQwen 32B-SFT, and an optional third-stage model EduQwen 32B-SFT-RL2: (1) RL optimization that implements progressive difficulty training, focuses on challenging examples, and employs extended reasoning rollouts; (2) a subsequent SFT phase that leverages the RL-trained model to synthesize high-quality training data with difficulty-weighted sampling; and (3) an optional second round of RL optimization. EduQwen 32B-RL1, EduQwen 32B-SFT, and EduQwen 32B-SFT-RL2 are an application-driven family of open-source pedagogical LLMs built on a dense Qwen3-32B backbone. These models remarkably achieve high enough accuracy on the Cross-Domain Pedagogical Knowledge (CDPK) Benchmark to establish new state-of-the-art (SOTA) results across the interactive Pedagogy Benchmark Leaderboard and surpass significantly larger proprietary systems such as the previous benchmark leader Gemini-3 Pro. These dense 32-billion-parameter models demonstrate that domain-specialized optimization can transform mid-sized open-source LLMs into true pedagogical domain experts that outperform much larger general-purpose systems, while preserving the transparency, customizability, and cost-efficiency required for responsible educational AI deployment.