Efficient Language Model Training

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 109

All topics

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Apr 28, 2026Fan Jiang, Yu Zhao, Chenyang Lyu +5Multilingual Language Models Mixture-of-Experts Language Models
Alibaba International Digital Commerce
Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
Apr 21, 2026Chaitanya Dwivedi, Binxuan Huang, Himanshu Gupta +3Efficient Language Model Training Language Model Scaling
1Amazon Stores Foundation AI · 3Anthropic · 2Carnegie Mellon University
FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs
Apr 20, 2026Yun Hong, Yan Zhou, Yang FengSpoken Dialogue Systems Efficient Language Model Training
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
Mar 16, 2026Qiyuan Chen, Xian Wu, Yi Wang +1Communication-Efficient Distributed Training Zeroth-Order Optimization
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.
Stabilizing Native Low-Rank LLM Pretraining
Feb 12, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLanguage Model Pretraining Low-Rank Matrix Decomposition
Concordia University, Montreal, Canada · Mila Quebec AI Institute, Montreal, Canada · Sorbonne University, CNRS, Paris, France
Revisiting the Shape Convention of Transformer Language Models
Feb 6, 2026Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin +2Efficient Transformer Inference Transformer FFNs
MediaTek Research · National Taiwan University
Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining
Jan 21, 2026Jingru Li, Yibo Fan, Huan LiStochastic Optimization Language Model Pretraining
College of Artificial Intelligence, Nankai University
Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT Distillation Selective Knowledge Distillation
1The University of British Columbia · 2LinkedIn · 3Canada Research Chair in NLP and ML
EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training
Nov 13, 2025Qingao Yi, Jiaang Duan, Jun Zhang +5Communication-Efficient Distributed Training Gradient Compression
University of Shanghai for Science and Technology, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Zhejiang University, Hangzhou, China
A Survey on Efficient Vision-Language-Action Models
Oct 27, 2025Zhaoshu Yu, Bo Wang, Pengpeng Zeng +7Model Compression Robotic Data Collection
School of Computer Science and Technology, Tongji University, China · School of Computing and Artificial Intelligence, Southwest Jiaotong University, China · School of Computer Science and Engineering, University of Electronic Science and Technology of China, China +1
Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation
Sep 26, 2025Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El Haddad +2CoT Distillation LLM Fine-Tuning
1Diabolocom · 5MICS, CentraleSupélec, Université Paris-Saclay · 2Artefact Research Center +2
Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
Sep 25, 2025Shiju Wang, Yujie Wang, Fangcheng Fu +6Long-Context Language Modeling Efficient Language Model Training
Tsinghua University · Peking University · Shanghai Jiao Tong University +1
Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
Aug 13, 2025Maxime Heuillet, Yufei Cui, Boxing Chen +2RL for Language Model Reasoning Efficient Language Model Training
Universit´e Laval (IID), Canada · Mila - Qu´ebec AI Institute, Canada · Huawei Noah’s Ark Lab (Montreal Research Center), Canada +1
PLoRA: Efficient Concurrent LoRA Training for Large Language Models
Aug 4, 2025Minghao Yan, Zhuang Wang, Zhen Jia +2LLM Fine-Tuning Efficient Language Model Training
University of Wisconsin-Madison · Amazon Web Services
Curriculum-Guided Layer Scaling for Language Model Pretraining
Jun 13, 2025Karanpartap Singh, Neil Band, Ehsan AdeliLanguage Model Pretraining Efficient Language Model Training
1Stanford University.
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
May 26, 2025Melis Ilayda Bal, Volkan Cevher, Michael MuehlebachLanguage Model Pretraining Efficient Language Model Training
Max Planck Institute for Intelligent Systems, Tübingen, Germany
A Survey of Transformer-based Language Models with Focus on Efficiency
May 15, 2024Wazib Ansar, Saptarsi Goswami, Amlan ChakrabartiLLM Compression Energy-Efficient ML
A. K. Choudhury School of IT University of Calcutta, Kolkata, India · Department of Computer Science Bangabasi Morning College Kolkata, India
DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration
Nov 8, 2023Martin Kuo, Jianyi Zhang, Dongting Li +1Language Model Pretraining Efficient Language Model Training
Center for Computational Evolutionary Intelligence, Duke University
Almost Free State Prediction Separation
Date pendingJohn Langford, Nathan Godey, Giovanni Monea +5Language Model Pretraining Efficient Language Model Training
Microsoft · Cornell University

Apr 28, 2026, cs.CLJ/K move · Enter open · S save

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling

Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi+4

Alibaba International Digital Commerce

Multilingual Language Models Mixture-of-Experts Language Models Activation Sparsity Efficient Language Model Training Open-Weight Language Models Sparse Neural Networks

Open paper AskPDF arXiv

We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-\textsc{Instruct} variants, which surpass the performance of competing models possessing

3

14\times

more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights.

Momentum

Related topics

Latest papers 109