Efficient Language Model Training

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 109

All topics

PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits
Jul 31, 2026Namkyung Yoon, Sanghong Kim, Hwangnam KimContinual Learning for LLMs LLM Fine-Tuning
School of Electrical Engineering, Korea University Seoul 02841, Republic of Korea
SDO: Structure-Aware Data Organization for Efficient LLM Post-Training
Jul 29, 2026Jinliang Gao, Ning Yang, Hai Wang +2Training Data Curation Efficient Language Model Training
Institute of Automation, Chinese Academy of Sciences · National University of Defense Technology
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
Jul 27, 2026Darina Gold, Alexander Schwirjow, Viktor Haag +4On-Device Language Model Inference Training Data Curation
IIS Fraunhofer
DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
Jul 24, 2026He ZhangSupervised Fine-Tuning Large Language Model-Guided Optimization
Tsinghua University
Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Jul 21, 2026Nuemaan MalikDeep Learning Optimization Mixture-of-Experts Language Models
Independent Researcher
PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning
Jul 20, 2026Hang Zhang, Warren J. GrossLLM Fine-Tuning Efficient Language Model Training
Department of Electrical and Computer Engineering, McGill University Montreal, QC, Canada
OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research
Jul 18, 2026Tavish Mankash, Vardhaman Kalloli, Keshava Prasad +1Language Model Pretraining Small Language Models
FAIRC · Plaksha University
Long-Context Fine-Tuning with Limited VRAM
Jul 16, 2026Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko +1Fine-Tuning KV-Cache Offloading
BMW Group
LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget
Jul 16, 2026Changhai Zhou, Kieran Liu, Yuhua Zhou +17Efficient Language Model Training Long-Context Modeling
MindLab · Fudan University
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
Jul 6, 2026Jingwei Zuo, Cong Zeng, Ilyas Chahed +6Memorization in Language Models Neural Network Memorization
Technology Innovation Institute Abu Dhabi, UAE
Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
Jul 2, 2026Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao +3Communication-Efficient Distributed Training High-Performance Computing
Salesforce AI Research
Parameter Golf: What Really Works?
Jul 1, 2026Prashanna Mani Paudel, Shivanand Venkanna SheshappanavarLLM Evaluation LLM Compression
Geometric Intelligence Research Lab., Department of Electrical Engineering and Computer Science University of Wyoming
Token Geometry
Jul 1, 2026Kathan ShahLanguage Model Pretraining LLM Fine-Tuning
Preprint.
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
Jun 30, 2026Dohyeon Kwon, Youngjin ParkMixture-of-Experts Language Models LLM Compression
AXOps Team, TeamSparta Inc., Seoul, South Korea
BluTrain: A C++/CUDA Framework for AI Systems
Jun 23, 2026Adhitya Charan, Adwaid Suresh, Anuj Kumar +19GPU Acceleration High-Performance Computing
Blubridge AI
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
Jun 22, 2026Dikshant Kukreja, Kritarth Prasad, Avinash Anand +6Automatic Differentiation Backpropagation
Puch AI · Singapore Institute of Technology · Nanyang Technological University +2
L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling
Jun 20, 2026Yin LiLanguage Model Pretraining Small Language Models
University of Birmingham
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale
Jun 13, 2026Ang Li, Ben Liu, Bin Han +215Efficient Language Model Training Linear Attention
A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget
Jun 11, 2026Joe DwyerLLM Evaluation Efficient Language Model Training
Department of Computer Information Science ECPI University Virginia Beach, VA 23462
Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisation
Jun 10, 2026Joshua Ong Jun Leang, Zheng Zhao, Mihaela Cătălina Stoian +5Automated Theorem Proving Efficient Language Model Training
Imperial College London · University of Edinburgh · Nanyang Technological University +1
Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey
Jun 9, 2026Vanessa Schmidt, Huy Hoang Nguyen, Cédric Jung +2Efficient Language Model Training LLM Training
Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University, 52074 Aachen, Germany · AIT Austrian Institute of Technology GmbH, 1210 Vienna, Austria · Automation and Control Institute, Technische Universität Wien (TUW), 1040 Vienna, Austria
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
Jun 5, 2026Rohan ShravanMixture-of-Experts Language Models Efficient Language Model Training
The School of AI Bengaluru, India
q0: Primitives for Hyper-Epoch Pretraining
Jun 2, 2026Bishwas Mandal, Shmuel Berman, Akshay Vegesna +1Language Model Pretraining Efficient Language Model Training
Q Labs · Princeton University
PortBERT: Navigating the Depths of Portuguese Language Models
Jun 1, 2026Raphael Scheible-Schmitt, Henry He, Armando B. MendesLanguage Model Pretraining Efficient Language Model Training
School of Computation, Information and Technology, Technical University of Munich, Munich, Germany · IS2E - Intelligent Systems, Science and Engineering, LIACC polo on Azores University, Ponta Delgada, Portugal
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Language Model Scaling Laws Efficient Language Model Training
University of Luxembourg · University of Twente · Eindhoven University of Technology +1
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
May 30, 2026Qiao Xiao, Boqian Wu, Patrik Okanovic +6Language Model Pretraining Efficient Language Model Training
Eindhoven University of Technology · University of Luxembourg · University of Twente +1
PithTrain: A Compact and Agent-Native MoE Training System
May 29, 2026Ruihang Lai, Hao Kang, Haozhan Tang +6Mixture-of-Experts Language Models Efficient Language Model Training
1Carnegie Mellon University · 3NVIDIA · 2Xlue
Towards Efficient LLMs Annealing with Principled Sample Selection
May 29, 2026Yuanjian Xu, Jianing Hao, Wanbo Zhang +2Language Model Pretraining Efficient Language Model Training
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Microsoft Research Asia, Beijing, China · Fudan University, Shanghai, China
D $^3$ : Dynamic Directional Graph-Constrained Data Scheduling for LLM Training
May 29, 2026Yuanjian Xu, Jianing Hao, Guang Zhang +1Efficient Language Model Training LLM Training
1HKUST-GZ · Microsoft Research Asia · 2Microsoft Research
MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection
May 28, 2026Haowen Wang, Yaxin Du, Jian Yang +9Data Selection Efficient Language Model Training
1Beihang University · 2IQuest Research · 3Shanghai Jiao Tong University +2

Jul 31, 2026, cs.CLJ/K move · Enter open · S save

PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits

Namkyung Yoon, Sanghong Kim, Hwangnam Kim

School of Electrical Engineering, Korea University Seoul 02841, Republic of Korea

Continual Learning for LLMs LLM Fine-Tuning Efficient Language Model Training

Open paper AskPDF arXiv

Recent language models achieve strong performance across a variety of tasks, but conventional adaptation applies updates uniformly across training samples regardless of their local update benefit. We propose PARALLEL, a prefrontal-aligned reinforcement inspired approach for language-model learning. Inspired by the complementary roles of goal-related and uncertainty-related control, PARALLEL represents these forms of information as separate controller signals and combines them with the current model representation. A reinforcement-inspired controller assigns sample-dependent update intensity using immediate utility-cost feedback. PARALLEL therefore learns when and how strongly to adapt to each sample, prioritizing beneficial updates while limiting unnecessary parameter changes. PARALLEL uses available updates more efficiently than selective baselines while retaining 94.1--99.2% of Full-adaptation performance. Beyond multiple-choice reasoning, experiments on XSum and CNN/DailyMail show that PARALLEL retains 96.9--98.6% of the ROUGE-1 and ROUGE-2 scores achieved by Full adaptation and 98.8--98.9% of the corresponding ROUGE-L scores. When compared at the same cumulative adaptation time or GPU energy, PARALLEL achieves higher ARC accuracy and exhibits a more stable late-stage adaptation trajectory than Full adaptation in the representative run. These results show that learning when and how strongly to update each sample supports stable and efficient post-deployment stream adaptation while avoiding unnecessary updates.

Momentum

Related topics

Latest papers 109