Warm Starts

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 6Week of Sep 21

Latest papers 26

All topics

SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining
Sep 28, 2026Qiulin Shang, Binyu Wang, Yongqi Qiao +3Batch Large Language Model Pretraining
Peking University · Nanjing University
AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Sep 8, 2026Ziyang Ma, Zhikang Niu, Wenming Tu +30Modern Generative Audio Models Audio Editing
When Does Online Adaptation Pay on the Edge? A Leakage-Free Evaluation of Warmup, Learning-Rate Selection, and Resource Trade-offs for Time-Series Forecasting
Sep 1, 2026Takumi Fujimoto, Hiroaki NishiNon-Stationarity Time Series Forecasting
School of Science for Open and Environmental Systems, Graduate School of Science and Technology, Keio University, Yokohama, Japan
Simple-OPD: Demystifying Warm-up for On-policy Distillation
Aug 7, 2026Tao Liu, Taiqiang Wu, Mao Zheng +5Uni-Opd Warm Starts
1Tsinghua University · 2The University of Hong Kong · 3LLM Department, Tencent
Physics-Informed Neural Operator for Warm-Starting Background-Decomposed and Preconditioned PSFD: Enabling Scalable 3-D EUV Mask Simulation
Jul 28, 2026Doyun Kim, Werner GillijnsLithography Warm Starts
IMEC, Kapeldreef 75, 3001 Leuven, Belgium
Guiding Posterior Exploration with Optimizer-Derived Geometry
Jul 28, 2026Moritz Schlager, Emanuel Sommer, Thomas Möllenhoff +1Adaptive Sampling Bayesian Neural Networks
TUM, MCML · LMU Munich, MCML · RIKEN AIP
WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training
Jul 12, 2026Jianhao Ma, Yuxin ChenBatch Warm Starts
Department of Statistics and Data Science, University of Pennsylvania
When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model
Jun 19, 2026Carson Rodrigues, Oysturn Vas, Isaiah Abner DCosta +1Mllm-Based Warm Starts
Celabe · University of Waterloo · University of Queensland (EAIT)
From Compression to Deployment: Real-Time and Energy-Efficient FastGRNN on Ultra-Constrained Microcontrollers
Jun 15, 2026Emre Can KizilatesCortex Neural Network Inference
Independent Researcher, Izmir, Turkey
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Jun 15, 2026Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6Curvature-Aware Spectral Framework Transformer Architectures
Pluralis Research
Stage-1 Controls the Entropy Regime, Not the Outcome
Jun 8, 2026Jianxiong ShenMulti-Stage Training Warm Starts
Architecture Shapes Transfer Specificity in Implicit Neural Representations
Jun 5, 2026D Yang EngMultilayer Perceptrons Reynolds-Averaged Navier-Stokes
Finite-Iteration Local Dynamics and Warm Starts for Alternating Power Iteration in Spiked Tensor PCA
Jun 2, 2026Yanjin Xiang, Zhihua ZhangNewton Warm Starts
Peking University
Physics-Informed Modeling and Control of Emergent Behaviors in Robot Swarms
Jun 1, 2026Zixuan Jin, Wenzhuo Zhang, Shuxian Quan +4Swarms Physics-Informed Learning
School of Computer and Communication Engineering, University of Science and Technology Beijing, 100083, Beijing, China. · Shunde Innovation School, University of Science and Technology Beijing, 528399, Foshan Guangdong, China.
Information-Directed Offline-to-Online Reinforcement Learning
May 28, 2026Keru ChenModel-Based Reinforcement Learning Mutual Information
School of Electrical, Computer and Energy Engineering, Arizona State University
How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
May 25, 2026Zhitao Zhu, Xili Wang, Shizhe Wu +2Large Language Model Training Batch
Peking University · Meituan
Human-in-the-Loop Contextual Bandits for Short-Term Rental Dynamic Pricing: Structural Equivalence of Historical Warm-Up and Approval-Gated Live Learning
May 22, 2026Oleg MiroshnichenkoDynamic Pricing Contextual Bandit Framework
When is Warmstarting Effective for Scaling Language Models?
May 13, 2026Neeratyoy Mallik, Maciej Janowski, Johannes Hog +4Warm Starts Large Language Model Training
University of Freiburg · Zuse School ELIZA · University of Technology Nuremberg +3
FreeMOCA: Memory-Free Continual Learning for Malicious Code Analysis
May 10, 2026Zahra Asadi, Haeseung Jeon, Sohyun Han +3Malware Replay-Based Continual Learning
Division of Artificial Intelligence & Software Ewha Womans University · Department of Computer Science University of Texas at El Paso
Continuous First, Discrete Later: VQ-VAEs Without Dimensional Collapse
May 7, 2026Xinyu Zhao, Nikita Karagodin, Hamed Hassani +3Vector-Quantized Variational Autoencoder Codebooks
MIT
ZAYA1-8B Technical Report
May 6, 2026Robert Washbourne, Rishi Iyer, Tomas Figliolia +15Deepseek Mixture-Of-Experts
Zyphra San Francisco, CA
First Shape, Then Meaning: Efficient Geometry and Semantics Learning for Indoor Reconstruction
May 5, 2026Remi Chierchia, Léo Lebrat, David Ahmedt-Aristizabal +3Neural Reconstruction Efficient Planning
School of Electrical Engineering & Robotics, Queensland University of Technology, Australia · Imaging and Computer Vision Group, CSIRO Data61, Australia
$μ$ pscaling small models: Principled warm starts and hyperparameter transfer
Feb 11, 2026Yuxin Ma, Nan Chen, Mateo Díaz +3Hyperparameter Neural Network Training
Department of Applied Mathematics and Statistics Johns Hopkins University
Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay
Feb 6, 2026Binghui Li, Zilin Wang, Fengling Chen +3Batch Scaling Laws
Center for Machine Learning Research, Peking University · School of Mathematical Sciences, Peking University · AI for Science Institute, Beijing
Warm-Starting Iterative Gaussian Processes for Faster Sequential Inference
Nov 20, 2025Alan Yufei Dong, Jihao Andreas Lin, José Miguel Hernández-LobatoGaussian Process Warm Starts
Department of Engineering, University of Cambridge
Why Do We Need Warm-up? A Theoretical Perspective
Oct 3, 2025Foivos Alimisis, Rustem Islamov, Aurelien LucchiWarm Starts Batch
University of Basel, Switzerland

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

Qiulin Shang, Binyu Wang, Yongqi Qiao, Songde Rao+2

Peking University · Nanjing University

Batch Large Language Model Pretraining Warm Starts Autoregressive Language Models

Open paper AskPDF arXiv

Learning-rate (LR) scheduling plays a central role in large language model (LLM) pretraining, yet current practice still relies heavily on hand-crafted heuristics such as Warmup-Cosine-Decay and Warmup-Stable-Decay. Because these schedules are fixed in advance, they cannot adapt to evolving optimization dynamics. Online learned scheduling within the Learning to Optimize (L2O) framework offers a dynamic alternative, but remains brittle at LLM scale due to noisy signals, delayed feedback, and the risk of catastrophic divergence. We propose SOLAR (State-driven Online Learning rAte scheduleR), a stabilized framework for reliable online LR adaptation. SOLAR uses a base schedule as a reference and learns bounded, state-dependent residual corrections for individual parameter groups. Each correction re-anchors to the base at every step, allowing the policy to adapt the LR without relearning the warmup-decay profile. A lightweight state representation and progress-aware reward guide online learning, while a Circuit-Breaker restores training after rare unsafe actions. Across autoregressive language-model pretraining, SOLAR improves final perplexity over tuned static schedules and automatic LR tuners for dense models from 60M to 1B, AdamW and Muon, and two MoE settings up to 3B. Matched 130M controls show that adding base anchoring and action bounds improves a global PPO controller from 27.09 to 23.74 final PPL, while group-wise control reaches 22.87 on the same two seeds. A residual policy trained on a 60M proxy can also be frozen and reused at larger dense scales without target PPO updates, remaining effective across a fourfold base-LR range. These results establish SOLAR as a practical learned LR controller for LLM pretraining.

Momentum

Related topics

Latest papers 26