Hyperparameter

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 50

All topics

How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
Oct 5, 2026Matthias Vigl, Nikita Pond, Jackson Barr +5Model Size Hyperparameter
Technical University of Munich · SLAC National Accelerator Laboratory · University College London +4
On Hyperparameter Tuning on the Test Set
Oct 5, 2026Matteo Fregonara, Tom Viering, Jan van GemertHyperparameter
Delft University of Technology
How Does Local Landscape Geometry Evolve in Language Model Pre-Training?
Sep 30, 2026Zhanpeng Zhou, Yuhan Sun, Bingrui Li +4Large Language Model Pretraining Batch
Shanghai Jiao Tong University · Tsinghua University · Peking University
Hyperparameter Scaling Laws Across MoE Sparsity
Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Hyperparameter Mixture-Of-Experts Architectures
Ling Team, Ant Group
TeMo: Temperature Modulation for Multimodal Contrastive Learning
Sep 7, 2026Dhimitrios Duka, Bernt Schiele, Hilde Kuehne +1Multimodal Contrastive Learning Contrastive Learning
MPI for Informatics, SIC · Tuebingen AI Center/University of Tuebingen · MIT-IBM Watson AI Lab
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
Sep 1, 2026Zhiliang Chen, Sebastian Ament, David Eriksson +4Hyperparameter Large Language Model Training
Meta · National University of Singapore · Atomic Machines; work done at Meta
Convergence rates for the RMSprop optimizer with full control of the hyperparameters
Aug 31, 2026Steffen Dereich, Arnulf JentzenAdam Stochastic Convex Optimization
Small-Scale Experiments: Are We There Yet?
Aug 12, 2026Nicholas Lourie, Kyunghyun Cho, Karen Ullrich +1Scaling Laws Model Size
1FAIR at MSL Meta · 2New York University
Dynamics Models for Offline Hyperparameter Selection in Real-World RL
Aug 11, 2026Jordan Coblin, Han Wang, Martha White +1Model-Based Reinforcement Learning Hyperparameter
University of Alberta · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair at Amii
Predicting Deep Neural Network Training Outcomes from Early Training Telemetry
Aug 4, 2026Ranjita Naik, Anh D. Nguyen, Pankaj Kumar SinghNeural Network Training Epoch
Georgia Institute of Technology
On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds
Aug 4, 2026Jiaxin Deng, Junbiao PangSharpness-Aware Minimization Hessian
Beijing University of Technology
Towards Practical Algorithm Selection for Unsupervised Domain Adaptation in Medical Imaging
Jul 30, 2026Yiheng Xiong, Luisa Gallée, Daniel Santak Wolf +2Domain Adaptation Medical Imaging Datasets
Section of Experimental Radiology, Ulm University Medical Center, Germany · Visual Computing Group, Ulm University, Germany
Efficient Heteroscedastic Bayesian Optimization for Risk-Aware AutoRL
Jul 29, 2026Mingxuan Che, Tsung-Yuan Tseng, Theresa Eimer +2Multi-Objective Bayesian Optimization Algorithms Bayesian Optimization
Leibniz University Hannover · CARIAD SE · University of Wuppertal +2
Variational Quantum Conditional Boltzmann Machines for Time-Series Forecasting: Architectures, Symmetric Hyperparameter Evaluation, and a Nonlinear Benchmark
Jul 27, 2026Gerhard Hellstern, Danyal Maheshwari, Martin Zaefferer +2Hybrid Quantum-Classical Pipeline Time Series Forecasting
Centre of Finance DHBW Stuttgart Stuttgart, Germany · Zentrum für Digitale Innovationen DHBW Ravensburg Ravensburg, Germany · DATEV eG Nürnberg, Germany
Scale Weight Decay and Train Better
Jul 26, 2026Anuj ApteWeight Decay Batch
Global Technology Applied Research, JPMorganChase, New York, NY 10001
Do emulated quantum circuits change what CNNs look at? Performance and explainability comparison in medical image classification
Jul 23, 2026Guillermo Rubiños Rodríguez, Martín Ottavianelli, Mateo Alonso +4Hybrid Quantum-Classical Pipeline Quantum Circuits
Instituto Tecnol´ogico de Galicia, Cant´on Grande 9, Planta 3, 15003 A Coru˜na, Spain · Centro Singular de Investigaci´on en Tecnolox´ıas Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Santiago de Compostela, 15782, Spain · Department of Neuroradiology, LMU University Hospital, Ludwig Maximilian University of Munich, Marchioninistraße 15, 81377 Munich, Germany +1
Effects of width-dependent model hyperparameters and $\ell_2$ -regularization on the loss landscape of two-layer ReLU networks
Jul 18, 2026Haruka Eshima, Makoto YamadaRectified Linear Unit Networks Anisotropic Loss Landscapes
Okinawa Institute of Science and Technology, Okinawa, Japan
Which Hyperparameters Matter? A Game-Theoretic Framework for Interpretable Hyperparameter Sensitivity Analysis
Jul 17, 2026Nyi Nyi Aung, Heepeom Shin, Abigail Lawlor +1Hyperparameter Shapley Value
Department of Mechanical and Industrial Engineering Louisiana State University Baton Rouge, USA · Department of Electrical and Computer Engineering Louisiana State University Baton Rouge, USA
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
Jul 11, 2026Petros Karypis, Sean O'Brien, Shreyas Kadekodi +2Rotary Position Embedding Positional Encoding
UC San Diego · Independent Researcher
Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures
Jul 9, 2026Hafsa Mateen, Radu Timofte, Dmitry IgnatovBatch Neural Architecture Search
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Beyond Heuristic Tuning: Power-Calibrated LLM Watermarking
Jul 6, 2026Xiaopu Wang, Zelin He, Chengyuan Liu +1Large Language Model Watermarks Watermarking
Department of Statistics, Pennsylvania State University
TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning
Jul 6, 2026Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev +1Tabular Learning Ensemble
1Yandex · 2HSE University
Hyperparameter Transfer in Graph Neural Networks
Jul 6, 2026Gage DeZoort, Boris HaninGraph Neural Networks Hyperparameter
Department of Operations Research and Financial Engineering · School of Engineering and Applied Science · Princeton University +1
MMAO-Cls: Metabolic Multi-Agent Optimization for Joint Feature Selection and Classifier Tuning
Jul 1, 2026Jinliang Xu, Liping MaFeature Selection General Grid Search Framework
independent researcher in Beijing, China · Department of Disease Control and Prevention, The Seventh Medical Center of Chinese PLA General Hospital, Beijing, China
Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
Jun 26, 2026Ruixuan Huang, Yipei Wang, Wenyi Fang +7Large Language Model Training Large Language Models Fail
1HKUST · 2Huawei · 3Independent Researcher
How Good Can Linear Models Be for Time-Series Forecasting?
Jun 25, 2026Lang Huang, Jinglue Xu, Luke DarlowTime Series Forecasting Time Series
1Sakana AI, Tokyo, Japan · 2National Institute of Informatics, Japan
Solve for the Hyperparameter, Skip the Search: Kolmogorov-Optimal Scaling Laws for Spline Regression
Jun 22, 2026Yong Yi Bay, Kathleen A. YearickKernel Ridge Regression General Grid Search Framework
PhD, University of Illinois at Urbana-Champaign
Dual-Network PINNs for Optimal Control: A Reproducible Benchmark on the Mass-Spring-Damper System
Jun 13, 2026Abdeladhim Tahimi, Rinaldo Vieira da Silva JuniorParametric Physics-Informed Neural Network Optimal Control
Centro de Engenharias e Ciˆencias Agr´arias, Universidade Federal de Alagoas, Brazil
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
Jun 4, 2026Yongwei Zhou, Juncheng Diao, Junlin Shang +2Hyperparameter Pretraining
MeiTuan · University of Chinese Academy of Sciences · Harbin Institute of Technology
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
May 22, 2026Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +3Transformer Architectures Mixture-Of-Experts
Adobe Research
Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate
May 20, 2026Dayal Singh Kalra, Maissam BarkeshliHyperparameter Large Language Model Training
Department of Computer Science, University of Maryland, College Park · Department of Physics, University of Maryland, College Park · Joint Quantum Institute, University of Maryland, College Park +1
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrLarge Language Model Benchmarks LLM Inference Optimization
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
May 19, 2026Zhiyu Xu, Lean Wang, Yuanxin Liu +5Cross-Modal Vision-Language Foundation Models
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · The University of Hong Kong · WeChat AI, Tencent Inc., China
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
May 13, 2026Paul Jeha, Anastasiia Sedova, Louis Béthune +4Multilingual Language Models Hyperparameter
1Apple · 2DTU
Hyperparameter Transfer for Dense Associative Memories
May 11, 2026Roi Holtzman, Dmitry Krotov, Boris HaninDense Associative Memory Hyperparameter
Rudolf Peierls Centre for Theoretical Physics, University of Oxford, Oxford OX1 3PU, UK · Dynamical Mind, IBM Research · Princeton
Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models
May 8, 2026Fan Zhou, Tim Van de CruysClassifier-Free Guidance Reward Gradients
KU Leuven
Distributionally-Robust Learning to Optimize
May 7, 2026Vinit Ranjan, Jisun Park, Bartolomeo StellatoDistributionally-Robust Optimization Convex Optimization
Department of Operations Research and Financial Engineering, Princeton University. · Department of Operations Research and Financial Engineering, Princeton University, and Research Institute of Mathematics, Seoul National University. · Research Institute of Mathematics, Seoul National University.
A Comprehensive Analysis of Tokenization and Self-Supervised Learning in End-to-End Automatic Speech Recognition applied on French Language
May 5, 2026Thibault Bañeras-Roux, Mickael Rouvier, Jane Wottawa +1Automatic Speech Recognition Subword Tokenization
Nantes University LS2N Nantes, France · Avignon University LIA Avignon, France · Le Mans University LIUM Le Mans, France
Enhancing RL Generalizability in Robotics through SHAP Analysis of Algorithms and Hyperparameters
May 4, 2026Lingxiao Kong, Cong Yang, Oya Deniz Beyan +1Shapley Additive Explanations Hyperparameter
Fraunhofer Institute for Applied Information Technology FIT, Germany · Soochow University, China · University of Cologne, Germany +1
Large margin classifier with graph-based adaptive regularization
May 3, 2026Vítor M. Hanriot, Turíbio T. Salis, Luiz C. B. Torres +2Classifier Class Imbalance
Graduate Program in Electrical Engineering - Universidade Federal de Minas Gerais - Av. Antônio Carlos 6627, 31270-901, Belo Horizonte, MG, Brazil · Department of Computer and Systems - Universidade Federal de Ouro Preto - João Monlevade 35931-022, Brazil.
Learning Rate Transfer in Normalized Transformers
Apr 29, 2026Boris Shigida, Boris Hanin, Andrey GromovBatch Transformer Architectures
Meta Superintelligence Labs · Princeton University
MesonGS++: Post-training Compression of 3D Gaussian Splatting with Hyperparameter Searching
Apr 29, 2026Shuzhao Xie, Junchen Ge, Weixiang Zhang +103D Gaussian Large-Scale 3D Editing Dataset
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Harbin Institute of Technology, Shenzhen, China +6
Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer
Apr 28, 2026Penghao Kuang, Haoyi Wu, Kewei TuTransformer Architectures Hyperparameter
School of Information Science and Technology, ShanghaiTech University
First-Order Softmax Weighted Switching Gradient Method for Distributed Stochastic Minimax Optimization with Stochastic Constraints
Mar 6, 2026Zhankun Luo, Antesh Upadhyay, Sang Bin Moon +1Decentralized Optimization Stochastic Optimization
School of Electrical and Computer Engineering, Purdue University
$μ$ pscaling small models: Principled warm starts and hyperparameter transfer
Feb 11, 2026Yuxin Ma, Nan Chen, Mateo Díaz +3Hyperparameter Neural Network Training
Department of Applied Mathematics and Statistics Johns Hopkins University
Selecting Hyperparameters for Tree-Boosting
Feb 5, 2026Floris Jan Koster, Fabio SigristGradient Boosted Decision Tree General Grid Search Framework
Seminar for Statistics, ETH Zurich, Switzerland
Shapley-Inspired Feature Weighting in $k$ -means with No Additional Hyperparameters
Aug 11, 2025Richard J. Fawley, Renato Cordeiro de AmorimK-Means Shapley Value
School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe, UK.
Homogeneity Bias in Open-Weight LLMs Is Robust to Decoding Hyperparameters
Jan 4, 2025Messi H. J. LeeLarge Language Model Bias Gender
Independent Researcher · Seoul, Republic of Korea
Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates
Date pendingSaumya Goyal, Rohith Rongali, Ritabrata Ray +1Gradient Descent Hyperparameter
Machine Learning Department, Carnegie Mellon University · Electrical and Computer Engineering Department, Carnegie Mellon University.
ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks
Date pendingZan Chaudhry, Naoko MizunoBatch Hyperparameter
Laboratory of Structural Cell Biology, National Institutes of Health, Bethesda, MD, 20892 USA · Ragon Institute, Cambridge, MA, 02139

Oct 5, 2026, hep-exJ/K move · Enter open · S save

How to scale your HEP ML models: A recipe for robust architecture comparisons at scale

Matthias Vigl, Nikita Pond, Jackson Barr, Alexander Froch+4

Technical University of Munich · SLAC National Accelerator Laboratory · University College London+4

Model Size Hyperparameter Mle-Bench Lite

Open paper AskPDF arXiv

Much of the recent progress in machine learning domains such as language models has come from scaling laws that predict performance as a function of training effort. In high-energy physics (HEP) similar behavior has now been observed. To aid further study, we present a systematic procedure to derive robust scaling laws and compare design choices on the relevant budget axes for HEP tasks. We first validate the full scaling trajectory on toy problems and then apply the procedure to multi-task transformers on the ~11 billion-jet ATLAS JetSet2 dataset, in both the compute- and data-constrained regimes. For the latter, we predict, to the best of our knowledge for the first time, the jointly optimal model size, training horizon, learning rate and batch size under early stopping. At compute-optimal scaling, we recover a near-equal

\sqrt{C}

dependence of model and dataset size, and find that auxiliary objectives lower the primary jet-classification loss at equal compute budget. Expanding the inputs toward lower-level data systematically lowers the loss while leaving the scaling exponent nearly unchanged. The onset of the power-law regime is itself set by scale: below a threshold in dataset size the loss carries little information about high-compute scaling, underscoring the value of large, high-quality full-simulation datasets as a foundation for scaling studies and the development of foundation models in HEP.

Momentum

Related topics

Latest papers 50