Model Pretraining

Follow this topic Ask about this topic

Recent momentum

-38%

31 papers in the last 28 days · 0.5% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

12 new papers

A weekly snapshot of new work published in Model Pretraining.

Period ending 2026-09-14

11 new papers

A weekly snapshot of new work published in Model Pretraining.

Period ending 2026-09-07

13 new papers

A weekly snapshot of new work published in Model Pretraining.

402 papers

Latest in Model Pretraining

Browse all topics →

HRM-Text: Efficient Pretraining Beyond Scaling
May 20, 2026Guan Wang, Changling Liu, Chenyu Wang +6Large Language Model Pretraining Model Pretraining
Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing
May 19, 2026Valentina Njaradi, Clémentine Dominé, Rachel Swanson +2Model Pretraining Unlabeled Data
Quantifying the Pre-training Dividend: Generative versus Latent Self-Supervised Learning for Time Series Foundation Models
May 19, 2026Noam Major, Kathy Razmadze, Yoli ShavitTime Series Foundation Models Self-Supervised
A Bitter Lesson for Data Filtering
May 19, 2026Christopher Mohri, John Duchi, Tatsunori HashimotoLarge Behavior Model Model Pretraining
MAM-CLIP: Vision-Language Pretraining on Mammography Atlases for BI-RADS Classification
May 19, 2026Halil Ibrahim Gulluk, Olivier GevaertMammography Medical Vision-Language Models
Generation of Heterogeneous PET Images from Uniform Organ Activity Maps Using a Pretrained Domain-Adapted Diffusion Model
May 18, 2026Suya Li, Kaushik Dutta, Debojyoti Pal +2Positron Emission Tomography Medical Image Generation
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
May 18, 2026Ali Zindari, Rotem Mulayoff, Sebastian U. StichLow-Rank Adaptation Finetuning
Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic
May 18, 2026Thomas Sommariva, Francesca Morandi, Simone Calderara +1Finetuning Model Pretraining
Protein Fold Classification at Scale: Benchmarking and Pretraining
May 18, 2026Dexiong Chen, Andrei Manolache, Mathias Niepert +1Protein Data Bank Protein
TabH2O: A Unified Foundation Model for Tabular Prediction
May 18, 2026Pascal Pfeiffer, Dmitry Gordeev, Mathias Müller +6Tabular Data Model Pretraining
Generating Pretraining Tokens from Organic Data for Data-Bound Scaling
May 18, 2026Zichun Yu, Chenyan XiongSynthetic Data Model Pretraining
AMO: Adaptive Muon Orthogonalization
May 18, 2026Xinlin Zhuang, Panyi Ouyang, Yichen Li +7Muon Optimizer Adam
Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
May 18, 2026Athanasios Glentis, Dawei Li, Chung-Yiu Yau +1Large Language Model Pretraining Adam
Surface-Form Neural Sparse Retrieval: Robust Fuzzy Matching for Industrial Music Search
May 18, 2026Paul Greyson, Zhichao Geng, Wei Zhang +1Retrieval Layer Conversational Music Recommendation
MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation
May 18, 2026Ronyu Zhang, Aosong Cheng, Gaole Dai +8Continual Test-Time Adaptation Model Pretraining
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization
May 17, 2026Gunjan Balde, Soumyadeep Roy, Mainack Mondal +1Large Language Model Adaptation Summarization
Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density
May 17, 2026Jingru Fei, Kun Yi, Alex Xing Wang +3Time Series Foundation Models Model Pretraining
How Do Electrocardiogram Models Scale?
May 17, 2026Jiawei Li, Fabio Bonassi, Ming Jin +4Electroencephalography Foundation Models Electronic Health Record Foundation Models
Extending Pretrained 10-Second ECG Foundation Models to Longer Horizons
May 16, 2026Wei Tang, Jinpei Han, Kangning Cui +10Pediatric-Adult Electrocardiogram Alignment Electrocardiogram Classification
Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space
May 15, 2026Valeria Ruscio, Eli-Shaoul Khedouri, Keiran ThompsonAsymmetry Model Pretraining
ITGPT: Generative Pretraining on Irregular Timeseries
May 15, 2026Antoine Honoré, Ming XiaoTime-Series Classification Time Series
Neutral-Reference Prompting for Vision-Language Models
May 15, 2026Senmao Tian, Xiang Wei, Shunli ZhangTransfer Learning Model Pretraining
Toward LLMs Beyond English-Centric Development
May 15, 2026Sho Takase, Ukyo HondaMultilingual Large Language Models Large Language Model Bias
Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Study
May 15, 2026Alexander Hackett, Srikanth Thudumu, Ginny Fisher +1Frozen Encoder Contrastive Learning
TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale
May 14, 2026Anurup GanguliModel Pretraining Catastrophic Forgetting
SpectraFlow: Unifying Structural Pretraining and Frequency Adaptation for Medical Image Segmentation
May 14, 2026Zhiquan Chen, Haitao Wang, Guowei Zou +1Medical Image Segmentation Model Pretraining
DUET: Dual-Paradigm Adaptive Expert Triage with Single-cell Inductive Prior for Spatial Transcriptomics Prediction
May 13, 2026Junchao Zhu, Ruining Deng, Junlin Guo +11Spatial Transcriptomics Single-Cell Rna Sequencing
Enhanced and Efficient Reasoning in Large Learning Models
May 13, 2026Leslie G. ValiantModel Pretraining Principles
Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study
May 13, 2026Wuao Liu, Mustafa Chasmai, Subhransu Maji +1Passive Acoustic Monitoring Acoustic Representation
Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
May 13, 2026Ruan Visser, Trienko Grobler, Marcel DunaiskiSubword Tokenization Model Pretraining
Adaptive Kernel Density Estimation with Pre-training
May 13, 2026Ruitong Zhang, Ke DengDensity Ratio Estimation Distributional Learning
Scaling Laws for Mixture Pretraining Under Data Constraints
May 12, 2026Anastasiia Sedova, Skyler Seto, Natalie Schluter +1Model Pretraining Scaling Laws
Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs
May 12, 2026Guinan Su, Yanwu Yang, Xueyan Li +1Parallel Artificial Intelligence Agents
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
May 12, 2026Rian Touchent, Eric de la ClergerieCausal Attention Model Pretraining
Anomaly-Aware Vision-Language Adapters for Zero-Shot Anomaly Detection
May 12, 2026Muhammad Aqeel, Maham Nazir, Uzair Khan +2Zero-Shot Anomaly Detection Vision-Language Model Adaptation
A Transfer Learning Evaluation of Deep Neural Networks for Image Classification
May 12, 2026Nermeen Abou Baker, Nico Zengeler, Uwe HandmannTransfer Learning Model Pretraining
Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training
May 12, 2026Yu-Hang Wu, Qin-Yuan Liu, Qiu-Yang Zhao +3Model Pretraining Circuit Tracing
MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification
May 12, 2026Bo Zheng, Yudong Chen, Zihua Xiong +4Tabular Learning Model Pretraining
Inducing Spatial Locality in Vision Transformers through the Training Protocol
May 11, 2026Eduardo Santiago Toledo, Asael Fabian MartínezVision Transformer Imagenet
PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models
May 11, 2026Xinyu Guo, Bin Xie, Wei Chai +4Vision-Language-Action Models Vision-Language Model Adaptation
Locking Pretrained Weights via Deep Low-Rank Residual Distillation
May 11, 2026Keitaro Sakamoto, Pierre Ablin, Federico Danieli +1Model-Agnostic Defense Model Weights
On Improving Graph Neural Networks for QSAR by Pre-training on Extended-Connectivity Fingerprints
May 11, 2026Sam Money-Kyrle, Markus Dablander, Thierry Hanser +3Graph Neural Networks Structure-Based Drug Design
Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm
May 11, 2026Haoyu Wang, Yifan Shang, Zhongxiang Sun +3Model Pretraining Procedural Knowledge
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Generative Pretrained Transformers Large Language Model Decoder
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
May 11, 2026Xu Guo, Runyu Peng, Jian Tong +4Large Language Model Pretraining Model Pretraining
Annotations Mitigate Post-Training Mode Collapse
May 11, 2026Jacob Mitchell Springer, Madhu Advani, Lukas Aichberger +7LLM Post-Training Methods Model Pretraining
MambaNetBurst: Direct Byte-level Network Traffic Classification without Tokenization or Pretraining
May 11, 2026Gayan K. Kulatilleke, Siamak Layeghy, Mahsa Baktashmotlagh +1Network Traffic Bidirectional Mamba
SABER: A Scalable Action-Based Embodied Dataset for Real-World VLA Adaptation
May 10, 2026Narsimha Menga, Parikshit Sakurikar, Amirreza Rouhi +6Robotic Datasets Egocentric Video
FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning
May 10, 2026Xing Han, Shravan Chaudhari, Tanvi Ranade +2Mixture-of-Experts Models Unseen-Task Adaptation
Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
May 9, 2026Christopher M. Bryant, Hao LiuScaling Laws Large Models
Predicting Large Model Test Losses with a Noisy Quadratic System
May 9, 2026Chuning Li, Chris J. MaddisonLarge Models Loss Function
From pre-training to downstream performance: Does domain-specific pre-training make sense?
May 9, 2026Felix KronesMultimodal Pretraining Model Pretraining
SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization
May 9, 2026Yan Sun, Guoxia Wang, Jinle Zeng +6Large Language Model Pretraining Model Pretraining
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Large Language Model Compression Large Language Model Pretraining
Prototype Guided Post-pretraining for Single-Cell Representation Learning
May 8, 2026Sachini Weerasekara, Natasha Darras, Sagar Kamarthi +2Spatial Transcriptomics Model Pretraining
When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretraining
May 8, 2026Ivan Karpukhin, Andrey SavchenkoModel Pretraining Hyperparameter Optimization
Hierarchical Mixture-of-Experts with Two-Stage Optimization
May 8, 2026Gleb Molodtsov, Alexander Miasnikov, Aleksandr BeznosikovMixture-of-Experts Models Load Balancing
HTPO: Towards Exploration-Exploitation Balanced Policy Optimization via Hierarchical Token-level Objective Control
May 8, 2026Xincheng Yao, Ruoqi Li, Cheng Chen +4Reinforcement Learning With Verifiable Reward Large Language Model Reasoning
Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation
May 8, 2026Junjie Yu, Yue Wang, Zihan Deng +3Model Pretraining Finetuning
Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models
May 8, 2026Bincheng Peng, Guang Li, Ping Liu +2Chain-Of-Thought Distillation Imagenet

May 20, 2026 · cs.CLJ/K move · Enter open · S save

HRM-Text: Efficient Pretraining Beyond Scaling

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou+5

1Sapient Intelligence · 2MIT

The current pretraining paradigm for large language models relies on massive compute and internet-scale raw text, creating a significant barrier to foundational research. In contrast, biological systems demonstrate highly sample-efficient learning through multi-timescale processing, such as the functional organization of the frontoparietal loop. Taking this as inspiration, we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recurrent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving execution layers. To stabilize this deep recurrence for language modeling, we introduce MagicNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pretraining, we train exclusively on instruction-response pairs using a task-completion objective and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500 budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x less estimated compute than standard baselines, HRM-Text performs competitively with 2-7B parameter open models. These results demonstrate that co-designing architectures and objectives can radically reduce the compute-to-performance ratio, making pretraining from scratch accessible to the broader research community.

Large Language Model Pretraining Model Pretraining Neural Scaling Laws

Open paper Ask about it PDF