Pretraining

Ask about this topic

Momentum

60 papers in the last four weeks, up 122% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 428

All topics
CardsList

  1. Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

    Oct 7, 2026Sangyoon Bae, Sk Miraj Ahmed, Shinjae Yoo +1Pretraining

    Interdisciplinary Program in Artificial Intelligence Seoul National University Seoul, 08826, South Korea · Computational Science Initiative Brookhaven National Laboratory Upton, New York, 11973, USA · Department of Psychology Seoul National University Seoul, 08826, South Korea

  2. Sequential Pretraining Favors Large Models

    Oct 7, 2026Mohnish Harwani, Yujia ZhengPretrainingModel Size

    Purdue University · University of Illinois Urbana–Champaign

  3. Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems

    Oct 6, 2026Kasper Helverskov Petersen, Rasmus Hannibal Tirsgaard, François R J Cornet +2Joint-Embedding Predictive ArchitecturesPretraining

    Department of Applied Mathematics and Computer Science Technical University of Denmark

  4. Differentially Private Mixing of Public Datasets Improves Private Learning

    Oct 5, 2026Yufei Chen, Tejumade Afonja, Anvith Thudi +1Pretraining

    University of Toronto and Vector Institute, Toronto, Canada · CISPA Helmholtz Center for Information Security, Saarbrucken, Germany

  5. EMG-FM-Bench: A Comprehensive Benchmark for Foundation Model Transfer and Adaptation on Electromyography

    Oct 5, 2026Tianhao Wu, Xu Wu, Amirmohammad Radmehr +4Electroencephalography Foundation ModelsElectrocardiogram Foundation Models

    University of Georgia · University of Oklahoma · University of Massachusetts Amherst

  6. Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

    Oct 5, 2026Hong Ha Le, Jackie Lok, Atsushi Nitanda +1Gumbel-Softmax RelaxationPretraining

    National University of Singapore · Princeton University · Nanyang Technological University

  7. RepICL: Reusable In-Context Prediction Across Heterogeneous Representation Spaces

    Oct 5, 2026Yu-Hsiang Liu, Kuan-Yu Chen, Chih-Sheng Chen +3Few-Shot LearningIn-Context Learning

    SinoPac Holdings, Taipei, Taiwan

  8. What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining

    Oct 4, 2026Yekun Chai, Haoyi XiongEpochPretraining

    FloatAI · ETH Zurich · Independent Researcher

  9. Local Support Learning

    Oct 1, 2026Assaf Ben-Kish, Akarsh Kumar, James Glass +1Catastrophic ForgettingLarge Language Model Memory

    Tel Aviv University · MIT CSAIL

  10. Artifact Annotations Partially Substitute for Per-User Calibration: SAFE-EDA and a Normalization-Controlled Evaluation of Wrist-EDA Affect Recognition

    Oct 1, 2026Haochen Chai, Xinbi Luo, Zining Liu +1Surface ElectromyographyNormalization

    College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China

  11. Localizing Transfer Between Memorization Tasks

    Sep 30, 2026Yimiao Yu, Florentin GuthTransfer LearningPretraining

    Center for Data Science New York University · Center for Computational Neuroscience Flatiron Institute

  12. Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining

    Sep 30, 2026Lukas Thede, Shengzhuang Chen, Stefan Winzeck +3ReplayPretraining

    University of Tübingen, Tübingen AI Center · Helmholtz Munich · Munich Center for Machine Learning (MCML) +3

  13. CORD: Learning Reusable Degradation Representations Across Heterogeneous Physical Systems

    Sep 30, 2026Haibo Li, Zhiguo ZengDegradationPredictive Maintenance

    CentraleSupélec, Université Paris-Saclay

  14. How Does Local Landscape Geometry Evolve in Language Model Pre-Training?

    Sep 30, 2026Zhanpeng Zhou, Yuhan Sun, Bingrui Li +4Large Language Model PretrainingBatch

    Shanghai Jiao Tong University · Tsinghua University · Peking University

  15. Comparative study of adapting pre-trained models for driving behavior video captioning

    Sep 30, 2026Sayak Mallick, Philipp Geiger, Augustin KelavaAutonomous DrivingVision-Language Model Adaptation

    University of Tübingen · Bosch Center for Artificial Intelligence

  16. QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code

    Sep 30, 2026Alexey Chernysh, Orkhan Ekhtibarov, Dmitry ZmitrovichAlgorithmic TradingCode Generation

  17. When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events

    Sep 30, 2026Daniel Schoess, Florian von WangenheimTime Series Foundation ModelsData-Driven Forecasting

    ETH Zurich

  18. Alleviating Hallucination in Reasoning Tasks with Training-Free Uncertainty-Guided Steering

    Sep 30, 2026Litian Liu, Qiqi Hou, Yubing Jian +5Large Language Model HallucinationLarge Language Model Uncertainty

    Qualcomm AI Research · UC Santa Barbara

  19. Lasting Effects of Abstract Pretraining Beyond Perplexity

    Sep 30, 2026Zachary Shinnick, Hemanth Saratchandran, Damien Teney +1Large Language Model PretrainingPretraining

    Australian Institute for Machine Learning (AIML), Adelaide University · Idiap Research Institute · Metacognition AI

  20. What Pretraining and Midtraining Make Learnable from Rewards?

    Sep 29, 2026Chiwun Yang, Xiaoyu LiReinforcement Learning Post-TrainingPretraining

    City University of Hong Kong · University of New South Wales

  21. Rethinking Representations for World-Action Modeling

    Sep 29, 2026Haoyi Jiang, Liu Liu, Xinjiang Wang +12Efficient World-Action ModelPretraining

    Huazhong University of Science & Technology · Horizon Robotics · Fudan University +2

  22. Pretraining Latent Information Feedback Transformers with Teacher Supervision

    Sep 29, 2026Dor Tirosh, Ido Amos, Mor GevaTransformer ArchitecturesPretraining

    Blavatnik School of Computer Science and AI, Tel Aviv University · The Hebrew University of Jerusalem

  23. Flattening the Connectome Spectrum: A Spectral Filter for FC Induces a Pretraining Target for fMRI Encoders

    Sep 29, 2026Giovanni Marraffini, Victoria Shevchenko, Carlo Alberto Barbano +1Functional ConnectivityHuman Connectome Project

    UNITO · MIND

  24. Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing

    Sep 29, 2026Guannan Lai, Han-Jia YeLarge Language Model RoutingLLM Inference Optimization

    School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University

  25. Loss-Guided Pretraining Data Selection for Time-Series Foundation Models

    Sep 29, 2026Yike Li, Shaoxu Song, Jianmin WangTime Series Foundation ModelsPretraining

    Tsinghua University

  26. Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradients

    Sep 29, 2026Aram Davtyan, Pablo Acuaviva, Sebastian Stapf +1Decentralized LearningPretraining

    u b University of Bern Computer Vision Group, Institute of Computer Science University of Bern, Switzerland

  27. FLOORA: A Human-Aligned Domain-Specific Language Model for Architectural Design

    Sep 28, 2026Sahand Rezaei-Shoshtari, Patryk Wozniczka, Shu Ishida +14Floor Plan GenerationLayouts

    Autodesk Research

  28. Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

    Sep 28, 2026Yixin Peng, Er Jin, Diego Collarana +1Relational Deep LearningTemporal Graph Neural Networks

    RWTH Aachen University, Aachen, Germany · Fraunhofer FIT, Sankt Augustin, Germany

  29. Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

    Sep 28, 2026Alexandre Declèves, Etienne Boursier, Nicolas FlammarionPretrainingModel Fine-Tuning

    TML, EPFL, Lausanne, Switzerland · INRIA, LMO, Université Paris-Saclay, Orsay, France

  30. Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks

    Sep 28, 2026Etienne Boursier, Nicolas FlammarionFeature LearningModel Fine-Tuning

    INRIA, LMO, Universit´e Paris-Saclay, Orsay, France · EPFL, Lausanne, Switzerland

  31. SentZero: An Enhanced Sentence-Centric Vision-Language Pretraining for Multi-Task Zero-Shot Chest X-Ray Analysis

    Sep 28, 2026Hangyul Yoon, Hyungyung Lee, Edward Choi +1Chest X-RayRecent Vision-Language Models

    Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST)

  32. Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training

    Sep 28, 2026Junlin Chen, Daize Dong, Huanwei Di +9Block Sparse Flash AttentionMatched Fp16 Intermediate

    Rutgers University · Carnegie Mellon University · Oracle +2

  33. What masking geometry works best for EEG foundation models?

    Sep 27, 2026Pierre Guetschel, Bruno Aristimunha, Yassine El Ouahidi +3Electroencephalography Foundation ModelsPretraining

    Donders Institute for Brain, Cognition and Behaviour Radboud University Nijmegen, The Netherlands · Yneuro, University of California San Diego, Paris, France · Lab-STICC, IMT Atlantique Brest, France +2

  34. Self-Play Pretraining with Zero Data

    Sep 24, 2026Aditya Cowsik, Kfir Dolev, Michael Y. Li +4PretrainingSelf-Play

    Independent Researcher · Tel Aviv University · Stanford University +1

  35. Med-AR: Autoregressive Vision-Language Pretraining for Long-Tailed Chest X-Ray Classification and Uncertainty-Aware Evaluation

    Sep 24, 2026Janhavi Prabhu, Sahil, Akshay V +3Chest X-RayMultimodal Clinical Data

    Qure.ai, Bangalore, India

  36. Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions

    Sep 23, 2026Michael Lawrence Castanares, Princess Ventures, Allan TanComputerized Adaptive TestingPedagogical Frameworks

    Predictive Systems Inc, Ortigas, Pasig City, Philippines, 1605 · Better Labs Oy, Helsinki, Finland

  37. A generalizable structural brain MRI foundation model built through dual-priority federated pretraining

    Sep 23, 2026Zhen Yu, Yang Liu, Xiahai Zhuang +1Functional Magnetic Resonance ImagingWireless Foundation Models

    National Institute of Health Data Science, Peking University, Beijing, 100871, China. · Institute of Medical Technology, Peking University Health Science Center, Beijing, 100871, China. · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, 100871, China. +2

  38. A Scaling Study for fMRI Foundation Models

    Sep 23, 2026Wenhao Ye, Xuanye Pan, Junfeng Xia +3Model SizeFunctional Magnetic Resonance Imaging

    Southern University of Science and Technology · Shenzhen University · SLAI +1

  39. MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction

    Sep 22, 2026Fiona Kekwick, Matthew Baugh, Bernhard Kainz +2Multimodal Clinical DataMultimodal Pretraining

    Department of Computing, Imperial College London, London, UK · FAU Erlangen-Nürnberg, Erlangen, Germany · Department of Brain Sciences, Imperial College London, London, UK +1

  40. Foundation model embeddings capture pre-diagnostic changes on screening mammograms

    Sep 22, 2026Kalina P. Slavkova, Eric Brattain, Aditya Gowd +6MammographyMultidisciplinary Tumor Boards

    HOPPR, Chicago, IL, USA · University of Utah School of Medicine, Salt Lake City, UT, USA · 4DMedical, Woodland Hills, CA, USA +1

  41. Learning to Fluctuate: Statistical Foundations for Causal Tabular Pretraining

    Sep 22, 2026Zhiheng ZhangCausalPretraining

    School of Statistics and Data Science Shanghai University of Finance and Economics

  42. Signed Graph Pre-Training and Prompt Learning

    Sep 22, 2026Zihan Mei, Rong Pan, Yuzhou Chen +1Graph Representation LearningPersistent Homology

    Arizona State University · University of California, Riverside

  43. A Deployment Study of Identity-Gated Drone Gesture Control

    Sep 22, 2026Diyari Mohammed Salih, Ilyes Chaabeni, Naima Ait OufroukhAutonomous DronesAerial Navigation

    1M2 Smart Aerospace and Autonomous Systems, Université Paris-Saclay, Évry, France. · 2IBISC Laboratory, Université Paris-Saclay, Évry, France.

  44. PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics

    Sep 21, 2026Henry KvingePretrainingPermutation

    Pacific Northwest National Laboratory · University of Washington

  45. Toward a foundation model for forest point clouds

    Sep 21, 2026Yuanwen Yue, Stefano Puliti, Damien Robert +7ForestsForest Biomass

    ETH Zurich · University of Oxford · Norwegian Institute of Bioeconomy Research (NIBIO) +1

  46. Video-STLayout Pre-training

    Sep 21, 2026Akash Abdu Jyothi, Greg MoriFine-Grained Video UnderstandingVision Encoders

    School of Computing Science Simon Fraser University Burnaby, Canada

  47. Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

    Sep 20, 2026Fırat Öncel, Salman Hussain Ali, Mirco Ravanelli +2Replay-Based Continual LearningCatastrophic Forgetting

    Concordia University · Mila – Quebec AI Institute · Université de Montréal +3

  48. QUALS: Corpus Equilibrium for Universal Forecasting via Pattern Quantization and Learnability Synchronization

    Sep 17, 2026Yujie Li, Zezhi Shao, Chengqing Yu +7Zero-Shot ForecastingPretraining

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · School of Data Science and Engineering, East China Normal University, Shanghai, China

  49. CSWAM: Better Causal Semantic Representations for Out-of-Distribution Generalization in World Action Models

    Sep 16, 2026Tianbin Liu, Jian Zhu, Taiyi Su +5Efficient World-Action ModelWorld Models

    AIRC, Midea Group · Tongji University

  50. Procedural Pretraining for Molecular Property Prediction

    Sep 15, 2026Moritz Friedemann, Zachary Shinnick, Philip Torr +1Molecular Property PredictionPretraining

    University of Oxford · Adelaide University · Slater Labs

  51. Measuring Annotation Efficiency for Handwritten Devanagari Recognition: Sample-Complexity Curves for Four Pretraining Regimes

    Sep 15, 2026Manglesh Kumar Pandey, Sumit Kumar BanshalHandwritten Text RecognitionPretraining

    Alliance School of Liberal Arts and Sciences, Alliance University Bengaluru, India · Department of Computer Science and Engineering, Alliance University Bengaluru, India

  52. Which Pretext Task Transfers? Self-Supervised Pretraining Objectives for Lung Ultrasound

    Sep 15, 2026Moein Heidari, Junbo Rao, Jai Choraria +3UltrasoundSelf-Supervised Learning

    University of British Columbia, Vancouver, BC, Canada · Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA

  53. Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent

    Sep 14, 2026Alex Buna, Fanghui Liu, Patrick RebeschiniPretrainingModel Fine-Tuning

    Department of Statistics, University of Oxford, UK · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

  54. Parameter-Efficient Adaptation of Pretrained Language Models for Time-Series Forecasting

    Sep 14, 2026Tamanna Kumavat, Georg Brunner, Kyriakos FlourisForecasting BackboneTime Series Forecasting

    University of Zurich, Switzerland · Computer Vision Laboratory, ETH Zurich, Switzerland · MRC Biostatistics Unit, University of Cambridge, United Kingdom

  55. BRIDGE-EEG: Bridging Self-Supervised Pretraining and Efficient Deployment for Cross-Dataset EEG Classification

    Sep 14, 2026Meghna Roy Chowdhury, Chengwei Zhou, Haotian Yu +2Electroencephalography Foundation ModelsElectroencephalography

    Purdue University, Indiana, USA · Case Western University, Ohio, USA

  56. Self-supervised Pre-training Helps Retinal Disease Progression Modelling Most When Data Is Scarce

    Sep 14, 2026Ifeoma Veronica Nwabufo, Julius Gervelmeyer, Sarah Müller +1Self-Supervised LearningLongitudinal Imaging

    Hertie Institute for AI in Brain Health, Faculty of Medicine, University of Tübingen, Germany · Tübingen AI Center, University of Tübingen, Germany

  57. Pre-Trained Low-Rank Tensor Decomposition for Multi-Dimensional Image Recovery

    Sep 14, 2026Bing-Zhang Fu, Zhi-Long Han, Ting-Zhu Huang +2Tensor DecompositionLow-Rank Structure

    School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · Pazhou Laboratory (Huangpu), Guangzhou, Guangdong 510335, China

  58. Towards a knowledge-enhanced single-cell foundation model

    Sep 14, 2026Hanqing Zhang, Jie Bao, Mei Ma +7Regulatory NetworksPretraining

    College of Animal Science and Technology, China Agricultural University, Beijing 100193, China · College of Information and Electrical Engineering, China Agricultural University, Beijing 100193, China

  59. Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

    Sep 14, 2026Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang +3Reinforcement Learning ControlModel Predictive Control

    Honda Research Institute USA, San Jose, CA, USA · Skild AI, San Mateo, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA

  60. Enemray: Toward Capable Language Models for Hassaniya

    Sep 13, 2026Cheikh AhmedLanguage AcquisitionPretraining