Model Pretraining

Follow this topic Ask about this topic

Recent momentum

-38%

31 papers in the last 28 days · 0.5% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

12 new papers

A weekly snapshot of new work published in Model Pretraining.

Period ending 2026-09-14

11 new papers

A weekly snapshot of new work published in Model Pretraining.

Period ending 2026-09-07

13 new papers

A weekly snapshot of new work published in Model Pretraining.

402 papers

Latest in Model Pretraining

Browse all topics →

TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
May 7, 2026Shang-Ling Hsu, Mark Tenzer, Cyrus Shahabi +1Mobility Model Pretraining
EMO: Pretraining Mixture of Experts for Emergent Modularity
May 7, 2026Ryan Wang, Akshita Bhagia, Sewon MinMixture-Of-Expert Language Models Mixture-of-Experts Models
Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less
May 7, 2026Yuxing Liu, Jianyu Wang, Tong ZhangSupervised Fine-Tuning Model Pretraining
DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency
May 7, 2026Shuyang Jiang, Nan Yu, Yiming Zhang +2Architecture-Agnostic Loss Function Model Pretraining
Efficient Pre-Training with Token Superposition
May 7, 2026Bowen Peng, Théo Gigant, Jeffrey QuesnelleModel Pretraining Loss Function
3D MRI Image Pretraining via Controllable 2D Slice Navigation Task
May 7, 2026Yu Wang, Qingchao ChenMagnetic Resonance Imaging Model Pretraining
Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models
May 7, 2026Alex O. Davies, Telmo de Menezes e Silva Filho, Nirav AjmeriTabular Foundation Models Tabular Data
TinySSL: Distilled Self-Supervised Pretraining for Sub-Megabyte MCU Models
May 7, 2026Bibin WilsonSelf-Supervised Chain-Of-Thought Distillation
On the Blessing of Pre-training in Weak-to-Strong Generalization
May 7, 2026Wei Yao, Wang Zhaoyang, Gengze Xu +5Weak Supervision Model Pretraining
In-Context Positive-Unlabeled Learning
May 7, 2026Siyan Liu, Yi Chang, Manli Cheng +2Positive--Unlabeled Learning Unlabeled Data
Learngene Search Across Multiple Datasets for Building Variable-Sized Models
May 6, 2026Boyu Shi, Junbo Zhou, Chang Liu +3Model Pretraining Vision Transformer
Memory-Efficient Continual Learning with CLIP Models
May 5, 2026Ryan King, Gang Li, Bobak Mortazavi +1Domain Incremental Learning Frozen Contrastive Language-Image Pre-Training Visual Encoder
Distribution-Free Pretraining of Classification Losses via Evolutionary Dynamics
May 5, 2026Meng Xiang, Yan PeiCross-Entropy Losses Loss Function
ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
May 5, 2026Jiaxi Li, Lu Yin, Li Shen +5Low-Rank Factorization Model Pretraining
Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discovery
May 5, 2026Bo Ye, Kai Gan, Tong Wei +1Generalized Category Discovery Model Pretraining
CERSA: Cumulative Energy-Retaining Subspace Adaptation for Memory-Efficient Fine-Tuning
May 5, 2026Jingze Ge, Xue Geng, Yun Liu +6Parameter-Efficient Fine-Tuning Model Pretraining
LUMINA: A Grid Foundation Model for Benchmarking AC Optimal Power Flow Surrogate Learning
May 4, 2026Hongwei Jin, Keunju Song, Zeeshan Memon +5Ac Optimal Power Flow Accurate Surrogate Models
Spoken Language Identification with Pre-trained Models and Margin Loss
May 3, 2026Zhihua Fang, Liang He, Weiwu JiangModel Pretraining Encoding Models
Fine-Tuning Pre-Trained Code Models for AI-Generated Code Detection
May 2, 2026Jany-Gabriel Ispas, Sergiu NisioiMachine-Generated Text Detection Code Quality
The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles
May 2, 2026Hiun Kim, Tae Kwan Lee, Taeryun WonLarge Language Model Pretraining Model Pretraining
Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders
May 2, 2026Ailiang Lin, Zhuoyun Li, Keyu Mao +2Domain-Invariant Text Embeddings In-Context Learning
Developing a Strong Pre-Trained Base Model for Plant Leaf Disease Classification
May 2, 2026David J. RichterTomato Diseases Transfer Learning
Rhamba: Region-Aware Hybrid Attention-Mamba Framework for Self-Supervised Learning in Resting-State fMRI
May 2, 2026Ruthwik Reddy Doodipala, Pankaj Pandey, Pratheek Eranki +3Brain Magnetic Resonance Imaging Self-Supervised
LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
May 1, 2026Shashank Kapadia, Deep Naryan Mishra, Sujal Reddy Alugubelli +4Chain-Of-Thought Distillation Model Pretraining
Let ViT Speak: Generative Language-Image Pre-training
May 1, 2026Yan Fang, Mengcheng Lan, Zilong Huang +7Large Vision Language Models Vision Encoders
Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies
May 1, 2026Yi Wang, Xinchen Li, Pengwei Xie +13Flow-Matching Vision-Language-Action Robot Policies
Alethia: A Foundational Encoder for Voice Deepfakes
Apr 30, 2026Yi Zhu, Brahmi Dwivedi, Jayaram Raghuram +1Audio Deepfake Detection Deepfake Detection
Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training
Apr 30, 2026Jasmine Brazilek, Juliana SeawellLLM Post-Training Methods Model Pretraining
Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
Apr 30, 2026Hyeonseo Jang, Jaebyeong Jeon, Joong-Won Hwang +1Prompt Tuning Vision-Language Model Adaptation
A generalised pre-training strategy for deep learning networks in semantic segmentation of remotely sensed images
Apr 30, 2026Yuan Fang, Yuanzhi Cai, Jagannath Aryal +4Remote Sensing Image Segmentation Remote Sensing
UniBCI: Towards a Unified Pretrained Model for Invasive Brain-Computer Interfaces
Apr 30, 2026Binjie Hong, Rui Xiong, Liyuan Han +1Brain-Computer Interface Neural Recordings
Comparative Analysis of AutoML and BiLSTM Models for Cyberbullying Detection on Indonesian Instagram Comments
Apr 29, 2026Raihana Adelia Putri, Aisyah Musfirah, Anggi Puspita Ningrum +3Indonesian E-Commerce Reviews Bidirectional Long Short-Term Memory
Staged Factorial Screening for Budget-Constrained Micro-Pretraining
Apr 27, 2026Felipe Chavarro PolaniaToken Budgets Exploratory Factor Analysis
Training Machine Learning Models on Encrypted Data: A Privacy-Preserving Framework using Homomorphic Encryption
Apr 25, 2026Alexandre Marques, Beatriz Sá, Rui Botelho +1Homomorphic Encryption Privacy-Preserving Framework
CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging
Apr 24, 2026Ashwin Kumar, Robbie Holland, Corey Barrett +8Medical Vision-Language Models Medical Image Generation
PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training
Apr 23, 2026Harsh Kumar, Rahul Maity, Tanmay Joshi +4Language Model Evasion Attacks Multiple Attack Surfaces
Generalizing Numerical Reasoning in Table Data through Operation Sketches and Self-Supervised Learning
Apr 23, 2026Hanjun Cho, Gahyun Yoo, Hanseong Kim +1Table Reasoning Datasets Tabular Data
Decoupled DiLoCo for Resilient Distributed Pre-training
Apr 23, 2026Arthur Douillard, Keith Rush, Yani Donchev +14Model Pretraining Large Language Model Pretraining
Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training
Apr 23, 2026Yoshinori NomuraLarge Language Model Pretraining Model Pretraining
How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Apr 22, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisRecurrence Large Models
Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance
Apr 22, 2026Amandeep Kaur, Mirali Purohit, Gedeon Muhawenayo +2Geospatial Foundation Models Model Pretraining
Serialisation Strategy Matters: How FHIR Data Format Affects LLM Medication Reconciliation
Apr 22, 2026Sanjoy PatorModel Pretraining Handoff
Projected Gradient Unlearning for Text-to-Image Diffusion Models: Defending Against Concept Revival Attacks
Apr 22, 2026Aljalila Aladawi, Mohammed Talha Alam, Fakhri KarrayText-To-Image Diffusion Models Exact Unlearning
Image Generators are Generalist Vision Learners
Apr 22, 2026Valentin Gabeur, Shangbang Long, Songyou Peng +22Visual Generation Image Generation
Mask World Model: Predicting What Matters for Robust Robot Policy Learning
Apr 21, 2026Yunfan Lou, Xiaowei Chi, Xiaojie Zhang +9Robot Policies Future Video Prediction
Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?
Apr 21, 2026Niclas Doll, Jasper Schulze Buschhoff, Shalaka Satheesh +3Large Language Model Adaptation Model Pretraining
Generative Texture Filtering
Apr 21, 2026Rongjia Zheng, Shangwei Huang, Lei Zhu +2Model Pretraining Supervised Fine-Tuning
Learning Posterior Predictive Distributions for Node Classification from Synthetic Graph Priors
Apr 21, 2026Jeongwhan Choi, Jongwoo Kim, Woosung Kang +1Graph Neural Networks Graph Representations
Post-training is (Massive) Supervised Learning
Apr 20, 2026Michael Hassid, Yossi Adi, Roy SchwartzLarge Language Model Post-Training LLM Post-Training Methods
2D Pre-Training for 3D Pose Estimation
Apr 19, 2026Liyao Jiang, Ruichen Chen, Keith G. Mills3D Pose Estimator Model Pretraining
Channel-Level Semantic Perturbations: Unlearnable Examples for Diverse Training Paradigms
Apr 18, 2026Bo Wang, Jia Ni, Mengnan Zhao +2Model Pretraining Unlabeled Data
Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencoder
Apr 18, 2026Bowen Peng, Yongxiang Liu, Jie Zhou +4Multimodal Pretraining Synthetic Aperture Radar
Incentivizing Parametric Knowledge via Reinforcement Learning with Verifiable Rewards for Cross-Cultural Entity Translation
Apr 18, 2026Jiang Zhou, Xiaohu Zhao, Xinwei Wu +8Reinforcement Learning With Verifiable Reward Model Pretraining
Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs
Apr 17, 2026Jun-Liang Lin, Kamesh Madduri, Mahmut Taylan KandemirMulti-Gpu Systems Parallel
In Search of Lost DNA Sequence Pretraining
Apr 17, 2026Zhijiang Tang, Jiaxin Qi, Yan Cui +3Model Pretraining Sequence
PolarMAE: Efficient Fetal Ultrasound Pre-training via Semantic Screening and Polar-Guided Masking
Apr 17, 2026Meng Lv, Yapeng Li, Hang Su +2Prenatal Ultrasound Multimodal Pretraining
Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization
Apr 16, 2026Fu Feng, Yucheng Xie, Ruixiao Shi +2Model Pretraining Model Weights
Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining
Apr 14, 2026Junfeng Xia, Wenhao Ye, Xuanye Pan +3Brain Model Pretraining
Using Deep Learning Models Pretrained by Self-Supervised Learning for Protein Localization
Apr 13, 2026Ben Isselmann, Dilara Göksu, Heinz Neumann +1Microscopy Self-Supervised
To View Transform or Not to View Transform: NeRF-based Pre-training Perspective
Mar 30, 2026Hyeonjun Jeong, Juyeb Shin, Dongsuk KumNeural Radiance Fields Autonomous Driving Perception

May 7, 2026 · cs.LGJ/K move · Enter open · S save

TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond

Shang-Ling Hsu, Mark Tenzer, Cyrus Shahabi, Khurram Shafique

University of Southern California · Novateur Research Solutions

Human mobility differs from text and from generic time series in three structural ways: visits are tuple-valued events whose meaning depends on the joint distribution over location, time, and activity; users carry persistent signatures across trajectories; and visits are not independent across users, since co-location at shared places is a primary signal. Existing pre-training recipes for mobility import objectives from language modeling, treating trajectories as sentences and visits as tokens, an analogy that fails against each of the three properties above. These properties define a broader class, multi-entity spatiotemporal event streams (MESES), spanning enterprise authentication logs, electronic health records, and other event-stream domains where entities share infrastructure, schedules, or contexts. We make the properties precise as three axioms that any pre-training framework for MESES should satisfy, and introduce TraXion, whose objectives and architecture are jointly designed to meet them. A single TraXion checkpoint per dataset beats task-specific baselines on every task across six public mobility datasets covering anomaly detection, next-POI recommendation, next-visit prediction, and social-link prediction. The same recipe, applied unchanged to enterprise authentication logs and ICU mortality prediction, matches or exceeds prior work on both, showing that event streams from domains as different as mobility, security, and healthcare can be modeled under a single framework.

Mobility Model Pretraining Next Activity Prediction

Open paper Ask about it PDF