Feature Learning

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 72

All topics

Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks
Sep 28, 2026Etienne Boursier, Nicolas FlammarionFeature Learning Model Fine-Tuning
INRIA, LMO, Universit´e Paris-Saclay, Orsay, France · EPFL, Lausanne, Switzerland
Hidden not Deleted: How Networks Suppress Entangled Features
Sep 23, 2026Akash Samanta, Manish Pratap Singh, Debasis ChaudhuriConcept Erasure Entanglement
Techno India University Salt Lake, Kolkata, India · DRDO Young Scientist Laboratory - CT Chennai, India
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Sep 22, 2026Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2Neural Tangent Kernel Feature Learning
Arnold-Sommerfeld-Center for Theoretical Physics, Ludwig-Maximilians-Universität München, Munich, Germany · Department of Applied Physics, Standford University, Standford, CA, USA · Department of Mathematics, University of Cambridge, Cambridge, UK
Structured Features Overfit Where Random Features Grok
Sep 14, 2026Chon-Fai Kam, Miloud Bessafi, Frederic CadetOverparameterization Overfitting
University Paris City & University of Reunion, Paris, France; Dipartimento di Fisica e Chimica · Emilio Segr`e, Universit`a degli Studi di Palermo, Italy · EnergyLab, University of Reunion, Saint-Denis, France +2
Where Decoder Cosine Similarity Fails for SAE Feature Flow Discovery
Sep 14, 2026Hendrik Droste, Christian Medeiros Adriano, Kathrin Korte +1Sparse Autoencoder Features Feature Learning
Hasso Plattner Institute, Germany · IT University of Copenhagen, Denmark
A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagram
Sep 9, 2026Alexis D. PlascenciaSparse Autoencoder Features Phase Transitions
GIFT: Guided Intermediate Feature Training via Action-Oriented Structural Supervision for Robotic Manipulation
Sep 3, 2026Yupeng Zheng, Xiang Li, Songen Gu +11Robotic Manipulation Feature Learning
Institute of Automation, Chinese Academy of Sciences · Tsinghua University · Fudan University +2
Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs
Aug 25, 2026Amirhesam Abedsoltan, Enric Boix-Adsera, Fivos Kalogiannis +1Feature Learning Multilayer Perceptrons
Department of Computer Science and Engineering, UC San Diego · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania · Halıcıoğlu Data Science Institute, UC San Diego
Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention
Aug 11, 2026Vicente OpazoFeature Learning Low-Rank Structure
CENIA
Hidden Gauge Controls Feature Specialization in ReLU Networks
Aug 7, 2026Tongxi WangFeature Learning Rectified Linear Unit Networks
School of Future Technology Southeast University Nanjing, China
Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
Aug 6, 2026Björn Ladewig, Ibrahim Talha Ersoy, Karoline WiesnerFeature Learning Anisotropic Loss Landscapes
Institute of Physics and Astronomy, University of Potsdam
DAS-PMVC: A Framework for Partial Multi-View Clustering via Dual Alignment and Structure Enhancement
Jul 30, 2026Shubin Ma, Liang Zhao, Chuanye He +4Multi-View Multimodal Alignment
Dalian University of Technology Dalian, Liaoning, China · Inspur Group Co., Ltd. Jinan, Shandong, China · China University of Mining and Technology Xuzhou, Jiangsu, China +1
Toward Multi-Modal Deep Learning for Pulmonary Disease Classification: A Texture-Based Machine Learning Pilot Study on Public Chest X-Ray Data
Jul 29, 2026Yogisri Pujitha ChinthotiMultimodal Deep Learning Multimodal Clinical Data
Independent Researcher, Bengaluru, India
Bit-Accurate FPGA Evaluation of Learned Feature Gating in a Fixed-Point Fourier-Feature Automatic Modulation Classifier
Jul 27, 2026Gawthaman Senthilvelan, Luthira AbeykoonField-Programmable Gate Arrays Classifier
University of Toronto · Toronto, Canada
SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignment
Jul 16, 2026Saad Ejaz, Miguel Fernandez-Cortizas, Javier Civera +2Geometric Machine Learning Latent Representation Alignment
Automation and Robotics Research Group, Interdisciplinary Centre for Security, Reliability, and Trust (SnT), University of Luxembourg, Luxembourg · I3A, Universidad de Zaragoza
From Preimage Search To Source-Grounded Feature Inversion
Jul 14, 2026Kaixiang ShuInverse Design Feature Learning
Independent Researcher
The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning
Jul 3, 2026Joan BrunaFeature Learning Chaos
Courant Institute School of Mathematics, Computing and Data Science, New York University
SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model
Jun 29, 2026Tyler LaBonte, Vidya MuthukumarFeature Learning Rectified Linear Unit Networks
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology · School of Electrical and Computer Engineering, Georgia Institute of Technology
From Spatial to Spectral: An Efficient, Frequency-Guided Feature Representation Learner for Small Object Detection
Jun 22, 2026Yuhan Rui, Shihan Qiao, Yibin Lou +7Feature Pyramid Feature Learning
Southern University of Science and Technology, Shenzhen, China · University of Pennsylvania
Dual Dimensionality for Local and Global Attention
Jun 17, 2026Zhiyuan Wang, Xuan Luo, Sirui Zeng +1Transformer Encoder Representational Capacity
UC Santa Barbara
Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
Jun 10, 2026Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky +3Improving Sparse Autoencoders Feature Learning
T-Tech
Bernstein-Schur Kernels: Random Features by Sketched Modulation and Radial Randomization
Jun 8, 2026Taha BouhsineAtom-Averaged Features Kernel Hilbert Spaces
Azetta AI
Muon Learns More Robust and Transferable Features than Adam
Jun 8, 2026Tianyu Ruan, Fengzhuo Zhang, Shuche Wang +1Adam Feature Learning
Yale University · University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, CAS +1
Interactions Between Crosscoder Features: A Compact Proofs Perspective
Jun 8, 2026Dmitry Manning-Coe, Thomas Read, Anna Soligo +4Unsupervised Dictionary Learning Cross-Encoder
1Anthony J. Leggett Institute for Condensed Matter Theory · 2MATS · 3UK AI Security Institute (AISI) +4
Low-Frequency Shortcuts in Texture-Driven Visual Learning
Jun 2, 2026Utku Şirin, Cathy Hou, David Alvarez-Melis +1Graph-Coherent Low-Frequency Ones Feature Learning
Harvard University
Unlocking Feature Learning in Gated Delta Networks at Scale
Jun 2, 2026Yifeng Liu, Quanquan GuTransfer Learning Feature Learning
University of California, Los Angeles
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
Jun 2, 2026Seungu Kang, Songkuk KimKnowledge Distillation Feature Learning
Yonsei University, Seoul 03722, Republic of Korea
Query Lens: Interpreting Sparse Key-Value Features with Indirect Effects
May 30, 2026Hwiyeong Lee, Ingyu Bang, Uiji Hwang +2Feature Learning Model Activations
1Hanyang University, Seoul, Republic of Korea.
On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders
May 29, 2026Elana Simon, Etowah Adams, James ZouSparse Autoencoder Features Model Activations
1Stanford University · 2Columbia University
Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data
May 28, 2026Collin Cranston, Zhichao Wang, Todd Kemp +1Kernel Method Spectral Decomposition
Department of Mathematics University of California, San Diego, USA · ICSI and Department of Statistics University of California, Berkeley, USA · ICSI, LBNL and Department of Statistics University of California, Berkeley, USA
Feature Learning in Wide Neural Networks under $μ$ P: Identifiability and Sparse-Dictionary Decomposition of the Mean-Field Limit
May 23, 2026Akmal XodarevTwo-Layer Neural Networks Feature Learning
Independent Researcher · Tashkent, Uzbekistan
Feature Lottery? A Bifurcation Theory of Concept Emergence
May 22, 2026Fuming YangFeature Learning Bifurcation
MIT
Aligned Training: A Parameter-Free Method to Improve Feature Quality and Stability of Sparse Autoencoders (SAE)
May 18, 2026Michał Brzozowski, Neo Christopher ChungSparse Autoencoder Features Feature Learning
Samsung AI Center, Warsaw, Poland · University of Warsaw, Poland
VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation
May 18, 2026Linan ZHU, Zihao Zhai, Xiao Han +4Emotion Recognition Automatic Speaker Verification
1Zhejiang University of Technology · 2ETH Zurich, Switzerland
Canonical Regularisation of Wide Feature-Learning Neural Networks
May 18, 2026George Whittle, Pranav Vaidhyanathan, Juliusz Ziomek +2Feature Learning Regularization
Department of Engineering Science University of Oxford
Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent
May 18, 2026Behrad Moniri, Hamed HassaniTwo-Layer Neural Networks Feature Learning
University of Pennsylvania, Philadelphia, PA, USA
How does feature learning reshape the function space?
May 18, 2026João Lobo, Bruno Loureiro, Long Tran-Than +1Feature Learning Two-Layer Neural Networks
Department of Computer Science, University of Warwick, United Kingdom · Departement d’Informatique, École Normale Supérieure, PSL & CNRS · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China. Part of work was done at Department of Computer Science, and Centre for Discrete Mathematics and its Applications (DIMAP), University of Warwick, United Kingdom
Toy Combinatorial Interpretability Models Reveal Lottery Tickets in Early Feature Space
May 18, 2026Alon Bebchuk, Nir ShavitLotteries Feature Learning
Tel-Aviv University · MIT · Red Hat AI
GQA-μP: The maximal parameterization update for grouped query attention
May 14, 2026Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7Model Weights Parameter-Efficient Fine-Tuning Methods
UC Davis & MBZUAI IFM · USC & MBZUAI IFM · USC +4
Learning with Shallow Neural Networks on Cluster-Structured Features
May 14, 2026Elisabetta Cornacchia, Laurent MassouliéLow-Dimensional Structure Feature Learning
INRIA, DI/ENS, PSL.
Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model
May 14, 2026Arie Wortsman-Zurich, Hugo Tabanelli, Yatin Dandi +2Feature Learning Scaling Laws
Département d’Informatique, Ecole Normale Supérieure, PSL & CNRS · Information Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
May 13, 2026Yatin Dandi, Matteo Vilucchio, Luca Arnaboldi +2Feature Learning Deep Learning
Information Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)
FeatCal: Feature Calibration for Post-Merging Models
May 13, 2026Yanggan Gu, Shuo Cai, Zihao Wang +7Continual Model Merging Direct Feature Fusion
The Hong Kong Polytechnic University (PolyU) · The Chinese University of Hong Kong · PolyU-Daya Bay Technology and Innovation Research Institute
The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge
May 13, 2026Ryoya Awano, Taiji SuzukiFeature Learning Stochastic Gradient Descent
University of Tokyo · Center for Advanced Intelligence Project, RIKEN
Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2
May 13, 2026Stuart Bladon, Brinnae BentHuman Visual Cortex Functional Magnetic Resonance Imaging
Duke University, Durham, NC, USA.
AGOP as Explanation: From Feature Learning to Per-Sample Attribution in Image Classifiers
May 12, 2026Raj Kiran Gupta KatakamGradient-Weighted Class Activation Mapping Feature Learning
Credit Karma, San Francisco, CA, USA
Fused Gromov-Wasserstein Distance with Feature Selection
May 12, 2026Harlin Lee, Ying Yu, Mingxin Li +1Gromov--Wasserstein Feature Selection
School of Data Science and Society · University of North Carolina at Chapel Hill · Division of Computational Social Science +5
FeatMap: Understanding image manipulation in the feature space and its implications for feature space geometry
May 11, 2026Elias B. Krey, Nils Neukirch, Nils StrodthoffFeature Learning Equivariant Learning
Division AI4Health Carl von Ossietzky Universität Oldenburg Oldenburg, Germany
Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks
May 11, 2026Minh-Toan Nguyen, Jean BarbierFeature Learning Two-Layer Neural Networks
ICTP Trieste, Italy
Scalable Gaussian process inference via neural feature maps
May 11, 2026Anthony StephensonGaussian Process Kernel Method
School of Mathematics University of Bristol
Selection Plateau and a Sparsity-Dependent Hierarchy of Pruning Features
May 10, 2026Guangqi Li, Yongxin LiSparsity Structured Pruning
Zaozhuang University, Zaozhuang, Shandong, China
The Geometric Structure of Models Learning Sparse Data
May 8, 2026Thomas Walker, T. Mitchell Roddenberry, Ahmed Imtiaz Humayun +2Feature Learning Deep Network
Rice University · Rice University† · Now at Google Research +1
Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer
May 8, 2026Clarissa Lauditi, Cengiz Pehlevan, Blake BordelonDeep Network Neural Network
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Center for Mathematical Sciences and Applications, Harvard University +1
Features have life history. And we should care
May 7, 2026Philipp Stecher, Sandro Radovanović, Vlasta Sikimić +1Feature Learning Life
University of Tübingen · University of Belgrade · Eindhoven University of Technology
The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks
May 7, 2026Taehun Cha, Daniel Beaglehole, Adityanarayanan Radhakrishnan +1Feature Learning Deep Network
Department of Mathematics Korea University · Computer Science and Engineering UC San Diego · MIT Mathematics Broad Institute of MIT and Harvard
EvoTSC: Evolving Feature Learning Models for Time Series Classification via Genetic Programming
Apr 28, 2026Xuanhao Yang, Bing Xue, Mengjie ZhangTime-Series Classification Genetic Algorithms
Centre for Data Science and Artificial Intelligence & School of Engineering and Computer Science, Victoria University of Wellington, PO Box 600, Wellington 6140, New Zealand
Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking
Apr 28, 2026Yongzhong XuFeature Learning Grokking
Diffusion-Guided Feature Selection via Nishimori Temperature: Noise-Based Spectral Embedding
Apr 27, 2026Vasiliy S. Usatyuk, Denis A. Sapozhnikov, Sergey I. EgorovFeature Selection Feature Learning
R&D Department T8 LLC Moscow, Russia · Department of Computer Science South-West State University Kursk, Russia
Convergent Evolution: How Different Language Models Learn Similar Number Representations
Apr 22, 2026Deqing Fu, Tianyi Zhou, Mikhail Belkin +2Language Modeling Feature Learning
University of Southern California · UC San Diego
Structural Instability of Feature Composition
Apr 18, 2026Yunpeng ZhouFeature Learning Steering
Whiteknights House, Whiteknights, Reading, RG6 6UR

Sep 28, 2026, stat.MLJ/K move · Enter open · S save

Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks

Etienne Boursier, Nicolas Flammarion

INRIA, LMO, Universit´e Paris-Saclay, Orsay, France · EPFL, Lausanne, Switzerland

Feature Learning Model Fine-Tuning Continual Fine-Tuning Pretraining Rectified Linear Unit Networks Learn+1

Open paper AskPDF arXiv

Fine-tuning pre-trained models on specialized tasks with scarce data is central to modern deep learning. Despite its empirical success, theoretical understanding of fine-tuning remains limited. We introduce a Gaussian multi-index setting to study fine-tuning from pre-trained weights, where the teacher network has

m+1

features,

m

of which are learned during pre-training and one of which must be learned during fine-tuning. For two-layer ReLU networks, we show that two-timescale training, i.e., updating the outer weights infinitely faster than the hidden ones, learns the new task-specific feature while preserving the pre-trained ones in the model representation. Moreover, only

\mathcal{O}(d)

fine-tuning samples are required for this recovery, independently of the number of pre-trained features. In contrast, with random initialization, the same number of samples is insufficient to recover the target parameters. Our results therefore demonstrate that pre-training can induce an implicit bias with a clear statistical advantage over random initialization, enabling feature learning from scarce fine-tuning data.

Momentum

Related topics

Latest papers 72