Feature Learning

Feature learning, the process by which machine learning models automatically discover useful representations from raw data, aims to improve model performance and generalization. Current research focuses on understanding the dynamics of feature learning in various architectures, including convolutional neural networks (CNNs), transformers, and multi-layer perceptrons (MLPs), often employing techniques like sparse modeling, attention mechanisms, and low-rank adaptations to enhance efficiency. These advancements are impacting diverse fields, from image analysis and natural language processing to biomedical signal processing and anomaly detection, by enabling more accurate and efficient models for complex tasks.

Papers

October 31, 2024

Provable Benefit of Cutout and CutMix for Feature Learning
Junsoo Oh, Chulhee Yun
Feature Learning Two Layer Neural Network Cut and Approximate Provable Benefit Multi Label Feature Selection

October 23, 2024

Feature Learning in Attention Mechanisms Is More Compact and Stable Than in Convolution
Baiyuan Chen
Vision Transformer Attention Mechanism Direct Convolution Convolutional Layer Feature Learning

October 11, 2024

Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data
Binghui Li, Yuanzhi Li
Native Robustness Adversarial Example General Analysis Adversarial Training Adversarial Perturbation Feature Learning Structured Data Model Robustness

October 8, 2024

MTFL: Multi-Timescale Feature Learning for Weakly-Supervised Anomaly Detection in Surveillance Videos
Yiling Zhang, Erkut Akdag, Egor Bondarev, Peter H. N. De With
Feature Learning Video Anomaly Surveillance Video Anomaly Detection Performance Weakly Supervised Anomaly Detection Spatial Temporal Video

September 29, 2024

Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
Bikang Pan, Wei Huang, Ye Shi
Vision Language Model Practical Method Theoretical Analysis Feature Learning Vision Language Foundation Model

September 28, 2024

Sparse Modelling for Feature Learning in High Dimensional Data
Harish Neelam, Koushik Sai Veerella, Souradip Biswas
Dimensionality Reduction Feature Extraction High Dimensional Data Feature Learning Sparse Model Variable Selection Sparse Optimization

September 27, 2024

Feature Estimation of Global Language Processing in EEG Using Attention Maps
Dai Shimizu, Ko Watanabe, Andreas Dengel
Attention Map Feature Learning EEG Feature Task Based

September 26, 2024

How Feature Learning Can Improve Neural Scaling Laws
Blake Bordelon, Alexander Atanasov, Cengiz Pehlevan
Scaling Law Feature Learning Reproducing Kernel Hilbert Space Neural Scaling Law

September 23, 2024

Automatic Feature Learning for Essence: a Case Study on Car Sequencing
Alessio Pellegrino, Özgür Akgün, Nguyen Dang, Zeynep Kiziltan, Ian Miguel
Case Study Feature Learning Constraint Programming Main CAR Sale Factor Modeling Language Constraint Model Constraint Solver Constraint Language

September 9, 2024

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
Siyu Chen, Heejune Sheen, Tianhao Wang, Zhuoran Yang
Large Language Model Transformer Megatron Decepticons Context Learning Feature Learning Training Dynamic Transformer Performance Head Transformer

August 16, 2024

AdaRank: Disagreement Based Module Rank Prediction for Low-rank Adaptation
Yihe Dong
Multimodal Model Feature Learning AdaPtation Performance

July 31, 2024

DD-rPPGNet: De-interfering and Descriptive Feature Learning for Unsupervised rPPG Estimation
Pei-Kai Huang, Tzu-Hsien Chen, Ya-Ting Chan, Kuan-Wen Chen, Chiou-Ting Hsu
Feature Learning Remote Photoplethysmography Catastrophic Interference rPPG Estimation

July 16, 2024

Learning Multi-view Anomaly Detection
Haoyang He, Jiangning Zhang, Guanzhong Tian, Chengjie Wang, Lei Xie
Multi View Feature Learning Task Relatedness Neighborhood Attention

July 13, 2024

Imbalanced Graph-Level Anomaly Detection via Counterfactual Augmentation and Feature Learning
Zitong Wang, Xuexiong Luo, Enfeng Song, Qiuqing Bai, Fu Lin
Graph Neural Network Feature Learning Anomaly Detection Performance Anomalous Graph Graph Level Anomaly Detection Counterfactual Augmentation

July 1, 2024

On Feature Learning for Titi Monkey Activity Detection
Aditya Ravuri, Jen Muir, Neil D. Lawrence
Feature Learning Environmental Sound Speech Detection Raw Audio

June 27, 2024

Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think
Luke Sernau
Neural Tangent Kernel Feature Learning Sequential Optimization Infinite Width Width Network

June 12, 2024

Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis
Paolo Bonetti, Alberto Maria Metelli, Marcello Restelli
Multi Task Learning Feature Learning Feature Aggregation Aggregated Data Two Stage Aggregation Task Clustering

June 5, 2024

Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers
Federico Bassetti, Marco Gherardi, Alessandro Ingrosso, Mauro Pastore, Pietro Rotondo
Deep Learning Convolutional Layer Structured Output Feature Learning Deep Linear Deep Architecture

June 3, 2024

How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networks
Mo Zhou, Rong Ge
Gradient Descent Gradient Based Neural Tangent Kernel Feature Learning Two Layer Neural Network Local Convergence

May 27, 2024

Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets
Arthur Jacot, Alexandre Kaiser
Feature Learning Low Dimensional Representation Dimensional Input Fully Connected Neural Network Leaky ReLU Hamiltonian Function Bottleneck Structure

Feature Learning

Papers

Provable Benefit of Cutout and CutMix for Feature Learning

Feature Learning in Attention Mechanisms Is More Compact and Stable Than in Convolution

Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data

MTFL: Multi-Timescale Feature Learning for Weakly-Supervised Anomaly Detection in Surveillance Videos

Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

Sparse Modelling for Feature Learning in High Dimensional Data

Feature Estimation of Global Language Processing in EEG Using Attention Maps

How Feature Learning Can Improve Neural Scaling Laws

Automatic Feature Learning for Essence: a Case Study on Car Sequencing

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers

AdaRank: Disagreement Based Module Rank Prediction for Low-rank Adaptation

DD-rPPGNet: De-interfering and Descriptive Feature Learning for Unsupervised rPPG Estimation

Learning Multi-view Anomaly Detection

Imbalanced Graph-Level Anomaly Detection via Counterfactual Augmentation and Feature Learning

On Feature Learning for Titi Monkey Activity Detection

Infinite Width Models That Work: Why Feature Learning Doesn't Matter as Much as You Think

Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis

Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers

How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networks

Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets