Representation Learning

Follow this topic Ask about this topic

Representation learning studies how to turn raw data—such as images, text, graphs, time series, and sensor signals—into features that make important patterns easier to use. Common approaches include contrastive and masked prediction, autoencoders, predictive latent models, sparse coding, and methods that impose structure such as geometry, hierarchy, or disentangled subspaces. Good representations can reduce label needs and support transfer across tasks, domains, and modalities, with evaluations spanning linear probes, retrieval, classification, and downstream control. A central tension is that compression and invariance can discard information a later task needs, so no single objective or representation geometry works best across settings.

Latest papers

799 papers130 papers in the last four weeks, up 97% on the four weeks before. 0.8% of all new papers.

Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization
May 7, 2026Andy Zeyi Liu, Elliot Paquette, John SousRepresentation Learning Decoder-Only Language Models
Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA +1
Feature Starvation as Geometric Instability in Sparse Autoencoders
May 6, 2026Faris Chaudhry, Keisuke Yano, Anthea MonodRepresentation Learning Sparse Autoencoders
Imperial College London South Kensington Campus, London SW7 2AZ, UK · Institute for Statistical Mathematics 10-3 Midori-cho, Tachikawa Tokyo 190-8562, Japan
Reference-based Category Discovery: Unsupervised Object Detection with Category Awareness
May 6, 2026Yichen Li, Qiankun Liu, Ying FuRepresentation Learning Object Detection
Department of Computer Science Beijing Institute of Technology · University of Science and Technology Beijing
Velox: Learning Representations of 4D Geometry and Appearance
May 6, 2026Anagh Malik, Dorian Chan, Xiaoming Zhao +3Representation Learning Point Cloud Learning
Apple · University of Toronto
PHALAR: Phasors for Learned Musical Audio Representations
May 5, 2026Davide Marincione, Michele Mancusi, Giorgio Strano +4Audio Representation Learning Representation Learning
Department of Computer Science, Sapienza University of Rome, Italy · Moises Systems, Inc. · Paradigma, Inc.
Graph Neural Network based Hierarchy-Aware Embeddings of Knowledge Graphs: Applications to Yeast Phenotype Prediction
May 5, 2026Filip Kronström, Alexander H. Gower, Daniel Brunnsåker +2Graph Neural Networks Representation Learning
Department of Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg, Sweden · Department of Life Sciences, Chalmers University of Technology, Sweden · Department of Industrial Biotechnology, KTH Royal Institute of Technology, Sweden +1
Disentangling Shared and Task-Specific Representations from Multi-Modal Clinical Data
May 5, 2026He Lyu, Huolin Zeng, Junren Wang +7Disentangled Representation Learning Representation Learning
Department of Anesthesiology and West China Biomedical Big Data Center West China Hospital, Sichuan University, Chengdu, China · Med-X Center for Informatics, Sichuan University, Chengdu, China · Department of National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, China +4
Understanding Self-Supervised Learning via Latent Distribution Matching
May 5, 2026Fabian A Mikulasch, Friedemann ZenkeRepresentation Learning Self-Supervised Learning
Friedrich Miescher Institute for Biomedical Research, 4056 Basel, Switzerland · Faculty of Science, University of Basel, 4033 Basel, Switzerland
Evaluating Tabular Representation Learning for Network Intrusion Detection
May 4, 2026Muhammad Usman Butt, Andreas Hotho, Daniel SchlörRepresentation Learning Network Intrusion Detection
Julius-Maximilians-Universität Würzburg Würzburg, Germany · Data Science Chair Center for Artificial Intelligence and Data Science Julius-Maximilians-Universität Würzburg Würzburg, Germany
HELIX: Hybrid Encoding with Learnable Identity and Cross-dimensional Synthesis for Time Series Imputation
May 4, 2026Fengming Zhang, Wenjie Du, Huan Zhang +2Time Series Imputation Representation Learning
Center for Energy and Environmental Policy Research, Beijing Institute of Technology, Beijing 100081, China · School of Management, Beijing Institute of Technology, Beijing 100081, China · Beijing Lab for System Engineering of Carbon Neutrality, Beijing 100081, China +1
Benchmarking Wireless Representations: High-Dimensional vs. Compressed Embeddings for Efficiency and Robustness
May 3, 2026Murilo Batista, Shirin Salehi, Saeed Mashdour +3Representation Learning Embedding Compression
Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University, Aachen, Germany · Centre for Telecommunications Studies, Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro 22541-041, Brazil
ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs
May 3, 2026Marah Halawa, Olaf HellwichRepresentation Learning Algorithmic Fairness
Technische Universität Berlin
Behavior-Grounded Lane Representation Learning for Multi-Task Traffic Digital Twins
May 3, 2026Rei Tamaru, Pei Li, Bin RanRepresentation Learning Digital Twins
Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, WI, USA · Department of Civil and Architecture Engineering and Construction Management, University of Wyoming, Laramie, WY, USA
Latent State Design for World Models under Sufficiency Constraints
May 3, 2026Keon Woo KimRepresentation Learning Predictive Representation Learning
Om Labs
Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations
May 2, 2026Pratyush Acharya, Nuraj Rimal, Habish DhakalTransformer Interpretability Representation Learning
MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation
May 2, 2026Pham Khanh Chi, Quoc Phong Dao, Thuat Nguyen +3Representation Learning Language Model Distillation
1Hanoi University of Science and Technology · 2Monash University · University of Oregon
Continuous Temporal Representations of Event-Based Signals via Interference-Based Wave Modeling
May 2, 2026Magnus BengtssonRepresentation Learning Surface Electromyography
Department of Engineering, University of Borås, Borås, Sweden
Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probing
May 1, 2026Leonid BedratyukRepresentation Learning Neural Representation Geometry
SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings
May 1, 2026Priyam Mazumdar, Yurii Halychanskyi, Steven Guo +2Representation Learning TTS Synthesis
University of Illinois Urbana-Champaign, USA · National Center for Supercomputing Applications, USA
SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control
May 1, 2026Stavros Orfanoudakis, Pedro P. VergaraRepresentation Learning Policy Optimization
Delft University of Technology, The Netherlands
Physiology-Aware Masked Cross-Modal Reconstruction for Biosignal Representation Learning
May 1, 2026Hao Zhou, Simon A. Lee, Cyrus Tanade +12Representation Learning Cross-Modal Representation Learning
Samsung Research America · Algorithm Team, Digital Health Lab, Samsung Research America · The Pennsylvania State University +1
Near-optimal and Efficient First-Order Algorithm for Multi-Task Learning with Shared Linear Representation
May 1, 2026Shihong Ding, Fangyu Du, Cong FangShared Representation Learning Representation Learning
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University, China · School of Mathematical Sciences, Peking University, China · Institute for Artificial Intelligence, Peking University, China
Combined Dictionary Unfolding Network with Gradient-Adaptive Fidelity for Transferable Multi-Source Fusion
May 1, 2026Ge Luo, Jun-Jie Huang, Qi Yu +6Multi-Source Image Fusion Representation Learning
College of Computer Science and Technology, National University of Defense Technology, Changsha, China · College of Surveying and Geoinformatics, Tongji University, Shanghai, China · School of Computer and Information, Hefei University of Technology, Hefei 230009, China
Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements
Apr 30, 2026Genki Kinoshita, Shu Nakamura, Ryo Kawahara +3Latent Action Learning Representation Learning
1Kyoto University · 2Kyoto Institute of Technology · 3RIKEN
Interpretable EEG Microstate Discovery via Variational Deep Embedding: A Systematic Architecture Search with Multi-Quadrant Evaluation
Apr 29, 2026Saheed Faremi, Andrea Visentin, Luca LongoVariational Autoencoders Electroencephalography
School of Computer Science and Information Technology, University College Cork, Western Road, Cork, T12 XF62, Ireland · Artificial Intelligence and Cognitive Load Research Lab, University College Cork, Ireland · Insight RI Research Centre for Data Analytics, University College Cork, Ireland
Robust Representation Learning through Explicit Environment Modeling
Apr 28, 2026Yuli Slavutsky, David M. BleiRepresentation Learning Domain Generalization
Department of Statistics Columbia University · Departments of Statistics and Computer Science Columbia University
Learning Generalizable Multimodal Representations for Software Vulnerability Detection
Apr 28, 2026Zeming Dong, Yuejun Guo, Qiang Hu +5Representation Learning Software Vulnerability Detection
University of Luxembourg · 2Luxembourg Institute of Science and Technology · 3Tianjin University
MIPIC: Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chaining
Apr 27, 2026Phung Gia Huy, Hai An Vu, Minh-Phuc Truong +4Representation Learning Sentence Embeddings
1Hanoi University of Science and Technology · University of Oregon · 3Monash University
Gromov-Wasserstein Methods for Multi-View Relational Embedding and Clustering
Apr 26, 2026Rafael Pereira Eufrazio, Eduardo Fernandes Montesuma, Charles Casimiro CavalcanteRepresentation Learning Multidimensional Scaling
Instituto Federal de Educação, Ciência e Tecnologia do Ceará, Canindé, Ceará, Brazil · Federal University of Ceara, Fortaleza, Ceará, Brazil · Sigma Nova, Paris, France
Transformer as an Euler Discretization of Score-based Variational Flow
Apr 26, 2026Huadong LiaoRepresentation Learning Transformer

May 7, 2026J/K move · Enter open · S save

Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

Andy Zeyi Liu, Elliot Paquette, John Sous

Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA+1

Representation Learning Decoder-Only Language Models Neural Network Interpretability Neural Network Training Dynamics LLM Training

Open paper AskPDF arXiv

Training loss and throughput can hide distinct internal representation in language-model training. To examine these hidden mechanics, we use spectral measurements as practical and operational diagnostics. Using a controlled family of decoder-only models adapted from the modded NanoGPT codebase, we introduce an empirical protocol based on activation covariance and per-sample gradient SVD spectra. This dual-view reveals three empirical findings and one mechanistic explanation. First, batch size acts as a latent determinant of representation geometry: runs that reach equal loss settle into systematically distinct activation spectra. Second, the activation covariance tail measured early in training reliably forecasts downstream token efficiency. Third, movement of the activation spectrum head (leading modes), together with gradient spectra, characterizes underlying learning-dynamics changes, separating learning-side architectural improvements from primarily execution-side gains. These predictive and diagnostic signals persist across the 12-, 36-, and 48-layer model tiers. Finally, a mechanistic model proves the main observations and explains how activation covariance spectra correlate with task-aligned feature learning.