Representational Collapse

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 39

All topics

Analysis of Quantized and Efficiently Adapted Protein Language Models
Sep 30, 2026Ilan Yaniv Zeisler, Sebastian Clancy, Pouriya Bayat +8Large Language Model Quantization Language Modeling
Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, ON, Canada. · Department of Electrical and Computer Engineering, University of Toronto, Toronto, Ontario, Canada. · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada. +2
Aligned Data Can Induce Misalignment via Context Confusion
Sep 29, 2026Yavuz Bakman, Duygu Nur Yaldiz, Baris Askin +4Large Language Model Alignment Training-Inference Mismatch
University of Southern California · Carnegie Mellon University · Amazon +1
Early Learning Shapes Later Directions Of Representation Change In Continual Learning
Sep 28, 2026Yuantao Deng, Jinnuo Liu, Kaizhen Tan +1Neural Network Representations Continual Learning
New York University · Xi’an Jiaotong University
$λ$ -JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning
Sep 28, 2026Berker Demirel, Clémentine Dominé, Valentino Maiorca +3Self-Supervised Learning Regularization
Institute of Science and Technology Austria (ISTA) Am Campus 1, 3400 Klosterneuburg, Austria
Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders
Sep 21, 2026Rithin Nagaraj, Rupa Laalasa Oruganti, Prerna Subhashchandra Kunder +1Transformer Architectures Transformer Attention
Department of Computer Science and Engineering PES University Bangalore, India
Misaligned Clinical Risk Classification and Cost Asymmetry in Open-Weight Large Language Models
Sep 21, 2026Star S. D. Liu, Xiyu Ding, Robert B. Barrett +3Risk Trade-Off
Biomedical Informatics and Data Science, Johns Hopkins University School of Medicine, Baltimore, MD, USA · Microsoft, Redmond, WA, USA · National Institutes of Health, Bethesda, MD, USA
How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models
Sep 3, 2026Dun Li Chan, Emily Liu, Niyathi Allu +1Representational Collapse
INTI International College Penang · Independent Researcher · FPT University
Representational separation between unitary and channel quantum generative models via shared classical randomness at shallow depth
Aug 5, 2026Arunava Majumder, Marius Krumm, Hendrik Poulsen Nautrup +1Quantum Circuits Quantum Processor
University of Innsbruck, Department of Theoretical Physics, Technikerstraße 21a, A-6020 Innsbruck, Austria.
Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models
Aug 1, 2026Okan S. Coskun, Florian Rottach, Carsten Eickhoff +1Multiple-Choice Questions Representational Collapse
University of Tubingen · The University of Texas at Austin
The Learning Objective Governs Perceptual Narrowing: A Cross-Lingual, Layer-Wise, Ten-Seed Study of Self-Supervised Speech Encoders
Aug 1, 2026Sejin YooSpeech Encoder Grapheme-To-Phoneme
Independent Researcher
Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering
Jul 29, 2026Yu Wang, Sharon LiMultimodal Model Representational Collapse
Department of Computer Sciences, University of Wisconsin-Madison
Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models
Jul 28, 2026Antyabha Rahman, Akshaj Gurugubelli, Omar Ankit +2Large Reasoning Models Supervised Finetuning
University of New South Wales · Algoverse AI Research · University of Waterloo +1
What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
Jul 25, 2026Chen-Yi Lu, Yueh-Shao Chen, Somali ChaterjiNegation Representational Collapse
Purdue University, West Lafayette, IN 47907, USA
What Does a Chemical Language Model Know About Molecules?
Jun 22, 2026Christian Kenneth, Etowah Adams, Liam Bai +1Molecules Language Modeling
Independent · Department of Systems Biology, Columbia University, New York · Generate:Biomedicines, Massachusetts +1
The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classification
Jun 12, 2026Arnav GuptaSaturation Intrinsic Dimensionality
Independent Researcher, Nepal
Detecting Explanatory Insufficiency in Learned Representations: A Framework for Representational Vigilance
Jun 11, 2026Jacques Raynal, Pierre Slangen, Elsa Raynal +1Representation Learning Representational Collapse
Laboratory of Bioengineering and Nanosciences (LBN), University of Montpellier, France · EuroMov Digital Health in Motion, University of Montpellier, IMT Mines Alès, Alès, France · Certified Sophrologist, Sensorimotor Practice, Montpellier, France +1
When Role-playing, Do Models Believe What They Say?
Jun 9, 2026Benjamin Sturgeon, David Africa, Sid BlackRole-Playing Agents Language Model Computes
MATS
Bootstrap Theory of Representational Emergence: Explanatory Insufficiency as a Driver of Representation Learning and World Models
Jun 5, 2026Jacques Raynal, Pierre Slangen, Elsa Raynal +1Representational Capacity Representational Collapse
Laboratory of Bioengineering and Nanosciences (LBN), University of Montpellier, Montpellier, France · 2EuroMov Digital Health in Motion, University of Montpellier, IMT Mines Alès, Alès, France · 3Certified Sophrologist and Dental Assistant, Sensorimotor Practice, Montpellier, France +1
Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal
Jun 3, 2026Elan BarenholtzSurprisal Language Modeling
Machine Perception & Cognitive Robotics Laboratory Department of Psychology / Center for Complex Systems Florida Atlantic University
Feature Geometry of LoRA Adapters: A Sparse Autoencoder Analysis of Representational Divergence in Fine-Tuned Language Models
May 27, 2026Prasanth K KLow-Rank Adaptation Adapters Sparse Autoencoder Features
Independent AI Safety Researcher
Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagery
May 26, 2026Yifan Jiang, Ruoxi Ning, Sheng Yao +1Representational Collapse Relevance
University of Waterloo · Vector Institute
Feature Lottery? A Bifurcation Theory of Concept Emergence
May 22, 2026Fuming YangFeature Learning Bifurcation
MIT
TextReg: Mitigating Prompt Distributional Overfitting via Regularized Text-Space Optimization
May 20, 2026Lucheng Fu, Ye Yu, Yiyang Wang +4Overfitting Text-Driven
Georgia Institute of Technology · University of Illinois Urbana-Champaign
Mechanistic Interpretability of EEG Foundation Models via Sparse Autoencoders
May 13, 2026William Lehn-Schiøler, Magnus Ruud Kjær, Rahul Thapa +10Electroencephalography Foundation Models Improving Sparse Autoencoders
BrainCapture, Kongens Lyngby, Denmark · DTU Health Tech, Technical University of Denmark, Kongens Lyngby, Denmark · DTU Compute, Technical University of Denmark, Kongens Lyngby, Denmark +5
Towards Visually Grounded Multimodal Summarization via Cross-Modal Transformer and Gated Attention
May 12, 2026Abid Ali, Diego Molla-Aliod, Usman NaseemCross-Modal Attention Summarization
School of Computing, Macquarie University, Sydney, Australia
A Mechanistic Investigation of Supervised Fine Tuning
May 12, 2026Ruhaan ChopraSupervised Finetuning Model Activations
Independent Researcher
XTinyU-Net: Training-Free U-Net Scaling via Initialization-Time Sensitivity
May 10, 2026Alvin Kimbowa, Moein Heidari, David Liu +1U-Net Representational Collapse
School of Biomedical Engineering, The University of British Columbia, Vancouver, Canada · Department of Radiology, The University of British Columbia, Vancouver, Canada · Department of Medicine, The University of British Columbia, Vancouver, Canada
Mechanistic Analysis of Alignment Algorithms in Language Models
May 9, 2026Aarush Sinha, Ishan Garg, Veeraraju Elluru +2Large Language Model Alignment Preference Alignment Learning
University of Copenhagen · Independent · IIT Jodhpur +2
Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs
May 7, 2026Fahd Seddik, Fatemeh FardLatent Thoughts Reasoning Skills
Department of Computer Science University of British Columbia Kelowna, BC, Canada
AHPA: Adaptive Hierarchical Prior Alignment for Diffusion Transformers
May 5, 2026Ruibin Min, Yexin Liu, Aimin Pan +5Diffusion Alignment Diffusion Transformers
Sun Yat-sen University · The Hong Kong University of Science and Technology · VNET Group +2
Lost in State Space: Probing Frozen Mamba Representations
Apr 30, 2026Bhagyashree Wagh, Akash SinghToken Embeddings Recurrent State
University of Washington · University of Florida
When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Tasks
Apr 29, 2026Chung-Hsiang Lo, Lu Li, Diji Yang +4Structured Context Intrinsic Dimensionality
1Northeastern University · University of Pennsylvania · 3UC Santa Cruz +3
Monitoring Neural Training with Topology: A Footprint-Predictable Collapse Index
Apr 28, 2026Alexander KalinowskiNeural Collapse Topology
SUNY Empire
Why Are Some Emotions Harder for LLMs? Uncovering the Causal Mechanisms of Emotion Inference via Sparse Autoencoders
Apr 28, 2026Bangzhao Shu, Arinjay Singh, Mai ElSheriefEmotion Recognition Large Language Models Fail
Northeastern University
Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
Apr 26, 2026Giansalvo CirrincioneMulti-Head Attention Transformer Architectures
aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France
To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
Apr 23, 2026Nevena Lazić, Liam Fowl, András György +1Transformer Encoder Transformer Architectures
REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning
Apr 19, 2026Seungmin Lee, Jeonghwan Lee, Hyunkuk Lim +2Domain-Adapted Diffusion Regularization
PwC Korea GenAI Team, Seoul, South Korea
Decomposing the Depth Profile of Fine-Tuning
Apr 19, 2026Jayadev BillaLayer-Wise Model Fine-Tuning
Can Fine-Tuning Erase Your Edits? On the Fragile Coexistence of Knowledge Editing and Adaptation
Nov 8, 2025Yinjie Cheng, Paul Youssef, Christin Seifert +2Multimodal Knowledge Editing Model Fine-Tuning
School of Computer Science University of Sheffield Sheffield, UK · Mathematics and Computer Science Marburg University Marburg, Germany

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Analysis of Quantized and Efficiently Adapted Protein Language Models

Ilan Yaniv Zeisler, Sebastian Clancy, Pouriya Bayat, Saaim Raad+7

Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, ON, Canada. · Department of Electrical and Computer Engineering, University of Toronto, Toronto, Ontario, Canada. · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada.+2

Large Language Model Quantization Language Modeling Quantized Model Fine-Tuning Protein Representational Collapse

Open paper AskPDF arXiv

Background: Protein language models (PLMs) are increasingly used for sequence generation and property prediction, but their size makes fine-tuning and deployment expensive. The effects of quantization and parameter efficient fine-tuning on performance, representations and generation remain insufficiently characterized. Results: We evaluated 4-bit quantization and low-rank adapter fine-tuning (QLoRA) across ESM-2, ESMC, ProtBERT, ProtT5, Ankh, Ankh3 and Profluent-E1. Across protein prediction tasks, many model-task pairs retained more than 90% of full fine-tuning performance. Peak GPU memory savings approached 90% for the largest models, although performance and efficiency varied by model, dataset and training configuration. QLoRA often preserved early-layer representations while inducing task-specific adaptations in middle and late layers, resembling full fine-tuning with smaller representational changes. Training speed and power effects were more varied. For unconditional generation with ProLLaMA, ProtGPT2, ProGen2, ProteinGLM and ESM3, 4-bit quantization largely preserved predicted structural and sequence-level properties, but token-level analysis revealed model-dependent shifts in autoregressive output distributions. Conclusion: QLoRA and 4-bit quantization reduce PLM computational requirements, particularly GPU memory usage. Our results support QLoRA as a first-pass strategy for memory limited adaptation, reserving full fine-tuning for challenging tasks, unstable architectures or low validation recovery. For generative PLMs, sequence-level and structural metrics should be complemented with distributional analysis, since downstream predictions alone may miss quantization-induced shifts. These approaches can broaden access to large-scale protein modelling while requiring model- and task-specific validation.

Momentum

Related topics

Latest papers 39