Parameter Identifiability

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 77

All topics

Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
Apr 26, 2026Giansalvo CirrincioneTransformer Interpretability Representation Collapse
aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France
On the Identifiability of User Adaptation in Co-Adaptive Neural Interfaces
Apr 24, 2026Philip WaggonerBrain-Computer Interfaces Parameter Identifiability
Stanford University
SOLIS: Physics-Informed Learning of Interpretable Neural Surrogates for Nonlinear Systems
Apr 16, 2026Murat Furkan Mansur, Tufan KumbasarNeural Surrogate Modeling Parameter Estimation
AI and Intelligent Systems Laboratory, Istanbul Technical University, 34469, Istanbul, Türkiye
A Complete Symmetry Classification of Shallow ReLU Networks
Apr 15, 2026Pranavkrishnan RamakrishnanShallow Neural Networks Neural Representation Geometry
IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video
Mar 17, 2026Rasul Khanbayov, Mohamed Rayan Barhdadi, Erchin Serpedin +1Parameter Identifiability System Identification
Hamad Bin Khalifa University · Texas A&M University
Provable Subspace Identification of Nonlinear Multi-view CCA
Feb 27, 2026Zhiwei Han, Stefan Matthes, Hao ShenRepresentation Identifiability Parameter Identifiability
Chair of Data Processing, Technical University of Munich, Germany · fortiss GmbH, Munich, Germany
Stability and Concentration in Nonlinear Inverse Problems with Block-Structured Parameters: Lipschitz Geometry, Identifiability, and an Application to Gaussian Splatting
Feb 10, 2026Joe-Mei Feng, Hsin-Hsiung KaoInverse Rendering Parameter Identifiability
Computer Science and Information Engineering, Tamkang University, New Taipei City, Taiwan (R.O.C.) · Department of Information Management, Central Police University, Taoyuan, Taiwan
Complete Identification of Deep ReLU Networks through Łukasiewicz Logic
Jan 30, 2026Yani Zhang, Helmut BölcskeiParameter Identifiability ReLU Neural Networks
Chair for Mathematical Information Science, ETH Zürich
Uncertainty-aware Causal Decision Making via Effect Bound Decomposition
Jan 30, 2026Md Musfiqur Rahman, Ziwei Jiang, Hilaf Hasson +1Causal Effect Estimation Parameter Identifiability
Electrical and Computer Engineering, Purdue University · Computer Science, Johns Hopkins University · Cohesity
Nonnegative matrix factorizations and related compositional models: Equivalence, identifiability, and an application on the grain-size analysis of sediments
Dec 25, 2025Qianqian Qi, Peter G. M. van der Heijden, Maarten A. PrinsLatent Variable Models Parameter Identifiability
Hangzhou Dianzi University, China · Utrecht University, the Netherlands · University of Southampton, UK +1
Variational Physics-Informed Ansatz for Reconstructing Hidden Interaction Networks from Steady States
Dec 6, 2025Kaiming LuoParameter Identifiability
School of Information Science and Technology, Fudan University, Shanghai, 200438, China
The Alexander-Hirschowitz theorem for neurovarieties
Nov 24, 2025A. Massarenti, M. MellaPolynomial Neural Networks Parameter Identifiability
Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors
Oct 6, 2025Zhiwei Han, Stefan Matthes, Hao ShenRepresentation Learning Parameter Identifiability
fortiss GmbH, Munich, Germany · Technical university of Munich
Identifiable Convex-Concave Regression via Sub-gradient Regularised Least Squares
Jun 22, 2025William ChungParameter Identifiability Nonparametric Regression
Department of Decision Analytics and Operations, City University of Hong Kong, Hong Kong
Identifiability Analysis of Linear ODE Systems with Hidden Confounders
Oct 29, 2024Yuanyuan Wang, Biwei Huang, Wei Huang +2Parameter Identifiability Causal Identifiability
The University of Melbourne · University of California, San Diego
Symbolic Recovery of Differential Equations: The Identifiability Problem
Oct 15, 2022Philipp Scholl, Aras Bacho, Holger Boche +1Parameter Identifiability Autonomous Scientific Discovery
Ludwig-Maximilians-Universität München Munich Germany · Technische Universität München Munich Center for Quantum Science and Technology (MCQST) Munich Quantum Valley (MQV) Munich Germany · Ludwig-Maximilians-Universität München University of Tromsø DLR-German Aerospace Center Munich Center for Machine Learning (MCML) Munich Germany
Diffusion learning reveals viable parameter manifolds and compensation geometry in biological dynamical systems
Date pendingRuilin Zhang, Louis Tao, Zhuo-Cheng XiaoDynamical Systems Parameter Identifiability
Peking University–Tsinghua University–National Institute of Biological Sciences Joint Graduate Program, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China · Center for Bioinformatics, School of Life Sciences, Peking University, Beijing 100871, China · Center for Quantitative Biology, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China +2

Apr 26, 2026, cs.LGJ/K move · Enter open · S save

Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers

Giansalvo Cirrincione

aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France

Transformer Interpretability Representation Collapse Parameter Identifiability Self-Attention Symmetry Breaking Transformer Attention+2

Open paper AskPDF arXiv

A widely cited result by Dong et al. (2021) showed that Transformers built from self-attention alone, without skip connections or feed-forward layers, suffer from rapid rank collapse: all token representations converge to a single direction. The proposed remedy was the MLP. We show that this picture, while correct in the regime studied by Dong, is incomplete in ways that matter for architectural understanding. Three results are established. First, layer normalisation is precisely affine-rank-neutral: it preserves the affine rank of the token representation set exactly. The widespread claim that LN "plays no role" is imprecise; the correct statement is sharper. Second, residual connections generically obstruct rank collapse in real Transformers such as BERT-base, in a measure-theoretic sense, without contribution from the MLP. The MLP's irreplaceable function is different: generating feature directions outside the linear span of the original token embeddings, which no stack of attention layers can produce. Third, a phenomenon distinct from rank collapse is identified: head-channel non-identifiability. After multi-head attention sums per-head outputs through the output projection, individual contributions cannot be canonically attributed to a specific head; n(H-1)d_k degrees of freedom per layer remain ambiguous when recovering a single head from the mixed signal. The MLP cannot remedy this because it acts on the post-summation signal. A constructive partial remedy is proposed: a position-gated output projection (PG-OP) at parameter overhead below 1.6% of the standard output projection. The four collapse phenomena identified in the literature -- rank collapse in depth, in width, head-channel non-identifiability, and entropy collapse -- are unified under a symmetry-breaking framework, each corresponding to a distinct symmetry of the Transformer's forward pass.

Momentum

Related topics

Latest papers 77