Intrinsic Dimensionality

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, down 20% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 37

Sep 28, 2026cond-mat.mtrl-sci

Measuring trainable degrees of freedom in materials graph neural networks: a random-subspace intrinsic dimension analysis

Final predictive accuracy is the standard basis for comparing graph neural networks (GNNs) in materials-property prediction, but it does not show how strongly performance depends on access to trainable parameter-space directions. Here, we introduce trainable-degree dependence as a complementary characterization of materials GNN learning. Using random-subspace intrinsic-dimension analysis, we train CGCNN, ALIGNN, and DimeNet++ in randomly oriented parameter subspaces across six prediction tasks and measure how performance recovers as independent trainable degrees of freedom are restored. The resulting recovery curves separate endpoint accuracy from the trainable-dimensional demand required to recover it. They reveal distinctions that final errors alone miss: metallic classification and log-bulk-modulus regression recover near-reference performance from small fractional subspaces, formation-energy and band-gap prediction show stronger architecture dependence, and phonon prediction is most sensitive to dimensional restriction. Dataset-size sweeps show that band-gap models require larger fractional subspaces as training data grows, whereas formation-energy and bulk-modulus responses are more stable. A width sweep shows that fractional thresholds can remain stable while absolute threshold dimensions increase with model size. Random-subspace analysis therefore provides a targeted stress test for how materials GNNs use their optimization space.

Shehroz Ahmad Shoaib, Kangming Li

Physical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · Department of Electrical Engineering, King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia.

Intrinsic Dimensionality Graph Neural Networks Materials Property Prediction

Sep 27, 2026cs.DS

Geometry-Adaptive Mechanisms for Private Synthetic Data

Generating differentially private synthetic data with meaningful Wasserstein utility guarantees is challenging in high dimensions. For datasets of size

n

on

[0,1]^d

with

d\ge2

, existing pure

\varepsilon

-differentially private mechanisms achieve expected

1

-Wasserstein error of order

(\varepsilon n)^{-1/d}

, reflecting the curse of dimensionality. While this rate is optimal in the worst case, it can be overly pessimistic when the data are supported on a lower-dimensional set. We formalize this through a multiscale packing-growth dimension

k

, which captures the geometric complexity of the support via the growth of packing numbers across scales. We propose \emph{Adaptive Pruned-PMM}, a pure

\varepsilon

-differentially private mechanism that combines private depth selection with our pruned variant of the Private Measure Mechanism (PMM) of He et al.\ (2023). The mechanism supports deeper, geometry-adapted hierarchies with expected running time

O\!\left(d(n+d)\log(\varepsilon n)\right)

, which is near-linear in

n

for fixed dimension and privacy budget. Under an external multiscale packing-growth condition with dimension

k

, we show that, for fixed positive privacy budgets and fixed geometry, the expected

1

-Wasserstein error is of order

(\varepsilon n)^{-1/k}

for

k>1

as

n

grows. We also prove a lower bound under a corresponding internal packing-growth condition, showing that the exponent

1/k

is sharp within this framework.

Raoof Zare Moayedi, Amir R. Asadi, Mohammad Hossein Yassaee +1

Sharif University of Technology · University of Birmingham · The Alan Turing Institute

Intrinsic Dimensionality Synthetic Data Generation Privacy-Preserving ML Differential Privacy Wasserstein Distance

Sep 24, 2026cs.CG

It's the Geometry, Not the Model: Effective Rank and Subspace Alignment in Functional Connectivity Classification

Resting-state functional connectivity (FC) is widely used to classify brain phenotypes and disorders. Most pipelines use the full connectome and seek gains through model design. We instead examine how FC geometry constrains classification and cross-site transfer. Across-subject FC variation concentrates in a small effective subspace, suggesting substantial redundancy in nominal dimensions. Across cohorts, these subspaces may differ in orientation even when their effective ranks are comparable, potentially limiting transfer. Across 2,330 subjects from HCP, ABIDE, and ADHD-200, effective-rank analysis reveals strong spectral concentration. Projection onto leading components at the effective-rank scale recovers most of the full-FC classification performance. In ABIDE, site-specific effective subspaces are weakly aligned, and their principal-angle overlap predicts pairwise transfer after covariate adjustment despite comparable per-site effective ranks. Controlled rotations that alter subspace orientation while preserving the mean and covariance spectrum drive transfer toward chance, whereas displacement-matched label-orthogonal rotations do not. These results identify subspace orientation as a key factor in transfer degradation under controlled perturbations. This study offers a geometric diagnostic of FC generalization and suggests evaluating cross-site harmonization by its ability to align effective subspaces alongside classification accuracy.

Xiao Fan, Jingyuan Li, Yubo Han +6

Intrinsic Dimensionality Functional Connectivity Cross-Domain Transfer Learning Representation Alignment Latent Space Alignment

Sep 15, 2026cs.LG

Symmetry without a manifold: intrinsic dimension on orbits

The standard geometric derivation of neural scaling exponents takes the intrinsic dimension of a data manifold as its input. On modular addition in

\mathbb{Z}_p

that derivation has no input. The exact algebraic solution is an orbit of

\mathbb{Z}_p

acting by isometries. Transitivity alone makes the ratio statistic underlying the standard dimension estimator a point mass, so the estimator is undefined, and here the two nearest neighbour distances coincide exactly. Breaking the symmetry at scale

ε

returns a number, but one that tracks

1/ε

with no scale free plateau. We show that the failure is general, since on any finite orbit of a group acting by isometries the estimator reports the resolution at which the set is probed rather than a dimension. What replaces the power law is exponential in hidden width,

L(h)=L_\infty+A\exp(-c\,h^α)

, with

R^2

between 0.982 and 0.995 against 0.857 to 0.906 for a power law admitting the same floor and fitted under the same protocol. Where the data supply is sufficient the rate belongs to the regulariser rather than to the group, since weight decay moves

c

by a factor of 47 while group order moves it by 1.10, a residual below seed to seed resolution, for every fixed

α

between 0.75 and 2. The critical width falls with group order rather than rising, against capacity counting that assigns a fixed number of neurons to each irreducible representation.

Chon-Fai Kam, Miloud Bessafi, Frédéric Cadet

Dipartimento di Fisica e Chimica, Universit`a degli Studi di Palermo, via Archirafi 36, I-90123 Palermo, Italy · EnergyLab, Universit´e de La R´eunion, F-97715 Saint-Denis, France · Universit´e Paris Cit´e and Universit´e de La R´eunion, BIGR, INSERM UMR S1134, F-75014 Paris, France +1

Intrinsic Dimensionality Neural Scaling Laws

Aug 31, 2026quant-ph

Fractal dimension predicts quantum kernel collapse in angle-encoded data

Angle-encoded quantum kernels on tabular data collapse when the feature map is wider than the intrinsic dimension of the data. We propose the correlation fractal dimension D2 as an a priori qubit budget: encode D2 coordinates chosen by FD-ASE instead of the PCA-95% width or all E attributes. On nine data sets and a statevector simulator (n= 32), a one-layer ZZ fidelity kernel at q=D2 stays geometrically alive while the same kernel at the PCA-95% width has already collapsed. The budget is map-dependent: product-state and IQP maps overshoot it; a second ZZ layer undershoots it. Packed dense-angle and re-uploading encodings still live at the fractal q, but not when PCA-95% features are stacked onto those qubits. Shrinking the angle bandwidth moves the ZZ knee later; stretching it kills the kernel earlier. On IBM Quantum (ibm_fez, 256 shots, n=8) the one-layer ZZ kernel at the fractal width matches the exact kernel (MAE 0.021); past that width both hardware and simulator have collapsed. The ceiling is a property of the map-data pair at a stated bandwidth, not of the classical table alone.

Ana Paula Appel

Quantum Feature Maps Intrinsic Dimensionality Quantum Machine Learning Quantum Kernel Methods

Aug 30, 2026cs.CL

A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddings

A token-embedding table holds a hub of short rows near its origin, and we show that this cluster biases what nearest-neighbor intrinsic-dimension (ID) estimators report. Because of the concentration of measure, a token is closer to the central cluster than to any other token, so its first two neighbors are both hub rows at nearly the same distance. As a result, the ID estimators such as TwoNN return a dimension far above the real ID. Measured one token at a time, dimension is a heavy-tailed distribution. Measured on the full vocabulary, it grows with the model's parameter count. However, when we remove the hub, the heavy tail disappears and the measured dimension collapses to a narrow range for eleven models, from GPT-2 to models such as K3 and GLM-4.7. The hub acts as a switch: a few hundred rows are enough to fully inflate the estimate. We reproduced an experiment stating that the intrinsic dimension (ID) of Pythia's token-embedding table grows with the parameter count, from

27

to

122

between 160M and 12B parameters. We show that this result disappears when the hub is removed: the table then reads

10

to

17

at every size. The hub contains a subset of the population that under-trained-token detectors flag, but on Pythia the hub that we detected and removed as a whole was updated during training: what seem to characterize these rows is simply their length, not an absence of updates. Finally, we show that normalizing the rows instead of removing them gives the same lower reading.

Alexandre Quemy

Hother Labs

Intrinsic Dimensionality Neural Representation Geometry Representation Geometry in Language Models

Aug 11, 2026cs.LG

FiGuRO: Intrinsic Dimension Estimation for Multi-Modal Data

Determining the complexity, or Intrinsic Dimension (ID), of data is fundamental to efficient and interpretable representation learning. This is particularly challenging in multi-modal settings when trying to learn disentangled representations for shared and private information. Existing techniques leave a critical gap: they are often static, uni-modal, or in the case of contrastive methods, adapt only to the shared ID implicitly. We introduce Fidelity-Guided Rank Optimization (FiGuRO), a framework for approximating the ID of uni- and multi-modal data under constraints of model capacity and hyperparameters. FiGuRO learns the dimensions of low-rank projections using truncated singular value decomposition and an algorithm that determines when to reduce or increase dimension and in which latent space. Disentanglement of shared and private information arises as an emergent property of this optimization, eliminating the need for complex auxiliary loss functions. We demonstrate that FiGuRO outperforms existing ID estimation techniques and is more robust to hyperparameter changes. Across simulations and real-world data, FiGuRO captures distinct ID scales and varying subspace ratios, and decomposes shared and private information successfully. Furthermore, we show that FiGuRO can be applied to modern uni-modal pretrained models, enabling efficient, post-hoc disentanglement of multi-modal representations.

Viktoria Schuster, Sana Tonekaboni, Caroline Uhler

Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA · Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard, Cambridge, MA, USA · Technical University of Denmark, Lyngby, Denmark +1

Intrinsic Dimensionality Representation Learning Multimodal Disentangled Representation Learning

Aug 11, 2026cs.LG

$β$ -VAEs as Effective Theories: Tolerance-Dependent Dimension

In a

β

-VAE, increasing the regularization strength acts as a spectral cutoff by collapsing low-utility latent coordinates. In the linear Gaussian VAE, the collapse order matches the ranking of reconstruction utilities exactly, because both are set by the PCA spectrum. We ask which parts of this picture survive in fully connected nonlinear VAEs trained on WorldClim. We find that nonlinear interactions shift and broaden collapse onsets, so thresholds no longer coincide exactly with utilities. However, the common ordering is preserved over the resolved ranks, so the spectral cutoff still acts as a utility cutoff and the effective-description logic carries through. The resulting effective-dimension curves reveal a head--tail tradeoff: increasing depth concentrates utility into the first few coordinates but worsens tail fidelity.

Johannes Hirn

Image Processing Laboratory (IPL), Universitat de València, Paterna, València 46980, Spain

Variational Autoencoders Intrinsic Dimensionality

Aug 11, 2026stat.ML

Iterative Erasure Count Is Not an Affine-Invariant Concept Dimension

How many directions does a neural representation use to encode a concept? A common answer repeatedly erases probe directions and reports the stopping count or cumulative removed rank. We show that both quantities can change under an information-preserving invertible reparameterization, so neither is intrinsically a concept dimension. We distinguish model-defined population quantities (generating dimension, sufficient linear dimension, and minimum guarding rank) from procedure-defined quantities such as stopping count and cumulative edit rank. In a population Gaussian construction, an invertible shear preserves the prediction problem and all three quantities, yet changes the cumulative Euclidean erasure count from one to two. The separation holds for Moore--Penrose ordinary least squares and every finite nonnegative ridge weight. For a two-output full-QR procedure matching our motivating video analysis, cumulative edit rank similarly changes from two to the ambient dimension four. Conversely, the complete cumulative metric-QR trajectory is affine-equivariant when its positive-definite metric, probe, regularizer, and tie-breaking are transported consistently; exact covariance is one corollary, not a canonical semantic metric. In a known-rank finite-sample Adam/QR calibration, identity mixing stops after one accepted update in all 20 large-sample runs, whereas each tested shear

a\in\{.5,.75,1,1.25,2\}

accepts at least two updates in all 20 runs. Controlled reparameterizations of frozen V-JEPA2 features preserve rank-zero predictions yet alter later Euclidean trajectories under practical optimization. These visual contact experiments are stress tests, not estimates of contact dimension. Iterative erasure therefore returns a procedure-relative estimand jointly determined by representation geometry and the full measurement procedure, not a semantic dimension by itself.

Tingan Jin, Shuhang Dong, Haosong Li +1

UCLA · Independent Researcher · Cal Poly Pomona

Intrinsic Dimensionality Neural Representation Geometry Concept Erasure

Jul 21, 2026cs.CV

ERank in Latent Space as an Image-Complexity and Richness Measure

We propose the effective rank (ERank) of the channel covariance of an image's deep feature map as a per-sample, label-free measure of visual richness, computed from a single forward pass through a frozen pretrained encoder. ERank counts how many decorrelated channel directions an image activates, and we characterize its properties, including its behavior under noise. Empirically, ERank orders images from plain to visually rich, correlates with codec bitrate, sharpness, and edge density, and correlates with human complexity annotations on IC9600 with

r = 0.72

. As a data-selection criterion, removing low-ERank samples improves super-resolution and removing high-ERank samples improves OCR, in both pretraining and finetuning, while selection does not help classification, segmentation, or denoising. ERank is thus a cheap richness signal, useful exactly when task difficulty is governed by input richness.

Maksim Smirnov, Grigory Kononov, Anastasiia Linich +2

Intrinsic Dimensionality Image Quality Assessment

Jul 21, 2026cs.CR

GLID: Gated Local Intrinsic Dimension Repairs the Blind Spots of Face-Forgery Detectors

Fine-tuned foundation-model detectors dominate face-forgery benchmarks, yet they stay blind to generator families absent from training. We present GLID, a detector that repairs this blind spot with geometry instead of data. GLID treats the patch tokens of a single image as a sample from a manifold and estimates their local intrinsic dimension (LID) at several depths of a frozen vision transformer. This 12-dimensional, training-free signal enters a fine-tuned detector through a confidence gate whose strength is calibrated purely in-distribution. On a 16-axis cross-generator benchmark, GLID reaches 0.805 mean AUC, first among retrained state-of-the-art baselines and never significantly behind the strongest of them on any axis. It lifts the generation axes by +0.084 AUC while moving reenactment by only -0.005. Two empirical laws explain the design. First, forged faces bend the token manifold at family-specific depths: GAN artifacts peak at the last layer, diffusion artifacts peak mid-network, and the pattern survives four backbones, three dimension estimators, and non-face imagery. Second, fine-tuning absorbs auxiliary gains exactly where training data covers: injecting 1% target-family images erases a +0.100 gain, so geometric signals matter precisely where data is unavailable. The deterministic signal also cuts the cross-seed spread of accuracy 5.5x. Code, preregistered analysis gates, and per-image scores accompany the paper.

Guang Yang, Fengchen Liu

University of California, Los Angeles · University of California, Berkeley

Intrinsic Dimensionality Deepfake Detection

Jul 7, 2026cs.LG

Physics-Informed Neural Embeddings of PDE Solution Families

We introduce a physics-informed framework for learning finite-dimensional embeddings of solution families of partial differential equations. The method uses a multihead Physics-Informed Neural Network in which a shared body learns a latent manifold representing the solution space, while linear heads reconstruct individual solutions associated with different initial conditions. A head-orthogonalization penalty removes degeneracies in the latent representation and stabilizes the principal-component spectrum across training realizations. Because the initial condition is built into the network output by construction, these principal components measure the additional variability the network learns on top of the initial profile, not the full solution itself. We apply the method to the one-dimensional viscous Burgers equation, with the heat and wave equations as robustness checks. For a latent dimension

n_b=20

, the learned manifolds exhibit pronounced effective dimensional reduction: for Burgers dynamics, only

2

-

4

principal components capture about

95\%

of the latent-space variance, while

4

-

7

capture about

99\%

, depending on the initial-condition family; the same qualitative compression holds for the heat and wave equations. We also split the wavenumber axis into bands (``Fourier shells'') and measure how much each band contributes to every principal component. The resulting frequency profile is invariant under the change-of-basis freedom that the orthogonalization penalty leaves in the latent space, and is therefore reproducible across independent training runs. More broadly, this establishes the learned spectral profiles and principal components as robust observables of solution-manifold geometry.

Raul Jimenez, Svitlana Mayboroda, Pavlos Protopapas +3

Institute of Cosmos Sciences (ICC), University of Barcelona, Martí i Franquès 1, ES-08028, Barcelona, Spain. · Department of Mathematics, ETH Zurich, Rämistrasse 101, 8092, Zürich, Switzerland. · Institute for Applied Computational Science, Harvard University, Cambridge, MA, USA. +1

Intrinsic Dimensionality PDE Solving Manifold Learning Physics-Informed ML

Jul 2, 2026cs.CV

Transformer Geometry Observatory TGO-II: Representational Similarity Observatory

While Vision Transformers have achieved remarkable success across computer vision and language applications, the geometric evolution of their internal representations throughout training remains insufficiently understood. Existing analyses primarily focus on attention mechanisms and downstream performance, leaving the evolution of representation geometry largely unexplored. In this work, we present Transformer Geometry Observatory-II (TGO-II), a representation geometry analysis framework designed to investigate how Transformer representations evolve during supervised training. TGO-II analyzes Vision Transformer (ViT-Small/16) representations using Centered Kernel Alignment (CKA), Singular Vector Canonical Correlation Analysis (SVCCA), Two-Nearest Neighbor Intrinsic Dimensionality (TwoNN-ID), and token covariance analysis. Our experiments reveal three key observations. First, both CKA and SVCCA progressively decrease throughout training, indicating increasing representational specialization across Transformer layers. Second, intrinsic dimensionality consistently increases before stabilizing, suggesting progressive expansion of the representation manifold into a larger set of locally accessible degrees of freedom. Third, token covariance and coupling analyses demonstrate that strong token interaction structure persists throughout training, challenging the hypothesis that increasing representational complexity arises primarily from progressive token independence. These findings suggest that representation complexity and layer specialization emerge simultaneously during training. Manifold expansion appears to occur without token decoupling. Together, these observations motivate a new hypothesis in which Vision Transformers increase representational complexity through progressively richer transformations while preserving strong token interaction structure during learning.

Kaustubh Kapil, Kishor P. Upla

Department of Electronics Engineering Sardar Vallabhai National Institute of Technology (SVNIT), Surat, India

Intrinsic Dimensionality Representational Similarity Analysis Neural Representation Geometry Visual Representation Learning Vision Transformer

Jul 1, 2026stat.ML

Function-Counting Theory for Low-Dimensional Data Structures

The success of deep learning models in classification and regression is widely attributed to the low-dimensional structure that real-world data tend to exhibit, despite their high-dimensional representation. This work attempts to provide a mathematical framework for binary classification on low-dimensional data, building on Cover's (1965) function-counting theory. With our framework, we aim to address the question of how the low-dimensional structure of the data affects the classification capabilities of learning models. Cover's theory relies on a general position assumption that blinds it to the underlying data structure. We refine this assumption to account for the low-dimensionality of the data and derive dichotomy counts that reflect the data structure. We further extend Cover's separation capacity and problem of generalization to the low-dimensional setting, enabling the impact of the underlying data structure on both to be analyzed.

Konstantin Häberle, Helmut Bölcskei

ETH Zurich

Intrinsic Dimensionality Binary Classification Statistical Learning Theory Generalization Bounds

Jun 29, 2026cs.LG

I-BBS: Coordinate-Free Inference of Latent Sub-Manifolds Using Random Distance Matrix Theory

Bogomolny, Bohigas and Schmit (BBS) found that the spectrum of the pairwise distance matrix on N points sampled from a smooth d-dimensional manifold encodes a signature of the underlying geometry. We develop I-BBS (Inference-BBS), a coordinate-free method that identifies a low-dimensional latent sub-manifold embedded in a high-dimensional ambient distance matrix alone, without accessing an ambient high-dimensional vector space. It therefore applies even when that space is only partly observable or undefined. We model the ambient embedding by two classes of generative noise, model-based and model-free. The noise mixes the latent signal with off-manifold components, so the eigenvalues reorganise collectively and the latent geometry cannot be read off eigenvalue by eigenvalue. We recover it instead from two integer-stable signatures that survive the noise: the multiplicity of the top non-Perron multiplet, which fixes

d

, and a parameter-free law for how the multiplet positions shrink as the noise grows. On synthetic spheres

S^1

,

S^2

and

S^3

these integer signatures are far more stable under noise than the continuous spectral slope, and a blind test recovers both the manifold and the noise model from a single distance matrix. Applications to neural-network representations and to the dynamic training regime are developed in two companion papers.

Igor Halperin

Intrinsic Dimensionality Random Matrix Theory Manifold Learning

Jun 18, 2026cs.LG

Effective Dimension Governs Generalization in Quantum Kernel Vision Models

Recent quantum vision models-quantum vision transformers and quantum convolutional networks-report two striking but unexplained empirical phenomena: (i) ansatze with more, or more uniformly distributed, entanglement generalize better, and (ii) injecting quantum noise can improve test accuracy rather than degrade it. These observations are currently treated as curiosities, discovered by grid search and explained, if at all, by hand. We show that both are manifestations of a single, measurable quantity: the \emph{effective dimension}

d_{\rm eff}

of the (noise-shaped) quantum feature kernel. Working primarily with quantum-kernel vision models-a quantum feature map read out by a kernel classifier-we give a spectral account in which entanglement structure and quantum noise are two knobs that move

d_{\rm eff}

; in an overfitting regime, contracting

d_{\rm eff}

acts as ridge-like regularization. We analyze the mechanism: an \emph{exact} decomposition of the depolarized kernel

K_p=(1-p)^2K+\tfrac{p(2-p)}{D}\mathbf{1}\mathbf{1}^\top

with

d_{\rm eff}(K_p)\to1

, a contraction result (and its boundary) for amplitude damping, a kernel-machine capacity bound, and a capacity/alignment risk decomposition; the monotone contraction operative in our entangled experiments is verified empirically, not proven in general. Along the one-parameter depolarizing family the collapse is instead exact by construction; we use it only to confirm the kernel decomposition to machine precision and at up to

12

qubits, not as evidence for

d_{\rm eff}

. Amplitude damping contracts

d_{\rm eff}

and lifts test accuracy by up to

+13\%

along an inverted-U sweet spot; the effect's sign flips between the over- and under-fitting regimes; noise injection matches an explicit spectral-filtering frontier. Our results organize two reported anecdotes into a single measurable principle for designing quantum-vision models.

Jian Xu, Delu Zeng, John Paisley +1

RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1

Quantum Feature Maps Intrinsic Dimensionality Quantum Machine Learning Quantum Kernel Methods

Jun 17, 2026cs.CV

Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory

Despite the widespread adoption of Vision Transformers (ViTs) and their success across numerous computer vision applications, the fundamental understanding of their dimensional and representational geometry remains relatively underexplored. To address this gap, we introduce Transformer Geometry Observatory (TGO), a systematic framework of experiments and analysis pipelines designed to investigate the representational geometry and dynamics of Vision Transformers. TGO-I, the first installment of the framework, focuses on the spectral geometry of ViT representations. Using a ViT-Small/16 model trained on ImageNet-100, we analyze Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, Spectral Anisotropy, covariance structure, eigenspectra, and singular value spectra throughout training. Our results reveal a consistent increase in dimensional utilization, accompanied by decreasing anisotropy, increasing spectral entropy, increasing participation ratio, and progressively flatter eigenspectra. Contrary to the common intuition that training should concentrate information into a small number of dominant directions, we observe a progressive redistribution of variance across representational dimensions. This phenomenon is particularly pronounced in the final CLS token representation, which exhibits the highest effective dimensionality and lowest anisotropy within the network.

Kaustubh Kapil, Kishor P. Upla

Department of Electronics Engineering Sardar Vallabhai National Institute of Technology (SVNIT), Surat, India

Intrinsic Dimensionality Neural Representation Geometry Vision Transformer

Jun 14, 2026cs.LG

The Data Manifold under the Microscope

A significant gap exists between theory and practice in deep learning. Generalization and approximation error bounds are often derived for simplified models or are too loose to be informative. Many rely on the manifold hypothesis and on geometric regularity such as intrinsic dimension, curvature, and reach. Progress requires insight into data-manifold geometry and suitable benchmarks, yet existing options are polarized: analytic manifolds with known geometry but limited applicability, or real-world datasets where geometry is only coarsely estimable. We introduce a benchmarking framework for studying data geometry. We repurpose and extend dSprites and COIL-20 with additional transformation dimensions and dense, axis-aligned sampling, and pair them with finite-difference estimators that recover curvature, reach, and volume at near-ground-truth accuracy in a regime where general-purpose estimators are unreliable or difficult to deploy. The framework is intended as a controlled testbed, useful as a calibration environment for geometric estimators and a sandbox for probing theoretical assumptions. To illustrate its use, we present two application studies, namely assessing the scaling behavior of the bounds of Genovese et al. and Fefferman et al., and tracking the layer-wise geometry of a

β

-VAE, highlighting the behavior of current bounds and the value of controlled benchmarks for guiding and validating future theory. A reference implementation is available at https://github.com/koulakis/manifold-microscope.

Marios Koulakis, Constantin Seibold

Independent Researcher, Germany · Diagnostic and Interventional Radiology, University Clinic Heidelberg, Heidelberg, Germany.

Intrinsic Dimensionality Synthetic Benchmark Generation Manifold Learning Generalization Bounds

Jun 6, 2026cs.LG

Orthogonality and Dimensionality in Airline Cluster Analysis using PCA and Kernel PCA

This methodological study analyzes the effects of collinearity, effective dimensionality, and cluster stability in a 2023 study of US airline profit cycles from 1995 to 2020 by Renold et al., which uses k-means clustering, principal component analysis, and system dynamic modelling.We replicate their clustering experiment in three spaces -- the original 7-dim. raw-variable space, a 3-dim. PC score space, and a 4-dim. PC score space using their dataset. We show that the six-cluster taxonomy is geometrically robust: k-means in 3-PC space produces bit-for-bit identical cluster assignments relative to 7D raw space. As a nonlinearity check we apply kernel PCA under six kernels spanning three families plus a linear baseline. The kernels confirm an intrinsically linear manifold with no detectable curvature. The silhouette criterion reveals that the dataset structurally supports only three clusters, not six. Collinearity in the raw 7D space suppresses the silhouette signal. A kernel ridge regression check confirms no nonlinear accuracy gain over linear ridge once the COVID19 year is excluded. Together, these results argue for clustering on PC scores rather than raw variables in collinearity-prone panel data.

Andreas Schlapbach

Swiss Federal Railways (SBB), Switzerland

Intrinsic Dimensionality Principal Component Analysis Clustering

Jun 4, 2026stat.ML

Effective Dimensionality as an Operator Invariant for Physics-Preserving Constraint Adaptation in Physics-Informed Neural Networks

Physics-Informed Neural Networks inherently suffer from task interference because they rely on a shared parameter space to satisfy both governing differential equations and boundary conditions. We analyze this structural conflict using the Fisher Information Matrix to quantify the effective degrees of freedom (

d_{eff}

) in a physics-constrained model. Unlike the classical

d_{eff}

which measures how many parameter directions are informed by data against a statistical prior, our

d_{eff}

measures the dimension of the parameter directions unconstrained by the differential operator. For operators with finite-dimensional kernel, we show that

d_{eff}

converges to the kernel dimension exactly, independent of network width, depth, or activation function, recasting it from a fit diagnostic into a structural invariant of the underlying continuous operator. For operators with infinite-dimensional kernel,

d_{eff}

instead measures the network's finite-dimensional representational bandwidth for that kernel rather than recovering an integer invariant. Importantly,

d_{eff}

also serves as an a priori structural diagnostic. Driving

d_{eff}

of a well-posed problem to zero certifies that the physics and boundary constraints have absorbed the network's free directions. Building on this characterization, we introduce subspace projection strategies for boundary adaptation. Rather than retraining from scratch, we project parameter updates into the null space of the pre-trained physics operator so that new boundary conditions are satisfied without disturbing the learned physics. Gradient-based fine-tuning can match or exceed this but needs more wall-clock time and tuning, whereas subspace projection delivers near-equivalent quality in seconds to minutes. We validate on linear and nonlinear operators, demonstrating accurate adaptation to initial and boundary shifts and unencountered constraint types.

Cornelius Otchere, Michael Shields

Department of Civil and Systems Engineering, Johns Hopkins University, Baltimore, Maryland, USA

Intrinsic Dimensionality Physics-Informed ML

Jun 2, 2026cs.LG

IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimension

Self-supervised learning (SSL) has emerged as a powerful paradigm for learning meaningful representations from unlabeled data. However, the standard protocol for evaluating these representations, linear probing, is computationally expensive, sensitive to hyperparameters, and provides limited insight into the geometric structure of the representation space. In this work, motivated by connections between neural network generalization and intrinsic dimension (ID) we propose IdEst, a method for estimating the ID of SSL representations via the Minimum Spanning Tree dimension estimator (

\mathrm{dim}_\mathrm{MST}

). Across diverse datasets, architectures, and SSL pretraining objectives, we show that IdEst strongly correlates with downstream linear probe performances. Furthermore, we demonstrate that IdEst enables efficient hyperparameter selection, significantly reducing the computational cost compared to supervised alternatives. Our results highlight intrinsic dimensionality as a principled geometric proxy for assessing SSL representations, complementing standard supervised probing protocols.

Julie Mordacq, Vicky Kalogeiton, Steve Oudot

Inria Saclay 2LIX, CNRS, École Polytechnique, IP Paris

Intrinsic Dimensionality Self-Supervised Learning Representation Probing

May 22, 2026cs.LG

Automated Random Embedding for Practical Bayesian Optimization with Unknown Effective Dimension

Bayesian optimization is widely employed for optimizing complex black-box functions but struggles with the curse of dimensionality. Random embedding, as a dimension reduction strategy, simplifies tasks that possess the effective dimension by optimizing within a low-dimensional subspace. However, determining the effective dimension of a task in advance remains a significant challenge, which influences the selection of the subspace dimensionality and the optimization performance. Traditional methods use fixed subspace dimensions provided by experts or rely on trial and error to estimate subspace dimensions with resources consumed. To this end, this paper proposes an automated random embedding for high-dimensional Bayesian optimization with unknown effective dimension, called Dynamic Shared Embedding Bayesian Optimization (DSEBO). DSEBO starts with a low dimension and switches to a higher subspace if the solutions in the current subspace show preliminary convergence. DSEBO dynamically determines the dimension of the next subspace based on the quality of the solutions in different subspaces and shares the queried solutions with the new subspace for a better initialization. Theoretically, we derive a regret bound for DSEBO and demonstrate that DSEBO can better balance approximation and optimization errors. Extensive experiments on functions with dimensionality of varying magnitudes and real-world tasks with unknown effective dimensions reveal that, compared with state-of-the-art methods, alternating optimization across different subspaces results in significant improvements in high-dimensional optimization, both in terms of optimization regret and time.

Hong Qian, Xiang Shu, Xiang Xia +5

Shanghai Institute of AI for Education, and School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · Ant Group, Hangzhou 310013, China · Nanjing University, Nanjing 210023, China

Intrinsic Dimensionality Bayesian Optimization Dimensionality Reduction

May 20, 2026cs.LG

Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective

Characterizing precisely the asymptotic generalization error of neural networks using parameters that can be estimated efficiently is a crucial problem in machine learning, which relies heavily on heuristics and practitioners' intuition to make key design choices. In order to mitigate this issue, we introduce the Representation Gap, a metric closely related to the generalization error, but admitting better-behaved asymptotic dynamics. Focusing on equivariant diffusion models and leveraging results from optimal quantization and point-process theory, we derive a precise asymptotic equivalent of the Representation Gap and show that it is governed by a single parameter, the \textit{intrinsic dimension} of the task, which is easy to interpret, efficient to estimate, and can be linked to the equivariances of common neural network architectures. We show that this asymptotic dynamic also extends to a broader range of tasks and training algorithms. Finally, we demonstrate empirically that our asymptotic law and intrinsic dimension estimation are accurate on a wide range of synthetic datasets, where these quantities are known, as well as on more realistic datasets, where we obtain results consistent with the related literature.

David Perera, Victor Moura, Lais Isabelle Alves dos Santos +2

Universidade Federal de Minas Gerais · Queen Mary University of London

Intrinsic Dimensionality Neural Network Generalization

May 15, 2026cs.LG

Intrinsic Wasserstein Rates for Score-Based Generative Models on Smooth Manifolds

Score-based generative models are trained in high-dimensional ambient spaces, yet many data distributions are supported on low-dimensional nonlinear structures. We prove that, for compact

d

-dimensional smooth manifolds

\mathcal{M} \subset [0,1]^D

with

d > 2

and

β

-Hölder densities strictly positive on

\mathcal{M}

, a variance-preserving SGM estimator attains the intrinsic Wasserstein--1 sample exponent

\tilde{\mathcal{O}}(D^{\mathcal{O}_β(d)}n^{-(β+1)/(d+2β)})

, up to logarithmic factors and explicit geometry and density factors. The full nonasymptotic bound explicitly isolates the finite-order geometry envelope, Hölder radius, density lower bound, ambient dependence, and finite-order correction terms. The analysis separates score approximation into a large-noise tangent-cell regime and a small-noise projection-centered, de-Gaussianized Laplace regime. The key technical ingredient is a ReLU implementation of nearest-projection coordinates via finite intrinsic anchors and Gauss--Newton iterations, rather than approximating the manifold projection as a black-box high-dimensional smooth map. Consequently, for families with polynomially controlled geometry and density lower bounds, the constructed score-network parameters have polynomial ambient dependence.

Guoji Fu, Taiji Suzuki, Wee Sun Lee +1

National University of Singapore · The University of Tokyo · RIKEN Center for Advanced Intelligence Project +2

Intrinsic Dimensionality Generative Modeling Score-Based Generative Modeling Wasserstein Distance

May 11, 2026cs.LG

Rank Is Not Capacity: Spectral Occupancy for Latent Graph Models

Graph representation learning has become a standard approach for analyzing networked data, with latent embeddings widely used for link prediction, community detection, and related tasks. Yet a basic design choice, the latent dimension, is still treated as a brittle hyperparameter, fixed before training and tuned by held-out performance. Learned factors are also identifiable only up to rotation and rescaling, so the nominal rank rarely coincides with the quantity that governs model behavior. We propose Spectral Prefix Extraction and Capacity-Targeted Representation Analysis (Spectra), which replaces rank as the unit of analysis with the spectrum of a learned positive semidefinite kernel, trace-normalized so that spectra are comparable across fits. The normalized eigenvalues form a distribution on the simplex, and their Shannon effective rank acts both as a summary of learned capacity and as a controllable training-time coordinate: a single scalar shapes this realized dimension during training, and bisection targets any desired value within the rank cap. To theoretically support that, we show local regularity and monotonicity of the realized-dimension profile. Across collaboration, social, biological, and infrastructure networks, Spectra traces performance--capacity frontiers that make the trade-off between predictive accuracy and realized dimension visible. It performs competitively with strong link-prediction baselines, yields aligned lower-capacity views of the same fitted model through spectral prefixes, and provides a principled handle on capacity in the overparameterized regime. Capacity thus becomes a property of the fitted model rather than a hyperparameter of the training.

Nikolaos Nakis, Panagiotis Promponas, Konstantinos Tsirkas +4

Human Nature Lab, Yale University · Department of Electrical and Computer Engineering, Yale University · Department of Statistics and Data Science, Yale University +1

Representation Geometry Intrinsic Dimensionality Graph Representation Learning

May 6, 2026cs.CV

Local Intrinsic Dimension Unveils Hallucinations in Diffusion Models

Diffusion models are prone to generating structural hallucinations - samples that match the statistical properties of the training data yet defy underlying structural rules, resulting in anomalies like hands with more than five fingers. Recent research studied this failure mode from several viewpoints, offering partial explanations to their occurrence, such as mode interpolation. In this work, we propose a complementary perspective that treats hallucinations as instabilities on the model-induced manifold. We begin by showing that a hallucination filter based on such instabilities matches or exceeds the performance of the recently proposed temporal one. By tracing the source of these instabilities, we identify local intrinsic dimension (LID) as their primary driver and propose Intrinsic Quenching (IQ), a direct corrective mechanism that deflates it to alleviate hallucinations. IQ consistently outperforms standard hallucination reduction baselines across a wide array of benchmarks and offers a highly promising solution for enforcing anatomical consistency in downstream medical imaging tasks.

Bartlomiej Sobieski, Matthew Tivnan, Dawid Płudowski +4

Centre for Credible AI, Warsaw University of Technology · University of Warsaw · Massachusetts General Hospital +1

Intrinsic Dimensionality Hallucination Mitigation Diffusion Models

May 5, 2026cs.DS

Provable Accuracy Collapse in Embedding-Based Representations under Dimensionality Mismatch

Embedding-based representations in Euclidean space

\mathbb{R}^d

are a cornerstone of modern machine learning, where a major goal is to use the \emph{smallest dimension} that faithfully captures data relations. In this work, we prove sharp dimension--accuracy tradeoffs and identify a fundamental information-theoretic limitation: unless the embedding dimension

d

is chosen close to the ground-truth dimension

D

, accuracy undergoes a sudden collapse. Our main result shows that this phenomenon arises even in standard contrastive learning settings, where supervision is limited to a set of

m

anchor--positive--negative triplets

(i,j,k)

encoding distance comparisons

\mathrm{dist}(i,j) < \mathrm{dist}(i,k)

. Specifically, given triplets realizable by an unknown ground-truth embedding in

D

dimensions, we prove that there exists constant

c < 1

, such that \emph{every embedding of dimension at most

cD

violates half of the triplets}, yielding accuracy as low as a trivial one-dimensional solution that ignores the input. We complement our information-theoretic bounds with strong computational hardness results: under the Unique Games Conjecture, even if the given triplets are nearly realizable in

D=1

dimension, no polynomial-time algorithm -- \textit{regardless of its dimension} -- can achieve accuracy above the trivial

50\%

baseline.

Dionysis Arvanitakis, Vaggos Chatziafratis, Yiyuan Luo

Northwestern University, IL, USA · University of California

Intrinsic Dimensionality Contrastive Learning Dimensionality Reduction Information-Theoretic Lower Bounds

May 4, 2026eess.AS

Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models

Self-supervised speech models (S3Ms) achieve strong downstream performance, yet their learned representations remain poorly understood under natural and adversarial perturbations. Prior studies rely on representation similarity or global dimensionality, offering limited visibility into local geometric changes. We ask: how do perturbations deform local geometry, and do these shifts track downstream automatic speech recognition (ASR) degradation? To address this, we present GRIDS, a framework using Local Intrinsic Dimensionality (LID) across layer-wise representations in WavLM and wav2vec 2.0. We find that LID increases for all low signal-to noise ratio (SNR) perturbations and diverges at high SNR: benign noise converges toward the clean profile, while adversarial inputs retain early-layer LID elevation. We show LID elevation co-occurs with increased WER, and that layer-wise LID features enable anomaly detection (AUROC 0.78-1.00), opening the door to transcript-free monitoring in S3Ms.

Sandra Arcos-Holzinger, Sarah M. Erfani, James Bailey +1

University of Melbourne, School of Computing and Information Systems, Australia · Monash University, Department of Data Science and Artificial Intelligence, Australia · Johns Hopkins University, Center for Language and Speech Processing, USA

Intrinsic Dimensionality Unsupervised Anomaly Detection Self-Supervised Speech Representation Learning Anomaly Detection

May 2, 2026cs.LG

Local Hessian Spectral Filtering for Robust Intrinsic Dimension Estimation

While diffusion models enable new approaches for estimating Local Intrinsic Dimension (LID), existing methods fail in high-dimensional spaces where noise from vast normal directions overwhelms the tangent signal. We propose Local Hessian Spectral Dimension (LHSD), which resolves this by applying spectral filtering to the log-density Hessian, explicitly cutting off large eigenvalues associated with normal directions to count zero-curvature tangent directions. Implemented using Stochastic Lanczos Quadrature (SLQ), LHSD avoids full Hessian construction, achieving linear scalability with dimension

D

. Experiments on synthetic and real data confirm LHSD's superior robustness and its utility in detecting memorization in large-scale diffusion models. The code is available at github.com/geosada/LHSD

Genki Osada

LY Corporation.

Intrinsic Dimensionality Spectral Methods

Apr 30, 2026cs.LG

When Does Structure Matter in Continual Learning? Dimensionality Controls When Modularity Shapes Representational Geometry

To preserve previously learned representations, continual learning systems must strike a balance between plasticity, the ability to acquire new knowledge, and stability. This stability-plasticity dilemma affects how representations can be reused across tasks: shared structure enables transfer when tasks are similar but may also induce interference when new learning disrupts existing representations. However, it remains unclear when and why structural separation influences this trade-off. In this study, we examine how network architecture, task similarity, and representational dimensionality jointly shape learning in a sequential task paradigm inspired by transfer-interference studies. We compare a task-partitioned modular recurrent network with a single-module baseline by systematically varying task similarity (low, medium, high) and the scale of weight initialization, which induces different learning regimes that we empirically characterize through the effective dimensionality of the learned representations. We find that architecture has minimal impact in high-dimensional regimes where representations are sufficiently unconstrained to accommodate multiple tasks without strong interference. In contrast, in lower-dimensional (rich) regimes, architectural separation is decisive: modular networks exhibit graded alignment of task-specific subspaces with overlap for similar tasks, partial orthogonalization for moderately dissimilar tasks, and stronger separation for dissimilar tasks. This graded geometry is absent in the single network baseline. Our findings suggest that representational dimensionality acts as a key organizing variable governing when structural separation becomes functionally relevant, and highlight adaptive geometry as a central principle for designing continual learning systems.

Kathrin Korte, Joachim Winter Pedersen, Eleni Nisioti +1

IT University of Copenhagen, Denmark

Intrinsic Dimensionality Continual Learning Modular Neural Networks