Hidden coordinates are not uniquely determined by a language model's input--output function, so representation-derived measurements should be invariant to function-preserving changes of basis. This study shows that column-permutation parallel analysis violates function-preserving reparameterization invariance because its reference distribution and selected component count can change while the model function and observed covariance spectrum remain fixed. More generally, a data-internal reference procedure cannot simultaneously preserve every coordinate marginal, remain orthogonally equivariant, and remove cross-coordinate covariance. Empirically, across five models, three retrieval domains, and 75 transformations, median component-count disagreement is 0.79 and median fixed-threshold decision disagreement is 0.26. A centering-only control isolates the reference-driven effect, with 1,141 of 1,200 component counts changing despite an unchanged observed spectrum, whereas independent parallel analysis seeds change none of the corresponding decisions. By contrast, orthogonally invariant comparator scores remain numerically stable with similar held-out discrimination. Together, these results show that parallel analysis-derived component counts and decisions can reflect hidden-coordinate choice rather than a well-defined property of the model.
Figures & tables
Reference procedure
Exact marginals
O(d) -equivariant
Different covariance
Independent column permutations
Yes
No
Yes
Common row permutation
Yes
Yes
No
Trace-matched isotropic Gaussian
No
Yes
Yes
Table 1: Assumptions and consequences for representative data-internal reference procedures. “Different covariance” asks whether the procedure can produce Z⊤Z=X⊤X .
Model
Architecture
Normalization
Transformation class
Qwen2.5-0.5B
Qwen2
RMSNorm
O(d)
Qwen2.5-1.5B
Qwen2
RMSNorm
O(d)
Qwen3-0.6B
Qwen3
RMSNorm
O(d)
SmolLM2-1.7B
Llama-style
RMSNorm
O(d)
Pythia-1B
GPT-NeoX
LayerNorm
{R∈O(d):R1=1}
Table 2: Models and exact residual-stream transformation classes. Parameter counts are those encoded in the public model identifiers.
Endpoint
Median
IQR
Range
PA component-count disagreement
0.79
(0.725, 1.000)
(0.59, 1.00)
Decision disagreement
0.26
(0.160, 0.345)
(0.00, 0.44)
Within-question ordering reversal
0.10
(0.060, 0.140)
(0.02, 0.24)
Absolute component-count change
2
(1, 4)
(0, 12)
Table 3: Stability under analytically function-preserving, numerically verified reparameterizations. The first three rows summarize 75 model–domain–transformation comparison-level rates. The final row pools 7,500 matched context transformations. IQR and range are descriptive, not confidence limits.
Figure 1: Heterogeneity across the 15 model–domain combinations. Each point is the median across the five fixed transformations; horizontal bars show the minimum and maximum across those transformations. The ranges describe the five evaluated transformations and are not confidence intervals for an underlying transformation population.
Figure 2: Distribution and downstream consequences of component-count changes. Top left. raw selected component counts for 1,500 original contexts and 7,500 matched reparameterized contexts. Top right. absolute changes for the 7,500 matched context transformations. Bottom. decision disagreement and within-question ordering reversal for each of the 75 model–domain–transformation comparisons; horizontal segments mark medians. Decisions use the score threshold selected on the original calibration split.
Table 4: Post-hoc controls across 12 model–domain combinations. Rates are fractions of 100 evaluation contexts. The reparameterization rate is summarized over five transformations for the seed comparison. The seed control uses one independent PA rerun. The preprocessing comparison uses the same single transformation (seed 101) and permutation seed in its standardized and centering-only arms.
Figure 3: Reparameterization-induced versus independent-seed no-reparameterization disagreement for selected component counts (left.) and fixed-threshold decisions (right.). Each point is one of 12 model–domain combinations; the dashed line denotes equality.
Figure 4: Reference-distribution isolation across 12 model–domain combinations. Under centering alone, the observed covariance spectrum is exactly invariant to the orthogonal reparameterization, while the column-permutation reference changes. Points are model–domain disagreement rates under transformation seed 101; horizontal segments mark medians.
Score
Median AUROC
Range
PA-selected subspace score
0.703
(0.637, 0.813)
Cumulative-variance principal-subspace score
0.697
(0.664, 0.824)
Ridge-shrinkage subspace score
0.712
(0.683, 0.827)
Answer log-probability
0.689
(0.527, 0.839)
Table 5: Held-out context discrimination across 15 model–domain combinations. Ranges show the minimum and maximum AUROC across combinations.
Figure 5: Left. held-out discrimination. Each point is one of 15 model–domain combinations, and each horizontal segment is the median. Right. sensitivity to function-preserving reparameterization. Each point is one of 75 model–domain–transformation comparisons. PA-selected subspace values are median absolute score changes across evaluation contexts; the right-orthogonally invariant comparator-score values are maximum absolute changes and therefore provide a conservative numerical stability check.
Choice
Values
Median rate
Permutation replicates
50 / 100 / 200
0.870 / 0.865 / 0.865
Reference quantile
0.90 / 0.95 / 0.99
0.880 / 0.870 / 0.885
Selection rule
Consecutive from largest / count all
0.870 / 0.870
Layer aggregation
Quarter / half / three-quarter / median
0.925 / 0.980 / 0.935 / 0.870
Table 6: Baseline-centered slices of the full factorial sensitivity analysis in four representative model–domain settings. Each row varies one choice while holding the others at 50 replicates, the 0.95 permutation-reference quantile, consecutive-from-largest selection, and median-across-layer aggregation. Intervals are question-level bootstrap 95% intervals for the median across four representative model–domain combinations.
Figure 6: Context-level PA component-count disagreement rates across analysis choices. Points are the four representative model–domain–transformation settings; horizontal segments mark medians. In each panel, non-displayed choices are held at the main experimental setting.
Learned representations are often invariant to rotational transformations, leaving individual dimensions non-identifiable and interchangeable. We study how Matryoshka Representation Learning (MRL) induces a task-aligned privileged basis distinct from variance-based or regularizer-induced orderings. In the linear setting, we prove that full-prefix MRL recovers the ordered principal directions, and can be computed efficiently using shared statistics. Empirically, we demonstrate that MRL yields consistent per-dimension structure aligned with task signal, where coordinate magnitude reflects informativeness.
Arghamitra Talukder, Philippe Chlenski, Itsik Pe'er
Representational similarity measures (RSA, CKA, Procrustes) are chosen for the transformations they treat as irrelevant, and all of them take the set of conditions as given. That set is fixed by two decisions, which inputs are admitted and how they are grouped, both taken before the matrix a measure operates on exists: no invariance property reaches them. We measure what they are worth where the arrangement the conditions ought to have is known independently of the systems compared, the acoustic organisation of vowels, across fifteen languages and twenty-seven representations. Changing which categories are compared, at a fixed number of them, moves the gap between two representations by more than twice the effect being measured, while changing how many moves it not at all. In this setting rankings of models are identified under these choices, and rankings across languages are not.
Partial Least Squares (PLS) regression extracts a few outcome-aligned directions in a high-dimensional X and is widely used across applied science, but inference on the resulting fit is either expensive, biased and discouraged, or absent. We reduce inference to held-out OLS refits of the supervised subspace, a primitive shared by PLS, supervised PCA, and linear probes, and supply two tests using held-out correlations: a Nadeau-Bengio corrected asymptotic t-test as a fast approximation, and a permutation test with comparable power, finite-sample valid under outcome-predictor independence and iid rows. Held-out predictions are unchanged under any orthogonal rebasing of the supervised span, so an interpretable basis such as varimax inherits the joint claim but not a per-axis p-value; per-component claims come from a fixed-sequence test on the PLS extraction order. We validate on synthetic geometries, two NIR chemometric datasets, and cross-lingual word-embedding regressions; the exact test also transfers to supervised PCA and a ridge probe. The proposed tests have more power than CV-permutation-Q^2, at a fraction of its cost. A pre-run check on n and the spectrum of X says when the approximation is safe. We release a Rust library with Python, R, and Julia bindings, plus a Python text pipeline.
Paweł Lenartowicz, Hubert Plisiecki
Society for Open Science (Stowarzyszenie na Rzecz Otwartej Nauki) Centre for Brain Research, Jagiellonian University · Society for Open Science (Stowarzyszenie na Rzecz Otwartej Nauki) IDEAS Research Institute