Hidden coordinates are not uniquely determined by a language model's input--output function, so representation-derived measurements should be invariant to function-preserving changes of basis. This study shows that column-permutation parallel analysis violates function-preserving reparameterization invariance because its reference distribution and selected component count can change while the model function and observed covariance spectrum remain fixed. More generally, a data-internal reference procedure cannot simultaneously preserve every coordinate marginal, remain orthogonally equivariant, and remove cross-coordinate covariance. Empirically, across five models, three retrieval domains, and 75 transformations, median component-count disagreement is 0.79 and median fixed-threshold decision disagreement is 0.26. A centering-only control isolates the reference-driven effect, with 1,141 of 1,200 component counts changing despite an unchanged observed spectrum, whereas independent parallel analysis seeds change none of the corresponding decisions. By contrast, orthogonally invariant comparator scores remain numerically stable with similar held-out discrimination. Together, these results show that parallel analysis-derived component counts and decisions can reflect hidden-coordinate choice rather than a well-defined property of the model.
Figures & tables
Reference procedure
Exact marginals
O(d) -equivariant
Different covariance
Independent column permutations
Yes
No
Yes
Common row permutation
Yes
Yes
No
Trace-matched isotropic Gaussian
No
Yes
Yes
Table 1: Assumptions and consequences for representative data-internal reference procedures. “Different covariance” asks whether the procedure can produce Z⊤Z=X⊤X .
Model
Architecture
Normalization
Transformation class
Qwen2.5-0.5B
Qwen2
RMSNorm
O(d)
Qwen2.5-1.5B
Qwen2
RMSNorm
O(d)
Qwen3-0.6B
Qwen3
RMSNorm
O(d)
SmolLM2-1.7B
Llama-style
RMSNorm
O(d)
Pythia-1B
GPT-NeoX
LayerNorm
{R∈O(d):R1=1}
Table 2: Models and exact residual-stream transformation classes. Parameter counts are those encoded in the public model identifiers.
Endpoint
Median
IQR
Range
PA component-count disagreement
0.79
(0.725, 1.000)
(0.59, 1.00)
Decision disagreement
0.26
(0.160, 0.345)
(0.00, 0.44)
Within-question ordering reversal
0.10
(0.060, 0.140)
(0.02, 0.24)
Absolute component-count change
2
(1, 4)
(0, 12)
Table 3: Stability under analytically function-preserving, numerically verified reparameterizations. The first three rows summarize 75 model–domain–transformation comparison-level rates. The final row pools 7,500 matched context transformations. IQR and range are descriptive, not confidence limits.
Figure 1: Heterogeneity across the 15 model–domain combinations. Each point is the median across the five fixed transformations; horizontal bars show the minimum and maximum across those transformations. The ranges describe the five evaluated transformations and are not confidence intervals for an underlying transformation population.
Figure 2: Distribution and downstream consequences of component-count changes. Top left. raw selected component counts for 1,500 original contexts and 7,500 matched reparameterized contexts. Top right. absolute changes for the 7,500 matched context transformations. Bottom. decision disagreement and within-question ordering reversal for each of the 75 model–domain–transformation comparisons; horizontal segments mark medians. Decisions use the score threshold selected on the original calibration split.
Table 4: Post-hoc controls across 12 model–domain combinations. Rates are fractions of 100 evaluation contexts. The reparameterization rate is summarized over five transformations for the seed comparison. The seed control uses one independent PA rerun. The preprocessing comparison uses the same single transformation (seed 101) and permutation seed in its standardized and centering-only arms.
Figure 3: Reparameterization-induced versus independent-seed no-reparameterization disagreement for selected component counts (left.) and fixed-threshold decisions (right.). Each point is one of 12 model–domain combinations; the dashed line denotes equality.
Figure 4: Reference-distribution isolation across 12 model–domain combinations. Under centering alone, the observed covariance spectrum is exactly invariant to the orthogonal reparameterization, while the column-permutation reference changes. Points are model–domain disagreement rates under transformation seed 101; horizontal segments mark medians.
Score
Median AUROC
Range
PA-selected subspace score
0.703
(0.637, 0.813)
Cumulative-variance principal-subspace score
0.697
(0.664, 0.824)
Ridge-shrinkage subspace score
0.712
(0.683, 0.827)
Answer log-probability
0.689
(0.527, 0.839)
Table 5: Held-out context discrimination across 15 model–domain combinations. Ranges show the minimum and maximum AUROC across combinations.
Figure 5: Left. held-out discrimination. Each point is one of 15 model–domain combinations, and each horizontal segment is the median. Right. sensitivity to function-preserving reparameterization. Each point is one of 75 model–domain–transformation comparisons. PA-selected subspace values are median absolute score changes across evaluation contexts; the right-orthogonally invariant comparator-score values are maximum absolute changes and therefore provide a conservative numerical stability check.
Choice
Values
Median rate
Permutation replicates
50 / 100 / 200
0.870 / 0.865 / 0.865
Reference quantile
0.90 / 0.95 / 0.99
0.880 / 0.870 / 0.885
Selection rule
Consecutive from largest / count all
0.870 / 0.870
Layer aggregation
Quarter / half / three-quarter / median
0.925 / 0.980 / 0.935 / 0.870
Table 6: Baseline-centered slices of the full factorial sensitivity analysis in four representative model–domain settings. Each row varies one choice while holding the others at 50 replicates, the 0.95 permutation-reference quantile, consecutive-from-largest selection, and median-across-layer aggregation. Intervals are question-level bootstrap 95% intervals for the median across four representative model–domain combinations.
Figure 6: Context-level PA component-count disagreement rates across analysis choices. Points are the four representative model–domain–transformation settings; horizontal segments mark medians. In each panel, non-displayed choices are held at the main experimental setting.
Society for Open Science (Stowarzyszenie na Rzecz Otwartej Nauki) Centre for Brain Research, Jagiellonian University · Society for Open Science (Stowarzyszenie na Rzecz Otwartej Nauki) IDEAS Research Institute