cs.LGFeb 27, 2026
SaveWho Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representations
Organizations: Mila - Québec AI Institute & Université de Montréal · Max-Planck-Institute for Intelligent Systems, ELLIS Institute Tübingen, University of Tübingen
Abstract
Identifiability in representation learning is commonly evaluated using standard metrics (e.g., MCC, DCI, R^2) on synthetic benchmarks with known ground-truth factors. These metrics are assumed to reflect recovery up to the equivalence class guaranteed by identifiability theory. We show that this assumption holds only under specific structural conditions: each metric implicitly encodes assumptions about both the data-generating process (DGP) and the encoder. When these assumptions are violated, metrics become misspecified and can produce systematic false positives and false negatives. Such failures occur both within classical identifiability regimes and in post-hoc settings where identifiability is most needed. We introduce a taxonomy separating DGP assumptions from encoder geometry, use it to characterise the validity domains of existing metrics, and release an evaluation suite for reproducible stress testing and comparison.
Figures & tables
Figure 1: Every identifiability metric fails under at least one common evaluation setting. We test four desiderata ( Properties 1 , 2 , 3 and 4 ) using controlled synthetic encoders that isolate metric behaviour from optimisation artefacts. (P1) Latent correlation: conflates correlation with identifiability (FP to ); -D penalises it (FN ). (P2) Factor dropping: -D reports perfect disentanglement even when of factors are lost. (P3) Overcompleteness: inflates for entangled encoders; -D deflates for disentangled ones. (P4) Null encoder: all metrics inflate as grows, with scaling in the order of . Only is robust across (P1), (P3), and (P4), but shares the (P2) limitation. No single metric is trustworthy across all settings.
| Sym. | Meaning | Range |
| Scaling parameters | ||
| # i.i.d. paired samples | – | |
| Dim. of | – | |
| # ground-truth factors | – | |
| Complexity parameters | ||
| Pairwise correlation ( ); off-diagonal entries of | ||
Table 1: (overcompleteness ratio), (sample ratio), and (representation-to-sample ratio).
Figure 2: conflates correlation with identifiability. Under E 3 , increases with and approaches the score of the perfectly disentangled encoder E 1 at high correlation, despite the encoder remaining entangled. better separates E 1 from E 3 but collapses to near-zero scores making it hard to distinguish from a non-identifiable encoder. The bias sharpens with increasing . See Fig. 14 for all metrics.
Figure 3: Regression-based metrics detect single-factor redundancy. Left ( ): every dropped factor is informative; follows and -D declines steadily. -P/S report even at (false positive). Right ( ): at , the dropped factor is redundant ( ); and -D plateau near , correctly recognising ossless compression, then decline as informative factors are removed. -P/S remain at throughout both panels and cannot distinguish the two.
Figure 4: Sweeping reveals encoder-specific violations of Property 3 . E 5 (elementwise linear duplication) is the only encoder for which all metrics remain stable. -D increases for entangled E 7 as grows. collapses for nonlinear E 6 . -P decreases for disjoint E 8 . , .
Figure 5: Null-encoder scores reveal that (columns), not (rows), governs false-positive inflation. Each cell shows the metric score of a random encoder E9 that carries no information about ; any score above is a false positive. See § F.3 for the theoretical analysis and Fig. 18 for the Gaussian null (nearly identical).
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Short description |
| DCI | Measures Disentanglement (D), Completeness (C), and Informativeness (I) by assessing how well latent dimensions predict ground-truth factors using supervised regressors ( Eastwood and Williams, 2018 ) . |
| MCC | Evaluates alignment between learned and ground-truth latent variables via an optimal one-to-one matching that maximizes pairwise correlations (P=pearson, S=spearman, RDC=Randomized Dependence Coefficient ( Lopez-Paz et al., 2013 ) ). |
| Quantifies the proportion of variance in ground-truth factors explained by the learned representation through linear regression. | |
| T-MEX | Assesses disentanglement by measuring how selectively latent variables respond to interventions on ground-truth factors ( Yao et al., 2025 ) . |
| MIG | Computes the gap between the top two mutual information scores between a factor and latent variables [CITE]. |
| InfoMEC | Measures equivalence classes of representations by evaluating how much information about the ground-truth factors is preserved under invertible transformations ( Hsu et al., 2023 ) (M=modularity, E=explicitness, C=compactness). |
Table 2: Common evaluation metrics for causal representation learning with ground-truth factors.
| Metric | P1 | P2 | P3 | P4 |
| -P | ✗ | ✗ | ✗ | ✗ |
| -S | ✗ | ✗ | ✗ | ✗ |
| ✓ | ✗ | ✓ | ||
| -D | ✗ | |||
| MIG | ✗ | ✗ | ✗ | |
| T-MEX | ✗ | ✗ |
Table 3: No single metric satisfies all four properties. Each cell shows whether a metric satisfies (✓), partially satisfies ( ), or violates (✗) the corresponding property. Superscripts reference the relevant subsection. See Appx. G for MI-based metrics.
Figure 6: overestimates identifiability as increases.
Figure 7: -D is not stable for .
Figure 8: MIG reports even for the ideal – E 1 case.
Figure 9: -D and -S are quite stable against increasing non-linearity strength, hence reliable for evaluating E 2 .
Figure 10: MI-based metrics struggle to evaluate E 2 .
Figure 11: Sample sensitivity of four main metrics (DCI-D, MCC-P, MCC-S, R²) across all DGP × encoder combinations ( , ). MCC-P and MCC-S are sample-efficient: they stabilise by in most settings. DCI-D requires to converge, with wide confidence intervals at , particularly under entangled encoders (E2, E3). The overcomplete encoder E7 ( ) inflates DCI-D variance at low because twice as many codes must be estimated from the same number of samples.
Figure 12: Full metric suite (adding InfoM, MIG, MCC-RDC, T-MEX to the main four). InfoM and MIG produce NaN at across all DGP × encoder combinations (visible as missing lines), making them unusable below . MCC-RDC converges slowly under nonlinear encoders (E2), lagging behind MCC-P and MCC-S until . T-MEX likewise returns NaN universally at small . The variance heatmap confirms that R² under E2 is the single worst-case cell (std at ), while correlation-based metrics remain stable (std ). Overall, metrics split into two reliability tiers: correlation-based measures (MCC-P, MCC-S) are robust across sample sizes, while predictor-based (R², DCI-D) and information-theoretic (InfoM, MIG, T-MEX) metrics require for trustworthy estimates.
Figure 13: Disentangling the effects of correlation and entanglement ( ). An ideal metric would vary only along columns (increasing , i.e. worse entanglement) and be constant along rows (changing ). -P and -S show clear row-wise gradients, confirming violation of Property 1 . -D is more stable but collapses to near-zero even under moderate entanglement (for all ).
Figure 14: Full metric suite across dimensionalities. The additional metrics reveal two distinct behaviours. MIG (grey) and MCC-RDC (cyan) degrade sharply with increasing under E1 at : MIG drops from at to at , while MCC-RDC follows a similar decline, indicating that these metrics conflate inter-factor correlation with non-identifiability. InfoM and T-MEX are absent at (NaN) but appear at ; T-MEX (yellow) remains flat near 1.0 under E1 regardless of , showing complete sign- and correlation-invariance. Under E3, the metric spread widens substantially: InfoM (dark blue) tracks DCI-D but at lower absolute values, while MCC-RDC drops more steeply than MCC-P, suggesting that the RDC kernel is sensitive to the interaction between entanglement and inter-factor correlation. The key takeaway is that correlation-based metrics (MCC-P, MCC-S) and T-MEX are robust to the sign of , while tree-based (DCI-D) and MI-based (MIG) metrics are sensitive to it, particularly at low .
Figure 15: Full metric suite: – heatmaps at . Extension of Fig. 13 to all metrics. An ideal metric is constant along rows (varying at fixed ); row-wise gradients indicate spurious sensitivity to inter-factor correlation. , .
Figure 16: Full metric suite: – heatmaps at . Extension of Fig. 6 to all metrics. Same layout as Fig. 15 ; qualitative conclusions carry over from to . .
Figure 17: Metric scores as a function of the number of retained factors across all DGP types. Extension of Fig. 3 to and . Under , exceeds because the probe partially predicts dropped factors from correlated retained ones. Under ( , ), metric behaviour is indistinguishable from despite the first omission being lossless: no metric detects the multi-factor redundancy. -P/S remain at across all panels. , .
Figure 18: False positives under a Gaussian null encoder. Same layout as Fig. 5 (uniform null) but with . The false-positive pattern is nearly identical: -P/S scores are governed by , not , confirming that the inflation is independent of the null distribution.
Figure 19: At moderate overcompleteness ( ),a metrics distinguish entanglement from redundancy. Overcomplete disentangled encoders ( E 5 – E 8 ) score near on -D, , and , whereas the entangled encoder E 7 is correctly penalised by -D and . , . See Fig. 20 for all DGPs.
Figure 20: Overcomplete encoders across all DGP types ( , ). Extension of Fig. 19 from to . At this smaller , -D and still separate disentangled overcomplete encoders ( E 5 – E 8 ) from the entangled baseline E 7 , though the gap is narrower than at .
Figure 21: Metrics conflate factor predictability with disentanglement under functional dependencies. Full metric suite under E 1 comparing (varying ) with (deterministic constraint ). Under , regression-based metrics ( , -D) penalise the encoder when the dependent factor is harder to predict from the retained code, despite perfect elementwise recovery. , . See Fig. 22 for .
Figure 22: Predictability vs. disentanglement at . Same setup as Fig. 21 with factors. The conflation between factor predictability and measured disentanglement persists at higher dimensionality.
Figure 23: Full metric suite for the dropped-factor experiment across all DGP types. Extension of Fig. 17 with additional metrics (MIG, InfoMEC, -RDC, T-MEX). MI-based metrics decline with fewer retained factors even under at , failing to recognise lossless compression of the redundant factor.
Figure 24: Effect of inflating the number of ground-truth factors under . The representation dimension is fixed at while increases by adding duplicated ground-truth factors. -P/S remain constant as they match only codes; and -D decline because the probe must predict an increasing number of factors from the same codes.
Figure 25: Redundancy and compression under . The encoder compresses factors (with a single-factor constraint , ) into codes. and -D plateau near at , correctly recognising that the omitted factor carries no independent information. -P/S report at all compression levels, unable to distinguish lossless from lossy omission.
Figure 26: Full metric suite: scores across encoding types and DGP types. Each panel compares matched-dimension encoders ( E 1 – E 3 ) with overcomplete encoders ( E 5 – E 8 ) under – . MI-based metrics (MIG, InfoMEC) and T-MEX are included alongside the main metrics. , .
Figure 27: Full metric suite for the overcomplete sweep. Extension of Fig. 4 to all metrics (MIG, InfoMEC, -RDC, T-MEX). MI-based metrics decline for distributed codes ( E 8 ) similarly to , while T-MEX is more robust to overcompleteness. , .
Figure 28: Matched-dimension entangled ( E 3 ) versus overcomplete entangled ( E 7 ). Full metric suite comparing the two entangled geometries as increases. -D inflates for E 7 at high , scoring substantially above the matched-dimension baseline E 3 despite equivalent identifiability status. , .
Figure 29: Metric inflation under null encoders. Full metric suite showing scores when the representation is independent of . -RDC exhibits persistent inflation that does not vanish with increasing . MI-based metrics (MIG, InfoMEC) also return non-trivial scores. remains closest to the expected value of . , .
Figure 30: Convergence of null-encoder scores with increasing . Each panel shows one metric under E9 ; scores should converge to as grows. converges fastest. -P/S retain elevated scores at large when is large, consistent with the floor ( § F.3 ). .
Figure 31: Metric scores across encoder types under correlated factors ( ). Overlay of all metrics for varying under . -P/S increase with under entangled encoders ( E 3 ), while and -D are less affected by the correlation structure. , .
Figure 32: Metric scores as a function of under different configurations. Each panel shows one metric; overlapping curves from different pairs with the same ratio confirm that , not or individually, governs metric behaviour in the undercomplete regime. -P/S report regardless of ; and -D increase approximately linearly. .
Figure 33: Ratio-collapse analysis grouped by encoder type. Same data as Fig. 32 , reorganised by encoder geometry. Curves from sweeps over (fixed ) and over (fixed ) overlap at matched , confirming the ratio as the governing quantity across encoder types.
Figure 34: Comprehensive parameter sweep (part A). Full metric suite across DGP types and encoder geometries, sweeping scaling and complexity parameters. Extends the targeted analyses of § 3.1 , 3.2 , 3.3 and 3.4 to a broader parameter range. , .
Figure 35: Comprehensive parameter sweep (part B). Continuation of Fig. 34 for additional parameter configurations and encoder–DGP combinations.
Figure 36: Full metric suite: false-positive phase diagram under Gaussian null. Extension of Fig. 18 to all metrics. -RDC shows the highest inflation across the grid. MI-based metrics (MIG, InfoMEC) also inflate at moderate .
Figure 37: Full metric suite: false-positive phase diagram under uniform null. Extension of Fig. 5 to all metrics. The pattern closely mirrors the Gaussian null ( Fig. 36 ), confirming that the false-positive floor is distribution-agnostic and governed by .
More in this topic
Representation Identifiability
Representation identifiability studies when learned internal codes can be uniquely recovered from observations or behavior, and which ambiguities remain.
See all 41 papers