Causal representation learning (CRL) models aim to transform high-dimensional data into a latent space, enabling interventions to generate counterfactual samples or modify existing data based on the causal relationships among latent variables. To facilitate the development and evaluation of these models, a variety of synthetic and real-world datasets have been proposed, each with distinct advantages and limitations. For practical applications, CRL models must perform robustly across multiple evaluation directions, including reconstruction, disentanglement, causal discovery, and counterfactual reasoning, using appropriate metrics for each direction. However, this multi-directional evaluation can complicate model comparison, as a model may excel in some direction while under-performing in others. Another significant challenge in this field is reproducibility: the source code corresponding to published results must be publicly available, and repeated runs should yield performance consistent with the original reports. In this study, we critically analyzed the synthetic and real-world datasets currently employed in the literature, highlighting their limitations and proposing a set of essential characteristics for suitable datasets in CRL model development. We also introduce a single aggregate metric that consolidates performance across all evaluation directions, providing a comprehensive score for each model. Finally, we reviewed existing implementations from the literature and assessed them in terms of reproducibility, identifying gaps and best practices in the field.
Figures & tables
Figure 1: Fundamental causal junction types that constitute the building blocks of causal graphs.
Figure 2: a) Causal graphs for CelebA(SMILE) and CelebA(BEARD), two commonly used subsets of the CelebA dataset for CRL model development. b) Illustration of an unseen variable X acting as a potential confounder, which may explain the dependencies observed between age and gender in CelebA(BEARD), and between gender and smile in CelebA(SMILE). c) Causal graphs for additional CelebA subsets constructed and utilized by Huang et al. (2025) , with subset names consistent with those used in their original study.
Figure 3: ausal graphs for Pendulum, Flow Noise, Shadow(SunLight), and Shadow(PointLight), four commonly used synthetic datasets for CRL model development.
Figure 4: Illustration showing that if a CRL model performs well in all directions except one, its real-world applicability is compromised, analogous to a car with a single flat tire impairing overall system performance.
Figure 5: Reconstruction performance of different models. Although CausalVAE outperforms the other two models in MIC and TIC metrics, its reconstruction ability is inferior, highlighting a critical limitation for practical applications and underscoring the often-overlooked importance of reconstruction in CRL models
Disentanglement
Counterfactual
Models
Reconstruction↑
IRS↑
JEMMIG↑
MIC↑
TIC↑
FID↓
IS↑
KID↓
β -VAE
0.9863±0.0013
0.7965±0.0249
0.2556±0.0312
0.4133±0.1510
0.2802±0.1353
15.3481±3.3664
1.2137±0.0732
0.0136±0.0031
ConditionalVAE
0.9881±0.0018
0.7559±0.0265
0.1930±0.0050
0.4331±0.0262
0.3174±0.0274
14.6477±3.9643
1.2083±0.0582
0.0117±0.0021
CausalVAE
0.8654±0.0092
0.7087±0.0303
0.3538±0.0601
0.7409±0.0840
0.6038±0.0729
169.1046±14.0410
2.0548±0.2224
0.1995±0.0227
Table 1: Performance of various models in different directions on pendulum dataset
Figure 6: a) Radar plot for visual comparison of model performance across multiple directions and metrics. The shape and enclosed area are sensitive to the order of axes, making quantitative comparison potentially inconsistent. b) Origami plot for the same experiment, where the enclosed area for each model is insensitive to axis order, providing a more robust and reliable metric for overall performance comparison.
Figure 7: Different MIC values obtained by the β -VAE model trained on the Flow Noise dataset, where each result corresponds to a distinct assignment between the generative factors and the latent variables. This variation demonstrates the model’s high sensitivity to the permutation ambiguity inherent in unsupervised CRL frameworks.
Pendulum
Flow Noise
CausalVAE
All
Boundries out
Top 5
CausalVAE
All
Boundries out
Top 5
MIC
95.1±2.4
69.1± 21.7
74.7±6.9
81.2±5.9
72.1±1.3
57.3±26.6
61.1±22.7
74.1±1.0
TIC
81.6±1.9
56.0±19.5
60.7±6.6
66.9±6.7
56.4±1.6
47.9±25.4
51.4±21.9
64.6±1.2
Table 2: MIC and TIC values of CausalVAE reported by the original study (column CausalVAE), and the results after our reimplementation.
Causal representation learning seeks latent features that reveal underlying causal variables and mechanisms for robust prediction, intervention, and transfer.
We evaluate methods for causal representation learning (CRL) on a simple, real-world system that satisfies the basic problem setup of CRL. The system consists of a controlled optical experiment producing a variety of measurements where the underlying causal factors---the control inputs to the experiment---are known, providing a ground truth. We select methods representative of different approaches to CRL and find that they all fail to consistently recover the underlying causal factors. To understand the failure modes of the evaluated algorithms, we perform an ablation on the data by substituting the real data-generating process with a simpler synthetic equivalent. The results reveal a reproducibility problem, as most methods already fail on this synthetic ablation despite its simple data-generating process. Additionally, we observe that common assumptions on the mixing function are crucial for the performance of some of the methods but do not hold in the real data. Our efforts highlight the contrast between the theoretical promise of the state of the art and the challenges in its application. We hope the benchmark serves as a simple, real-world sanity check to further develop and validate methodol- ogy, bridging the gap towards CRL methods that work in practice.
Juan L. Gamella, Simon Bing, Jakob Runge
Seminar for Statistics, ETH Zurich · Technische Universität Berlin · Department of Computer Science, University of Potsdam +1
Causal representation learning (CRL) and traditional representation learning have largely developed along different trajectories. Traditional representation learning has been driven mainly by applications and empirical objectives, whereas CRL has focused more on theoretical questions, particularly identifiability. This difference in emphasis has created a gap between the two fields in terminology, problem formulation, and evaluation, limiting communication and sometimes leading to disconnected or redundant efforts. In this paper, we argue that these two fields should be brought into dialogue rather than treated as separate paradigms. To this end, we introduce a unified formulation in which the representation learning is characterized by two components: a task component, which specifies what information the learned representation is required to preserve, and a constraint component, which specifies what structure is imposed on the latent space. Under this formulation, the benefits run in both directions. CRL provides theoretical tools for understanding when structured latent constraints are useful or necessary, while traditional representation learning offers practical insights on task design and objective choice that can improve the development of CRL methods. To illustrate this interaction, we experimentally study how different task components affect the behavior of CRL methods under different structured constraints. Results on CausalVerse show that the effectiveness of causal constraints depends strongly on the tasks with which they are paired.
Yan Li, Yuewen Sun, Shaoan Xie +4
1Mohamed bin Zayed University of Artificial Intelligence · 2Carnegie Mellon University
Causal representation learning aims to discover robust features by exploiting the causal structure underlying data generation. Existing methods require specifying the causal structure a priori, yet different structures demand fundamentally incompatible invariance constraints, and misspecification leads to representations that discard predictive information. We introduce SaCRL, a framework that jointly identifies the causal structure and learns the corresponding invariant representation without prior structural knowledge. Our approach formulates structure selection as a soft optimization over candidate invariances using HSIC-based violation metrics, with adaptive weights that automatically concentrate on the achievable structure. We provide theoretical guarantees for structure identification, including under random-feature approximation, invariance satisfaction, and out-of-distribution generalization. Empirically, SaCRL recovers the true structure on synthetic and semi-synthetic Bayesian-network benchmarks, outperforms fixed-invariance baselines on Colored MNIST, achieves state-of-the-art accuracy on three DomainBed benchmarks (PACS, VLCS, OfficeHome), and degrades gracefully under structural misspecification and limited environment diversity. Code is available at: https://github.com/ArmanBehnam/sacrl.
Arman Behnam, Binghui Wang
Department of Computer Science Illinois Institute of Technology Chicago, IL, USA