Standard identifiability theory assumes one-dimensional latent factors, but learned representations can encode latent information across multidimensional blocks whose internal coordinates are not uniquely determined. When such blocks are used to represent and intervene on higher-level concepts, it is unclear whether their causal meaning survives the transformations left unresolved by block-identifiability. We connect block-identifiability to constructive causal abstraction. We show that block-diagonal invertible transformations followed by compatible coordinate permutations form the maximal class that preserves every constructive causal abstraction built on those blocks. This guarantee assumes that the correspondence between blocks and concepts is updated with the representation. A small synthetic experiment with additive autoencoders shows that high-level interventions can be transferred between independently learned representations while approximately preserving their intended effects.
Figures & tables
Figure 1: From ground-truth factors to a constructive causal abstraction of a learned representation. The ground-truth causal factors S take values s , which determine the network input and the resulting activation vector o=f(s) . A featurizer g , such as a sparse autoencoder (SAE), maps these activations to the learned representation x=g(o) . We treat the coordinates of x as variables of the low-level structural causal model (SCM) L and partition them into blocks V(b) . The constructive causal abstraction (τ,ω) maps low-level states and complete-block interventions to states and interventions in the high-level SCM H , where each human-interpretable concept Z(b) is computed from its corresponding block. Blue and pink indicate this block-to-concept correspondence. The ground-truth factors S generate the activations but need not coincide with the concepts in H .
Figure 2: Concept-directed steering and transport between independently trained additive autoencoders. Starting from a base observation, the high-level intervention J⋆=do(Z(1)=s1⋆) in H is implemented in the reference model L0 by editing block 1. Here, π1 maps that block to Z(1) , and x0(1)⋆ is the closest block value satisfying π1(x0(1)⋆)=s1⋆ ; block 2 remains unchanged. The fitted transformation Tr=σr∘Dr transfers the edit to recipient model Lr , where Dr maps corresponding blocks and σr accounts for their ordering. The decoders G0 and Gr produce the edited outputs o0⋆ and or⋆ . For evaluation, the exact factor setting sgen=(s1⋆,u1,s2,u2) changes only s1 , setting it to the intervention value s1⋆ , and the data-generating map f produces the comparison observation ogen=f(sgen) . Both decoded outputs are compared with this same observation, as indicated by the dotted lines. Green marks the intervention, while blue and pink identify the two blocks throughout.
Model (seed)
Recon.
Z(1)
Z(2)
Reference (0)
0.007312
0.007407
0.007378
Recipient (2)
0.007927
0.011931
0.010487
Recipient (3)
0.007762
0.009342
0.010307
Recipient (5)
0.007825
0.010572
0.010985
Recipient (10)
0.007779
0.009505
0.010923
Table 1: Reconstruction and concept-intervention accuracy across retained additive autoencoders. Rows show one reference additive autoencoder and four independently trained recipient additive autoencoders, identified by training seed. “Recon.” is the normalized mean squared error (NMSE) between each model’s decoded reconstructions and held-out synthetic input observations; only models with reconstruction NMSE at most 0.01 are retained. The Z(b) columns report intervention error. An intervention on Z(b) is implemented as a block edit in the reference model and transported to each recipient model. Each decoded edited output is compared with the exact synthetic observation obtained by changing only sb to the intervention value while keeping ub and (s3−b,u3−b) fixed. Each NMSE pools 5,000 examples; lower values are better.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Term
Notation
Meaning
Defined in
Representation (feature vector)
x=g(o)
Featurizer output whose coordinates form VL
§ 2
Activation vector
o
Activation read from the network and supplied to the featurizer g
§ 2
Featurizer
g
Maps network activations to x ; examples include SAEs and block-sparse featurizers
§ 2
Learned feature
X∈VL
Coordinate variable of the featurizer output
§ 2
Concept
Z(b)∈VH
High-level variable, possibly encoded across several learned coordinates
§ 2
Low-level model
L , VL
Representation-level SCM on the featurizer output, whose coordinates carry the block partition
§§ 2 and 1
Appendix
Table 2: Core terminology and notation used in the paper.
Causal representation learning seeks latent features that reveal underlying causal variables and mechanisms for robust prediction, intervention, and transfer.
We provide explicit, finite-sample guarantees for learning causal representations from data with a sublinear number of environments. Causal representation learning seeks to provide a rigourous foundation for the general representation learning problem by bridging causal models with latent factor models in order to learn interpretable representations with causal semantics. Despite a blossoming theory of identifiability in causal representation learning, estimation and finite-sample bounds are less well understood. We show that causal representations can be learned with only a logarithmic number of unknown, multi-node interventions, and that the intervention targets need not be carefully designed in advance. Through a careful perturbation analysis, we provide a new analysis of this problem that guarantees consistent recovery of (a) the latent causal graph, (b) the mixing matrix and representations, and (c) unknown intervention targets.
Causal abstractions formalize when a high-level structural causal model (SCM) captures the interventional behavior of a lower-level SCM. Existing applications of this notion largely follow a hypothesis-testing paradigm: an expert proposes a candidate high-level model and then evaluates if the low-level system implements it. We study the complementary problem of learning a high-level model directly from low-level measurements. Our contributions leverage hypotheses from low-rank causal discovery, and can be summarized as follows: (1) we show that observations generated by a low-rank graph induce latents that form a causal abstraction, (2) we provide identifiability results about these latents, and (3) we propose a practical objective to learn this high-level SCM.
Théo Saulus, Simon Lacoste-Julien, Dhanya Sridhar
Mila - Quebec AI Institute · Université de Montréal · Canada CIFAR AI Chair
Causal representation learning (CRL) is the process of recovering causally-related latent variables from high-dimensional observations. As a label-free inference method, CRL is particularly attractive for applications where data labels are unavailable or impractical to obtain. While there has been significant progress in understanding the identifiability guarantees of CRL, such guarantees often hold under highly stylized assumptions, which temper the direct application to real-world problems. This paper has a two-fold objective for interventional CRL. First, it establishes identifiability guarantees for substantially weaker interventional assumptions, resulting in block disentanglement of the causal variables, where the block structure depends on the realistically available intervention mechanisms. Secondly, the block disentanglement framework is used for embodied visual state estimation, in which the objective is to recover the latent physical variables of a robotic system directly from visual data (images and videos) without labeled data. These two components are critically complementary. The block disentanglement theory delineates identifiability guarantees under weakened assumptions, and the application demonstrates that the resulting objective remains effective in a controlled embodied setting despite further assumption violations, providing a theory-to-practice bridge needed to translate the promise of label-free CRL into practical problems.