When Does Label-Free Deconfounding of Document Embeddings Work?
Organizations: ETH Zurich · Department of Computational Linguistics, University of Zurich · Department of Computational Linguistics University of Zurich · The University of Hong Kong
Abstract
Document embeddings encode spurious attributes that can distort semantic comparisons; yet supervised erasure requires metadata that is often unavailable. We systematically investigate label-free deconfounding across corpora and embedding models. Our results show that label-free methods improve semantic utility over unprocessed embeddings, while simple geometric approaches recover much of the supervised benefit, sometimes matching metadata-supervised erasure in linear confounder recoverability and semantic utility. We identify two distinct causes of ineffective erasure: poor proxy-confounder alignment and insufficient intervention coverage. Crucially, confounder recoverability and semantic utility are distinct dimensions of deconfounding success: utility can improve despite residual confounder information, while stronger suppression does not necessarily improve utility. Moreover, reduced linear recoverability can coexist with substantial nonlinear recoverability. Together, these findings distinguish confounder discovery from intervention effectiveness and highlight the importance of evaluating nuisance suppression alongside semantic preservation.
Figures & tables
| Linked groups | |||||||
|---|---|---|---|---|---|---|---|
| Corpus | Nuisance (classes) | C | Docs | pairs | triples | Ret. pairs | Semantic evaluation |
| SemEval | Language (DE/EN/ES/ZH) | 4 | 856 | 428 | 0 | 428 | Retrieval, graded sim. |
| SwiLTra | Language (DE/FR/IT) | 3 | 8,352 | 0 | 2,784 | 8,352 | Retrieval |
| Fedlex | Language (DE/FR/IT) | 3 | 8,352 | 0 | 2,784 | 8,352 | Retrieval |
| Bills–Tweets | Source (bills/tweets) | 2 | 9,933 | – | – | – | None (leakage only) |
| SCOTUS | Source (Oyez/Lexis/Wiki) | 3 | 7,506 | 1,286 | 633 | 3,185 | Retrieval |
| E5-small | MiniLM | ||||||
| SemEval | SwiLTra | Fedlex | Bills–Tweets | SCOTUS | CAP | SCOTUS | |
| C=4 | C=3 | C=3 | C=2 | C=3 | C=3 | C=3 | |
| Panel A: linear-probe nuisance ROC-AUC (one-vs-rest macro average for ) | |||||||
| Raw | 1.000 .000 | .999 .000 | 1.000 .000 | 1.000 .000 | .999 .000 | 1.000 .000 | .986 .000 |
| PCA-1 | .927 .006 | .906 .004 | .911 .006 | .678 .014 | .882 .004 | .892 .008 | .986 .000 |
| -means LEACE | .797 .050 | .577 .011 | .644 .016 | .612 .011 | .699 .085 | .879 .012 | .950 .047 |
| SwiLTra (DE/FR/IT) | SCOTUS (3 sources) | ||||||
| Condition | Lin. AUC | MLP AUC | R@10 | Lin. AUC | MLP AUC | R@10 | |
| A. Direction selection (Directions-LEACE; except A4) | |||||||
| A1 | MI-selected SAE directions | .616 .010 | .838 .007 | .914 .004 | .727 .013 | .797 .014 | .774 .005 |
| A2 | Random SAE directions | .630 .010 | .842 .007 | .909 .003 | .742 .002 | .814 .005 | .772 .005 |
| A3 | Random non-SAE directions | .819 .015 | .913 .007 | .923 .003 | .887 .010 | .916 .006 | .771 .006 |
| A4 | Non-SAE directions, displ.-matched | .781 .012 | .893 .007 | .911 .003 | .861 .009 | .894 .006 | .767 .004 |
| Linear AUC | Utility | |||||||
|---|---|---|---|---|---|---|---|---|
| Setting | Displ. | SAE | PCA | PCA SAE | SAE | PCA | PCA SAE | |
| SwiLTra (DE/FR/IT) | .36 | 22–23 | .616 | .600 | .016 [ .021, .011] | .914 | .882 | .032 [ .037, .026] |
| SCOTUS (3 sources) | .30 | 22 | .727 | .677 | .049 [ .065, .033] | .774 | .774 | +.001 [ .003, +.005] |
| SemEval | .42 | 18–21 | .702 | .684 | .018 [ .035, .000] | .807 | .814 | +.007 [ .001, +.015] |
| CAP | .33 | 20–21 | .713 | .687 | .026 [ .035, .017] | .284 | .215 | .069 [ .135, .003] |
| Fedlex (DE/FR/IT) | .36 | 21 | .674 | .665 | .009 [ .015, .002] | .939 | .926 | .013 [ .019, .008] |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Docs per class | Docs | Pairs | Unpaired | ||
| SwiLTra | DE | FR | IT | |||
| DE–FR | 2,784 | 2,784 | – | 5,568 | 2,784 | 0 |
| DE–IT | 2,784 | – | 2,784 | 5,568 | 2,784 | 0 |
| FR–IT | – | 2,784 | 2,784 | 5,568 | 2,784 | 0 |
| DE–FR–IT | 2,784 | 2,784 | 2,784 | 8,352 | 8,352 | 0 |
| Fedlex | DE | FR | IT | |||
| Encoder | Hugging Face model | Dim. |
|---|---|---|
| MiniLM | sentence-transformers/all-MiniLM-L6-v2 | 384 |
| E5-small a | intfloat/multilingual-e5-small | 384 |
| GIST-small | avsolatorio/GIST-small-Embedding-v0 | 384 |
| GIST-base | avsolatorio/GIST-Embedding-v0 | 768 |
| GIST-large | avsolatorio/GIST-large-Embedding-v0 | 1024 |
| E5-base a | intfloat/multilingual-e5-base | 768 |
| Component | Setting |
|---|---|
| SAE training | Adam; learning rate ; batch size 1,024; 200 epochs. Initially and ; decoder rows have unit norm after each step. AuxK: , ; a latent is dead after 20 epochs without firing. |
| Pseudo-groups | Scikit-learn -means, . Maximize silhouette on at most 4,000 training documents; ties favor smaller . |
| LEACE | Continuous concept columns are standardized using training statistics. |
| Orthogonal bases | Uncentered decoder-span removal: SVD cut-off . Mean-preserving variants: basis cut-off times the largest singular value. |
| Sweeps | Figure 1 : , as above, . Frozen-transfer PCA sweep: the same grid without 64. |
| Probes | Logistic regression: at most 5,000 iterations; regularization . Standardized features; coordinates are dropped if their SD is below times the median SD. MLP: hidden layers , at most 300 iterations, early stopping. |
| SemEval | SwiLTra DE–FR | SCOTUS O–L | CAP | Fedlex | |||||||||||
| Encoder | PCA | KM | SAE | PCA | KM | SAE | PCA | KM | SAE | PCA | KM | SAE | PCA | KM | SAE |
| MiniLM | |||||||||||||||
| GIST-small | |||||||||||||||
| GIST-base | |||||||||||||||
| GIST-large | |||||||||||||||
| E5-small | |||||||||||||||
Representation Disentanglement
Representation disentanglement studies how to separate distinct factors of variation in learned features so models can interpret, control, and transfer them.
See all 103 papers