Causal Effect Estimation with Latent Textual Treatments

Omri Feldman; Amar Venugopal; Jann Spiess; Amir Feder

cs.CLFeb 17, 2026

Causal Effect Estimation with Latent Textual Treatments

Authors: Omri Feldman, Amar Venugopal, Jann Spiess, Amir Feder

Organizations: School of Computer Science and Engineering, The Hebrew University of Jerusalem · Department of Economics, Stanford University · Graduate School of Business, Stanford University

Abstract

Understanding the causal effects of text on downstream outcomes is a central task in many applications. Estimating such effects requires researchers to run controlled experiments that systematically vary textual features. While large language models (LLMs) hold promise for generating text, producing and evaluating controlled variation requires more careful attention. In this paper, we present an end-to-end pipeline for the generation and causal estimation of latent textual interventions. Our work first performs hypothesis generation and steering via sparse autoencoders (SAEs), followed by robust causal estimation. Our pipeline addresses both computational and statistical challenges in text-as-treatment experiments. We demonstrate that naive estimation of causal effects suffers from significant bias as text inherently conflates treatment and covariate information. We describe the estimation bias induced in this setting and propose a solution based on covariate residualization. Our empirical results show that our pipeline effectively induces variation in target features and mitigates estimation error, providing a robust foundation for causal effect estimation in text-as-treatment settings.

Figures & tables

Figure 1: An overview of our methodology. We start with a dataset of text documents, with labels classifying the semantic concept we want to intervene on. We run probes on SAE representations and identify the most persistent and semantically relevant SAE features that serve as our hypotheses. We then steer texts to generate quasi-counterfactuals that are used in downstream experiments. Finally, we present a novel residualization approach for CATE estimation in such text-as-treatment setting, alongside theoretical guarantees and empirical evidence.

Figure 2: Our residualization pipeline. We assume that natural language is generated from latent semantic concepts, which serve as inputs to the LLM. By applying SAE steering interventions, we modify the model’s generation process and produce quasi-counterfactual texts. We then measure their ex-post intensity $I$ , and use embedding model for representing the text use as controls. Finally, using our measured intensity, we residualize the embeddings to remove the target concept $\mathbf{C}_{1}$ , which ensures that the treatment information is separated from the rest of text.

Figure 3: CATE simulation from Dataset C, using Llama-3.1-8B-Instruct, layer 23 feature 53435, and all-MiniLM-L6-v2 embeddings. Top: distribution of true (black) and estimated CATE based on raw (red) and residualized (blue) covariates. Bottom: scatter of true vs. estimated CATE values based on raw (red) and residualized (blue) covariates, with the 45-degree line in black. Residualization performed via dimension-by-dimension strategy.

Figure 4: Examples of features intensity as measures by the mean cosine similarity for Gemma-2-9b-IT. Left: a high intensity features that follow a linear trend with a relatively high slope, making it a good candidate for causal interventions. Center: a medium intensity feature with a reversed ‘L’ shape, where there is no response to steering at first, followed by a sudden spike, making their suitability for downstream experiments limited. Right: a low intensity feature with a constant slope despite increasing steering factors, making it completely ineffective for experimentation. Plots include only valid texts and steering factors with more than 75% validation rate.

Gemma-2-9B-IT
Layer Position	$I^{*}(\phi)$	$J^{*}(\phi)$	$IC(\phi)$
Early (9)	0.44	2.59	1.16
Middle (20)	1.71	2.41	4.20
Late (31)	0.92	2.51	2.25
Llama 3.1 8B Instruct
Early (7)	0.81	2.33	1.91

Table 1: Mean Intensity (

I^{*}

), Coherence (

J^{*}

), and IC scores grouped by model and layer depth (layer number in parentheses), after removing invalid texts. Middle-layer features are generally preferred for causal interventions as they achieve high intensity scores while maintaining high coherence scores.

Figure 5: Distributions of the out-of-sample predictive accuracy of treatment between raw and residualized embeddings, where residualization is conducted dimension-by-dimension (solid) and by dropping the first principle component (dashed), taken over all tested LLMs, datasets, SAE features, and embedding models.

Panel of Figure 6 — Figure 6: (a) Distributions of the performance gap in predictive accuracy between raw and residualized embeddings across PCA components. (b) IC scores and accuracy improvements in the first principal component (correlation $r=0.463$ ).

Figure 7: Distribution of magnitude of bias (solid) and RMSE (dashed) for raw (red) and residualized (blue) covariates, across all tested DGP functional forms, SAE features, LLMs, datasets, and embedding models. Residualization performed via dimension-by-dimension strategy.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

LLM	$\|$ Bias $\|$ ( $\mathbf{X}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )	RMSE ( $\mathbf{X}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )
Gemma 2 9B IT	$4.25\pm 0.7$	$\mathbf{1.22\pm 0.9}$	$1.18\pm 0.7$	$4.58\pm 0.5$	$\mathbf{2.06\pm 0.9}$	$2.12\pm 0.8$
Llama 3.1 8B Instruct	$\mathbf{3.89\pm 1.0}$	$1.44\pm 1.1$	$\mathbf{1.01\pm 0.7}$	$\mathbf{4.36\pm 0.7}$	$2.39\pm 1.0$	$\mathbf{1.97\pm 0.7}$
Qwen 2.5 7B Instruct	$4.08\pm 1.0$	$1.58\pm 1.1$	$1.13\pm 0.7$	$4.47\pm 0.7$	$2.48\pm 1.1$	$2.13\pm 0.8$

Appendix

Table 2: CATE estimation performance by LLM (averages

\pm

standard deviations).

Embedding Model	$\|$ Bias $\|$ ( $\mathbf{X}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )	RMSE ( $\mathbf{X}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )
EmbeddingGemma-300m	$4.43\pm 0.7$	$1.56\pm 1.0$	$1.17\pm 0.8$	$4.69\pm 0.5$	$2.45\pm 0.9$	$2.1\pm 0.8$
all-MiniLM-L6-v2	$3.99\pm 0.9$	$\mathbf{1.20\pm 0.9}$	$1.08\pm 0.7$	$4.46\pm 0.7$	$\mathbf{2.14\pm 0.9}$	$2.09\pm 0.7$
all-mpnet-base-v2	$\mathbf{3.80\pm 1.0}$	$1.47\pm 1.2$	$\mathbf{1.07\pm 0.6}$	$\mathbf{4.26\pm 0.7}$	$2.33\pm 1.1$	$\mathbf{2.03\pm 0.7}$

Appendix

Table 3: CATE estimation performance by embedding model (averages

\pm

standard deviations).

Dataset	$\|$ Bias $\|$ ( $\mathbf{X}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )	RMSE ( $\mathbf{X}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )
Local Govt Speech	$\mathbf{3.89\pm 1.0}$	$\mathbf{1.0\pm 0.7}$	$\mathbf{0.78\pm 0.5}$	$\mathbf{4.33\pm 0.8}$	$\mathbf{1.94\pm 0.8}$	$\mathbf{1.74\pm 0.7}$
Political Ads	$4.38\pm 0.5$	$2.14\pm 1.1$	$1.37\pm 0.8$	$4.67\pm 0.4$	$2.95\pm 1.0$	$2.27\pm 0.9$
Us vs. Them	$3.99\pm 1.0$	$1.17\pm 0.8$	$1.2\pm 0.6$	$4.43\pm 0.7$	$2.1\pm 0.9$	$2.24\pm 0.6$

Appendix

Table 4: CATE estimation performance by dataset (averages

\pm

standard deviations).

Feature Depth	$\|$ Bias $\|$ ( $\mathbf{X}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )	RMSE ( $\mathbf{X}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )
Early	$\mathbf{3.88\pm 0.9}$	$1.59\pm 1.3$	$\mathbf{0.95\pm 0.7}$	$\mathbf{4.37\pm 0.6}$	$2.49\pm 1.2$	$\mathbf{1.97\pm 0.8}$
Middle	$4.27\pm 0.7$	$1.42\pm 0.8$	$1.15\pm 0.7$	$4.6\pm 0.5$	$2.34\pm 0.8$	$2.08\pm 0.8$
Late	$4.0\pm 1.1$	$\mathbf{1.26\pm 1.0}$	$1.17\pm 0.6$	$4.39\pm 0.8$	$\mathbf{2.12\pm 1.0}$	$2.14\pm 0.7$

Appendix

Table 5: CATE estimation performance by layer position (averages

\pm

standard deviations).

LLM	$\|$ Bias $\|$ ( $\mathbf{X}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )	RMSE ( $\mathbf{X}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )
Gemma 2 9B IT	$2.33\pm 1.9$	$\mathbf{0.27\pm 0.2}$	$\mathbf{0.62\pm 0.3}$	$2.91\pm 1.7$	$\mathbf{1.18\pm 0.4}$	$\mathbf{1.51\pm 0.5}$
Llama 3.1 8B Instruct	$\mathbf{1.76\pm 0.8}$	$0.47\pm 0.5$	$1.07\pm 0.5$	$2.88\pm 0.9$	$1.57\pm 0.7$	$2.24\pm 0.6$
Qwen 2.5 7B Instruct	$1.78\pm 1.2$	$379.38\pm 1278.9$	$0.8\pm 0.7$	$\mathbf{2.7\pm 1.1}$	$1.74e5\pm 5.0e5$	$1.71\pm 0.8$

Appendix

Table 6: CATE estimation performance by LLM (averages

\pm

standard deviations).

Embedding Model	$\|$ Bias $\|$ ( $\mathbf{X}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )	RMSE ( $\mathbf{X}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )
EmbeddingGemma-300m	$2.2\pm 1.4$	$219.95\pm 1058.4$	$0.99\pm 0.8$	$2.98\pm 1.2$	$9.07e4\pm 3.6e5$	$1.95\pm 0.8$
all-MiniLM-L6-v2	$1.84\pm 1.3$	$\mathbf{1.0\pm 1.4}$	$0.76\pm 0.4$	$2.8\pm 1.3$	$\mathbf{2.0\pm 1.3}$	$1.77\pm 0.7$
all-mpnet-base-v2	$\mathbf{1.79\pm 1.3}$	$206.55\pm 889.2$	$\mathbf{0.73\pm 0.4}$	$\mathbf{2.68\pm 1.3}$	$1.05e5\pm 4.1e5$	$\mathbf{1.72\pm 0.7}$

Appendix

Table 7: CATE estimation performance by embedding model (averages

\pm

standard deviations).

Dataset	$\|$ Bias $\|$ ( $\mathbf{X}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )	RMSE ( $\mathbf{X}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )
Local Govt Speech	$3.14\pm 1.1$	$568.13\pm 1531.9$	$1.01\pm 0.7$	$3.84\pm 0.8$	$2.61e5\pm 6.0e5$	$2.07\pm 0.8$
Political Ads	$\mathbf{0.88\pm 0.7}$	$\mathbf{0.38\pm 0.5}$	$\mathbf{0.59\pm 0.4}$	$\mathbf{1.85\pm 1.0}$	$\mathbf{1.32\pm 0.7}$	$\mathbf{1.52\pm 0.7}$
Us vs. Them	$2.48\pm 1.1$	$0.97\pm 1.3$	$1.01\pm 0.5$	$3.37\pm 0.9$	$1.9\pm 1.2$	$2.03\pm 0.6$

Appendix

Table 8: CATE estimation performance by dataset (averages

\pm

standard deviations).

Feature Depth	$\|$ Bias $\|$ ( $\mathbf{X}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	$\|$ Bias $\|$ ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )	RMSE ( $\mathbf{X}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{DD}}$ )	RMSE ( $\mathbf{\widetilde{X}}^{\textrm{Drop}}$ )
Early	$2.38\pm 1.0$	$227.84\pm 1007.8$	$1.03\pm 0.6$	$3.35\pm 0.8$	$1.04e5\pm 4.0e5$	$2.1\pm 0.7$
Middle	$4.50\pm 0.1$	$0.48\pm 0.1$	$0.69\pm 0.1$	$4.77\pm 0.1$	$1.51\pm 0.3$	$1.84\pm 0.2$
Late	$\mathbf{0.55\pm 0.4}$	$\mathbf{0.21\pm 0.3}$	$\mathbf{0.46\pm 0.4}$	$\mathbf{1.38\pm 0.5}$	$\mathbf{1.02\pm 0.5}$	$\mathbf{1.24\pm 0.5}$

Appendix

Table 9: CATE estimation performance by layer position (averages

\pm

standard deviations).

Causal Effect Estimation

Causal effect estimation measures how outcomes would change under interventions, using experiments or observational data and explicit assumptions.

See all 264 papers

Supervised Learning Objectives for High-Dimensional Treatment Effects
May 26, 2026Nikita Dhawan, Arnav Paruthi, Andrew Kim +3Causal Effect Estimation Causal Inference
University of Toronto, Vector Institute · University of Toronto · Vanguard
The Confounder Trap: Treatment-Encoding Representations in Causal Inference with Text
Jul 28, 2026Marie Neubrander, Graham Tierney, Alexander VolfovskyCausal Effect Estimation Causal Representation Learning
Department of Statistical Science Duke University Durham, NC, 27705
Words as Difference Makers: How Large Language Models Determine Causal Structure in Text
Jun 21, 2026Wolfgang PietschCausal World Models LLM Interpretability
tum

May 26, 2026J/K move · Enter open · S save

Supervised Learning Objectives for High-Dimensional Treatment Effects

Nikita Dhawan, Arnav Paruthi, Andrew Kim, Lovedeep Gondara+2

University of Toronto, Vector Institute · University of Toronto · Vanguard

Causal Effect Estimation Causal Inference

Open paper AskPDF arXiv

Predicting the effect of interventions with many possible variations, e.g., therapeutic content that affects mental health outcomes or an earnings call transcript that drives movement in share price, is useful across several domains. However, classical causal estimators tend to assume that all possible interventions are observed, which is infeasible when interventions vary widely, for instance, in the space of all text strings. We adapt a well-known approach of recasting causal inference as a learning problem to address such settings, assuming realizability of the learned function and no unobserved confounding. We show that causal error decomposes into moment-balancing errors of increasing order and exploit structure of the conditional outcome in the confounders to design objectives that directly improve causal estimation. We also show how to project the effect of a high-dimensional treatment onto coarser attributes, which allows a single model to answer several causal questions without additional attribute-specific training. We empirically evaluate our estimators in settings with high-dimensional continuous, discrete, and text treatments. Our experiments demonstrate generalization to held-out treatments, benefits of balancing higher-order moments when required, and projected causal estimates that are competitive with attribute-specific ones.

Abstract

Causal Effect Estimation

Explore similar work

Supervised Learning Objectives for High-Dimensional Treatment Effects

The Confounder Trap: Treatment-Encoding Representations in Causal Inference with Text

Words as Difference Makers: How Large Language Models Determine Causal Structure in Text