When Does Pre-Trained Capacity Help Causal Learning? From Scalar Prediction to Counterfactual Post-Training

Qi Xu; Zijie Lei; Zhifei Deng; Zhigang Hua; Luke Simon

cs.LGOct 23, 2026

When Does Pre-Trained Capacity Help Causal Learning? From Scalar Prediction to Counterfactual Post-Training

Authors: Qi Xu, Zijie Lei, Zhifei Deng, Zhigang Hua, Luke Simon

Organizations: Meta Platforms, Inc., Sunnyvale, CA, USA

Abstract

How does pre-trained model capacity relate to performance after adaptation for causal tasks? We study two systems built on the Qwen 2.5 family: CausalLLM, which estimates scalar potential outcomes with treatment-specific heads, and CausalGen, which generates treatment-conditioned text or code using soft prefixes and factual supervised fine-tuning (SFT), counterfactual SFT, and preference alignment. Increasing backbone size from 0.5B to 1.5B yields mixed scalar-estimation results on IHDP, but improves CodeGen pass@1 from 0.267 to 0.575 and treatment-effect score from 0.231 to 0.515 after the complete generative pipeline. Within 0.5B CausalLLM, narrative inputs reduce mean sqrt(PEHE) from 1.842 to 1.780, a 3.4% improvement over tabular inputs; the 5.8% improvement over a no-LLM baseline combines changes in representation and input format. These descriptive comparisons motivate a distinction between available and convertible capacity. They do not isolate the contribution of post-training: the two systems differ in dataset, architecture, output modality, and supervision, and intermediate-stage evaluations are unavailable. With only two small backbones and three replications, the results support a hypothesis about capacity--adaptation alignment rather than a general scaling law or a demonstrated causal mechanism.

Figures & tables

Figure 1: Two adaptation regimes built on pre-trained LLMs. (a) CausalLLM adapts a shared encoder with lightweight treatment-specific scalar heads for ITE estimation. (b) CausalGen uses treatment-specific projections and soft prefixes followed by factual SFT, counterfactual SFT, and causal preference alignment. Only the 0.5B and 1.5B backbones and DPO-style preference alignment are evaluated in this paper; the additional 3B, GRPO, and DAPO options shown in the schematic are not evaluated. The systems differ in data, heads, output modality, supervision, and training procedure; their scaling contrast is descriptive.

	IHDP		Twins
Method	$\sqrt{\text{PEHE}}$ $\downarrow$	ATE error $\downarrow$	$\sqrt{\text{PEHE}}$ $\downarrow$	ATE error $\downarrow$
T-Learner	1.96 $\pm$ .05	0.21	.467 $\pm$ .012	.056
TARNet	2.02 $\pm$ .08	0.34	.454 $\pm$ .012	.012
Dual-head MLP (no LLM)	1.89 $\pm$ .03	0.31	.452 $\pm$ .007	.035
CausalLLM (Narrative)	1.78 $\pm$ .19	0.48	.450 $\pm$ .009	.012

Table 1: Scalar ITE estimation (3 reps, mean

\pm

std). “Dual-head MLP” is the no-LLM baseline shown as “MLP (no LLM)” in Fig. 2 ; CausalLLM (Narrative) uses the pre-trained LLM representation with the same treatment-specific prediction structure.

\downarrow

= lower is better.

Figure 2: Input-format comparison (IHDP, 0.5B). Narrative inputs reduce mean

\sqrt{\mathrm{PEHE}}

by 3.4% versus tabular CausalLLM. The 5.8% reduction versus the no-LLM MLP is a different, combined comparison; uncertainty for the format differences is not available.

	CodeGen		Jobs
Method	BLEU $Y(0)$	TE Score	BLEU $Y(0)$	TE Score
Single-head	0.151	—	0.193	—
CausalGen	0.374	0.354	0.440	0.312
$\Delta$ vs. single	2.5 $\times$		2.3 $\times$

Table 2: Counterfactual generation (3 reps). The final dual-head system has 2.3–2.5

\times

higher reference BLEU for treatment arm

Y(0)

Mode	pass@1	TE Score	CF Faith.	Swap
Subtract	0.100	0.294	0.373	0.611
Gate	0.167	0.354	0.429	0.592
Cross-Attn	0.408	0.363	0.379	0.654
Bilinear	0.200	0.213	0.377	0.430

Table 3: DeltaHead ablation (CodeGen, 0.5B, 3 reps).

	CausalLLM (IHDP)			CausalGen (CodeGen)
Size	$\sqrt{\text{PEHE}}$	ATE err.	RMSE	pass@1	TE	ATE dir.%
0.5B	1.780	0.475	1.247	0.267	0.231	73.3
1.5B	1.796	0.445	1.259	0.575	0.515	93.3

Table 4: Effect of increasing backbone size from 0.5B to 1.5B. Scalar ITE estimation on IHDP shows no consistent gain across error metrics, whereas counterfactual generation on CodeGen improves substantially. We treat this two-size comparison as a task-dependent capacity effect, not a scaling law.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Backbone family	Qwen 2.5; 0.5B and 1.5B in the size comparison
LoRA	Rank 16; scaling $\alpha=32$
Scalar heads (0.5B)	$896\to 256\to 128\to 1$ ; ReLU, LayerNorm, dropout
Generative projection	$d\to 2d\to d$ ; GELU; residual connection
Soft prefix	8 continuous embeddings
MLP optimizer	Adam; learning rate $10^{-3}$ ; 100 epochs

Appendix

Table 5: Documented experimental configuration.

Causal Inference

Causal inference studies how interventions and counterfactuals reveal cause-and-effect relationships beyond observed associations.

See all 318 papers

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
Apr 9, 2024Yu Zhou, Xingyu Wu, Jibin Wu +2LLM Evaluation Causal Discovery
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency
Feb 18, 2026Victoria Lin, Xinnuo Xu, Rachel Lawrence +4Counterfactual Reasoning in Language Models Causal Reasoning in Language Models
Carnegie Mellon University · Microsoft Research Cambridge · Microsoft Research India
CauSim: Scaling Causal Reasoning with Increasingly Complex Causal Simulators
May 9, 2026Nicolás Astorga, Anita Kriz, Mihaela van der SchaarStructural Causal Models Knowledge Augmentation for Language Models
DAMTP, University of Cambridge, Cambridge, UK

Apr 9, 2024J/K move · Enter open · S save

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng+1

Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China

LLM Evaluation Causal Discovery Causal Structure Learning Causal Reasoning in Language Models Causal Reasoning

Open paper AskPDF arXiv

The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual reasoning, as causality reveals the underlying data distribution. However, the lack of a comprehensive benchmark currently limits the evaluation of LLMs' causal learning capabilities. To fill this gap, this paper develops CausalBN-Bench based on data from the causal research community, enabling comparative evaluations of LLMs against traditional causal learning algorithms. To provide a comprehensive investigation, we offer three tasks of varying difficulties, including correlation, causal skeleton, and causality identification. Evaluations of 19 leading LLMs reveal that, while closed-source LLMs show potential for simple causal relationships, they significantly lag behind traditional algorithms on larger-scale networks (

>50

nodes). Specifically, LLMs struggle with collider structures but excel at chain structures, especially at long-chain causality analogous to Chain-of-Thought techniques. This supports the current prompt approaches while suggesting directions to enhance LLMs' causal reasoning capability. Furthermore, CausalBN-Bench incorporates background knowledge and training data into prompts to thoroughly unlock LLMs' text-comprehension ability during evaluation, whose findings indicate that, LLM understand causality through semantic associations with distinct entities, rather than directly from contextual information or numerical distributions.

Abstract

Causal Inference

Explore similar work

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency

CauSim: Scaling Causal Reasoning with Increasingly Complex Causal Simulators