How does pre-trained model capacity relate to performance after adaptation for causal tasks? We study two systems built on the Qwen 2.5 family: CausalLLM, which estimates scalar potential outcomes with treatment-specific heads, and CausalGen, which generates treatment-conditioned text or code using soft prefixes and factual supervised fine-tuning (SFT), counterfactual SFT, and preference alignment. Increasing backbone size from 0.5B to 1.5B yields mixed scalar-estimation results on IHDP, but improves CodeGen pass@1 from 0.267 to 0.575 and treatment-effect score from 0.231 to 0.515 after the complete generative pipeline. Within 0.5B CausalLLM, narrative inputs reduce mean sqrt(PEHE) from 1.842 to 1.780, a 3.4% improvement over tabular inputs; the 5.8% improvement over a no-LLM baseline combines changes in representation and input format. These descriptive comparisons motivate a distinction between available and convertible capacity. They do not isolate the contribution of post-training: the two systems differ in dataset, architecture, output modality, and supervision, and intermediate-stage evaluations are unavailable. With only two small backbones and three replications, the results support a hypothesis about capacity--adaptation alignment rather than a general scaling law or a demonstrated causal mechanism.
Figures & tables
Figure 1: Two adaptation regimes built on pre-trained LLMs. (a) CausalLLM adapts a shared encoder with lightweight treatment-specific scalar heads for ITE estimation. (b) CausalGen uses treatment-specific projections and soft prefixes followed by factual SFT, counterfactual SFT, and causal preference alignment. Only the 0.5B and 1.5B backbones and DPO-style preference alignment are evaluated in this paper; the additional 3B, GRPO, and DAPO options shown in the schematic are not evaluated. The systems differ in data, heads, output modality, supervision, and training procedure; their scaling contrast is descriptive.
IHDP
Twins
Method
PEHE↓
ATE error ↓
PEHE↓
ATE error ↓
T-Learner
1.96 ± .05
0.21
.467 ± .012
.056
TARNet
2.02 ± .08
0.34
.454 ± .012
.012
Dual-head MLP (no LLM)
1.89 ± .03
0.31
.452 ± .007
.035
CausalLLM (Narrative)
1.78 ± .19
0.48
.450 ± .009
.012
Table 1: Scalar ITE estimation (3 reps, mean ± std). “Dual-head MLP” is the no-LLM baseline shown as “MLP (no LLM)” in Fig. 2 ; CausalLLM (Narrative) uses the pre-trained LLM representation with the same treatment-specific prediction structure. ↓ = lower is better.
Figure 2: Input-format comparison (IHDP, 0.5B). Narrative inputs reduce mean PEHE by 3.4% versus tabular CausalLLM. The 5.8% reduction versus the no-LLM MLP is a different, combined comparison; uncertainty for the format differences is not available.
CodeGen
Jobs
Method
BLEU Y(0)
TE Score
BLEU Y(0)
TE Score
Single-head
0.151
—
0.193
—
CausalGen
0.374
0.354
0.440
0.312
Δ vs. single
2.5 ×
2.3 ×
Table 2: Counterfactual generation (3 reps). The final dual-head system has 2.3–2.5 × higher reference BLEU for treatment arm Y(0) .
Table 4: Effect of increasing backbone size from 0.5B to 1.5B. Scalar ITE estimation on IHDP shows no consistent gain across error metrics, whereas counterfactual generation on CodeGen improves substantially. We treat this two-size comparison as a task-dependent capacity effect, not a scaling law.
The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual reasoning, as causality reveals the underlying data distribution. However, the lack of a comprehensive benchmark currently limits the evaluation of LLMs' causal learning capabilities. To fill this gap, this paper develops CausalBN-Bench based on data from the causal research community, enabling comparative evaluations of LLMs against traditional causal learning algorithms. To provide a comprehensive investigation, we offer three tasks of varying difficulties, including correlation, causal skeleton, and causality identification. Evaluations of 19 leading LLMs reveal that, while closed-source LLMs show potential for simple causal relationships, they significantly lag behind traditional algorithms on larger-scale networks (>50 nodes). Specifically, LLMs struggle with collider structures but excel at chain structures, especially at long-chain causality analogous to Chain-of-Thought techniques. This supports the current prompt approaches while suggesting directions to enhance LLMs' causal reasoning capability. Furthermore, CausalBN-Bench incorporates background knowledge and training data into prompts to thoroughly unlock LLMs' text-comprehension ability during evaluation, whose findings indicate that, LLM understand causality through semantic associations with distinct entities, rather than directly from contextual information or numerical distributions.
Yu Zhou, Xingyu Wu, Jibin Wu +2
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
Despite their strong performance on reasoning benchmarks, large language models (LLMs) have proven brittle when presented with counterfactual questions, suggesting weaknesses in their causal reasoning ability. While recent work has demonstrated that labeled counterfactual tasks can be useful benchmarks of LLMs' causal reasoning, producing such data at the scale required to cover the vast potential space of counterfactuals is limited. In this work, we introduce double counterfactual consistency (DCC), a lightweight inference-time method for measuring and guiding the ability of LLMs to reason causally. Without requiring labeled counterfactual data, DCC verifies a model's ability to execute two important elements of causal reasoning: causal intervention and counterfactual prediction. Using DCC, we evaluate the causal reasoning abilities of various leading LLMs across a range of reasoning tasks and interventions. Moreover, we demonstrate the effectiveness of DCC as a training-free test-time rejection sampling criterion and show that it can directly improve performance on reasoning tasks across multiple model families.
Victoria Lin, Xinnuo Xu, Rachel Lawrence +4
Carnegie Mellon University · Microsoft Research Cambridge · Microsoft Research India
Despite surpassing human performance across mathematics, coding, and other knowledge-intensive tasks, large language models (LLMs) continue to struggle with causal reasoning. A core obstacle is the target data itself: causal systems are complex and often expressed in non-executable forms, while ground-truth answers to causal queries are inherently scarce. We introduce CauSim, a framework that turns causal reasoning from a scarce-label problem into a scalable supervised one. CauSim constructs increasingly complex causal simulators: executable structural causal models (SCMs), incrementally built by LLMs, that scale to globally complex systems while maintaining verifiable answers to causal queries. CauSim operates across representations by formalizing non-executable causal knowledge into code, enabling data augmentation, and translating executable SCMs into natural language, enabling supervision in previously difficult-to-supervise representations. We structure our research into two parts: (1) how to construct increasingly complex causal simulators, and (2) a systematic study of what CauSim enables, demonstrating generalization across representations, consistent gains from curriculum scaling and data volume, LLM self-improvement through self-generated simulators, and data augmentation via formalization of existing domain knowledge.
Nicolás Astorga, Anita Kriz, Mihaela van der Schaar