Causal Effects with Unobserved Unit Types in Interacting Human-AI Systems

William Overman; Sadegh Shirani; Mohsen Bayati

stat.MLMar 2, 2026

Causal Effects with Unobserved Unit Types in Interacting Human-AI Systems

Authors: William Overman, Sadegh Shirani, Mohsen Bayati

Organizations: Graduate School of Business, Stanford University

Abstract

We study experiments on interacting populations of humans and AI agents, where both unit types and the interaction network remain unobserved. Although causal effects propagate throughout the system, the goal is to estimate effects on humans. Examples include online platforms where human users interact alongside AI-driven accounts. We assume a human-AI prior that gives each unit a probability of being human. While humans cannot be distinguished at the unit level, the prior allows us to compute the average human composition within large subpopulations. We then model outcome dynamics through a causal message passing (CMP) framework and analyze sample-mean outcomes across subpopulations. We show that by constructing subpopulations that vary in expected human composition and treatment exposure, one can consistently recover human-specific causal effects. Our results characterize when distributional knowledge of population composition (without observing unit types or the interaction network) is sufficient for identification. We validate the approach on a simulated human-AI platform driven by behaviorally differentiated LLM agents. Together, these results provide a theoretical and practical framework for experimentation in emerging human-AI systems.

Figures & tables

Figure 1: Overview of the proposed framework. (a) A mixed population of humans and AI agents with unobserved unit types; each unit $i$ is associated with a known prior $Q_{i}\in[0,1]$ representing the probability of being human, and treatment is assigned at random. (b) Subpopulations are constructed by stratifying units along two axes: expected human composition (horizontal) and treatment exposure (vertical), yielding groups with systematic variation in both dimensions. (c) The experimental state evolution (ESE) is fitted to the aggregate outcome trajectories of these subpopulations, then used to project counterfactual outcomes under full treatment and full control with composition set to $q^{S}=1$ (human-only). The difference between these projected trajectories yields the estimated human total treatment effect (H-TTE).

	$\displaystyle F(\nu,$	$\displaystyle\;\pi^{S},q^{S},\pi;\theta)=\delta_{H}q^{S}+\delta_{A}(1-q^{S})$
		$\displaystyle+\big(\tau_{H}q^{S}+\tau_{A}(1-q^{S})\big)\pi^{S}+\bar{\alpha}\pi+\bar{\beta}\nu+\bar{\gamma}\pi\nu.$

Algorithm 1 Estimating H-TTE via state evolution

Figure 2: Estimated human total treatment effect (H-TTE) on engagement ( $Y_{i,t}\in\{0,\ldots,4\}$ ) across 16 rounds (4 warmup + 12 main). Algorithm 1 is shown under three prior-quality configurations ( $a\in\{0.7,0.8,0.9\}$ , $\sigma=0.15$ ), alongside five baselines. The ground-truth human TTE (black) stabilizes near $0.5$ , while baseline estimators remain near zero due to cancellation between positive human and negative AI responses and network interference. Lines show means and shaded bands denote $\pm 1$ standard error over 10 seeds.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Estimator	MAE	Final Err.	Est. TTE
Alg. 1 ( $a=0.7$ )	$0.101$	$0.062$	$0.605$
Alg. 1 ( $a=0.8$ )	0.037	$-0.061$	$0.475$
Alg. 1 ( $a=0.9$ )	$0.116$	$-0.159$	$0.389$
CMP	$0.483$	$-0.500$	$0.022$
CMP Basic	$0.512$	$-0.543$	$-0.007$
DinM	$0.527$	$-0.532$	$-0.022$

Appendix

Table 1: Estimator performance on engagement (

Y_{i,t}\in\{0,\ldots,4\}

), averaged over the 12 post-warmup rounds. Ground-truth human TTE is

+0.505

, while the population-average TTE is

+0.043

Causal Effect Estimation

Causal effect estimation measures how outcomes would change under interventions, using experiments or observational data and explicit assumptions.

See all 264 papers

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
May 25, 2026Junlin Yang, Dylan Zhang, Xiangchen Song +7Structural Causal Models LLM Agent Evaluation
Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University +2
CausalSteward: An Agentic Divide-Conquer-Combine Copilot for Causal Discovery
Jul 2, 2026Nicholas Tagliapietra, Gian Lorenzo Marchioni, Moritz Willig +3Causal Discovery Multi-Agent Collaboration
1 : Computer Science Department, TU Darmstadt, Germany · 2 : Bosch Center for Artificial Intelligence, Renningen, Germany · 5 : LUISS University, Rome, Italy +2
Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies
Apr 17, 2026Myke C. Cohen, Mingqian Zheng, Neel Bhandari +6Human-AI Interaction
Aptima, Inc. · Arizona State University · Carnegie Mellon University

May 25, 2026J/K move · Enter open · S save

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

Junlin Yang, Dylan Zhang, Xiangchen Song, Qirun Dai+6

Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University+2

Structural Causal Models LLM Agent Evaluation Causal Discovery AI Agent Benchmarks Causal Reasoning in Language Models

Open paper AskPDF arXiv

We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge

F_1

. Mixed observation-intervention strategies improve structural fidelity, while pure intervention remains difficult even for strong agents. We identify premature stopping as a major weakness and show that consistency verification mitigates it. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents' limits as experimental causal reasoners.

Abstract

Causal Effect Estimation

Explore similar work

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

CausalSteward: An Agentic Divide-Conquer-Combine Copilot for Causal Discovery

Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies