We study experiments on interacting populations of humans and AI agents, where both unit types and the interaction network remain unobserved. Although causal effects propagate throughout the system, the goal is to estimate effects on humans. Examples include online platforms where human users interact alongside AI-driven accounts. We assume a human-AI prior that gives each unit a probability of being human. While humans cannot be distinguished at the unit level, the prior allows us to compute the average human composition within large subpopulations. We then model outcome dynamics through a causal message passing (CMP) framework and analyze sample-mean outcomes across subpopulations. We show that by constructing subpopulations that vary in expected human composition and treatment exposure, one can consistently recover human-specific causal effects. Our results characterize when distributional knowledge of population composition (without observing unit types or the interaction network) is sufficient for identification. We validate the approach on a simulated human-AI platform driven by behaviorally differentiated LLM agents. Together, these results provide a theoretical and practical framework for experimentation in emerging human-AI systems.
Figures & tables
Figure 1: Overview of the proposed framework. (a) A mixed population of humans and AI agents with unobserved unit types; each unit i is associated with a known prior Qi∈[0,1] representing the probability of being human, and treatment is assigned at random. (b) Subpopulations are constructed by stratifying units along two axes: expected human composition (horizontal) and treatment exposure (vertical), yielding groups with systematic variation in both dimensions. (c) The experimental state evolution (ESE) is fitted to the aggregate outcome trajectories of these subpopulations, then used to project counterfactual outcomes under full treatment and full control with composition set to qS=1 (human-only). The difference between these projected trajectories yields the estimated human total treatment effect (H-TTE).
F(ν,
πS,qS,π;θ)=δHqS+δA(1−qS)
+(τHqS+τA(1−qS))πS+αˉπ+βˉν+γˉπν.
Algorithm 1 Estimating H-TTE via state evolution
Figure 2: Estimated human total treatment effect (H-TTE) on engagement ( Yi,t∈{0,…,4} ) across 16 rounds (4 warmup + 12 main). Algorithm 1 is shown under three prior-quality configurations ( a∈{0.7,0.8,0.9} , σ=0.15 ), alongside five baselines. The ground-truth human TTE (black) stabilizes near 0.5 , while baseline estimators remain near zero due to cancellation between positive human and negative AI responses and network interference. Lines show means and shaded bands denote ±1 standard error over 10 seeds.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Estimator
MAE
Final Err.
Est. TTE
Alg. 1 ( a=0.7 )
0.101
0.062
0.605
Alg. 1 ( a=0.8 )
0.037
−0.061
0.475
Alg. 1 ( a=0.9 )
0.116
−0.159
0.389
CMP
0.483
−0.500
0.022
CMP Basic
0.512
−0.543
−0.007
DinM
0.527
−0.532
−0.022
Appendix
Table 1: Estimator performance on engagement ( Yi,t∈{0,…,4} ), averaged over the 12 post-warmup rounds. Ground-truth human TTE is +0.505 , while the population-average TTE is +0.043 .
We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge F1. Mixed observation-intervention strategies improve structural fidelity, while pure intervention remains difficult even for strong agents. We identify premature stopping as a major weakness and show that consistency verification mitigates it. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents' limits as experimental causal reasoners.
Junlin Yang, Dylan Zhang, Xiangchen Song +7
Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University +2
Learning causal models from high-dimensional data is a significant challenge, particularly in real-world settings where violations of core assumptions lead to causal identifiability issues. Although massive amounts of prior knowledge are available, and contain valuable causal information, effectively integrating this knowledge into the causal discovery process remains an open problem. We introduce CausalSTeward (CAST), a novel human-in-the-loop framework for interactively assembling large causal models. CausalSteward is a multi-agent collaborative system that tackles high-dimensional causality through a divide-and-conquer approach where large clusters of variables are iteratively partitioned and then separately analyzed. Our framework fuses prior knowledge with a data-driven approach by using tailored tools such as retrieval augmented generation and conditional independence tests. Finally, we use this work to examine the capabilities and limitations of causal reasoning in multi-agent frameworks, and how the human-in-the-loop can contribute to accurate and trustworthy results.
Nicholas Tagliapietra, Gian Lorenzo Marchioni, Moritz Willig +3
1 : Computer Science Department, TU Darmstadt, Germany · 2 : Bosch Center for Artificial Intelligence, Renningen, Germany · 5 : LUISS University, Rome, Italy +2
AI design characteristics and human personality traits each impact the quality and outcomes of human-AI interactions. However, their relative and joint impacts are underexplored in imperfectly cooperative scenarios, where people and AI only have partially aligned goals and objectives. This study compares a purely simulated dataset comprising 2,000 simulations and a parallel human subjects experiment involving 290 human participants to investigate these effects across two scenario categories: (1) hiring negotiations between human job candidates and AI hiring agents; and (2) human-AI transactions wherein AI agents may conceal information to maximize internal goals. We examine user Extraversion and Agreeableness alongside AI design characteristics, including Adaptability, Expertise, and chain-of-thought Transparency. Our causal discovery analysis extends performance-focused evaluations by integrating scenario-based outcomes, communication analysis, and questionnaire measures. Results reveal divergences between purely simulated and human study datasets, and between scenario types. In simulation experiments, personality traits and AI attributes were comparatively influential. Yet, with actual human subjects, AI attributes -- particularly transparency -- were much more impactful. We discuss how these divergences vary across different interaction contexts, offering crucial insights for the future of human-centered AI agents.
Myke C. Cohen, Mingqian Zheng, Neel Bhandari +6
Aptima, Inc. · Arizona State University · Carnegie Mellon University