cs.AIOct 9, 2026

ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence

Authors: Yiwei Chen, Amila Weerasinghe, Siddharth Biswal, Utkrisht Rajkumar, Olcay Boz, Taylor Berg-Kirkpatrick, Rose Yu

Organizations: UC San Diego · Amazon

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system rather than merely answer a prompt. A useful evaluation must let agents choose experiments, interpret finite evidence, and make causal decisions, while keeping the environment coherent across repeated observations and interventions and preserving objective ground truth. We introduce ACED-BENCH (Active Causal Evidence and Decision Benchmark), a neuro-symbolic benchmark built around this requirement. The key idea is to separate the readable scientific surface from the formal causal world: LLMs generate natural-language scenarios and task text, while hidden probabilistic graphical models define the causal graph, data-generating process, intervention semantics, validation checks, and oracle answers. This yields scientific worlds that are natural for agents to investigate but exact enough for scalable, programmatic evaluation. The main leaderboard ACED-CORE contains 180 tasks: 60 set-valued causal ancestry/descendancy questions and 120 constrained causal-decision tasks. We also report ACED-QUAL, a 150-question yes/no qualification check using the same hidden-PGM interface. We evaluate seven contemporary LLMs and two earlier-generation models across three active agent designs. ACED-QUAL checks whether agents can use the interface to draw conclusions from evidence, while ACED-CORE tests whether they can decide what evidence to collect and when to commit to an answer. Evidence-ledger analysis localizes failures within the investigation: agents test the wrong intervention, lose relevant evidence, or continue after sufficient evidence has appeared. These results suggest that reliable AI co-scientists need explicit support for experiment design, evidence retention, answer commitment, and stopping.

Figures & tables

Figure 1: Overview of ACED-Bench on a hard causal-decision task. Each task is generated from a hidden PGM. The benchmark oracle has access to the causal graph, CPDs, and exact interventional answers. The LLM scientist agent only observes the scenario, variable metadata, non-intervenability annotations, and the question, and must answer under a four-turn budget.
Figure 2: Generation pipeline for the causal-decision portion of ACED-Core . A decision pattern and topic determine semantic roles, graph constraints, CPD construction, validation checks, and the final question. LLMs provide readable scientific surface forms, while code enforces causal validity and computes oracle answers.
RegimeAnswer formatTasks
ACED-CoreLists / decisions60 causal list + 120 decision (180 total)
ACED-QualYes/no150 causal-effect, marginal, conditional
Table 1: Evaluation regimes in ACED-Bench . ACED-Core is the main benchmark, while ACED-Qual is a qualification check for basic evidence use.
Figure 3: Accuracy on ACED-Core , the main hard benchmark. Results use the four-turn budgeted setting and combine causal ancestry/descendancy list queries with constrained causal-decision tasks. The budget applies to active agents; guess-only cells are omitted because they do not use interaction turns. Error bars are 95%95\% world-clustered bootstrap intervals.
Figure 4: Accuracy on ACED-Qual , the 150-question yes/no qualification check. Results use the same four-turn budgeted setting as ACED-Core . These questions test causal-effect, marginal, and conditional judgments from actively requested samples. The budget applies to active agents; guess-only cells are omitted because they do not use interaction turns. Error bars are 95%95\% world-clustered bootstrap intervals.
Figure 5: Evidence-ledger diagnostics for modular active-code trajectories on ACED-Core . The left panel reports final-answer accuracy, evidence acquisition, evidence retention, and ignored evidence. The right panel reports extra turns after the first strong-evidence turn.
Figure 6: Example evidence-ledger trace for a safety-constrained ACED-Core decision task. Strong evidence appears only after the agent compares candidate interventions on both the target and safety outcomes.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Interaction loop for structural questions, including the ACED-Qual qualification regime. The benchmark oracle samples from a hidden PGM, while the LLM scientist agent observes only the scenario, variable metadata, non-intervenability annotations, and question. The agent requests observational or interventional samples and answers structural statistical or causal queries, such as marginal, conditional, causal-effect, or causal-ancestry/descendancy questions.
Figure 8: Evidence acquisition versus evidence retention for modular active-code trajectories. The scatter compares ACED-Qual with ACED-Core ; bubble size indicates ignored evidence. The ledger is a diagnostic trajectory analysis, not a separate benchmark score.
Figure 9: Accuracy on ACED-Core under the Samp5k sample-budget ablation. Each task permits at most 5,000 sampled rows, changing the constrained resource from interaction turns to sample allocation.
More in this topic

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

Explore similar work

CardsList
Jul 5, 2026

CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention. Since scientific discovery fundamentally relies on uncovering causal relationships from observations, the capability of causal thinking, i.e., distinguishing causation from correlation and recognizing hidden biases, is essential to LLM agents. Although a number of benchmarks exist for AI Scientists, none explicitly incorporate challenges from selection bias, measurement error, and hidden confounders that widely exist in real-world scientific discovery. To this end, we present CausalGame, a benchmark that evaluates the causal thinking capabilities of LLM agents through interactive games. CausalGame asks LLM agents to actively design experimental protocols, collect observation data, and derive a final solution with an explanation report. To emulate realistic scientific discovery challenges, we design 14 scenarios that incorporate selection bias, measurement error, and hidden confounders. Across 30 LLM agents, none demonstrates reliable causal thinking: the best model reaches only 68.0% survival against analytical optima of 78-85%, and merely 5-7% of sessions receive credits on the causal-reasoning rubrics. CausalGame provides a scalable and controlled testbed for evaluating the causal thinking of AI Scientist agents.

Zhenhao Chen, Yongqiang Chen, Chenxi Liu +7

1MBZUAI · 2Carnegie Mellon University · 3TMLR Group, Hong Kong Baptist University +2

AI Agents for Scientific DiscoveryAI Agent BenchmarksCausal Reasoning in Language ModelsCausal Reasoning
AskPDF
Jul 9, 2026

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Large language models (LLMs) increasingly act as data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without causal data-generating structure or a range of causal data science tasks. Existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a real domain. We ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the memorization risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to imperfect observations. Abstention is tested directly: for targets the graph does not identify, the graded answer is to abstain. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding. Because the generator is fully synthetic, the exam is not a fixed dataset: composition, difficulty, and size are parameters, and fresh exams can be drawn at will.

Andrej Leban, Theo Saarinen, Yuekai Sun

Department of Statistics, University of Michigan, Ann Arbor, MI, United States

Structural Causal ModelsAI Agent BenchmarksCausal Reasoning in Language ModelsSynthetic Benchmark GenerationCausal ReasoningData Analysis Agents
AskPDF
May 25, 2026

CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge F1F_1. Mixed observation-intervention strategies improve structural fidelity, while pure intervention remains difficult even for strong agents. We identify premature stopping as a major weakness and show that consistency verification mitigates it. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents' limits as experimental causal reasoners.

Junlin Yang, Dylan Zhang, Xiangchen Song +7

Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University +2

Structural Causal ModelsLLM Agent EvaluationCausal DiscoveryAI Agent BenchmarksCausal Reasoning in Language Models
AskPDF