SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

Yusheng Zhao; Bowei He; Xue Liu; Mingxiao Li

cs.AIOct 27, 2026

SciGame: Let Agents Play the Scientific Discovery Games in the Real and Counterfactual World

Authors: Yusheng Zhao, Bowei He, Xue Liu, Mingxiao Li

Organizations: MBZUAI · McGill University · Shanghai AI Lab

Abstract

Large language model agents are increasingly used for scientific discovery. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors. We introduce SciGame, a controlled benchmark that converts scientific literature into structured scientific cards. Each card yields a pair of Original and dimensionally consistent Counterfactual environments with the same task interface. SciGame contains a Historical collection of canonical laws and a live Temporal collection curated from papers published after model knowledge cutoffs. It evaluates agents in candidate-guided Identification and open-ended Discovery settings. Each setting runs under Prior-only, Fixed, and Adaptive protocols. Experiments on five frontier models yield three findings. First, familiar priors hurt performance once the governing relation is altered. Second, adaptive experimentation helps the strongest models but harms weaker ones under limited budgets. Third, good explanations can hide quantitative failures. Recovering the exact symbolic form remains the main bottleneck, even when agents interpret evidence correctly. Current agents can investigate systems actively, but inducing exact laws in unfamiliar settings remains an open challenge.

Figures & tables

Benchmark	Active experiments	Numerical prediction	Paper grounding	Counterfactual	Temporal selection
DiscoveryWorld ( Jansen et al., 2024 )	$\checkmark$	$\checkmark$	—	—	—
FalsifyBench ( Bertolazzi et al., 2026 )	$\checkmark$	—	—	—	—
Auto-Discovery-Bench ( Chen et al., 2025a )	$\checkmark$	$\checkmark$	—	—	—
LLM-SRBench ( Shojaee et al., 2025 )	—	$\checkmark$	—	$\checkmark$	—
BoxingGym ( Gandhi et al., 2025 )	$\checkmark$	$\checkmark$	$\checkmark$	—	—
ActiveSciBench-Chem ( Kabra et al., 2026 )	$\checkmark$	$\checkmark$	—	$\checkmark$	—

Table 1: Comparison of scientific law discovery benchmarks and our work. “

\checkmark

” indicates supported and “—” unsupported. Paper grounding covers paper-derived tasks, Counterfactual includes mutated relation variants, and Temporal selection denotes date-based source curation.

Figure 1: Overview of the study pipeline. Scientific cards define the relations, variables, and applicability conditions used to build paired environments. The central panel illustrates the Adaptive protocol, where the Player designs experiments and chooses when to stop. Each episode is evaluated on predictions, explanations, and interaction trajectories.

Figure 2: Data curation pipeline . Part A describes the workflow on how to formulate accepted scientific cards from temporal sources. Part B describes how both historical and temporal corpus be curated as task family collections.

Figure 3: Comparison of observation protocols. All three methods share the initial task description and assigned model information. Fixed provides a batch of observations from $B$ pre-specified experiments upfront. Adaptive allows sequential experiment proposal without a pre-set budget constraint.

Figure 4: Evaluation pipeline.

		Historical												Temporal
		Identification						Discovery						Identification						Discovery
		Original			Counterfactual			Original			Counterfactual			Original			Counterfactual			Original			Counterfactual
Model	Method	ESR	PS	SES	ESR	PS	SES	ESR	PS	SES	ESR	PS	SES	ESR	PS	SES	ESR	PS	SES	ESR	PS	SES	ESR	PS	SES
DeepSeek-V4 Flash	Prior-only	88.3	0.950	0.707	0.0	0.136	0.401	31.7	0.574	0.684	0.0	0.100	0.218	40.3	0.467	0.563	0.0	0.158	0.552	0.0	0.078	0.439	0.0	0.062	0.379
	Fixed	91.7	0.967	0.839	48.3	0.660	0.563	53.3	0.867	0.757	0.0	0.338	0.426	51.4	0.639	0.649	33.3	0.440	0.624	12.5	0.505	0.587	9.7	0.378	0.564
	Adaptive	76.7	0.819	0.799	33.3	0.456	0.601	23.3	0.485	0.713	0.0	0.186	0.488	52.8	0.597	0.697	44.4	0.536	0.723	9.7	0.406	0.578	11.1	0.270	0.476

Table 2: Overall performance on both collections. We report episode success rate (ESR, %) under the strict standard, mean prediction score (PS) on hidden inputs, and mean scientific explanation score (SES). Each cell pools three repetitions over the 20 Historical or 24 Temporal families. Temporal papers were published between 30 April and 15 September 2026. Shaded cells are those referenced in the text.

Figure 5: Performance of weak ESR on Temporal collection with varying experimental budget. Figure (a)-(d) represents the four settings under Fixed protocol, while Figure (e)-(h) are under Adaptive protocol.

Figure 6: Fail modes for GPT-5.5, GPT-5.6 Sol and GPT-6 Astra on two collection.

Figure 7: Explanation Score for GPT-5.5, GPT-5.6 Sol and GPT-6 Astra on Discovery Counterfactual Temporal collection.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Adaptive interaction mechanism between the Player and the Oracle .

Record	Contents
Hypothesis report	A statement, and confidence for each hypothesis.
Experimental proposal	Experimental objective, apparatus, input settings, requested observables, replicate count, and numerical predictions under each active hypothesis.
Oracle response	Execution status and feedback message. Accepted proposals include quantized measurements with units and resolution metadata.
Final submission	hypothesis, executable predictor expressions $\hat{g}_{r}$ for all observables, scientific explanation, and applicability conditions.

Appendix

Table 3: Scientific fields recorded during an episode. Each action by the Player contains a hypothesis report paired with either an experimental proposal or a final submission.

Player	Knowledge cutoff	API release
GPT-5.5 ( OpenAI, 2026a )	1 December 2025	24 April 2026
GPT-5.6 Luna ( OpenAI, 2026b )	16 February 2026	9 July 2026
GPT-5.6 Terra ( OpenAI, 2026d )	16 February 2026	9 July 2026
GPT-5.6 Sol ( OpenAI, 2026c )	16 February 2026	9 July 2026
GPT-6 Astra ( OpenAI, 2026e )	30 April 2026	3 September 2026
Qwen-3.6 Plus	February 2026	2 Apr 2026

Appendix

Table 4: The Player models and dates.

Arity	Operators
At least 2	add , multiply , maximum , minimum , hypot
Exactly 2	subtract , divide , power , atan2
Exactly 1	negate , exp , expm1 , log , log1p , log2 , log10 , sqrt , cbrt , abs , sign , step , erf , erfc , sin , cos , tan , asin , acos , atan , sinh , cosh , tanh , asinh , acosh , atanh , floor , ceil , round

Appendix

Table 5: Expression operators and their arities. Arity is the number of child expressions.

Panel of Figure 9 — Figure 9: Success episode with varying experimental budget on Temporal collection.

				Tokens (M)				Cost (USD)
Model	Method	Episodes	Budget	Player in	Player out	Judge in	Judge out	Player	Judge	Total	$/episode
GPT-5.5	Prior-only	528	0	3.0	2.3	5.5	0.5	84	80	164	0.31
	Fixed ( $B=4$ )	528	4	3.4	2.8	6.9	0.5	101	92	193	0.37
	Adaptive	528	6.6 (0–84)	39.5	23.2	8.6	0.7	895	122	1,017	1.93
GPT-5.6 Luna	Prior-only	528	0	3.7	1.7	5.4	0.5	3	78	81	0.15
	Fixed ( $B=4$ )	528	4	5.3	3.1	6.7	0.5	5	92	96	0.18

Appendix

Table 6: Inference cost of the main benchmark. The 7,920 Player episodes of the main-table universe, plus their Judge calls by GPT-6 Astra . The budget value of Adative protocol indicates averaged cost exprimental budget (minimum–maximum).

		Historical								Temporal
		Identification				Discovery				Identification				Discovery
		Original		Counterfactual		Original		Counterfactual		Original		Counterfactual		Original		Counterfactual
Model	Method	Weak	Strict	Weak	Strict	Weak	Strict	Weak	Strict	Weak	Strict	Weak	Strict	Weak	Strict	Weak	Strict
DeepSeek-V4-Flash	Prior-only	95.0	88.3	0.0	0.0	51.7	31.7	0.0	0.0	45.8	40.3	5.6	0.0	0.0	0.0	0.0	0.0
	Fixed	96.7	91.7	58.3	48.3	86.7	53.3	16.7	0.0	61.1	51.4	36.1	33.3	45.8	12.5	27.8	9.7
	Adaptive	81.7	76.7	38.3	33.3	48.3	23.3	6.7	0.0	58.3	52.8	51.4	44.4	36.1	9.7	22.2	11.1

Appendix

Table 7: Episode success rate (ESR, %) on both collections. Weak and strict ESR are reported.

AI Agents for Scientific Discovery

AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.

See all 293 papers

CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
Jul 5, 2026Zhenhao Chen, Yongqiang Chen, Chenxi Liu +7AI Agents for Scientific Discovery AI Agent Benchmarks
1MBZUAI · 2Carnegie Mellon University · 3TMLR Group, Hong Kong Baptist University +2
Scaling Scientific Discovery Environments for Turn-Level Agentic RL
Jul 31, 2026Yucheng Xu, Keyi Zhang, Yuyang Yu +4Process Supervision Large Language Model-Guided Scientific Discovery
Shanghai AI Lab
Can LLMs Discover Scientific Laws in Real and Parallel Worlds?
Sep 1, 2026Yiming Huang, Ziche Liu, Junxia Cui +11Scientific Discovery Scientific Hypothesis Generation
University of California San Diego · University of California, Irvine · Cushing Academy +1

Jul 5, 2026J/K move · Enter open · S save

CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

Zhenhao Chen, Yongqiang Chen, Chenxi Liu, Junchi Yu+6

1MBZUAI · 2Carnegie Mellon University · 3TMLR Group, Hong Kong Baptist University+2

AI Agents for Scientific Discovery AI Agent Benchmarks Causal Reasoning in Language Models Causal Reasoning

Open paper AskPDF arXiv

Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention. Since scientific discovery fundamentally relies on uncovering causal relationships from observations, the capability of causal thinking, i.e., distinguishing causation from correlation and recognizing hidden biases, is essential to LLM agents. Although a number of benchmarks exist for AI Scientists, none explicitly incorporate challenges from selection bias, measurement error, and hidden confounders that widely exist in real-world scientific discovery. To this end, we present CausalGame, a benchmark that evaluates the causal thinking capabilities of LLM agents through interactive games. CausalGame asks LLM agents to actively design experimental protocols, collect observation data, and derive a final solution with an explanation report. To emulate realistic scientific discovery challenges, we design 14 scenarios that incorporate selection bias, measurement error, and hidden confounders. Across 30 LLM agents, none demonstrates reliable causal thinking: the best model reaches only 68.0% survival against analytical optima of 78-85%, and merely 5-7% of sessions receive credits on the causal-reasoning rubrics. CausalGame provides a scalable and controlled testbed for evaluating the causal thinking of AI Scientist agents.

Abstract

AI Agents for Scientific Discovery

Explore similar work

CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

Scaling Scientific Discovery Environments for Turn-Level Agentic RL

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?