Watts If? A Benchmark for Counterfactual Reasoning About the AI Data Center Buildout

Pankaj Chouhan; Rajarshi Das; Aditya Dubey; Manminder Tomar; Sreekar Reddy; Samta Sharma; Satyapragnya Kar; PR Panigrahi

cs.AIOct 15, 2026

Watts If? A Benchmark for Counterfactual Reasoning About the AI Data Center Buildout

Authors: Pankaj Chouhan, Rajarshi Das, Aditya Dubey, Manminder Tomar, Sreekar Reddy, Samta Sharma, Satyapragnya Kar, PR Panigrahi

Organizations: AlphaInfra · New York University · Envana Software Solutions

Abstract

An AI data center is committed years before it draws a watt: a few discrete choices (grid or on-site gas, evaporative or dry cooling, which county to file in) fix its capital, water draw, and carbon path, and LLMs are now asked to help screen them with no test of whether they track what each commitment rules out. We introduce DCBench-440, with two arms reported separately: 200 generative questions over STRIPS worlds for five US deal archetypes, with search-verified gold answers and no tested shortcut above 12% on any question type, and 240 multiple-choice questions grounded in quoted site records, emission equations, and stated scenarios. Across open-weight deployments, accuracy spans 12.5-71.5% (generative) and 58.3-92.5% (multiple choice). One skill fails everywhere: asked to list every action available at a decision gate, no model exceeds 4.0% exact match. The strongest list 72-85% of the actions that would change the project's state but omit most permitted actions that change nothing; with those excused, the best exact match is still 24%. Code, data, and other materials are available upon request.

Figures & tables

Type	The question	Gold answer	Graded by
App	every action applicable right now	the full applicable set, no-ops included	exact set match
Prog	what one action adds and deletes	$add(a)$ and $del(a)$	exact set match
Reach	a fact that can never hold	facts in no reachable state	membership
AReach	an action that can never apply	actions applicable in no reachable state	membership
Val	the first step of a plan that breaks	index of the first inapplicable action	exact index
Just	a plan step that buys nothing	actions whose removal leaves a valid plan	simulate the shortened plan

Table 1: The eight generative question types (from ACPBench Hard);

h^{\star}

is the optimal cost.

Panel of Figure 1 — Figure 1: A one-block world of the gulf_water domain and the eight generative-arm questions asked about it (benchmark instances have two or three blocks, and some add rental-chiller or prefabricated-block routes). The nine numbered pills are an optimal plan, read left to right and wrapping to the second row. Because the static fact (dry-cooling-viable) is absent, the dry-cooling branch is dead forever: only two actions apply at the start, (cooling-dry-selected) can never hold, and (water-contracted) is a landmark in this instance though it would not be if dry cooling were viable. The statics, not the wording, decide the answer.

Figure 2: Per-arm results. Each row is one deployment. In (a) and (b), a marker is its accuracy and the bar through it the 95% Wilson interval. (a) Generative arm. (b) Multiple-choice arm: all 240 questions (circles) and the 138 where the longest option is wrong (diamonds); lines mark chance and the longest option. (c) Generative App : share of gold actions listed that change the state (circles) or are no-ops (diamonds). Below the line: Groq Compound systems, tools on. OR: OpenRouter.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: The screening stage. Each threshold test is a static fact: it is true or false for a given site and no later action changes it. Passing requires every test to hold at once, so failing even one removes a route permanently — the same behavior our static predicates encode in the generative-arm domains (Section 3.1 ).

Panel of Figure 4 — Figure 4: The staging stage. A site that clears Figure 3 still moves through an ordered chain of commitments rather than a single go/no-go decision. The financial investment decision is a one-shot fork in our domains: the resource it spends (capital committed against a named risk) cannot be recovered by a later action, which is why an early stage gate can foreclose options several stages downstream.

Deployment	Generative (200)			Multiple choice (240)			LO-wrong	Bank 1	Bank 2
	Acc.	95% CI	$p$	Acc.	95% CI	$p$	(138)	(120)	(120)
Symbolic oracle	100.0	[98.1, 100.0]	–	–	–	–	–	–	–
GPT-OSS-120B (OR)	71.5	[64.9, 77.3]	–	91.7	[87.5, 94.5]	0.754	89.1	95.0	88.3
GPT-OSS-120B (Groq)	71.0	[64.4, 76.8]	1.000	90.8	[86.5, 93.9]	0.344	89.9	93.3	88.3
GPT-OSS-Safeguard-20B (OR)	71.0	[64.4, 76.8]	1.000	not run
GPT-OSS-Safeguard-20B (Groq)	70.0	[63.3, 75.9]	0.749	90.4	[86.0, 93.5]	0.180	89.1	91.7	89.2

Appendix

Table 2: Accuracy (%) on each arm with 95% Wilson intervals.

p

: exact McNemar test against the best open-weight deployment on that arm (paired by question). OR: OpenRouter. LO-wrong: the 138 multiple-choice questions on which the longest option is not the answer. Bank 1 and Bank 2 are the two independently authored 120-question banks. The arms differ in content, skills, prompting, and grading, so the two columns are not a format comparison. “Not run”: no multiple-choice run exists for that deployment.

Deployment	App	Prog	Reach	AReach	Val	Just	Land	NextA
Symbolic oracle	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
GPT-OSS-120B (OR)	0.0	88.0	96.0	48.0	76.0	100.0	80.0	84.0
GPT-OSS-120B (Groq)	0.0	92.0	92.0	36.0	88.0	100.0	84.0	76.0
GPT-OSS-Safeguard-20B (OR)	0.0	96.0	96.0	48.0	100.0	100.0	60.0	68.0
GPT-OSS-Safeguard-20B (Groq)	0.0	96.0	96.0	52.0	92.0	88.0	68.0	68.0
GPT-OSS-20B (OR)	0.0	84.0	96.0	36.0	96.0	100.0	60.0	76.0

Appendix

Table 3: Generative-arm accuracy (%) by question type, 25 questions per type, so each cell carries a 95% interval of up to

\pm

19 points. Best shortcut : the highest accuracy any world-model-free strategy reaches through the same parser and validators (for App : listing every grounded action). Shortcut bound : the generator’s analytic upper bound on guessing, used to select the least guessable questions.

Deployment	btm_ccgt	grid_queue	retrofit	gulf_water	permian_gas
GPT-OSS-120B (OR)	77.5	80.0	70.0	65.0	65.0
GPT-OSS-120B (Groq)	75.0	80.0	70.0	62.5	67.5
GPT-OSS-Safeguard-20B (OR)	77.5	67.5	75.0	70.0	65.0
GPT-OSS-Safeguard-20B (Groq)	77.5	67.5	67.5	70.0	67.5
GPT-OSS-20B (OR)	72.5	75.0	65.0	62.5	67.5
GPT-OSS-20B (Groq)	72.5	62.5	70.0	55.0	67.5

Appendix

Table 4: Generative-arm accuracy (%) by domain, 40 questions per domain.

Deployment	App (29)	ConsRec (34)	Inv (29)	Prog (38)	Reach (24)	Temp (24)	Trade (41)	Val (21)
GPT-OSS-120B (OR)	93.1	94.1	93.1	78.9	95.8	95.8	100.0	81.0
GPT-OSS-120B (Groq)	89.7	94.1	93.1	81.6	87.5	95.8	100.0	81.0
GPT-OSS-Safeguard-20B (OR)	not run
GPT-OSS-Safeguard-20B (Groq)	96.6	94.1	89.7	78.9	87.5	91.7	97.6	85.7
GPT-OSS-20B (OR)	96.6	94.1	93.1	81.6	87.5	100.0	100.0	85.7
GPT-OSS-20B (Groq)	93.1	94.1	93.1	84.2	83.3	95.8	100.0	85.7

Appendix

Table 5: Multiple-choice accuracy (%) by skill label, pooled over both banks (questions per skill in the header). Only App, Prog, Reach, and Val share a name with a generative question type, and none is the same task: multiple-choice App asks which stated option complies with a stated limit.

Deployment	Env.	Power	AI wkld.	Fin./siting	Cross	Permit
GPT-OSS-120B (OR)	92.5	97.5	85.0	97.5	85.0	92.5
GPT-OSS-120B (Groq)	90.0	95.0	82.5	97.5	87.5	92.5
GPT-OSS-Safeguard-20B (OR)	not run
GPT-OSS-Safeguard-20B (Groq)	87.5	97.5	82.5	97.5	82.5	95.0
GPT-OSS-20B (OR)	95.0	97.5	85.0	95.0	90.0	92.5
GPT-OSS-20B (Groq)	90.0	97.5	87.5	97.5	87.5	90.0

Appendix

Table 6: Multiple-choice accuracy (%) by content bucket, 40 questions per bucket.

Deployment	Strict	Excl.	P	R	F1	J	J $\geq$ .8	R	R	Omit.
		no-op						chg.	no-op	no-op (%)
GPT-OSS-120B (OR)	0.0	0.0	0.94	0.37	0.50	0.35	4.0	0.78	0.16	88
GPT-OSS-120B (Groq)	0.0	4.0	0.89	0.36	0.48	0.35	12.0	0.72	0.17	85
GPT-OSS-Safeguard-20B (OR)	0.0	24.0	0.91	0.44	0.57	0.42	4.0	0.85	0.21	91
GPT-OSS-Safeguard-20B (Groq)	0.0	20.0	0.89	0.39	0.51	0.36	0.0	0.82	0.15	90
GPT-OSS-20B (OR)	0.0	16.0	0.84	0.45	0.55	0.41	0.0	0.82	0.24	89

Appendix

Table 7: Listing every applicable action (generative App , 25 questions). Strict: exact set match, as the prompt asks (an action whose preconditions hold counts even if it changes nothing). Excl. no-op: exact match after removing such no-op actions from both the gold set and the answer. P, R, F1, J: mean precision, recall, F1, and Jaccard overlap. J

\geq

.8: share of questions with Jaccard at least 0.8. R chg./R no-op: recall on gold actions that change the state, and on no-op ones. Omit. no-op: share of all omitted gold actions that are no-ops. 66% of gold actions are no-ops. The last row lists every grounded action without reading the state: overlap metrics reward it, exact match does not.

Deployment	Correct	Wrong	Ungrounded	Unparseable	Out of tokens	Empty	App misses
GPT-OSS-120B (OR)	143	56	0	1	0	0	15/9/1
GPT-OSS-120B (Groq)	142	58	0	0	0	0	12/12/1
GPT-OSS-Safeguard-20B (OR)	142	57	1	0	0	0	12/13/0
GPT-OSS-Safeguard-20B (Groq)	140	58	2	0	0	0	12/13/0
GPT-OSS-20B (OR)	137	59	1	0	3	0	13/12/0
GPT-OSS-20B (Groq)	131	69	0	0	0	0	12/12/1

Appendix

Table 8: What happened to each of the 200 generative answers. Wrong: a parsed answer that names only objects in the world but fails the validator. Ungrounded: names a fact or action that does not exist in the world (or an index past the plan). Unparseable: no answer of the requested shape. Out of tokens: the response hit its token limit. Last column, for the 25 App answers that missed: omissions only / omissions and extra actions / extra actions only.

Deployment	Reasoning	Max out	Gen tokens	MCQ tokens	Cost	Tool rows
GPT-OSS-120B (OR)	default	16,384 a	0.40 / 0.24	0.07 / 0.07	0.14	0 / 0
GPT-OSS-120B (Groq)	default	16,384 a	0.40 / 0.22	0.07 / 0.06	– b	0 / 0
GPT-OSS-Safeguard-20B (OR)	default	16,384 a	0.40 / 0.42	–	0.15	0 / –
GPT-OSS-Safeguard-20B (Groq)	default	16,384 a	0.40 / 0.39	0.07 / 0.11	– b	0 / 0
GPT-OSS-20B (OR)	default	16,384 a	0.40 / 0.41	0.07 / 0.11	0.09	0 / 0
GPT-OSS-20B (Groq)	default	16,384 a	0.40 / 0.39	0.07 / 0.14	– b	0 / 0

Appendix

Table 9: Inference settings and cost. Every call used temperature 0 and one attempt per question, re-asked only after a provider error; the generative arm prompts with two worked examples from a held-out tutorial domain, the multiple-choice arm with none. Temperature 0 was not deterministic on these APIs: answers that exhausted their budget and were re-asked came back answered. Tokens in millions (prompt / completion). Cost in US

, both arms, as reported by OpenRouter (total

2.00). Tool rows: answers whose trace shows a server-side tool call (generative / multiple choice). a Or the model’s provider cap if lower. b Groq reports tokens but not cost. c The harness sets reasoning_effort=none for Qwen on Groq and caps its output at 2,048 tokens; OpenRouter leaves reasoning on.

Domain	Models	What binds	Facts	Actions	Applic.	Plan	States
btm_ccgt	Campus on its own gas turbines	Air permit, gas lateral, and fuel contract before any unit is energized	37.0	32.5	17.2	17.6	65,911
grid_queue	Utility interconnection	Screening, queue, phase-1 study, agreement, and security before any build; the three-to-seven-year path	35.5	35.0	14.0	15.8	7,362
retrofit	Densifying a live site	Feeder budget: a dense rack costs two feeders air-cooled, one liquid-cooled	34.2	31.2	12.4	9.9	6,946
gulf_water	Coastal site, scarce water	One-shot cooling fork, evaporative or dry; sets the site’s water draw	21.6	21.5	8.9	10.4	241
permian_gas	Gensets at well pads	Lease and gas-quality statics decide which pads can host a unit	36.9	50.6	11.8	11.9	45,433

Appendix

Table 10: The five generative-arm domains. Means over generated instances; each domain contributes 40 questions, five per type. Applicable: mean actions applicable in a sampled state, counting no-ops.

Counterfactual Reasoning in Language Models

Counterfactual reasoning in language models studies how models predict and explain outcomes when facts, evidence, actions, or contexts are changed.

See all 45 papers

InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers
Aug 13, 2026Nicoletta Tsiopani, Moysis Symeonides, George Pallis +1Energy-Efficient ML LLM Inference Serving
Department of Computer Science University of Cyprus
AI-driven Thermal-aware Data Center Capacity Planning
Oct 1, 2026Yixing Li, Mark Fenton, Matthew Kaufeler +3Surrogate Modeling
Cadence Design Systems
Locked In: Multi-Objective RL for the AI Data Center Build-Out
Oct 26, 2026Satyapragnya Kar, Rajarshi Das, PR PanigrahiMulti-Objective Reinforcement Learning Data Center Energy Management
New York University · AlphaInfra.ai, MQube Cognition · Envana Software Solutions

Aug 13, 2026J/K move · Enter open · S save

InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers

Nicoletta Tsiopani, Moysis Symeonides, George Pallis, Marios D. Dikaiakos

Department of Computer Science University of Cyprus

Energy-Efficient ML LLM Inference Serving Cloud Computing Data Center Energy Management Carbon-Aware Computing

Open paper AskPDF arXiv

The rapid growth of LLM inference is shifting sustainability concerns from one-time training to continuous serving, where infrastructure decisions shape energy use, carbon emissions, water consumption, and service quality. Yet operators often need to compare deployment alternatives before large-scale infrastructure is built, making direct measurement costly, slow, and sometimes infeasible. We present InFactPlanner, a trace-driven decision-support framework for what-if analysis of sustainable AI data center deployment for LLM inference across single and geo-distributed sites. InFactPlanner combines query traces, hardware-model profiles, candidate site configurations, PUE/WUE parameters, renewable generation models, and time-varying grid carbon intensity to estimate power, energy, carbon emissions, water use, latency, and server utilization. The framework abstracts low-level serving effects into configurable hardware-model profiles, enabling rapid comparison of site selection, capacity placement, hardware, model, renewable integration, and routing choices. We validate the energy accounting pipeline by reproducing reference LLM inference energy estimates with less than 10% deviation, evaluate scalability across multiple data centers and server counts, and demonstrate scenario-driven decision analyses for hardware selection, renewable placement, geographic deployment, and carbon-aware routing. Our results show that sustainability-optimal choices can differ from latency-optimal ones, and that the carbon value of deployment depends strongly on the local grid mix.

Abstract

Counterfactual Reasoning in Language Models

Explore similar work

InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers

AI-driven Thermal-aware Data Center Capacity Planning

Locked In: Multi-Objective RL for the AI Data Center Build-Out