Which Buildings Are Artificial Intelligence-Ready? A Measurement-Based Assessment Framework for AI Question Answering and Actuation

Wooyoung Jung

cs.AIOct 6, 2026

Which Buildings Are Artificial Intelligence-Ready? A Measurement-Based Assessment Framework for AI Question Answering and Actuation

Authors: Wooyoung Jung

Organizations: University of Arizona, Tucson, USA

Abstract

Agentic artificial intelligence (AI) systems are becoming the interface to buildings, answering questions and controlling operations, but a building's readiness for them has not been systematically assessed. This study proposes a framework to quantify it. First, a building's knowledge graph sets two ceilings. The answerable-readiness ceiling is the share of operational questions its data could answer, and the actuation-readiness ceiling is the share of control actions it exposes. Second, a reference AI agent's accuracy on a fixed set of these questions shows how much of the ceilings is realized. On a simulated office, the agent realizes 0.62 of a 0.64 answerable ceiling, so missing data, not the AI, limit readiness, except in naming a fault's cause. Across 37 public real-building graphs, the median answerable ceiling is 0.16, and in 15 of 45, unlinked sensors lower it. The framework turns "is this building AI-ready?" into an auditable, ranked retrofit question.

Figures & tables

Construct	What is scored	Evidence	Conditioned on instrumentation	Data vs. AI
SRI, SmartScore [ 17 , 19 ]	installed smart services and their functionality level	assessor inspection	no	no
Data readiness levels [ 22 ]	fitness of a dataset for analysis	data audit	no	no
Sensor coverage, metadata completeness [ 30 ]	points present or modelled	point counts	counts sensors, not answers	no
Application shapes [ 32 ]	whether a KG holds what an application needs	KG validation	KG only, not the series	no
Building KGQA benchmarks [ 38 , 34 ]	an AI’s retrieval from the KG	ground-truth queries	no	no
This study	the share of a question canon the data can answer, and the share a fixed agent realizes	KG, series, and a fixed agent	yes	yes

Table 1: The present study beside the constructs it is closest to.

Figure 1: Overview of the framework. The section numbers show where each step is defined.

Term	Symbol	Meaning
Canon		the declared set of operational questions, as domain–intent cells with target levels (question canon), and the declared control rungs of each actuable domain (actuation canon)
Cell	$(d,i)$	one domain–intent pair of the canon (Table 3 )
Level, target level	L1–L3	answer completeness, from L1 reporting to L2 judging to L3 attributing; the target level is the highest level a cell asks for
Rung, canon total	$c_{\Sigma}$	one level of one cell; the canon has 76 rungs
Stream class		a kind of measured or commanded point, defined as a set of Brick classes
Instrumentation profile	$I$	the set of stream classes a building exposes

Table 2: Key terms and symbols.

Domain	Status	Summary	Comparison	Verification	Diagnosis	Explanation
Thermal	2	2	3	3	3	3
Energy	1	2	3	–	3	3
Air quality	2	2	3	3	3	3
Equipment	2	2	3	3	3	3
Occupancy	2	2	2	–	2	–
Lighting	2	2	2	–	2	–

Table 3: The question canon: the target level (1–3) of each (domain, intent) cell. “–” marks a cell left out of the canon, for the reasons given in the text. The target levels sum to the canon total

c_{\Sigma}=76

Domain	Intent(s)	L1	L2	L3
Thermal	status, summary	T	T+Sp	–
Thermal	comparison	T	T	T+Sp+ActC+ActP
Thermal	verification	T	T+Sp	T+Sp+ActC+ActP
Thermal	diagnosis, explanation	T	T+Sp	T+Sp+SpC+ActC+ActP+SAT
Energy	status	Whole	–	–
Energy	summary	Whole	Whole+Sub	–

Table 4: The requirement table

R

: the stream classes a cell needs at each level. Each entry lists every class needed to reach that level. “–” marks a level above the cell’s target level.

Figure 2: From a zone’s points to a cell’s achieved level. (a) The usability rule (Section 4.6 ) drops series with large gaps or stuck values. Each point maps to a stream class (Table 16 ), and the stream classes of the points read form the instrumentation profile

I

, so one zone has three nested profiles. A setpoint or command point is also a handle, A1 or A2, for the actuation-readiness ceiling (Section 3.5 ). (b) The cell reaches the highest level whose required stream classes are all in the profile: L3 on the declared points, L2 on the linked points and L1 on the usable points.

Figure 3: Conceptual decomposition of building AI-readiness. The canon total splits into realized readiness, AI failure, unprobed credit and data limitation.

	Agent answers	Agent abstains
Answerable question	correct if right, AI failure if wrong	AI failure
Unanswerable question	overclaim, even if right	correct abstention

Table 5: How each answer is scored. Answerable questions enter the rung accuracy

a_{d,i,r}

, and unanswerable ones enter the overclaim rate.

Domain	A1 (supervisory)	A2 (direct)
Thermal	zone cooling/heating setpoint	damper / valve / reheat command
Air quality	airflow / fan-speed setpoint	outside-air damper command
Equipment	on/off, enable or start/stop command	compressor command, speed setpoint
Lighting	lighting-level setpoint	lighting on/off / luminance command

Table 6: The actuation canon: the Brick classes that make up each rung of each actuable domain.

Figure 4: AI-readiness of a building as a point in the plane of the two ceilings. The two points are illustrative.

Figure 5: Study design: the question each experiment answers and the data it uses. The top band and the column headings give the sections that describe them.

Figure 6: One question episode. A turn is one reply of the model, with the tools it calls and their results. An episode ends when the agent submits, when it replies without calling a tool, or after 25 turns. Table 19 follows one episode call by call.

Profile	Adds these stream classes	Basis	In BATS
Minimal	zone temperature	the least a zone controller reads	yes
Comfort	zone heating and cooling setpoints	a thermostat’s inputs	yes
Standard	occupancy	occupancy sensing	yes
Metered	whole-building meter, end-use submeters, zone lighting power, equipment power, outside air temperature	the building-level metering ASHRAE 90.1-2022 requires of a large building, plus equipment power, zone lighting power and outside air temperature	all but outside air temperature
Rich	scheduled setpoint, terminal actuation command and response, supply air temperature, run status, equipment load, zone CO 2 , outdoor-air flow	the points the Guideline 36 sequences read and write	none

Table 7: Instrumentation profiles of the sweep. Each profile holds the stream classes of the profiles above it and adds those listed. The last column gives which of the added classes the BATS large office includes.

Profile	$C_{\mathrm{ans}}(I)$	$n$	$A$	$A_{\mathrm{r}}$	$F$	$F/C_{\mathrm{ans}}(I)$	95% interval
minimal	7/76	72	0.921	0.085	0.007	0.08	0.080–0.089
comfort	12/76	114	0.981	0.155	0.003	0.02	0.152–0.158
standard	20/76	186	0.919	0.242	0.021	0.08	0.233–0.251
metered = rich	49/76	456	0.964	0.621	0.023	0.04	0.610–0.632

Table 8: The instrumentation sweep on the large office in Tampa, reference agent.

n

is the number of graded episodes,

F/C_{\mathrm{ans}}(I)

the AI failure as a share of the ceiling, and the interval the 95% bootstrap interval on

A_{\mathrm{r}}

. The unprobed share is zero at every profile, so AI failure is the rest of the ceiling,

F=C_{\mathrm{ans}}(I)-A_{\mathrm{r}}

Figure 7: The instrumentation sweep on the large office in Tampa, by domain, reference agent. The coloured segments are the realized readiness of each domain.

Canon	$c_{\Sigma}$	$C_{\mathrm{ans}}(I)$	$A_{\mathrm{r}}$	$F/C_{\mathrm{ans}}(I)$	Real median	Rank correlation
declared	76	0.645	0.621	0.036	0.158	1.000
no air quality	60	0.817	0.787	0.036	0.200	0.994
no occupancy, lighting	60	0.550	0.528	0.040	0.200	0.999
no level above L2	61	0.803	0.774	0.036	0.197	0.970
equal domain shares	76	0.708	0.684	0.034	0.125	1.000

Table 9: Headline results under alternative canons: the ceiling and

A_{\mathrm{r}}

of the large office at its metered profile, and the median ceiling of the 37 real KGs with linked series with the Spearman rank correlation of their ceilings against the declared canon.

Office, climate	$C_{\mathrm{ans}}(I)$	$A$	Thermal	Energy	Equipment	Occupancy	Lighting
Large, New York	49/76	0.952	0.95	0.93	0.97	0.96	0.96
Large, Buffalo	49/76	0.932	0.95	0.93	0.89	0.96	0.96
Medium, Tampa	49/76	0.940	0.95	0.93	0.88	0.96	1.00
Medium, New York	49/76	0.957	0.92	1.00	0.97	0.92	1.00
Medium, Buffalo	49/76	0.924	0.95	1.00	0.89	0.83	1.00
Small, Tampa	37/76	0.959	0.94	0.93	–	0.96	1.00

Table 10: The other eight buildings of the published corpus, reference agent, 1 repeat, 113 to 153 graded episodes each. Air quality has no stream in the release, and a dash marks a domain with no instance.

Figure 8: Other agents on the sweep building, metered profile, 22 classes, 1 repeat (66 episodes each). The interval is the Wilson 95% interval on accuracy.

Figure 9: Diagnosis outcomes, reference agent, 20 repeats per scenario and condition. Pair B is asked with the monitoring-sensor premise stated (Section 4.4 ). Without the separating stream an abstention is correct and any named cause is an overclaim. With it, only the true cause is correct.

Figure 10: Real buildings in the plane of the two ceilings, 37 KGs with linked series. Filled markers use the points that carry a series. An open marker, joined by a grey line, shows the higher ceiling the KG declares. Markers overlap where KGs share both ceilings.

Figure 11: Stream classes linked by the 37 real building KGs with linked series, coloured by the profile of the sweep that first adds each class. An outline shows the KGs that declare the class, where more declare it than link it.

Quantity	Value
Graphs in the census (Mortar + BTS)	45
Graphs with at least one linked series	37
$C_{\mathrm{ans}}(I)$ on linked data: minimum / median / maximum	0.000 / 0.158 / 0.500
Graphs whose declared $C_{\mathrm{ans}}(I)$ exceeds the linked $C_{\mathrm{ans}}(I)$	15 of 45
$C_{\mathrm{act}}(I)$ : minimum / median / maximum	0.000 / 0.125 / 0.625
Graphs with no actuation handle	13 of 37

Table 11: The real-building census: 45 KGs, 37 of them with linked series. “Declared” counts every typed point; “linked” counts points that carry a series reference.

Class	Count	What happened
agent error	116	44 abstentions although the data were present, most after failing to find a zone named by its label; 20 episodes without an answer at the turn limit; 14 right values placed outside the answer field; 13 occupied and unoccupied swapped; 25 other wrong values, entities, signs or units
grader strictness	4	a real-building room named by its full point path

Table 12: The 120 misses of the reference agent over 2,380 graded episodes (the sweep, the other eight buildings, the runs on the original KG and the real buildings).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

SRI domain	Service	Relation	Canon cell or axis
Heating	Heat emission control	A	thermal A1 (zone setpoint), A2 (valve or reheat command)
Heating	Emission control for TABS	A	thermal A1
Heating	Control of distribution fluid temperature	A	equipment A1
Heating	Control of distribution pumps	A	equipment A2 (speed command)
Heating	Heat generator control (all except heat pumps)	A	equipment A1, A2
Heating	Heat generator control (heat pumps)	A	equipment A2

Appendix

Table 13: The 30 SRI Method B services that relate to the canon. D = related through data; A = related through the actuation axis; TABS = thermally activated building systems; IAQ = indoor air quality; TBS = technical building systems.

Intent	L1	L2	L3
Thermal
status	What is Zone 3’s temperature now?	How far is Zone 3 above its cooling setpoint now?	–
summary	What was Zone 3’s mean temperature yesterday?	For how many hours yesterday was Zone 3 outside its setpoints?	–
comparison	Was Zone 3 warmer than Zone 5 at 15:00?	How much warmer was Zone 3 than the other zones on its air handler?	Was Zone 3 warmer than Zone 5 because its damper delivered less air?
verification	Did Zone 3 stay at or below $24\text{\,}\mathrm{\SIUnitSymbolCelsius}$ all day?	Did Zone 3 stay within its setpoints all day?	When Zone 3 ran above its cooling setpoint, did its damper open to correct it?
diagnosis	Which zone was warmest at 15:00?	Which zone was farthest above its own cooling setpoint at 15:00?	Zone 3 is too warm: is the cause a setpoint override, a stuck damper or warm supply air?

Appendix

Table 14: An example question at each level of every canon cell. Each example uses only the stream classes that Table 4 requires at its level. “–” marks a level above the cell’s target level. The questions posed to the agent are the question classes of D .

Domain	Candidate cause	What the data show	Separating stream
Thermal	schedule change	scheduled setpoint and setpoint in effect move together	SpC
Thermal	local override or controller offset	setpoint in effect moves, scheduled setpoint stays	SpC
Thermal	terminal-unit fault (stuck damper or valve)	the command moves or saturates while the position stays	ActC, ActP
Thermal	local load change	command and position move together in the correcting direction	ActC, ActP
Thermal	supply-side capacity loss	supply air temperature off its normal value, peer zones deviate together	SAT, T (peers)
Thermal	zone sensor fault	reading departs from peers with setpoints, command and position unchanged	T (peers), ActC, ActP

Appendix

Table 15: Declared cause sets and the stream that separates each cause from its rivals. “T (peers)” is the zone temperature of other zones on the same air system.

Stream	Brick classes	Condition
T	Zone_Air_Temperature_Sensor, Room_Air_Temperature_Sensor; generic Air_Temperature_Sensor, Temperature_Sensor	generic classes only on a zone or terminal unit
Sp	Zone and room air temperature setpoints; effective, occupied and unoccupied heating and cooling setpoints; generic Cooling, Heating and Temperature_Setpoint	generic classes only on a zone or terminal unit
SpC	as Sp	a KG that holds both a scheduled (occupied or unoccupied) and an effective setpoint
ActC	Damper_Position_Command and _Setpoint; Valve_Command; Heating_Command; Cooling_Command	on a zone or terminal unit
ActP	Damper_Position_Sensor; Valve_Position_Sensor; generic Position_Sensor and supply, discharge and air flow sensors	on a zone or terminal unit
SAT	Supply_Air_Temperature_Sensor, Discharge_Air_Temperature_Sensor	anywhere

Appendix

Table 16: Brick classes read as canon stream classes in the real-building census.

Domain	Intent	Target	minimal	comfort	standard	metered
Thermal	status	2	1	2	2	2
Thermal	summary	2	1	2	2	2
Thermal	comparison	3	2	2	2	2
Thermal	verification	3	1	2	2	2
Thermal	diagnosis	3	1	2	2	2
Thermal	explanation	3	1	2	2	2

Appendix

Table 17: Achieved level of every canon cell at each profile of the sweep (large office, Tampa; profiles as in Table 7 ). The column sums are the numerators of

C_{\mathrm{ans}}(I)

in Table 8 .

Tool	Input	Description given to the model
query_graph	sparql (string)	Run a SPARQL SELECT query over the building’s Brick graph. Returns up to 200 rows as JSON. Prefixes brick:, ref:, rdfs: and rdf: are predefined.
run_python	code (string)	Run standard-library Python 3 in an isolated process. GRAPH_TTL, CATALOG_CSV and VALUES_CSV are predefined file paths. Print what you need; stdout and stderr are returned.
submit_answer	answer (object), abstain (boolean, required), needed_streams (string)	Hand in the final answer. Call exactly once, as the last step. The answer is in the JSON shape the question gives; abstain is true if the available data cannot determine the answer; when abstaining, name the streams that would be needed.

Appendix

Table 18: The three tools, as declared to the model. In Section 4.1 they are the KG query, Python and answer tools.

Question (thermal summary, L1): What was the mean air temperature in zone Perimeter_mid_ZN_4 on 2018-01-02 (all 24 hourly values), in $\mathrm{\SIUnitSymbolCelsius}$ ? Answer shape {"value": <number>} .
Call	Tool	What the agent did	Result
1	KG query	looked for the zone’s temperature point with relation names the KG does not use	no rows
2	KG query	found the zone by its label	Zone_F3_Z05
3	Python	printed the first lines of the series catalogue	column names and 19 series
4	Python	filtered the catalogue to Zone_F3_Z05	the zone’s temperature series and its id
5	Python	read that series for 2 January and averaged the 24 values	24 values, mean 24.2313

Appendix

Table 19: One episode of the reference agent on the sweep building at the metered profile. The first query uses relation names the KG does not have and returns nothing, and the agent recovers by finding the zone by its label. The full transcript is in the released code.

Name in code	Cell, level	Requires	Answer	Question (paraphrased)	Score
t_status_1	thermal / status, L1	T	number, $\pm$ 0.5	air temperature of a zone at an hour	1.00
t_status_2 ‡	thermal / status, L2	T, Sp	number, $\pm$ 0.5	zone temperature minus its cooling setpoint at an hour	1.00
t_summary_1	thermal / summary, L1	T	number, $\pm$ 0.5	mean zone temperature over a day	1.00
t_summary_2 ‡	thermal / summary, L2	T, Sp	number, $\pm$ 1	hours in a day outside the zone’s own setpoint band	1.00
t_comparison_1	thermal / comparison, L1	T	yes/no	was one zone warmer than another at an hour	1.00
t_comparison_2	thermal / comparison, L2	T	number, $\pm$ 0.5	temperature of one zone minus another at an hour	0.89 (0.67–1.00)

Appendix

Table 20: The 64 question classes, with the reference agent’s score at the metered profile of the sweep. Each class is named as in the released code. A double dagger marks a class that one fixed answer passes in every instance on the sweep building.

Example	Question (abridged)	Ground truth (other instances)	Agent, 3 repeats
L1 number (t_status_1)	air temperature of DataCenter_basement_ZN_6 at 2018-01-02 08:00, $\mathrm{\SIUnitSymbolCelsius}$	27.0 (27.001, 24.0)	27 $\times$ 3
L2 difference (t_comparison_2)	Basement minus the HVAC zone DataCenter_top_ZN_6 at 2018-01-14 16:00, $\mathrm{\SIUnitSymbolCelsius}$	$-3.0$ ( $-0.899$ , $-3.001$ )	$-3$ $\times$ 3
yes/no (t_verification_2)	did Perimeter_mid_ZN_4 stay within its setpoints all of 2018-07-04	false (true, true)	false $\times$ 3
entity, lighting (l_comparison_2)	which of three zones used the most lighting energy on 2018-10-12	Core_bottom (Core_bottom, Core_mid)	Core_bottom $\times$ 3
two KG hops (t_comparison_2hop)	warmest zone served by AHU02 at 2018-10-09 11:00	Perimeter_mid_ZN_2 (Perimeter_top_ZN_1, Perimeter_mid_ZN_2)	Zone_F3_Z03 (its identifier) $\times$ 3
energy (e_summary_1)	highest hourly whole-building electricity use in 2018-10, J	$5.22\times 10^{9}$ ( $5.82\times 10^{9}$ , $4.29\times 10^{9}$ )	$5.22\times 10^{9}$ $\times$ 3

Appendix

Table 21: Worked examples on the sweep building at the metered profile, one per answer type and domain, and a typical miss. The ground truth of the instance shown is followed, in brackets, by those of the class’s other two instances. The agent’s answers are its three repeats.

Scenario	Injection	True cause / rival	Separating stream
A: thermostat offset	EnergyPlus fault model, 1 $\mathrm{\SIUnitSymbolCelsius}$ offset in the Core_bottom zone of the published faulted building from 15 June	local override or controller offset / schedule change	SpC
A: schedule change	the zone’s own heating and cooling setpoint schedules lowered by 1 $\mathrm{\SIUnitSymbolCelsius}$ from 15 June (re-simulated clean building)	schedule change / local override or controller offset	SpC
B: sensor drift	BuildStream telemetry injector, drift to $+1$ $\mathrm{\SIUnitSymbolCelsius}$ on the reported temperature of the Core_mid zone from 1 July to year end (re-simulated clean building)	zone sensor fault / local load change	ActP
B: internal-load step	an always-on equipment load of 40 $\mathrm{kW}$ in the Core_mid zone from 1 July (re-simulated clean building)	local load change / zone sensor fault	ActP

Appendix

Table 22: The four diagnosis scenarios, all in the large office in Tampa. Base streams are T and Sp in every scenario. The separating streams come from the supplement to the release.

Pair	Scenario	Zone T	Setpoint in effect	Peer T	Separating stream
A	thermostat offset	$-0.87$ $\mathrm{\SIUnitSymbolCelsius}$	$-1.00$ $\mathrm{\SIUnitSymbolCelsius}$	$+0.01$ $\mathrm{\SIUnitSymbolCelsius}$	SpC $+0.00$ $\mathrm{\SIUnitSymbolCelsius}$
A	schedule change	$-0.91$ $\mathrm{\SIUnitSymbolCelsius}$	$-1.00$ $\mathrm{\SIUnitSymbolCelsius}$	$+0.01$ $\mathrm{\SIUnitSymbolCelsius}$	SpC $-1.00$ $\mathrm{\SIUnitSymbolCelsius}$
B	sensor drift	$+0.75$ $\mathrm{\SIUnitSymbolCelsius}$	$0.00$ $\mathrm{\SIUnitSymbolCelsius}$	$0.00$ $\mathrm{\SIUnitSymbolCelsius}$	damper $+0.000$
B	internal-load step	$+0.67$ $\mathrm{\SIUnitSymbolCelsius}$	$0.00$ $\mathrm{\SIUnitSymbolCelsius}$	$0.00$ $\mathrm{\SIUnitSymbolCelsius}$	damper $+0.180$

Appendix

Table 23: Look-alike and separation checks: change against the clean building, mean over occupied weekday hours. The peer is the core zone on another floor.

Explore similar work

Aug 13, 2026cs.AI

RAIL: An Automatic Classifier of the Artificial Intelligence Readiness Level

Assessing the maturity of artificial intelligence technologies is essential for investment decisions, project management, and policy monitoring, yet the available readiness frameworks are heterogeneous and difficult to apply automatically: the adaptation of Technology Readiness Levels to AI lacks AI-specific gating criteria, the Machine Learning Technology Readiness Levels presuppose access to internal process artifacts, and AI/data readiness dimension models employ scales that resist direct comparison. This paper makes two contributions. First, we unify these three frameworks into the Unified AI Readiness Level (AIRL), a nine-level ordinal scale built on an environmental evidence ladder and complemented by dimensional caps (covering specification, data existence, data quality, data legality, expert knowledge, and algorithmic maturity) together with a generality-anchoring rule and explicit assignment disciplines, so that a readiness level becomes decidable from a natural-language description of the work alone. Second, we propose RAIL (Readiness Assessment via Independent LLM-experts), a panel-of-experts classifier that operationalizes the scale: one evidence agent and six independent dimension agents, each a large language model with a narrowly scoped mandate, deliver verdicts that a deterministic minimum rule aggregates and a chief expert reviews under asymmetric authority, confirming or lowering the panel's recommendation but never raising it above the caps. The method was tested in the analysis of several research works showing consistency and avoiding overestimation from monolithic LLM classifiers.

Juan Irving Vasquez, Juan Terven, Laura-Ivoone Garay-Jimenez

CIETEC-IPN Instituto Polit´ecnico Nacional M´exico City, M´exico · Instituto Polit´ecnico Nacional, UPIITA, M´exico City, M´exico

Readiness Artificial Intelligence Systems Agentic Evaluations Algorithmic Cores

Ask PDF

Jul 31, 2026cs.AI

InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk

Managing modern computing infrastructure has become a steadily harder problem due to the ever-increasing complexity. Recent advances in AI agents create a timely opportunity to automate infrastructure management tasks, but it remains unclear how well such agents can handle real-world infrastructure complexity. We present InfraBench, a benchmark suite for evaluating AI agents on realistic infrastructure tasks across the full system stack and full operational lifecycle with fine-grained risk assessment. Experiments with 15 agent-model configurations show that even the strongest agent cannot secure a full score across all tasks. Mean effective scores range from roughly 40% to 88% (with per-configuration standard errors of 6-12 points), repeating every task three times reveals that top configurations still pass only a fraction of their attempts, and per-check scoring exposes a general failure pattern: agents may routinely satisfy short-term objectives while leaving non-durable changes, broken distributed invariants, unsafe side effects, and uncleaned state behind. INFRABENCH, including its live leaderboard, tasks, and evaluation harness, is publicly available at infraben.ch.

Yuan Gao, Zeren Yang, Junnan Li +6

Wanxiang · University of Wisconsin–Madison · 2Iowa State University

Infrastructure Artificial Intelligence Agents Lifecycle Risk

Ask PDF

Jul 30, 2026cs.MA

Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness

AI agents are moving into production workflows where they retrieve information, call tools, maintain state, and act on behalf of users or organizations, but many release decisions still rely on capability signals, demos, or behavioral tests that do not show whether an agent is ready to operate under production constraints. Capability is therefore not production readiness. This paper introduces the ProofAgent Index (PAI), a governance readiness index for AI agents. PAI combines four dimensions of deployment evidence: Evaluation, Context, Compliance, and Governance. Evaluation measures observed behavior, Context measures the operating environment that shapes that behavior, Compliance measures alignment with applicable rules and controls, and Governance measures whether the organization can authorize, monitor, audit, and control the agent during operation. PAI is implemented inside ProofAgent Harness, an open source infrastructure for auditable AI agent evaluation and governance. Validation across two heavily regulated domains, healthcare and finance, shows that PAI carries held out readiness signal and separates higher risk from lower risk configurations. The results show that context engineering strongly changes reliability, capability improves behavior but does not determine readiness, and governance evidence must remain visible rather than averaged away. PAI reframes agent release from a faith based deployment decision into an auditable readiness decision.

Fouad Bousetouane

The University of Chicago, USA

Agentic Evaluations Artificial Intelligence Agents Governance

Ask PDF