APEXA: Execution-Integrity Enforcement for Multi-Agent LLM Automation of Synchrotron Data Reduction

Pawan K. Tripathi; Hemant Sharma; Andrew Chuang; Mathew J. Cherukara

cs.AISep 21, 2026

APEXA: Execution-Integrity Enforcement for Multi-Agent LLM Automation of Synchrotron Data Reduction

Authors: Pawan K. Tripathi, Hemant Sharma, Andrew Chuang, Mathew J. Cherukara

Organizations: X-ray Science Division, Advanced Photon Source, Argonne National Laboratory, Lemont, IL 60439, USA

Abstract

LLM agents can make expert scientific workflows accessible through natural language, but a plausible response does not establish that the requested computation ran. This matters at synchrotron facilities, where calibration and integration are multi-step operations over terabyte-scale datasets. We present APEXA, a deployed framework coordinating 61 tools for synchrotron data reduction. A deterministic execution-integrity guard prevents it from returning results for tool calls that did not execute, and the same tool layer enforces motor safety: across 50 adversarial scenarios and four models, it produced 0/200 violations against a simulated EPICS controller, versus 15/200 for a prompt-only baseline. We also introduce APEXA-Bench, 58 facility tasks organized by scientific correctness and physical consequence. On real APS beamline data, one prompt recovered detector geometry and integrated a full attenuation and exposure sweep. Trustworthy scientific agents need deterministic validation of execution, not model-side assurances. We release the framework, benchmark, and traces.

Figures & tables

Fig. 1 : Apexa reduces real high-energy diffraction data end to end from a single natural-language prompt. Panels are rendered from one CeO 2 calibration acquisition in the ai_tune beamline dataset (63 keV, $L_{sd}\!\approx\!900$ mm, 2880 $\times$ 2880 at 150 $\mu$ m): (a) raw detector frame; (b) after dark subtraction and bad-pixel masking; (c) autonomous MIDAS calibration with fitted CeO 2 Debye–Scherrer rings and recovered geometry ( $L_{sd}=895.80$ mm, beam centre $(1410,1447)$ px, $\lambda=0.1968$ Å, residual pseudo-strain $1.3\times 10^{-6}$ ); (d) azimuthal integration (2 $\theta$ – $\eta$ cake); (e) 1D lineout with indexed reflections. Robustness of the recovered geometry across the full attenuation and exposure sweep is shown in Fig. 3 . All values are read from the run manifests; none are hand-tuned.

Class	Failure cost	Reversible?	Example
I	none	N/A	physics knowledge query
C	compute time	retry	2D $\to$ 1D integration
P R	beam time	min–hr	motor to calib. position
P I	equip. damage	repair	runaway jog, over-limit move

TABLE I : Physical-consequence taxonomy. Each task is annotated with a class.

Fig. 2 : Apexa architecture, drawn around its trust boundary. A single stochastic reasoning agent (top) emits text tool calls through an institutional LLM gateway; all of them pass through a deterministic tool layer (green) before any tool runs. That layer, not the prompt, is where reliability is enforced: a format-tolerant parser absorbs cross-model tool-call drift, the execution-integrity guard refuses to surface any result not backed by an executed call, and five safety checks gate every EPICS motor command. The red paths are the failure the guard catches: the gateway can silently drop an emitted call, which the guard converts into an explicit non-result and a forced retry. Below sit the three MCP servers (core, Midas , EPICS) and, through Midas , the GPU/CPU engine abstraction, driving the analysis back ends with manifest-logged results.

Fig. 3 : Calibration on a real APS 1-ID dataset. (a) Mean pseudo-strain (the calibration residual; lower is better) vs. attenuation for CeO 2 and LaB 6 ; the two nonphysical calibrations (CeO 2 att5, LaB 6 att6) sit far above the convergence threshold and are autonomously flagged and rejected (

\times

). (b) Refined

L_{\rm sd}

for the converged calibrations is stable to

\pm 0.02

mm across both calibrants: APEXA recovers a reproducible physical geometry, and refuses to report the ones it does not. All values are read from the per-run calibration outputs ( ai_tune_bench.csv ); the

\sim

0.005 mm run-to-run variation in (b) is genuine scatter within the

\pm

0.02 mm stability band.

Fig. 4 : Deterministic motor-safety enforcement over the 50-scenario adversarial suite

\times

four models (200 trials). The same scenarios are run two ways: prompt-only (red), where the safety rules live only in a

\sim

700-token system prompt, and tool-enforced (green), where those rules run as five deterministic checks (soft-limit, limit-switch, slew-magnitude, velocity, jog-duration) in the EPICS server before any caput . Tool-enforced produces 0/200 violations (the green “0” at every bar) by construction; prompt-only lets 15/200 dangerous commands through, concentrated in large single-step slews (9) and out-of-range targets (4), with one each for prompt-injection and multi-axis. Bars show only the four scenario types where prompt-only violated; all other types were 0 under both. Counts from the adversarial suite run against a simulated EPICS IOC; failures spread across all four models.

Runtime Enforcement for AI Agents

Runtime enforcement for AI agents checks and constrains actions as they execute, helping prevent unsafe or unauthorized effects.

See all 276 papers

Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
Aug 27, 2026Lezhi Yu, Xiaogang Xu, Yuhua Zhou +2AI Agent Auditing AI-Assisted Scientific Research
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Zhejiang Lab, Hangzhou, China
APEX: Active Protection at Execution Boundaries for LLM Agents
Oct 3, 2026Xinran Zheng, Xin Fan Guo, Zhiqiang Hao +7AI Agent Security Prompt Injection Defense
Tsinghua University · Imperial College London · Nanjing University +6
SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks
Oct 23, 2026Tianyun Zhong, Wei Wang, Jialin Li +17Scientific Agent Benchmarks LLM Agent Evaluation
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4

Aug 27, 2026J/K move · Enter open · S save

Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research

Lezhi Yu, Xiaogang Xu, Yuhua Zhou, Shuibing He+1

College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Zhejiang Lab, Hangzhou, China

AI Agent Auditing AI-Assisted Scientific Research Scientific Reproducibility LLM Agent Reliability

Open paper AskPDF arXiv

LLM agents used for scientific experimentation must do more than generate executable code: they must implement the reference method faithfully, design experiments that test the paper's claims, and provide evidence supporting those claims. We show that agents often produce methodological hallucinations: silently reducing datasets or training budgets, replacing failed learning or generative components with lookup or oracle functions, or drawing conclusions from resource-limited settings where a method's claimed advantage disappears. To detect these failures, we introduce ABE-Ralph, a reference-anchored auditing framework that represents claims, protocols, required components, baselines, and metrics as structured experimental constraints, guides implementation through an 8-step workflow, and performs quantitative, qualitative, and code-level verification. Across 30 long-horizon reproduction runs covering 12 machine learning domains, ABE-Ralph achieves a 93% robust execution rate and identifies five scientific failure modes. In 23 NatureBench discovery tasks, ABE-Ralph matches or exceeds state-of-the-art performance on 5 tasks. These results show that reliable evaluation of AI scientists must assess whether the experimental design faithfully tests the intended claim and whether the resulting evidence supports it, rather than treating code execution or plausible metrics as evidence of scientific success.

Abstract

Runtime Enforcement for AI Agents

Explore similar work

Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research

APEX: Active Protection at Execution Boundaries for LLM Agents

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks