Evaluating Verified Autonomy in Quantum Engineering
Authors: Naixu Guo, Changhao Li, Siyu Cheng, Qicheng Tang, Binzhao Luo, Bikun Li, Yuxuan Du, Shihao Ru, Jiaqi Cai
Organizations: Centre for Quantum Technologies, National University of Singapore, 117543, Singapore · Unitary Foundation, San Francisco, CA, USA · Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA · School of Physics, Georgia Institute of Technology, Atlanta, GA 30332, USA · GaugeForge PTE. LTD., 049422, Singapore · Chicago Quantum Institute and Pritzker School of Molecular Engineering, University of Chicago, Chicago, Illinois 60637, USA · College of Computing and Data Science, Nanyang Technological University, Singapore 639798, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798, Singapore
Reliable quantum engineering is essential for turning quantum phenomena into practical technologies. As quantum platforms grow in scale and complexity, their characterization and operation require increasing human effort and coordination. Scientific artificial intelligence agents, which can plan experiments, operate instruments, and analyze observations, offer a promising route towards autonomous quantum engineering. Yet whether current agents can perform reliably in this setting has not been systematically established. To fill this gap, we developed Quantum-Harbor, a virtual laboratory that provides a controlled execution environment for agents to interact with quantum systems. This design enables direct verification of both the actions taken and the conclusions drawn. Building on this framework, we introduce QIQCBench, a benchmark of 49 expert-authored tasks spanning multiple layers including calibration and control, error correction and compilation, sensing and networking. Across 17 frontier agentic systems, QIQCBench reveals wide variation in verified performance. These results expose a substantial gap between demonstrating capability and achieving reliable operation, and establish Quantum-Harbor as a foundation for measuring progress towards verified autonomy in quantum engineering.
Figures & tables
Figure 1: AI agents for quantum engineering and representative workflows. a . Agent-operated quantum engineering. The agent iterates planning, execution, analysis, and adaptation while interacting with quantum laboratories. The workflow operates across the layers shown on the right, with individual tasks typically engaging multiple layers, and its outputs are submitted for independent verification. b . Decoder calibration for a lattice-surgery logical CNOT gate. The agent measures error correlations across the merge seam between surface-code patches and submits a retuned decoder, verified on fresh syndrome records. The public acceptance criteria require a distance-5 logical-memory error below 10−3 per cycle and, for eight logical-CNOT test circuits spanning both measurement bases, a mean error below 0.175 with each circuit-specific error below 0.185 . c . Surrogate modeling of a hidden 60-qubit parameterized quantum circuit. The agent collects measurement data and constructs a surrogate model. Once the measurement record is fixed, 40 held-out inputs are revealed, for which the agent predicts 1,770 two-qubit correlations for each input. The predictions are compared with hidden reference correlations, with accuracy evaluated by the root mean square error.
Figure 2: Architecture and evidence-bound grading in Quantum-Harbor. a. The agent environment contains the language model, work files, and agent-written code. Through the public interfaces, the agent consults hardware specifications and lab notebooks, submits experiment jobs, retrieves raw measurement records (bitstrings shown), and submits its final answer. Experiment jobs execute asynchronously. b. The hardware layer provides quantum hardware abstractions ( qtypes ) for transmons, neutral atoms, trapped ions, and other platforms. Experiments run on simulators governed by hidden device and noise parameters, or on replay and live-device backends where supported. An evidence log records experimental requests and outcomes for subsequent verification. c. The grader recomputes relevant quantities, binds submitted claims to the recorded experiments, and applies task-specific scoring criteria (pass/fail shown). For calibration tasks, submitted configurations can be evaluated on fresh simulated data. The shaded region and dashed boundary indicate components inaccessible to the agent, including hidden parameters, the private evidence log, and grading logic.
Figure 3: Task composition and quantum computing stack coverage of QIQCBench. a. The 49 tasks grouped into six scientific categories. Each column represents one task. Colors identify the hardware platform, with separate labels for platform-agnostic and device-less tasks. Evaluation types are indicated by filled squares for the 39 tasks assessed by a binary pass–fail evaluation, orange diamonds for the 7 racing tasks ranked by Elo rating, and open squares for the 3 infrastructure test cases excluded from science scores. b. Mapping of each task to five layers of the quantum computing stack, adapted from Ref. [ 58 ] : physical device, gates and calibration, error correction, logical and compilation, and application and algorithms. Dark blue cells mark the layers engaged by each task. Pale cells indicate unmapped layers. Columns share the task indices and category boundaries of panel a , showing how individual tasks connect multiple layers of quantum engineering.
Figure 4: Benchmark results and analysis of failure and pass patterns. a. Pass rate on the pass–fail tasks. Error bars: Wilson 68% intervals. b. Outcome composition of the trials, including passes, science failures, contract failures, and no-submission trials. c. Bradley–Terry Elo ratings over the 7 racing tasks (filled circles: games-weighted aggregate; open circles: per-task ratings). d. Per-task passes for every system, grouped by the scientific categories of Fig. 3 , easiest to hardest within each group. Only pass-fail tasks are shown here.
An autonomous research laboratory could conduct an entire study, from formulating questions to preparing manuscripts. Artificial intelligence (AI) offers one route towards this goal. Recent AI systems take on several research roles, but may draw conclusions that the evidence does not support. Here, we present an autonomous research laboratory in which agents based on large language models assess one another's work and can stop or redirect investigations. They can generate and reject research questions, require separate theoretical checks and conduct experiments designed to challenge their claims. We demonstrate these capabilities by investigating ensembles of quantum spins in diamond. The agents derived the minimum number of calibration settings needed to predict responses to a defined family of microwave pulse sequences. They checked the result computationally and designed experiments to distinguish competing predictions. An initial experiment agreed with the theoretical predictions, but repeated reference measurements showed that the signal was drifting during data collection. The agents therefore treated the result as inconclusive and revised the measurement protocol without relaxing the acceptance criteria. The revised experiment met those criteria and supported the theoretical predictions. This study demonstrates that collaborating AI agents can select research questions, derive theoretical results and test their predictions experimentally.
El Mustapha Mansouri, Eikichi Kimura, Masato Koga +1
School of Engineering, Institute of Science Tokyo, Yokohama, Kanagawa 226-8501, Japan
Quantum materials exhibit diverse electronic and functional phenomena, but their characterization requires extensive experimental expertise, iterative decision-making and accumulated laboratory knowledge that is difficult to document and transfer. Here we introduce Quantum Agent (QUANTA), an agentic artificial intelligence (AI) framework for autonomous quantum-materials characterization. Using a widely adopted Physical Property Measurement System (PPMS) as an experimental platform, QUANTA combines retrieval augmentation using scientific literature, experimental protocols and prior measurements with specialized agents for experimental planning, troubleshooting, and data interpretation. A separate deterministic control layer validates instrument actions. Guided by incoming data, QUANTA evaluates measurement quality, revises experimental plans and selects subsequent measurements. We demonstrate these capabilities through transport measurements of the topological Weyl semimetal TaP, as well as investigations of nonlinear responses in magnetic material Co2MnGa. These results establish QUANTA as a generalizable framework for evidence-driven, safeguarded and closed-loop quantum-materials characterization.
Quantum Measurement Group, MIT, Cambridge, MA 02139, USA · Department of Nuclear Science and Engineering, MIT, Cambridge, MA 02139, USA · Department of Electrical Engineering and Computer Science, MIT, Cambridge, MA 02139, USA +3
Quantum computers are moving from research laboratories to industrial machines accessible via the cloud and integrated into high-performance computing facilities. However, translating theoretical quantum protocols into hardware experiments remains a major bottleneck, requiring expertise across protocol design, compilation, simulation, and cloud execution. Here, we introduce an agentic workflow that automates this pipeline on neutral-atom quantum processors (here two Pasqal QPUs available on the cloud) while keeping the researcher in the loop for critical validation. In three case studies from many-body physics and optimization, the agent went from published paper or patent to a QPU campaign run overnight. In particular, human intervention was crucial to ensure scientific validity: the agent selected an inadequate observable in one experiment and constructed a plausible but incorrect hardware diagnosis in another, with both failures detected only through domain-expert review. Finally, we use a second agent to classify a corpus of 633 Rydberg-array arXiv papers and show that nearly half are implementable on present-day QPUs while identifying specific hardware upgrades needed for the rest. Together, these results demonstrate that agentic workflows provide a practical bridge between theoretical ideas and physical hardware, opening quantum experimentation to a much broader scientific community.
Constantin Dalyac, Alexandre Dauphin, Loïc Henriet +1
PASQAL SAS, 24 rue Emile Baudot, 91120 Palaiseau, France · Quantonation Ventures, Dallas, TX 75206, USA