cs.LGNov 8, 2026

Building heterogeneous multistep workflows in automated microscopy: reward design and knowledge transfer for autonomous structure property discovery

Authors: Sergei V. Kalinin

Organizations: Department of Materials Science and Engineering, University of Tennessee, Knoxville, TN 37996, USA · Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN 37996, USA

Abstract

Autonomous microscopy can turn fast acquisition into scientific progress when it decides not only where to measure, but also what information to acquire, at what scan rate, and when to recalibrate the instrument. We develop a budget constrained framework for such multistep structure-property workflows. It combines explicit scientific rewards, action costs, instrument observation models, transfer of pre experiment knowledge into the online campaign, and evaluation kept independent of the controller. The transferred knowledge includes predictor accuracy and computational cost, empirical learner maturity functions that weight model-derived rewards, and drift characteristics established before controller training. Using a real electron energy-loss spectroscopy dataset, we run two complementary campaigns. In the first, the controller chooses between imaging and spectroscopy, image size, and scan speed. In the second, the instrument drifts and the controller also decides when to recalibrate. The campaigns show that response discovery and structure-property learning favor different surrogate models and acquisition strategies. They also show that simple acquisition rules form strong, practical baselines. Independent error analysis reveals where model uncertainty is and is not a reliable learning signal. In the drift campaign, explicit registration greatly improves positional accuracy, and the framework measures what this costs in spectroscopy throughput. Together, these results provide a reproducible way to design, compare, and audit autonomous experiments, and they define what microscopes must provide for deployment: independently executable actions, characterized scan and detector responses, timestamped registration telemetry, and complete action records.

Explore similar work

Apr 5, 2026

PATHFINDER: Multi-objective discovery in structural and spectral spaces

Automated decision-making is becoming key for automated characterization including electron and scanning probe microscopies and nano indentation. Most machine learning driven workflows optimize a single predefined objective and tend to converge prematurely on familiar responses, overlooking rare but scientifically important states. More broadly, the challenge is not only where to measure next, but how to coordinate exploration across structural, spectral, and measurement spaces under finite experimental budgets while balancing target-driven optimization with novelty discovery. Here we introduce PATHFINDER, a framework for autonomous microscopy that combines novelty driven exploration with optimization, helping the system discover more diverse and useful representations across structural, spectral, and measurement spaces. By combining latent space representations of local structure, surrogate modeling of functional response, and Pareto-based acquisition, the framework selects measurements that balance novelty discovery in feature and object space and are informative and experimentally actionable. Benchmarked on pre acquired STEM EELS data and realized experimentally in scanning probe microscopy of ferroelectric materials, this approach expands the accessible structure property landscape and avoids collapse onto a single apparent optimum. These results point to a new mode of autonomous microscopy that is not only optimization-driven, but also discovery-oriented, broad in its search, and responsive to human guidance.
Mar 17, 2026

Novelty-Driven Target-Space Discovery in Automated Electron and Scanning Probe Microscopy

Modern automated microscopy faces a fundamental discovery challenge: in many systems, the most important scientific information does not reside in the immediately visible image features, but in the target space of sequentially acquired spectra or functional responses, making it essential to develop strategies that can actively search for new behaviors rather than simply optimize known objectives. Here, we developed a deep-kernel-learning BEACON framework that is explicitly designed to guide discovery in the target space by learning structure-property relationships during the experiment and using that evolving model to seek diverse response regimes. We first established the method through demonstration workflows built on pre-acquired ground-truth datasets, which enabled direct benchmarking against classical acquisition strategies and allowed us to define a set of monitoring functions for comparing exploration quality, target-space coverage, and surrogate-model behavior in a transparent and reproducible manner. This benchmarking framework provides a practical basis for evaluating discovery-driven algorithms, not just optimization performance. We then operationalized and deployed the workflow on STEM, showing that the approach can transition from offline validation to real experimental implementation. To support adoption and extension by the broader community, the associated notebooks are available, allowing users to reproduce the workflows, test the benchmarks, and adapt the method to their own instruments and datasets.
Jul 7, 2026

From Closed-Loop Optimization to Open Decision Making: Coupled Digital Twins for Predictive and Autonomous Microscopy

Automated experimentation is moving from closed-loop optimization toward open decision-making, where human or AI planners must forecast the consequences of candidate actions before executing them. Such forecasts require a model of both sides of the experiment: how the sample is likely to respond and what the instrument is likely to detect. We therefore introduce a coupled digital-twin framework that separates these roles and then links them. In this framework, the sample twin encodes material state inferred from prior knowledge and measurements till the moment. The instrument twin captures signal formation, feedback dynamics, and operating constraints based on prior knowledge. When coupled, the two twins estimate expected outcomes, uncertainty, and risk for candidate microscope operations. For amplitude-modulation scanning probe microscopy, we realize this framework with a physics-informed encoder of force-distance curves, a deterministic scanner model of cantilever and feedback dynamics, and sparse learned residual corrections. The encoder first recovers scanner-driving descriptors with sub-nanometer accuracy. The calibrated scanner then reproduces typical traces within a few nanometers and identifies operating-point noise amplification as the main source of mismatch. Supplementary phase analysis localizes residual error to the phase channel, which clarifies where added physics is needed. Together, these results establish coupled sample and instrument twins as a practical foundation for predictive microscope operation and autonomous experimental planning.