EAA: Automating materials characterization with vision language model agents

Ming Du; Yanqi Luo; Srutarshi Banerjee; Michael Wojcik; Jelena Popovic; Mathew J. Cherukara

cs.AIFeb 17, 2026

EAA: Automating materials characterization with vision language model agents

Authors: Ming Du, Yanqi Luo, Srutarshi Banerjee, Michael Wojcik, Jelena Popovic, Mathew J. Cherukara

Organizations: Advanced Photon Source Argonne National Laboratory Lemont, IL, USA · Data Science and Learning Division Argonne National Laboratory Lemont, IL, USA · Department of Radiation Oncology Northwestern University Chicago, IL, USA

Abstract

We present Experiment Automation Agents (EAA), a vision-language-model-driven agentic system designed to automate complex experimental microscopy workflows. EAA integrates multimodal reasoning, tool-augmented action, and optional long-term memory to support both autonomous procedures and interactive user-guided measurements. Built on a flexible task-manager architecture, the system enables workflows ranging from fully agent-driven automation to logic-defined routines that embed localized LLM queries. EAA further provides a modern tool ecosystem with two-way compatibility for Model Context Protocol (MCP), allowing instrument-control tools to be consumed or served across applications. We demonstrate EAA at an imaging beamline at the Advanced Photon Source, including automated zone plate focusing, natural language-described feature search, and interactive data acquisition. These results illustrate how vision-capable agents can enhance beamline efficiency, reduce operational burden, and lower the expertise barrier for users.

Figures & tables

Figure 1: The main components of EAA and their interactions. The task manager contains the chat loop or workflow, creates and holds the agent object, and maintains the context. New messages coming from the user, auto-generated by the workflow logic, or responded by the agent are added to the context. When the LLM agent drives the workflow, it sends a termination signal when the task is considered finished; The task manager then requests human input, based on which it either continues the workflow with updated instructions, enters the purely interactive chat mode, or exits. When a message is sent to the agent, it is sent with the entire context so that the VLM has conversational memory. Optionally, the agent can have long-term memory by saving the embeddings of notable messages into a vector store and retrieving them for future conversations and sessions. Tools from EAA’s tool library or provided through external MCP servers are registered to the agent’s tool manager through the task manager. The agent generates the schemas for the tools and attach them with messages sent to the VLM, and handles the tool calls made by the VLM.

Figure 2: Three levels of LLM involvement in experiment automation tools. Examples are enumerated for each level.

Figure 3: Example workflow diagrams of EAA task managers. (a) The chat loop, which is a generic workflow used for interactive conversations and agent-driven workflows. (b) The workflow of an automated focusing task as an example. This workflow falls in the hybrid workflow category defined in Section 2.1 , and involves programmed query with the agent about image overlap, logic-based registration and auto-reply composition, and call to a sub-task for feature tracking. In both diagrams, nodes with dashed borders are those involving agent interactions.

Figure 4: Trajectory of the focusing task. (a) 2D images acquired at all visited zone plate

z

-positions, used for estimating positional drift. The

z

-positions in mm are indicated at the top right corner of each image. The blue line in the image at

z=-198

indicate the vertical line that the agent was instructed to take line scans across. As the focus of the zone plate improves, images gradually become sharper. (b) The line profiles collected across the reference feature throughout the process. From red to green, the zone plate

z

-position moves from the initial -200 mm towards the final -193.7 mm. The positions of the line plots are relative, meaning the center of the profiles are aligned at

x=0

in the figure. (c) The trajectory of the process in the space of line profile FWHM and

z

-position. The initial point at

z=-200

mm is not shown because its FWHM is too large.

Figure 5: Trajectory and result of the feature search task. (a) The trajectory of the center of 2D image acquisitions in the sample space. Underlaid in the background is the SEM image of the sample (a test pattern), which is transformed to align with the trajectory plot. (b) The image returned by the last 2D scan tool call (the title and axis ticks are a part of the returned image), right before the agent concluded the process. The requested feature (Siemens star) is roughly centered in the FOV.

Figure 6: Input and acquired images during the interactive data acquisition session. (a) The image initially shown to the agent (agent did not see the dashed box). We subsequently took a screenshot of the area enclosed by the white dashed box and sent it to the agent, instructing it to take finer scans in that region. (b) Images captured by the agent with the 2D acquisition tool. These images are flipped vertically compared to (a) as (a) is a screenshot from the beamline control software and uses a different convention for plotting from the image acquisition tool of EAA. The scan step sizes are indicated at the top left corner. The image with $\Delta=2$ $\upmu$ m was acquired after the agent was instructed to perform a local scan of the user-selected feature. Images with smaller step sizes were acquired following the user’s instruction to perform 3 scans sequentially. The truncation on the right of the images with $\Delta$ = 0.5 and 0.2 $\upmu$ m is due to a problem in the hardware buffer that the tool depends on, irrelevant to EAA and the tools themselves.

Figure 7: Model benchmarking results for two tasks related to experiment automation and image analysis. The bar heights indicate the average, and the half lengths of the error bars indicate the standard deviations of the measured quantities calculated over 10 trials for each model. Temperature and reasoning effort for model prediction was set to 1.0. (a) The performance of GPT-4o, GPT-5, Gemini 2.5 Pro, and Gemini 3 Pro Preview when prompted to acquired 4 images on a square grid using a provided image acquisition tool. “Hit rate” is the proportion that the $4\times 10$ expected tool calls are made. “Latency” shows the average time that the agent took to finish the task in each trial. (b) The accuracy of identifying the position of a marker in an image, tested on GPT-4o, GPT-5, Gemini 2.5 Pro, Gemini 3 Pro Preview, and Gemma 3 (27B). The accuracy is measured by the error of the reported coordinates compared to the true coordinates. (c) shows the image with the marker that AI sees.

Laboratory Automation

Laboratory automation uses robotics, software, and AI to plan, execute, monitor, and analyze experiments with less manual intervention.

See all 61 papers

Agentic AI for operating scientific instruments for nanoscale characterization
Aug 25, 2026Zahra Ayar, Marcos Penedo, Mahdi Mehdikhani +3Laboratory Automation Agentic Workflows
Laboratory for Bio- and Nano-Instrumentation, Institute of Bioengineering, School of Engineering, Swiss Federal Institute of Technology Lausanne (EPFL), Lausanne 1015, Switzerland
BioProVLA-Agent: An Affordable, Protocol-Driven, Vision-Enhanced VLA-Enabled Embodied Multi-Agent System with Closed-Loop-Capable Reasoning for Biological Laboratory Manipulation
May 8, 2026Zhaohui Du, Zhe Wang, Dongzhan Zhou +9Multi-Agent Orchestration Long-Horizon Robotic Manipulation
Key Laboratory of Smart Manufacturing in Energy Chemical Process Ministry of Education, East China University of Science and Technology, Shanghai, CN. · Department of Computer Science and Engineering, East China University of Science and Technology, Shanghai, CN. · AI for Science Center, Shanghai AI Laboratory, Shanghai, CN. +2
Agentic self-driving microscopy benchmarks support qualification but do not necessarily generalize to unseen tasks
Aug 5, 2026Nathan S Johnson, Ian AbshireAI Agents for Scientific Discovery LLM Agent Orchestration
Carl Zeiss Research Microscopy Solutions, 5300 Central Boulevard, Dublin, CA 94568

Aug 25, 2026J/K move · Enter open · S save

Agentic AI for operating scientific instruments for nanoscale characterization

Zahra Ayar, Marcos Penedo, Mahdi Mehdikhani, Nahid Hosseini+2

Laboratory for Bio- and Nano-Instrumentation, Institute of Bioengineering, School of Engineering, Swiss Federal Institute of Technology Lausanne (EPFL), Lausanne 1015, Switzerland

Laboratory Automation Agentic Workflows AI Agent Safety

Open paper AskPDF arXiv

Operating a scientific instrument such as an atomic force microscope (AFM) requires continuous expert decision-making. A trained user defines the experimental intent, translates it into instrument commands, assesses incoming data, adjusts imaging parameters, and post-processes the final image. Existing automation usually addresses only parts of this workflow through hard-coded routines, task-specific controllers, or trained machine-learning models. Here we present an agentic-AI framework that operates the executable part of the AFM workflow using a general-purpose, tool-augmented large language model connected to instrument functions through the Model Context Protocol (MCP). The framework consists of 3 MCP-based agents: AFM Messenger converts natural-language instructions into checked instrument commands; AFM Pilot assesses image quality through a large language model (LLM) and, if necessary, adapts imaging parameters; and AFM Doctor diagnoses image artifacts and applies transparent post-processing from a pre-approved tool set. Because the language model performs image assessment rather than a fixed scalar objective or external optimizer, the same strategy can be applied across sample types and imaging modes without specific retraining. Safe hardware operation is enforced through an ambiguity check layer before execution. Benchmarking against fine-tuned and off-the-shelf tool-using models shows that this guarded execution layer, rather than model capability alone, reduces wrong-command execution to zero. In live experiments on different samples, AFM Pilot matched expert operators in image quality, iteration count, and tuning time, with no significant difference. These results demonstrate a safe route to agentic operation of scientific instruments, where experimental intent remains human-defined while command execution, image-based tuning, and post-processing are delegated to AI agents.

Abstract

Laboratory Automation

Explore similar work

Agentic AI for operating scientific instruments for nanoscale characterization

BioProVLA-Agent: An Affordable, Protocol-Driven, Vision-Enhanced VLA-Enabled Embodied Multi-Agent System with Closed-Loop-Capable Reasoning for Biological Laboratory Manipulation

Agentic self-driving microscopy benchmarks support qualification but do not necessarily generalize to unseen tasks