Adaptive Control of Scientific Instruments with Nonstationary Responses through Multimodal Agents

Guoyang Shi; Hongwen Li; Yiding Yu; Zhuang Fu; Siqi Ding; Zhengyuan Chen; JianGuo Chen; Hanyue Zhao

cs.AIOct 23, 2026

Adaptive Control of Scientific Instruments with Nonstationary Responses through Multimodal Agents

Authors: Guoyang Shi, Hongwen Li, Yiding Yu, Zhuang Fu, Siqi Ding, Zhengyuan Chen, JianGuo Chen, Hanyue Zhao,

Organizations: Beijing ENN Fusion Energy Science and Technology Co., Ltd., Beijing 101111, China · Beijing Key Laboratory of High Magnetic Field Spherical Torus Fusion Energy, Beijing 101111, China · Hebei Key Laboratory of Compact Fusion, Langfang 065000, China

Abstract

Controlling scientific instruments requires parameters to be adjusted progressively in response to feedback from successive operations. When unobserved operating conditions evolve with operational history, responses become nonstationary, and a static mapping from actions to responses becomes inadequate for decisions across shots. Shot-to-shot tuning of neutral beam injection (NBI) ion sources in magnetic confinement fusion devices is representative of this class of problems: the conditions of electrode surfaces and vacuum are difficult to observe directly, discharge outcomes vary between shots, and experiments are costly and carry operational risks. We propose a physically grounded multimodal agent with three independently auditable components. Deterministic algorithms extract symbolic physical quantities and visual evidence from multichannel waveforms; a memory with two layers organizes trajectories from the current task, experience from earlier tasks, and procedural knowledge, from which a multimodal large language model (MLLM) generates candidate actions; and an independent action feasibility check enforces action constraints before execution. In a strictly chronological replay of 455 shots from four EXL-50U platforms, physical symbols alone outperformed waveform images alone by 0.21-0.32 in macro-F1 for quality classification across all four evaluated models. Retrieved context increased the mean agreement with historical action directions by 22.4-42.7 percentage points, whereas the contribution of visual evidence varied across models. Under supervised operation on five ion sources, the system achieved a beam-extraction rate of 86.80% across 409 executed shots. These results demonstrate the engineering feasibility of the framework for supervised sequential tuning on a physical scientific instrument.

Figures & tables

Figure 1: Measured multichannel waveforms from three consecutive shots (shots 22541–22543) under identical control settings. Although all five control variables were held fixed, the recorded acceleration currents were 41.54 A, 0.00 A, and 33.65 A, respectively. The lower states $z_{1}$ – $z_{3}$ schematically represent unobserved operating conditions and denote history-dependent factors, beyond the current control variables, that may affect the response; the bottom annotation summarizes that the same action $a$ yields different responses $y$ as the latent state $z$ drifts.

Method	Handles latent- state drift	Requires a high- fidelity simulator	Uses unstructured knowledge	Local–global feature integration	Action-constraint mechanism	Physical-system closed loop
Ion-source/NBI static surrogate models and GA/PSO [ 20 , 22 , 23 , 24 , 25 ]	Usually no	No	No	Scalar features; no explicit integration	Static bounds and constraints can be encoded	No reported NBI tuning closed loop
Simulator-driven plasma RL [ 17 ]	Indirectly, through state observations	Yes	No	Local magnetic measurements and global shape objectives	Action bounds and device interlocks	Yes (plasma control)
Standard/safe Bayesian optimization [ 27 , 28 ]	Not addressed for latent-state drift	No	No	Point observations and a global surrogate	Conditional guarantees in safe variants	Applicable to online experiments
LLM agents for scientific experiments [ 31 , 32 ]	Through context or memory	No	Yes	Task-relevant inputs; no explicit integration	Can use semantic guardrails or external validators [ 38 ]	Yes (chemical experiments)
This work	History-conditioned dual-layer memory	No	Yes	Quantitative details and overall waveform morphology	Deterministic action feasibility check	Supervised shot-to-shot execution

Table 1: Qualitative comparison of the task conditions in this study with representative automated decision-making paradigms

Figure 2: Structure of the 50 kV hot-cathode arc-discharge positive ion source investigated in this study. The device consists primarily of a discharge-chamber backplate integrating the filaments and feedthrough assemblies, a multicusp-field discharge chamber in which the plasma is generated, and a three-electrode accelerator comprising the plasma, suppression, and ground grids.

Figure 3: Overall architecture of the physics-grounded multimodal agent. Deterministic signal grounding and annotated waveform rendering transform raw records into symbolic and visual evidence. Dual-layer memory supplies retrieved temporal context, which the MLLM integrates with the current evidence to propose a candidate action for the next shot. The candidate is passed unchanged to the PLC only after an independent feasibility check confirms every condition in $\mathcal{C}_{t}$ ; otherwise, it is rejected for regeneration or escalated to the operator. The executed shot is subsequently read back to update the archive and memory.

Figure 4: Symbolic grounding pipeline. After preprocessing and interval and event detection, the raw multichannel waveforms yield flat-top statistics, channel-missing flags, breakdown and oscillation events, and fault flags.

Figure 5: Typical acceleration-current waveform morphologies to be distinguished by the grounding operator: single flat top, multiple flat tops, beam oscillation, complete breakdown, and two types of sensor fault. The vertical axis of each panel is normalized separately and is intended only for morphological comparison.

Figure 6: Examples of annotations on measured acceleration-current waveforms: (a) beam flat-top identification for shot 11888; (b) high-frequency oscillation-dip identification for shot 12247. Dark gray denotes the raw waveform; the shaded band bounded by dotted lines marks the detected pulse interval, with the flat-top start $t_{s}$ , end $t_{e}$ , mean current $\bar{I}_{\mathrm{ACC}}$ , and width $\tau$ annotated in (a), and red triangles in (b) mark the detected oscillation points.

Class	Number of shots
Full breakdown	67
Insufficient pulse width	113
Long-pulse high current	80
Long-pulse low current	81
High-frequency oscillation	51
Fault noise	63

Table 2: Distribution of the six waveform classes in the 455-shot offline dataset

Configuration	Symbolic grounding	Visual evidence	Experience and knowledge retrieval	Feasibility check	Human feedback
Baseline	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$
Physical-only	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$
+Physical	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$
+RAG	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
Full	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 3: Five unified configurations for cross-model evaluation

Platform	Shots	Current MAE/A	Main-width MAE/ms	Flat-top-count accuracy	Flat-top event- count F 1	Breakdown-count accuracy	Breakdown event- count F 1
Test Bench 1	127	0.16	4.46	100.00%	100.00%	100.00%	100.00%
Test Bench 2	116	0.64	14.35	95.69%	96.08%	100.00%	100.00%
Experimental Bench 1	110	0.70	13.24	98.18%	97.44%	99.09%	97.56%
Experimental Bench 2	102	1.34	10.35	94.12%	96.20%	100.00%	100.00%
Pooled	455	0.68	10.42	97.14%	97.61%	99.78%	99.66%

Table 4: Independent validation of the symbolic grounding operator

\Phi

on four platforms

Model	Configuration	Quality-classification macro-F 1	Current MAE/A	Pulse-width MAE/ms	Flat-top-count MAE	Mean directional agreement
Qwen3.5-Plus	Baseline	0.47	3.67	138.19	0.27	36.7%
Qwen3.5-Plus	Physical-only	0.79	1.40	28.37	0.09	54.4%
Qwen3.5-Plus	+Physical	0.73	1.54	42.23	0.09	55.8%
Qwen3.5-Plus	+RAG	0.71	1.24	19.36	0.09	78.2%
Qwen3.5-Plus	Full	0.68	1.31	24.33	0.08	86.0%
MiniMax3	Baseline	0.57	3.91	107.80	0.24	35.6%

Table 5: Overall results for four models under five configurations

Ion source	Executed shots	With beam extraction	Without beam extraction	Beam-extraction rate
Test Bench 2	49	45	4	91.84%
Source 1, Experimental Bench 2	64	53	11	82.81%
Source 2, Experimental Bench 2	83	69	14	83.13%
Source 3, Experimental Bench 2	103	91	12	88.35%
Source 4, Experimental Bench 2	110	97	13	88.18%
All	409	355	54	86.80%

Table 6: Summary of physical operation results for five ion sources

Figure 7: Comparison of shot-level beam-extraction rates between manual tuning and supervised agent tuning (AI tuning in the figure) for four ion sources. Numbers inside the bars denote shots with beam extraction and physically executed shots, and labels above the bars give the corresponding extraction rates. Error bars are 95% Wilson intervals calculated from shot-level binomial samples and characterize the sampling uncertainty of each beam-extraction rate. Source IDs 5–8 on the horizontal axis are the identifiers in the raw records and correspond, respectively, to Sources 1–4 of Experimental Bench 2 in Table 6 .

Figure 8: Shot-to-shot response trajectories from three consecutive experiments under the target-oriented protocol. Bars indicate beam pulse width, lines indicate the median acceleration current within the valid interval, and red crosses indicate shots in which no valid beam interval was detected.

Metric	Trial 1	Trial 2	Trial 3	Total
Shot range	11394–11402	11884–11891	11948–11953	—
Number of shots	9	8	6	23
Shots with a valid beam interval	8	7	6	21
Valid-beam-interval rate	88.89%	87.50%	100.00%	91.30%
Median acceleration-voltage range/kV	26.39–33.36	21.06–44.54	22.43–38.44	21.06–44.54
Median acceleration-current range/A	16.60–26.88	10.31–23.46	10.89–28.56	10.31–28.56

Table 7: Waveform statistics from three consecutive experiments on the same ion source at Test Bench 2. Voltage, current, and pulse width are reported as ranges of shot-level statistics among shots with a valid beam interval.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Offline performance of four models under five configurations. From left to right, the panels report field-derived six-class macro-F 1 , mean directional agreement across five action parameters, quality-classification macro-F 1 , and acceleration-current MAE.

Model	Configuration	Current/A		Pulse width/ms		Flat-top count		Quality classification
		MAE	RMSE	MAE	RMSE	MAE	RMSE	Accuracy	F 1
Qwen3.5-Plus	Baseline	3.67	14.41	138.19	328.19	0.27	0.66	0.51	0.47
Qwen3.5-Plus	Physical-only	1.40	5.13	28.37	86.58	0.09	0.41	0.79	0.79
Qwen3.5-Plus	+Physical	1.54	7.08	42.23	121.89	0.09	0.34	0.73	0.73
Qwen3.5-Plus	+RAG	1.24	4.92	19.36	67.51	0.09	0.37	0.74	0.71
Qwen3.5-Plus	Full	1.31	5.05	24.33	77.34	0.08	0.36	0.84	0.68

Appendix

Table 8: Detailed recognition results for the four models. Continuous variables are reported as MAE/RMSE; quality classification is reported as Accuracy and macro-F 1 .

Model	Configuration	$V_{\rm ACC}$	$V_{\rm FIL}$	$V_{\rm ARC}$	$Q$	$V_{\rm DEC}$	Mean
Qwen3.5-Plus	Baseline	55.6%	47.9%	26.6%	18.7%	34.9%	36.7%
Qwen3.5-Plus	Physical-only	66.6%	69.2%	59.6%	39.8%	36.9%	54.4%
Qwen3.5-Plus	+Physical	66.6%	76.9%	56.9%	40.9%	37.8%	55.8%
Qwen3.5-Plus	+RAG	87.9%	82.2%	72.3%	75.4%	73.2%	78.2%
Qwen3.5-Plus	Full	89.7%	84.4%	85.1%	81.3%	89.7%	86.0%
MiniMax3	Baseline	49.2%	48.8%	23.7%	19.1%	37.4%	35.6%

Appendix

Table 9: Directional agreement (Accuracy) for five parameters across the four models. The mean is the arithmetic mean of the directional agreement for the five parameters.

Ion source	Manual tuning		Agent tuning		Difference (percentage points)
Ion source	Beam extracted/ executed	Extraction rate	Beam extracted/ executed	Extraction rate
ID 5 (Source 1)	377/436	86.47%	53/64	82.81%	$-3.7$
ID 6 (Source 2)	337/419	80.43%	69/83	83.13%	$+2.7$
ID 7 (Source 3)	361/419	86.16%	91/103	88.35%	$+2.2$
ID 8 (Source 4)	346/421	82.19%	97/110	88.18%	$+6.0$
Total	1421/1695	83.83%	310/360	86.11%	$\mathbf{+2.3}$

Appendix

Table 10: Comparison of shot-level beam-extraction rates between manual tuning and supervised agent tuning on four ion sources

Symbol	Physical quantity	Unit	PLC permissible range	Maximum per-shot change
$V_{\mathrm{ACC}}$	Acceleration voltage	kV	$5$ – $45$	Adjustment step: 1–5 kV
$V_{\mathrm{FIL}}$	Filament voltage	V	$5$ – $12$	Adjustment step: 0.1–0.5 V
$V_{\mathrm{ARC}}$	Arc voltage	V	$70$ – $130$	Adjustment step: 1–5 V
$Q$	Gas flow	sccm	$100$ – $700$	Adjustment step: 10–50 sccm
$V_{\mathrm{DEC}}$	Suppression voltage	V	$0$ – $3200$	Constrained during the check to 4%–10% of $V_{\mathrm{ACC}}$

Appendix

Table 11: Agent action parameters and PLC permissible constraints

Symbol	Meaning	Unit	Value and determination method
$w_{\mathrm{med}}$	Median-filter window width	samples	$15$
$3\sigma$	Measurement-spike rejection bound	dimensionless	$3$ ; $\lvert x-\mathrm{medfilt}\rvert$ is compared with the global standard deviation, rather than a local baseline
$\alpha$	Fraction trimmed from each end for the trimmed mean	%	$30$
$w_{\mathrm{trim}}$	Sliding-window width for the trimmed mean	samples	$21$
$\theta_{\mathrm{abs}}$	Absolute-amplitude threshold for candidate pulses	A	$1$ ; the acceleration-current threshold is identical across all four platforms
$r_{\mathrm{sel}}$	Relative coefficient for candidate pulses	dimensionless	$0.8$ ; relative to the maximum pulse mean

Appendix

Table 12: Signal-processing thresholds of the grounding operator

\Phi

Adaptive Control

Adaptive control lets systems update models, policies, or controller parameters online to maintain performance as dynamics and disturbances change.

See all 146 papers

Sep 26, 2026

Vision-Language Agents for Active Perception in Optics Laboratories

Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.

Ryan Lopez, Sachin Vaidya, Seou Choi +2

Department of Physics, Massachusetts Institute of Technology · Research Laboratory of Electronics, Massachusetts Institute of Technology · NSF Institute for Artificial Intelligence and Fundamental Interactions

Tool-Using Vision-Language Agents Active Perception Laboratory Automation

Ask PDF

Jun 15, 2026

LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

Current computer-use benchmarks primarily focus on software operation tasks in virtualized systems, whereas scientific instrumentation scenarios require coordinated control over complex interfaces, and feedback-driven parameter adjustment. However, directly evaluating agents on physical high-precision instruments is impractical due to high cost, safety risks, limited accessibility, and difficulty in ensuring reproducible evaluation. This motivates the need for a simulated yet realistic testbed that preserves the operational challenges of scientific instruments while enabling scalable and safe benchmarking. To this end, we introduce LabOSBench, a challenging benchmark for multimodal GUI agents built on a suite of web-based scientific-instrument simulators. Operating directly via a browser, LabOSBench avoids resource-heavy OS virtualization while supporting flexible task configuration and execution-based evaluation. Specifically, LabOSBench constructs 96 subtasks across eight instrument simulators, covering workflows from sample loading, alignment, parameter tuning, and data acquisition to result inspection. We evaluate general-purpose vision-language models, specialized GUI agent models, and advanced agentic frameworks at both subtask and end-to-end levels. Our experiments reveal that while existing agents can complete many structured GUI subtasks, they still struggle with feedback-driven operations and long-horizon workflow execution. Overall, LabOSBench provides a reproducible, low-cost testbed for advancing computer-using agents toward scientific-instrument control.

Anqi Zou, Han Deng, Chengyu Zhang +9

Shenzhen Loop Area Institute · Dalian University of Technology · The Chinese University of Hong Kong +1

Computer-Use Agent Benchmarks Computer-Use Agents Long-Horizon Agent Evaluation AI Agent Benchmarks Multimodal Agents

Ask PDF

Aug 25, 2026

Agentic AI for operating scientific instruments for nanoscale characterization

Operating a scientific instrument such as an atomic force microscope (AFM) requires continuous expert decision-making. A trained user defines the experimental intent, translates it into instrument commands, assesses incoming data, adjusts imaging parameters, and post-processes the final image. Existing automation usually addresses only parts of this workflow through hard-coded routines, task-specific controllers, or trained machine-learning models. Here we present an agentic-AI framework that operates the executable part of the AFM workflow using a general-purpose, tool-augmented large language model connected to instrument functions through the Model Context Protocol (MCP). The framework consists of 3 MCP-based agents: AFM Messenger converts natural-language instructions into checked instrument commands; AFM Pilot assesses image quality through a large language model (LLM) and, if necessary, adapts imaging parameters; and AFM Doctor diagnoses image artifacts and applies transparent post-processing from a pre-approved tool set. Because the language model performs image assessment rather than a fixed scalar objective or external optimizer, the same strategy can be applied across sample types and imaging modes without specific retraining. Safe hardware operation is enforced through an ambiguity check layer before execution. Benchmarking against fine-tuned and off-the-shelf tool-using models shows that this guarded execution layer, rather than model capability alone, reduces wrong-command execution to zero. In live experiments on different samples, AFM Pilot matched expert operators in image quality, iteration count, and tuning time, with no significant difference. These results demonstrate a safe route to agentic operation of scientific instruments, where experimental intent remains human-defined while command execution, image-based tuning, and post-processing are delegated to AI agents.

Zahra Ayar, Marcos Penedo, Mahdi Mehdikhani +3

Laboratory for Bio- and Nano-Instrumentation, Institute of Bioengineering, School of Engineering, Swiss Federal Institute of Technology Lausanne (EPFL), Lausanne 1015, Switzerland

Laboratory Automation Agentic Workflows AI Agent Safety

Ask PDF

Abstract

Adaptive Control

Explore similar work

Vision-Language Agents for Active Perception in Optics Laboratories

LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

Agentic AI for operating scientific instruments for nanoscale characterization