Organizations: Department of Physics, Massachusetts Institute of Technology · Research Laboratory of Electronics, Massachusetts Institute of Technology · NSF Institute for Artificial Intelligence and Fundamental Interactions
Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.
Figures & tables
Figure 1: Vision-language agent framework for active perception. The agent receives an experimental goal in natural language, the current observation (such as a camera image), and the full conversation history which includes past observations. The model then selects an optical-control action, a measurement action, or a completion decision in a closed feedback loop with the experimental setup. The three chosen optical systems highlight complementary visual reasoning capabilities required for performing optical alignment.
Figure 2: Vision-language-agent alignment of a Michelson interferometer. (a) Schematic of the Michelson interferometer setup. (b) Photograph of the corresponding laboratory setup with motorized mirror mounts used for closed-loop alignment. The white dotted line shows the intended laser beam path. (c) Mean summed distance of each beam from the center of camera over successive optimization iterations for three VLMs; shaded regions indicate the sample standard deviation across repeated runs. (d) Camera images during a representative GPT-5.6 Sol run, progressing from separated, clipped beam patterns to a centered overlap with linear interference fringes.
Figure 3: Vision-language-agent alignment of a two-mirror cavity. (a) Schematic of the two-mirror cavity, showing the laser, in-coupling mirror, out-coupling mirror, and a camera. (b) Photograph of the corresponding laboratory setup with a motorized mirror mount attached to the in-coupling mirror used for closed-loop alignment. The white dotted line shows the intended laser beam path. (c) Mean distance measured in motor steps between the center of the secondary beam and the main beam over optimization iterations for three VLMs; shaded regions indicate the sample standard deviation across repeated runs. (d) Camera images during a representative run. The green × marks the true location of the secondary beam and the red × marks where the agent thinks the beam is. These marks are for post-hoc analysis only; they were not provided to the agent.
Figure 4: Vision-language-agent alignment of a four-mirror relay. (a) Schematic of the four-mirror relay, showing the laser, four mirrors, and a camera. (b) Photograph of the corresponding laboratory setup with a robotic arm for pick-and-place and motorized mirror mounts attached to each mirror. The white dotted line shows the intended laser beam path. (c) The mean cumulative number of iterations performed to align each mirror across the two VLMs tested, measured from the start of each run. (d) Top-down views of the four-mirror relay showing typical camera placements (and beam images) at each stage, with the mirror being aligned highlighted in yellow.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Simulation reproduces laboratory alignment behavior and enables comparison with privileged optimization baselines. (a) Mean best-so-far Michelson interferometer alignment error, using the same error metric defined in the main text. Simulated and laboratory GPT-5.6 Sol trajectories show similar convergence, while GPT-5.6 Sol reaches a lower final error than BO and random search. (b) Mean best-so-far two-mirror cavity alignment error, using the same error metric and success threshold defined in the main text. Simulated and laboratory GPT-5.6 Sol trajectories again show similar convergence, while BO converges more slowly initially but ultimately reaches lower error. In (a-b), best-so-far error is shown because it is the natural performance measure for BO and random search. Shaded regions indicate sample standard deviation across runs. (c) Mean cumulative number of iterations, measured from the start of each run, required for GPT-5.6 Sol to align each successive mirror in the four-mirror relay in laboratory and simulation experiments; error bars indicate the sample standard deviation. The simulated and laboratory trajectories show similar sequential alignment behavior. Numerical baselines are not shown in this panel because all BO and random search runs failed to complete the relay: neither method propagated the beam to the fourth mirror, and BO therefore never obtained an informative final-camera signal from which to optimize. The laboratory results include the nine successful runs, and the simulation comparison uses the corresponding nine initial configurations.
Scientific laboratories increasingly rely on AI systems to reason about experiments, but the physical act of doing science remains largely outside their reach. AI can help read literature, generate hypotheses, and plan protocols, yet the execution of those protocols at the bench still requires a human operator. Vision-Language-Action (VLA) models provide one possible interface between written protocols and robot execution, but existing policies are trained mostly on household and tabletop demonstrations and rarely encounter the instruments, transparent liquids, or fixed protocol workflows found in scientific laboratories. Closing this gap requires both laboratory-specific supervision and a unified learning framework that can accommodate the diverse robot embodiments used to execute experimental protocols. We therefore identify data and embodiment as central bottlenecks alongside model design. To address the data side, we build RoboGenesis, a simulation-based workflow and data engine that composes configured laboratory workflows from atomic skills, validates and filters rollouts, and exports structured demonstrations across supported robot profiles. On the policy side, we present LabVLA, trained with a two-stage recipe: FAST action token pretraining first makes the Qwen3-VL-4B-Instruct backbone action aware before any continuous control is learned, and flow matching posttraining then attaches a DiT action expert under knowledge insulation. On the LabUtopia benchmark, LabVLA achieves the highest average success rate among all evaluated baselines under both in-distribution and out-of-distribution settings.
Baochang Ren, Xinjie Liu, Xi Chen +15
Zhejiang University · Shanghai AI Laboratory · Harbin Institute of Technology
We present Experiment Automation Agents (EAA), a vision-language-model-driven agentic system designed to automate complex experimental microscopy workflows. EAA integrates multimodal reasoning, tool-augmented action, and optional long-term memory to support both autonomous procedures and interactive user-guided measurements. Built on a flexible task-manager architecture, the system enables workflows ranging from fully agent-driven automation to logic-defined routines that embed localized LLM queries. EAA further provides a modern tool ecosystem with two-way compatibility for Model Context Protocol (MCP), allowing instrument-control tools to be consumed or served across applications. We demonstrate EAA at an imaging beamline at the Advanced Photon Source, including automated zone plate focusing, natural language-described feature search, and interactive data acquisition. These results illustrate how vision-capable agents can enhance beamline efficiency, reduce operational burden, and lower the expertise barrier for users.
Ming Du, Yanqi Luo, Srutarshi Banerjee +3
Advanced Photon Source Argonne National Laboratory Lemont, IL, USA · Data Science and Learning Division Argonne National Laboratory Lemont, IL, USA · Department of Radiation Oncology Northwestern University Chicago, IL, USA
Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception rather than a single static description. Frontier MLLMs collapse on ActiveVision: the highest-scoring model we evaluate, GPT-5.5 at the highest exposed reasoning-effort tier, solves only 10.6% of items and scores zero on 11 of the 17 tasks, and even Claude Fable 5, despite topping most reasoning and coding leaderboards, solves just 3.5%, far behind three human participants who average 96.1%. Furthermore, much of the gap persists even when models write and run their own vision code: such code is unreliable on realistic imagery, and catching its failures itself requires the active perception the models lack. Together, these results indicate that current MLLMs lack robust active visual observation, motivating architectures and training objectives that close the perception-reasoning loop.