Organizations: Department of Physics, Massachusetts Institute of Technology · Research Laboratory of Electronics, Massachusetts Institute of Technology · NSF Institute for Artificial Intelligence and Fundamental Interactions
Vision-language models (VLMs) are increasingly being used in scientific workflows, but their ability as agents to directly control laboratory experiments from visual feedback remains underexplored. This capability is important because many laboratory tasks do not naturally provide dense, pre-defined numerical objectives: informative signals can be sparse, intermittent, or visually ambiguous. A more general laboratory agent should instead be able to interpret visual observations, take actions to acquire useful feedback, and adapt its behavior based on the consequences of those actions. We study whether general-purpose VLMs can perform this kind of closed-loop scientific control using experimental optics as a testbed. We evaluate agents on three experimental systems that isolate distinct capabilities: a Michelson interferometer, a two-mirror cavity, and a four-mirror optical relay. The agents observe camera images, directly issue actuator and measurement commands, and retain their interaction history without receiving an engineered scalar objective during control. Across these experiments and matched simulations, we find that, given task-specific natural-language guidance, VLMs can estimate actuator-response relationships, resolve ambiguous observations through intervention, and actively create informative visual feedback when signals are sparse. These results suggest that pretrained multimodal models can serve as important decision-making agents within the experimental loop. Our work also establishes optics as a physically grounded testbed for visual reasoning and active perception in scientific agents.
Figures & tables
Figure 1: Vision-language agent framework for active perception. The agent receives an experimental goal in natural language, the current observation (such as a camera image), and the full conversation history which includes past observations. The model then selects an optical-control action, a measurement action, or a completion decision in a closed feedback loop with the experimental setup. The three chosen optical systems highlight complementary visual reasoning capabilities required for performing optical alignment.
Figure 2: Vision-language-agent alignment of a Michelson interferometer. (a) Schematic of the Michelson interferometer setup. (b) Photograph of the corresponding laboratory setup with motorized mirror mounts used for closed-loop alignment. The white dotted line shows the intended laser beam path. (c) Mean summed distance of each beam from the center of camera over successive optimization iterations for three VLMs; shaded regions indicate the sample standard deviation across repeated runs. (d) Camera images during a representative GPT-5.6 Sol run, progressing from separated, clipped beam patterns to a centered overlap with linear interference fringes.
Figure 3: Vision-language-agent alignment of a two-mirror cavity. (a) Schematic of the two-mirror cavity, showing the laser, in-coupling mirror, out-coupling mirror, and a camera. (b) Photograph of the corresponding laboratory setup with a motorized mirror mount attached to the in-coupling mirror used for closed-loop alignment. The white dotted line shows the intended laser beam path. (c) Mean distance measured in motor steps between the center of the secondary beam and the main beam over optimization iterations for three VLMs; shaded regions indicate the sample standard deviation across repeated runs. (d) Camera images during a representative run. The green × marks the true location of the secondary beam and the red × marks where the agent thinks the beam is. These marks are for post-hoc analysis only; they were not provided to the agent.
Figure 4: Vision-language-agent alignment of a four-mirror relay. (a) Schematic of the four-mirror relay, showing the laser, four mirrors, and a camera. (b) Photograph of the corresponding laboratory setup with a robotic arm for pick-and-place and motorized mirror mounts attached to each mirror. The white dotted line shows the intended laser beam path. (c) The mean cumulative number of iterations performed to align each mirror across the two VLMs tested, measured from the start of each run. (d) Top-down views of the four-mirror relay showing typical camera placements (and beam images) at each stage, with the mirror being aligned highlighted in yellow.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Simulation reproduces laboratory alignment behavior and enables comparison with privileged optimization baselines. (a) Mean best-so-far Michelson interferometer alignment error, using the same error metric defined in the main text. Simulated and laboratory GPT-5.6 Sol trajectories show similar convergence, while GPT-5.6 Sol reaches a lower final error than BO and random search. (b) Mean best-so-far two-mirror cavity alignment error, using the same error metric and success threshold defined in the main text. Simulated and laboratory GPT-5.6 Sol trajectories again show similar convergence, while BO converges more slowly initially but ultimately reaches lower error. In (a-b), best-so-far error is shown because it is the natural performance measure for BO and random search. Shaded regions indicate sample standard deviation across runs. (c) Mean cumulative number of iterations, measured from the start of each run, required for GPT-5.6 Sol to align each successive mirror in the four-mirror relay in laboratory and simulation experiments; error bars indicate the sample standard deviation. The simulated and laboratory trajectories show similar sequential alignment behavior. Numerical baselines are not shown in this panel because all BO and random search runs failed to complete the relay: neither method propagated the beam to the fourth mirror, and BO therefore never obtained an informative final-camera signal from which to optimize. The laboratory results include the nine successful runs, and the simulation comparison uses the corresponding nine initial configurations.
Advanced Photon Source Argonne National Laboratory Lemont, IL, USA · Data Science and Learning Division Argonne National Laboratory Lemont, IL, USA · Department of Radiation Oncology Northwestern University Chicago, IL, USA