cs.LGFeb 27, 2026

OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design

Authors: Yuyu Geng, Lei Sun, Yao Gao, Xinxin Hu, Zhonghua Yi, Xiaolong Qian, Weijian Hu, Jian Bai, 

Organizations: Zhejiang University · INSAIT, Sofia University · Xiaohongshu

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Optical design is the process of configuring optical elements to precisely manipulate light for high-fidelity imaging. It is inherently a highly non-convex optimization problem that relies heavily on human heuristic expertise and domain-specific knowledge. While Large Language Models (LLMs) possess extensive optical knowledge, their capabilities in leveraging the knowledge in designing lens system remain significantly constrained. This work represents the first attempt to employ LLMs in the field of optical design. We bridge the expertise gap by enabling users without formal optical training to successfully develop functional lens systems. Concretely, we curate a comprehensive dataset, named OptiDesignQA, which encompasses both classical lens systems sourced from standard optical textbooks and novel configurations generated by automated design algorithms for training and evaluation. Furthermore, we inject domain-specific optical expertise into the LLM through a hybrid objective of full-system synthesis and lens completion. To align the model with optical principles, we employ Group Relative Policy Optimization Done Right (DrGRPO) guided by Optical Lexicographic Reward for physics-driven policy alignment. This reward system incorporates structural format rewards, physical feasibility rewards, light-manipulation accuracy, and LLM-based heuristics. Finally, our model integrates with specialized optical optimization routines for end-to-end fine-tuning and precision refinement. We benchmark our proposed method against both traditional optimization-based automated design algorithms and LLM counterparts, and experimental results show the superiority of our method.

Figures & tables

Figure 1: Despite possessing foundational optical knowledge, general-purpose LLMs ( e.g. , ChatGPT) fail to generate physically realizable systems, regardless of the prompting strategy. In contrast, OptiAgent bridges this gap by producing designs that strictly satisfy both practical and physical constraints.
Figure 2: Overview of the OptiAgent framework. Given natural language instruction, the system integrates both optical prescription completion task and whole optical system design task, a physics-driven policy with hierarchical rewards, and a function call of Zemax in a closed loop to synthesize optical prescriptions L\mathcal{L} from specifications P\mathcal{P} . At inference, the generated initial structure L0\mathcal{L}_{0} undergoes Zemax local optimization for final refinement.
Figure 3: Statistics of EFFL, FoV, and F-number in our OptiDesignQA dataset. Our dataset covers a broad spectrum of specifications for commonly used optical lenses. Distinct peaks in F-number arises from the prevalence of standard engineering aperture settings in practical optical designs.
Figure 4: Layouts of representative designs from all evaluated methods. For each optical system, the RMS spot diagrams for three distinct FoVs are displayed. The first row illustrates the initial layouts produced by each method before fine-grained optimization in Zemax, while the second row shows the corresponding results after optimization. Ray colors blue, green, and red represent wavelengths of 0.485 μ\mu m, 0.588 μ\mu m, and 0.656 μ\mu m, respectively. Even when compared against significantly larger architectures like Qwen3-235B, our method utilizing Qwen3-4B as a foundation model exhibits better performance.
ModelSR(%) ↑\uparrowEFFL Relative Error ↓\downarrowInitial RMS ↓\downarrowFinal RMS ↓\downarrow
Avg(%)StdAvg(um)StdAvg(um)Std
Qwen3-4B-Instruct56.942.10.2912405.197578.91758.30227.46
ChatGPT-5.272.037.20.229942.3514033.07249.64172.65
Qwen3-235B-22B-Instruct77.854.70.193288.302107.08127.35176.32
Claude-Sonnet-4.581.835.30.225847.896113.31155.02164.47
OptiAgent (Ours)90.11.01.06×10−31.06\times 10^{-3}672.09138.1041.2376.14
Table 1: Performance comparison with LLMs on our OptiDesignQA test set. Best results in bold .
Reward CombinationSR(%) ↑\uparrowEFFL Relative Error ↓\downarrowInitial RMS ↓\downarrowFinal RMS ↓\downarrow
Avg(%)StdAvg(um)StdAvg(um)Std
Ray Tracing95.31.93.80×10−33.80\times 10^{-3}717.496.31422.71449.3
Ray Tracing + RMS0.00//////
Ray Tracing + RMS + LLM-as-judge83.728.22.06×10−172.06\times 10^{-17}621.35.01×10−145.01\times 10^{-14}305.11.25×10−141.25\times 10^{-14}
Ray Tracing →\rightarrow LLM-as-judge42.51.28.00×10−48.00\times 10^{-4}4574.01711.8344.9222.18
Ray Tracing →\rightarrow LLM-as-judge →\rightarrow RMS93.71.62.40×10−32.40\times 10^{-3}1809.21112.6127.3123.9
Table 2: Ablation study on the design of reward function. “+” means combining and “ →\rightarrow " signifies a threshold-based activation where the next stage is triggered only after the current criterion is satisfied.
Ratio of the masked dataSR(%) ↑\uparrowEFFL Relative Error ↓\downarrowInitial RMS ↓\downarrowFinal RMS ↓\downarrow
Avg(%)StdAvg(um)StdAvg(um)Std
028.841.32.17×10−172.17\times 10^{-17}268.11.24×10−141.24\times 10^{-14}601.85.50×10−145.50\times 10^{-14}
50%50\%64.327.52.06×10−172.06\times 10^{-17}637.44.91×10−144.91\times 10^{-14}256.431.2
100%100\%90.19.80.001672.1138.141.2376.14
Table 3: Ablation study on the Optical Prescription Completion. Different ratios of masked data are tested.
Train MethodSR(%) ↑\uparrowEFFL Relative Error ↓\downarrowInitial RMS ↓\downarrowFinal RMS ↓\downarrow
Avg (%)StdAvgStdAvgStd
SFT →\to RL85.028.5%28.5\%0.0003851.70.26200.920.9
SFT36.152.8%52.8\%0.275367.24083.1123.84244.18
RL ( Ours )90.10.98%0.98\%0.001672.1138.141.2376.14
Table 4: Ablation on training paradigm. Comparison between SFT, SFT followed by RL, and direct RL (Ours). Bold indicates the best performance.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 9
More in this topic

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 601 papers

Explore similar work

CardsList
Sep 24, 2026

LensDesigner: A Self-Improving Agent for Optical Lens Design

Optical lens design is a complex, non-convex optimization challenge that relies heavily on human experience and intuition. Existing optimized-based automatic lens design methods struggle to navigate this vast parameter space without meticulous manual tuning. In this paper, we present LensDesigner, an autonomous agent framework that mirrors the problem-solving workflow of expert opticians. To overcome the initial cold start problem, we construct LensLib100K, an extensive optical lens library, and employ Optics-Aware Retrieval to supply physically valid structural seeds. Within an interactive physical simulation environment, the agent executes macroscopic orchestration while receiving immediate optical feedback. Furthermore, we introduce a continuous self-evolving mechanism guided by a curriculum agent. By iteratively solving design tasks with progressively increasing difficulty, the agent autonomously extracts, accumulates, and reuses design heuristics, effectively evolving its optical lens design expertise over time. At the evaluation level, we introduce LensArena, a standardized evaluation benchmark comprising 120120 diverse optical design tasks, covering extreme configurations. Extensive experiments on this benchmark demonstrate that LensDesigner significantly outperforms publicly available baseline algorithms, achieving superior success rates and optimization efficiency. We hope this work sheds light on the emerging field of intelligent optics. The code will be publicly available.

Lei Sun, Haoran Liang, Dannong Xu +6

INSAIT, Sofia University “St. Kliment Ohridski” · Zhejiang University

Self-Improving AgentsAutomated Heuristic DesignConstrained Optimization
AskPDF
Feb 23, 2026

Agentic AI for Scalable and Robust Optical Systems Control

We present AgentOptics, an agentic AI framework for high-fidelity, autonomous optical system control built on the Model Context Protocol (MCP). AgentOptics interprets natural language tasks and executes protocol-compliant actions on heterogeneous optical devices through a structured tool abstraction layer. We implement 64 standardized MCP tools across 8 representative optical devices and construct a 410-task benchmark to evaluate request understanding, role-aware responses, multi-step coordination, robustness to linguistic variation, and error handling. We assess two deployment configurations--commercial online LLMs and locally hosted open-source LLMs--and compare them with LLM-based code generation baselines. AgentOptics achieves 87.7%--99.0% average task success rates, significantly outperforming code-generation approaches, which reach up to 50% success. We further demonstrate broader applicability through five case studies extending beyond device-level control to system orchestration, monitoring, and closed-loop optimization. These include DWDM link provisioning and coordinated monitoring of coherent 400 GbE and analog radio-over-fiber (ARoF) channels; autonomous characterization and bias optimization of a wideband ARoF link carrying 5G fronthaul traffic; multi-span channel provisioning with launch power optimization; closed-loop fiber polarization stabilization; and distributed acoustic sensing (DAS)-based fiber monitoring with LLM-assisted event detection. These results establish AgentOptics as a scalable, robust paradigm for autonomous control and orchestration of heterogeneous optical systems.

Zehao Wang, Mingzhe Han, Wei Cheng +12

Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA · NEC Laboratories America, Princeton, NJ 08540, USA · Duke Quantum Center and Department of Physics, Duke University, Durham, NC, USA 27708 +4

LLM Agent OrchestrationAI Agent BenchmarksLLM Agents
AskPDF
Aug 21, 2026

VortexChat: An agentic framework for autonomous multi-objective integrated photonic design

The advancement of modern integrated photonics is frequently bottlenecked by device design workflows that rely heavily on manual simulation and expert intuition. While inverse design offers an alternative, it remains constrained by expert supervision and a lack of end-to-end automation. To address these issues, we present VortexChat, an agentic framework for the autonomous, end-to-end inverse design of integrated photonic devices directly from natural language specifications. VortexChat couples a large language model (LLM) decision agent with topology generation, gradient-based refinement, and full-wave electromagnetic simulation. This closed-loop architecture enables the system to iteratively decompose design objectives, orchestrate computational tools, and update strategies based on feedback with minimal human intervention. Constrained by the absolute metrics of the Vortex100 Benchmark, VortexChat autonomously generates devices that strictly meet all predefined performance thresholds without any human-in-the-loop. As an experimental demonstration, we fabricated a broadband terahertz perfect vortex beam multiplexer, autonomously designed by VortexChat, with measurements confirming high-efficiency operation, high mode purity and low inter-channel crosstalk in agreement with full-wave simulations. These results demonstrate that an LLM agent can assume key aspects of expert decision-making in photonic inverse design while maintaining physical fidelity and fabrication feasibility, providing a scalable route towards autonomous design of complex integrated photonic systems.

Faqian Chong, Yulun Wu, Shilong Li +3

Innovative Institute of Electromagnetic Information and Electronic Integration, College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, 310027, China. · ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University, Hangzhou 310200, China. · International Joint Innovation Center, The Electromagnetics Academy at Zhejiang University, Zhejiang University, Haining 314400, China. +3

Inverse DesignTool-Augmented Language Model AgentsLLM Agent Orchestration
AskPDF