cs.LGFeb 27, 2026

OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design

Authors: Yuyu Geng, Lei Sun, Yao Gao, Xinxin Hu, Zhonghua Yi, Xiaolong Qian, Weijian Hu, Jian Bai, 

Organizations: Zhejiang University · INSAIT, Sofia University · Xiaohongshu

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Optical design is the process of configuring optical elements to precisely manipulate light for high-fidelity imaging. It is inherently a highly non-convex optimization problem that relies heavily on human heuristic expertise and domain-specific knowledge. While Large Language Models (LLMs) possess extensive optical knowledge, their capabilities in leveraging the knowledge in designing lens system remain significantly constrained. This work represents the first attempt to employ LLMs in the field of optical design. We bridge the expertise gap by enabling users without formal optical training to successfully develop functional lens systems. Concretely, we curate a comprehensive dataset, named OptiDesignQA, which encompasses both classical lens systems sourced from standard optical textbooks and novel configurations generated by automated design algorithms for training and evaluation. Furthermore, we inject domain-specific optical expertise into the LLM through a hybrid objective of full-system synthesis and lens completion. To align the model with optical principles, we employ Group Relative Policy Optimization Done Right (DrGRPO) guided by Optical Lexicographic Reward for physics-driven policy alignment. This reward system incorporates structural format rewards, physical feasibility rewards, light-manipulation accuracy, and LLM-based heuristics. Finally, our model integrates with specialized optical optimization routines for end-to-end fine-tuning and precision refinement. We benchmark our proposed method against both traditional optimization-based automated design algorithms and LLM counterparts, and experimental results show the superiority of our method.

Figures & tables

Figure 1: Despite possessing foundational optical knowledge, general-purpose LLMs ( e.g. , ChatGPT) fail to generate physically realizable systems, regardless of the prompting strategy. In contrast, OptiAgent bridges this gap by producing designs that strictly satisfy both practical and physical constraints.
Figure 2: Overview of the OptiAgent framework. Given natural language instruction, the system integrates both optical prescription completion task and whole optical system design task, a physics-driven policy with hierarchical rewards, and a function call of Zemax in a closed loop to synthesize optical prescriptions L\mathcal{L} from specifications P\mathcal{P} . At inference, the generated initial structure L0\mathcal{L}_{0} undergoes Zemax local optimization for final refinement.
Figure 3: Statistics of EFFL, FoV, and F-number in our OptiDesignQA dataset. Our dataset covers a broad spectrum of specifications for commonly used optical lenses. Distinct peaks in F-number arises from the prevalence of standard engineering aperture settings in practical optical designs.
Figure 4: Layouts of representative designs from all evaluated methods. For each optical system, the RMS spot diagrams for three distinct FoVs are displayed. The first row illustrates the initial layouts produced by each method before fine-grained optimization in Zemax, while the second row shows the corresponding results after optimization. Ray colors blue, green, and red represent wavelengths of 0.485 μ\mu m, 0.588 μ\mu m, and 0.656 μ\mu m, respectively. Even when compared against significantly larger architectures like Qwen3-235B, our method utilizing Qwen3-4B as a foundation model exhibits better performance.
ModelSR(%) ↑\uparrowEFFL Relative Error ↓\downarrowInitial RMS ↓\downarrowFinal RMS ↓\downarrow
Avg(%)StdAvg(um)StdAvg(um)Std
Qwen3-4B-Instruct56.942.10.2912405.197578.91758.30227.46
ChatGPT-5.272.037.20.229942.3514033.07249.64172.65
Qwen3-235B-22B-Instruct77.854.70.193288.302107.08127.35176.32
Claude-Sonnet-4.581.835.30.225847.896113.31155.02164.47
OptiAgent (Ours)90.11.01.06×10−31.06\times 10^{-3}672.09138.1041.2376.14
Table 1: Performance comparison with LLMs on our OptiDesignQA test set. Best results in bold .
Reward CombinationSR(%) ↑\uparrowEFFL Relative Error ↓\downarrowInitial RMS ↓\downarrowFinal RMS ↓\downarrow
Avg(%)StdAvg(um)StdAvg(um)Std
Ray Tracing95.31.93.80×10−33.80\times 10^{-3}717.496.31422.71449.3
Ray Tracing + RMS0.00//////
Ray Tracing + RMS + LLM-as-judge83.728.22.06×10−172.06\times 10^{-17}621.35.01×10−145.01\times 10^{-14}305.11.25×10−141.25\times 10^{-14}
Ray Tracing →\rightarrow LLM-as-judge42.51.28.00×10−48.00\times 10^{-4}4574.01711.8344.9222.18
Ray Tracing →\rightarrow LLM-as-judge →\rightarrow RMS93.71.62.40×10−32.40\times 10^{-3}1809.21112.6127.3123.9
Table 2: Ablation study on the design of reward function. “+” means combining and “ →\rightarrow " signifies a threshold-based activation where the next stage is triggered only after the current criterion is satisfied.
Ratio of the masked dataSR(%) ↑\uparrowEFFL Relative Error ↓\downarrowInitial RMS ↓\downarrowFinal RMS ↓\downarrow
Avg(%)StdAvg(um)StdAvg(um)Std
028.841.32.17×10−172.17\times 10^{-17}268.11.24×10−141.24\times 10^{-14}601.85.50×10−145.50\times 10^{-14}
50%50\%64.327.52.06×10−172.06\times 10^{-17}637.44.91×10−144.91\times 10^{-14}256.431.2
100%100\%90.19.80.001672.1138.141.2376.14
Table 3: Ablation study on the Optical Prescription Completion. Different ratios of masked data are tested.
Train MethodSR(%) ↑\uparrowEFFL Relative Error ↓\downarrowInitial RMS ↓\downarrowFinal RMS ↓\downarrow
Avg (%)StdAvgStdAvgStd
SFT →\to RL85.028.5%28.5\%0.0003851.70.26200.920.9
SFT36.152.8%52.8\%0.275367.24083.1123.84244.18
RL ( Ours )90.10.98%0.98\%0.001672.1138.141.2376.14
Table 4: Ablation on training paradigm. Comparison between SFT, SFT followed by RL, and direct RL (Ours). Bold indicates the best performance.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 9
More in this topic

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 601 papers

Explore similar work

CardsList

  1. LensDesigner: A Self-Improving Agent for Optical Lens Design

    Sep 24, 2026Lei Sun, Haoran Liang, Dannong Xu +6Self-Improving AgentsAutomated Heuristic Design

    INSAIT, Sofia University “St. Kliment Ohridski” · Zhejiang University

  2. Agentic AI for Scalable and Robust Optical Systems Control

    Feb 23, 2026Zehao Wang, Mingzhe Han, Wei Cheng +12LLM Agent OrchestrationAI Agent Benchmarks

    Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA · NEC Laboratories America, Princeton, NJ 08540, USA · Duke Quantum Center and Department of Physics, Duke University, Durham, NC, USA 27708 +4

  3. VortexChat: An agentic framework for autonomous multi-objective integrated photonic design

    Aug 21, 2026Faqian Chong, Yulun Wu, Shilong Li +3Inverse DesignTool-Augmented Language Model Agents

    Innovative Institute of Electromagnetic Information and Electronic Integration, College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, 310027, China. · ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University, Hangzhou 310200, China. · International Joint Innovation Center, The Electromagnetics Academy at Zhejiang University, Zhejiang University, Haining 314400, China. +3