LensDesigner: A Self-Improving Agent for Optical Lens Design

Lei Sun; Haoran Liang; Dannong Xu; Yao Gao; Yuyu Geng; Jinjin Gu; Kaiwei Wang; Danda Pani Paudel

cs.CVSep 24, 2026

LensDesigner: A Self-Improving Agent for Optical Lens Design

Authors: Lei Sun, Haoran Liang, Dannong Xu, Yao Gao, Yuyu Geng, Jinjin Gu, Kaiwei Wang, Danda Pani Paudel,

Organizations: INSAIT, Sofia University “St. Kliment Ohridski” · Zhejiang University

Abstract

Optical lens design is a complex, non-convex optimization challenge that relies heavily on human experience and intuition. Existing optimized-based automatic lens design methods struggle to navigate this vast parameter space without meticulous manual tuning. In this paper, we present LensDesigner, an autonomous agent framework that mirrors the problem-solving workflow of expert opticians. To overcome the initial cold start problem, we construct LensLib100K, an extensive optical lens library, and employ Optics-Aware Retrieval to supply physically valid structural seeds. Within an interactive physical simulation environment, the agent executes macroscopic orchestration while receiving immediate optical feedback. Furthermore, we introduce a continuous self-evolving mechanism guided by a curriculum agent. By iteratively solving design tasks with progressively increasing difficulty, the agent autonomously extracts, accumulates, and reuses design heuristics, effectively evolving its optical lens design expertise over time. At the evaluation level, we introduce LensArena, a standardized evaluation benchmark comprising $120$ diverse optical design tasks, covering extreme configurations. Extensive experiments on this benchmark demonstrate that LensDesigner significantly outperforms publicly available baseline algorithms, achieving superior success rates and optimization efficiency. We hope this work sheds light on the emerging field of intelligent optics. The code will be publicly available.

Figures & tables

Figure 1: Manual lens design is a highly experience-dependent task that typically requires years of accumulated human expertise. Similarly, LensDesigner mimics this trajectory through a self-evolving mechanism, continuously accumulating skills to advance its optical design capabilities.

Figure 2: Distribution of LensLib100K and LensArena. LensArena spans more challenging edge cases.

Figure 3: Main pipeline of LensDesigner . Given a design query, the framework retrieves the closest seeds from LensLib100K, followed by Macro Orchestration for systematic optimization and Optics-Aware Self-Evolving for continuous capability enhancement via a curriculum-based agent.

	Success Rate (%) $\uparrow$			Avg RMS ( $\mu m$ ) $\downarrow$
Method	Easy	Medium	Hard	Easy	Medium	Hard
[Existing Baselines]
LensNet Côté et al. (2021)	50.0	39.3	30.0	3.89	8.99	7.99
QGSO Gao et al. (2025a)	25.0	22.7	13.3	24.2	17.7	11.1
DeepLens Yang et al. (2024b)	4.2	1.5	0	29.6	41.1	-
[In Context Learning Baselines]

Table 1: Main results evaluated on the LensArena Benchmark.

Figure 4: Results on the LensArena benchmark. The evaluation demonstrates that our proposed method achieves superior structural rationality, a higher success rate, and a significantly lower RMS spot radius compared to the baselines.

Figure 5: Visualized results of lens layouts and spot diagrams from the LensArena benchmark. LensDesigner consistently yields optimal structural configurations and achieves the minimal RMS spot size, outperforming other methods in both design rationality and imaging quality.

Figure 7: Ablation study on the core components of LensDesigner . The violin plots illustrate the distribution of RMS spot radii across different model variants on the LensArena benchmark.

	Success Rate (%) $\uparrow$			Avg RMS ( $\mu m$ ) $\downarrow$
Method	Easy	Medium	Hard	Easy	Medium	Hard
w/o LensLib100K	25.0	39.3	26.7	89.6	194.4	293.4
w/ LensLib100K + Vanilla RAG	58.3	51.5	40.0	35.1	128.1	189.6
w/o Optical Tool Suite	66.7	63.6	26.7	37.1	54.7	68.9
w/o Macro Orchestration	80.0	69.7	33.3	21.8	34.2	87.9
[Self-Envolving]

Table 2: Ablation study on the core components of LensDesigner using the LensArena Benchmark. All variants are evaluated on the identical foundation model for fair comparison.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: A simple example of a lens system.

Guardrail Type	Trigger Condition and Action
Stop Surface Recognition	Rejects semi-diameter edits on non-stop surfaces.
Aperture Scaling Limits	Rejects semi-diameter scaling outside the $[0.5,3.0]$ ratio.
Dynamic F-number Sync	Synchronizes system F-number automatically after aperture edits.
Material Whitelist	Rejects non-catalog glasses (restricted to verified H- and D- types).
Loop Prevention	Rejects identical edits repeated $\geq 3$ times, prompting a strategy shift.

Appendix

Table 3: Physical guardrails implemented within the modify_lens tool.

Operand	Target	Weight	Physical Justification
RSCE (Per Field)	0	4 / 5 / 6	Escalating weights for challenging marginal fields.
EFFL / WFNO	User Spec	1.0 / 2.0	Weak anchoring to prevent macroscopic drift.
OPLT (Max TOTR)	Max Limit	2.0	Unilateral penalty applied only when exceeding limits.
MNCG	$2.0\,mm$	50.0	Hard constraint for minimum manufacturable glass thickness.

Appendix

Table 4: Custom Merit Function configuration for Zemax DLS optimization.

Figure 9: Additional visualized results of lens layouts and spot diagrams from the LensArena benchmark. LensDesigner demonstrates robust performance across a diverse range of design specifications, consistently producing optimized optical structures with superior imaging convergence compared to baseline methods.

Self-Improving Agents

Self-improving agents use experience and feedback to update their models, memories, skills, or workflows for better future task performance.

See all 167 papers

OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design
Feb 27, 2026Yuyu Geng, Lei Sun, Yao Gao +6Large Language Model-Guided Optimization
Zhejiang University · INSAIT, Sofia University · Xiaohongshu
A Self-Evolving Agentic Framework for Metasurface Inverse Design
Apr 1, 2026Yi Huang, Bowen Zheng, Yunxi Dong +4Inverse Design Self-Evolving Agents
Department of Electrical and Computer Engineering, University of Massachusetts Lowell
Online Automated Algorithm Design with Large Language Models
Sep 21, 2026Zhiyao Zhang, Yichen Li, Xingyu Wu +2Automated Algorithm Discovery Black-Box Optimization
Department of Data Science and Artificial Intelligence The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China

Feb 27, 2026J/K move · Enter open · S save

OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design

Yuyu Geng, Lei Sun, Yao Gao, Xinxin Hu+5

Zhejiang University · INSAIT, Sofia University · Xiaohongshu

Large Language Model-Guided Optimization

Open paper AskPDF arXiv

Optical design is the process of configuring optical elements to precisely manipulate light for high-fidelity imaging. It is inherently a highly non-convex optimization problem that relies heavily on human heuristic expertise and domain-specific knowledge. While Large Language Models (LLMs) possess extensive optical knowledge, their capabilities in leveraging the knowledge in designing lens system remain significantly constrained. This work represents the first attempt to employ LLMs in the field of optical design. We bridge the expertise gap by enabling users without formal optical training to successfully develop functional lens systems. Concretely, we curate a comprehensive dataset, named OptiDesignQA, which encompasses both classical lens systems sourced from standard optical textbooks and novel configurations generated by automated design algorithms for training and evaluation. Furthermore, we inject domain-specific optical expertise into the LLM through a hybrid objective of full-system synthesis and lens completion. To align the model with optical principles, we employ Group Relative Policy Optimization Done Right (DrGRPO) guided by Optical Lexicographic Reward for physics-driven policy alignment. This reward system incorporates structural format rewards, physical feasibility rewards, light-manipulation accuracy, and LLM-based heuristics. Finally, our model integrates with specialized optical optimization routines for end-to-end fine-tuning and precision refinement. We benchmark our proposed method against both traditional optimization-based automated design algorithms and LLM counterparts, and experimental results show the superiority of our method.

Abstract

Self-Improving Agents

Explore similar work

OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design

A Self-Evolving Agentic Framework for Metasurface Inverse Design

Online Automated Algorithm Design with Large Language Models