Designing Any Imaging System from Natural Language: Agent-Constrained Composition over a Finite Primitive Basis

Chengshuai Yang

cs.CVMar 26, 2026

Designing Any Imaging System from Natural Language: Agent-Constrained Composition over a Finite Primitive Basis

Authors: Chengshuai Yang

Organizations: NextGen PlatformAI C Corp, USA

Abstract

Designing a computational imaging system -- selecting operators, setting parameters, validating consistency -- requires weeks of specialist effort per modality, creating an expertise bottleneck that excludes the broader scientific community from prototyping imaging instruments. We introduce spec.md, a structured specification format, and three autonomous agents -- Plan, Judge, and Execute -- that translate a one-sentence natural-language description into a validated forward model with bounded reconstruction error. A design-to-real error theorem decomposes total reconstruction error into five independently bounded terms, each linked to a corrective action. On 6 real-data modalities spanning all 5 carrier families, the automated pipeline matches expert-library quality (98.1 +/- 4.2%). Ten novel designs -- composing primitives into chains from 3D to 5D -- demonstrate compositional reach beyond any single-modality tool.

Figures & tables

Panel of Figure 1 — Figure 1: Specification-driven design pipeline. A natural-language prompt is the sole user input. $\mathcal{A}_{P}$ generates a spec.md ; $\mathcal{A}_{J}$ validates in three sequential stages: (1) structural compilation (6 checks C1–C6), (2) Triad gate evaluation [ 3 ] ( $G_{1}$ – $G_{3}$ ), (3) cost and feasibility assessment. Two feedback paths handle failures: an automatic inner loop ( $\mathcal{A}_{P}\leftrightarrow\mathcal{A}_{J}$ , up to 3 rounds, dashed red) for structural/parametric corrections, and an outer loop (dashed blue) returning diagnostics to the user when automatic repair is exhausted. $\mathcal{A}_{E}$ executes reconstruction and outputs $\hat{A}$ with an explicit $\varepsilon$ -bound report.

Panel of Figure 2 — Figure 2: The spec.md specification format. Eight mandatory fields fully determine an imaging system design. Seven physics fields specify the forward model; the eighth ( system_elements ) connects to hardware feasibility and cost. Three examples span X-ray (CT), optical (CASSI), and spin (MRI) carriers. Each forward_model field is a primitive chain over the 11 FPB operators.

Modality	Chain	Carrier	PSNR (dB)	$\rho_{\text{time}}$	$\rho_{\text{recov}}$	Data
X-ray
CT	$\Pi\to D$	X-ray	$24.8\pm 2.4$	420	$\sim$ 1.0	Real
Optical photons
CASSI	$M\to W\to\Sigma\to D$	Photon	$24.3\pm 1.5$	310	0.85	Real
CACTI	$M\to\Sigma\to D$	Photon	$30.2\pm 1.1$	300	$\sim$ 1.0	Real
OCT	$P+P\to\Sigma\to D$	Photon	$30.5\pm 1.1$	290	0.86	Synth.

Table 1: Cross-modality validation (22 of 39 modalities), grouped by carrier family.

\rho_{\text{recov}}

: 4-scenario recovery ratio [ 3 ] . 95% CIs via bootstrap (

n{=}1000

). Bold = real measured data (challenging conditions: 60-view CT, 4-coil MRI);

\dagger

= self-reference metric (no ground truth);

*

= new system designs validated in the algorithm-comparison study (Extended Data Table 7 ).

\Phi_{z}

denotes depth-parameterised convolution:

C

applied with a depth-dependent PSF

\Phi(z)

; it is the convolution primitive

C

, not a 12th element of the FPB.

Panel of Figure 3 — Figure 3: Error decomposition by modality. Stacked bars show the relative contribution of each Theorem 1 error term for 6 representative modalities. For well-conditioned systems (CT, MRI, e-ptychography), parameter mismatch $\varepsilon_{\text{param}}$ dominates and is reducible by calibration. For scattering-dominated systems (DOT), unmodeled physics $\varepsilon_{\text{unmod}}$ dominates and requires tier-lifting. $\varepsilon_{\text{FPB}}<1\%$ in all cases. $\varepsilon_{\text{spec}}=\varepsilon_{\text{trans}}=0$ for 31/36 modalities; CASSI shows nonzero $\varepsilon_{\text{trans}}$ due to dispersion–accumulation chain ambiguity.

Figure 4: Design-to-real validation. (a) A natural-language prompt is translated into a structured spec.md by the Plan Agent. (b) The Judge Agent validates all gates and checks (3 Triad gates + 6 compiler checks), producing a certificate. (c) Reconstruction results on three real-data modalities with ground truth: CT ( $24.8$ dB, FISTA-TV on LoDoPaB), MRI ( $31.7$ dB, HybridCascade++ on M4Raw), and CASSI ( $24.3$ dB, GAP-TV on KAIST TSA [ 6 ] ). (d) Quality ratio (agent/expert PSNR) across 6 real-data modalities spanning all 5 carrier families: mean $98.1\pm 4.2$ %; theorem tightness ratio $\tau\in[1.8,5.2]$ , median 2.9.

Configuration	Valid	$\leq\varepsilon$	Redesign	Time	Dominant failure
Full ( $\mathcal{A}_{P}{+}\mathcal{A}_{J}{+}\mathcal{A}_{E}$ )	12/12	12/12	1.7 $\pm$ 0.6	4.8 min	—
No $\mathcal{A}_{J}$	8.2/12	6.4/12	0	2.9 min	Invalid spec, unbounded $\varepsilon$
No $\mathcal{A}_{E}$	12/12	9.6/12	1.7 $\pm$ 0.6	3.4 min	Suboptimal parameters
No $\mathcal{A}_{P}$ (manual)	10.4/12	10.4/12	0	52 min	Human error (14%)
Template only	—	3/12	0	0.5 min	No coverage (75%)

Table 2: Ablation study (5 trials

\times

12 modalities = 300 runs). “Valid”: passes all compiler checks. “

\leq\varepsilon

”: reconstruction within Theorem 1 error bound. No single agent is dispensable.

Established modalities ( $98.1\pm 4.2$ %)
	PSNR (dB)		Setup
Modality	Agent	Expert	Agent	Expert	Quality	LoC
CT (real)	24.8	24.0	25 min	7 d	103.3%	12 vs 340
MRI (real)	31.7	32.4	20 min	5 d	97.8%	14 vs 520
CASSI (real)	24.3	26.1	18 min	4 d	93.1%	11 vs 280
OCT	30.5	31.2	22 min	6 d	97.8%	13 vs 450

Table 3: Agent vs. established reconstruction methods. “Agent”: best-performing automated algorithm; “Expert”: established variational method (FISTA+TV [ 38 ] or GAP-TV). Bold = real measured data;

*

= new system designs. For novel systems (

*

), no system-specific expert baseline exists; “Expert” denotes the best generic variational method. Quality ratios

>

100% for novel systems reflect the advantage of system-specific algorithm selection over generic methods, not comparison with domain experts. Established modalities:

98.1\pm 4.2

%; novel systems:

114\pm 10

% (vs. generic baselines). Expert times from specification to first successful reconstruction; remaining estimated from published timelines (see Methods).

Panel of Figure 5 — Figure 5: Validation pyramid. Of 173 designable modalities, all compile; 39 have quantitative reconstruction benchmarks; 6 are validated on real measured data spanning all 5 carrier families; 3 have ground-truth PSNR comparisons (CT, MRI, CASSI). Each tier is a strict subset.

Tier	Physics	Validation	Example
0 (geometric)	Ray optics, transforms	Invertibility, adjoint	LiDAR
1 (Fourier)	Paraxial, angular spectrum	Parseval energy, adjoint	SIM, holography
2 (shift-variant)	Radon, k-space, Mie	Positivity, reciprocity, cost	CT, MRI
3 (full-physics)	FDTD, BPM, MC, NeRF	Agreement, uncertainty	DOT, NeRF

Table 4: Extended Data Table 1: Four-tier fidelity hierarchy. All 11 primitives are implemented at every tier.

Domain	Low tier	High tier	Correction	$\boldsymbol{\varepsilon_{\text{unmod}}}$	$\boldsymbol{\varepsilon_{\text{lift}}}$
DOT	Tier-2 (P 1 )	Tier-3 (P 3 RT)	Higher-order moments	7.8%	$<10^{-12}$
Coherent	Tier-1 (ang. spec.)	Tier-3 (FDTD)	Evanescent + scatter	147%	$<10^{-12}$

Table 5: Extended Data Table 2: Tier-lifting validation. Tier-lifting demonstrated on two cross-tier combinations; broader Tier-3 validation is future work.

Modality	Domain	Carrier	Tier	Chain	Status	Real?	Dom. $\varepsilon$
X-ray
CT	Medical	X-ray	1–2	$\Pi\to D$	FV	Yes	$\varepsilon_{\text{param}}$
CBCT	Medical	X-ray	2	$\Pi\to\Lambda\to D$	T	No	$\varepsilon_{\text{param}}$
CT (polychromatic)	Medical	X-ray	2	$\Pi\to\Lambda\to D$	T	No	$\varepsilon_{\text{unmod}}$
Phase contrast	Medical	X-ray	1–2	$\Pi\to P\to M\to D$	HO	No	$\varepsilon_{\text{param}}$
DEXA	Medical	X-ray	2	$M\to\Pi\to D$	T	No	$\varepsilon_{\text{param}}$

Table 6: Extended Data Table 3: Modality registry (39 validated canonical decompositions). Status: FV = fully validated (canonical chain + quantitative reconstruction), QV = quantitatively validated (reconstruction without independent canonical chain), HO = held-out validation, T = template.

\Phi_{z}

: depth-parameterised

C

(see Table 1 ). An additional

{\sim}

134 modalities pass all compiler checks but lack per-modality reconstruction benchmarks; the full list is in the repository.

Figure 6: Extended Data Figure 2: Algorithm sensitivity by modality. (a) Inter-algorithm PSNR Coefficient of Variation (CoV) for the top-5 published reconstruction algorithms per modality. Well-conditioned systems (blue, CoV

\approx

3.5–6.2%) show tight convergence among the top algorithms. SIM shows moderate CoV (15.0%). Ill-conditioned compressive systems (red, CoV

>

40%) show a wider spread, indicating greater sensitivity to algorithm selection. (b) Individual algorithm PSNR values (dots) with mean (horizontal bar) for each modality. Data from Extended Data Table 7 .

		CT		MRI
Method	Algorithm	PSNR	Type	Algorithm	PSNR
M1	iRadonMAP	36.9	Deep Learning	HUMUS-Net	37.3
M2	FBPConvNet	38.5	Deep Learning	PromptMR+	39.9
M3	DuDoTrans	42.1	Transformer	ReconFormer	40.1
M4	Score-CT	43.0	Diffusion	E2E-VarNet	40.5
M5	LEARN	43.1	Deep Unrolling	PromptMR	41.5

Table 7: Extended Data Table 4: Algorithm-comparison study (5 published algorithms

\times

5 modalities). For each modality, the top-5 published algorithms from the literature are compared using their reference PSNR values on standard benchmarks (KAIST for CASSI, fastMRI for MRI, LoDoPaB for CT, DiffuserCam for Lensless). Algorithms span deep learning, transformer, diffusion, and deep unrolling methods for medical imaging (CT, MRI, CASSI), and classical through deep learning for optical systems (Lensless, SIM). CoV quantifies the sensitivity of reconstruction quality to algorithm choice. PSNR values in dB.

Panel of Figure 7 — Figure 7: Extended Data Figure 1: Execute Agent calibration. Predicted vs. actual PSNR for 39 validated modalities. Shaded band: $\pm 1.8$ dB (overall MAE). Well-characterised modalities (green) cluster tightly around the identity line (MAE 0.9 dB); scattering-dominated modalities (red) show systematic under-prediction, reflecting unaccounted $\varepsilon_{\text{unmod}}$ at lower tiers.

Encoder	3D only	+Mask	+M+ $W_{\lambda}$	+M+ $W_{t}$	+M+ $W_{\lambda}$ + $W_{t}$
	(8:1)	(8:1)	(32:1)	(32:1)	(128:1)
Diffuser	17.0	20.3	24.6	26.9	38.1
Multi-lens	16.6	44.1	41.0	36.1	38.7
Meta-surface	16.6	16.2	24.6	25.8	38.0

Table S8: Depth encoder comparison: best PSNR (dB) across 3 encoders

\times

5 modulation combinations.

N_{z}{=}8

depth planes,

N_{\lambda}{=}4

spectral bands,

N_{t}{=}4

temporal frames, image size

128{\times}128

Explore similar work

Mar 13, 2026

Eleven Primitives and Three Gates: The Universal Structure of Computational Imaging

Computational imaging systems -- from coded-aperture cameras to cryo-electron microscopes -- span five carrier families yet share a hidden structural simplicity. We prove that every imaging forward model decomposes into a directed acyclic graph over exactly 11 physically typed primitives (Finite Primitive Basis Theorem) -- a sufficient and minimal basis that provides a compositional language for designing any imaging modality. We further prove that every reconstruction failure has exactly three independent root causes: information deficiency, carrier noise, and operator mismatch (Triad Decomposition). The three gates map to the system lifecycle: Gates 1 and 2 guide design (sampling geometry, carrier selection); Gate 3 governs deployment-stage calibration and drift correction. Validation across 12 modalities and all five carrier families confirms both results, with +0.8 to +13.9 dB recovery on deployed instruments. Together, the 11 primitives and 3 gates establish the first universal grammar for designing, diagnosing, and correcting computational imaging systems.

Chengshuai Yang, Xin Yuan

NextGen PlatformAI C Corp, USA. · School of Engineering, Westlake University, Hangzhou, China.

Computational Imaging Image Reconstruction Image Inverse Problems

Ask PDF

Jul 12, 2026

Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging

Computational imaging, which recovers hidden signals from indirect, noisy measurements, underpins quantitative discovery across scientific disciplines, yet building a correct reconstruction pipeline demands deep domain expertise and remains laborious even for domain scientists. We introduce Imaging-101, a benchmark of 57 expert-verified computational imaging tasks spanning six scientific domains, each grounded in a peer-reviewed paper and canonicalized into a standardized four-stage pipeline (preprocessing, forward physics modeling, inverse solver, and visualization) Three evaluation tracks (planning, function-level unit tests, and end-to-end reconstruction) probe distinct agent capabilities across the full pipeline. Evaluating seven frontier LLMs uncovers systematic challenges in applying coding agents to computational imaging that go beyond those exposed by general coding benchmarks, spanning algorithm selection, physical convention handling, and pipeline integration. These findings highlight concrete capability gaps and point toward skill-augmented, domain-specialized agents as a practical path to reliable computational imaging assistance.

Siyi Chen, Jiahe Ying, Yixuan Jia +9

Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technologies, Ministry of Education, Beijing, 102206, China

AI Agents for Scientific Discovery Computational Imaging LLM Agents Image Inverse Problems

Ask PDF

Jul 8, 2026

Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks

Vision-language models (VLMs) and agentic AI have shown strong performance on semantic visual tasks, but it remains unclear whether they can handle the physics and inverse problems that underlie computational imaging. We present ImagingBench, a benchmark of 20 computational imaging tasks spanning five categories: ray and wave optics, image signal processing, inverse reconstruction, computational sensing, and calibration. ImagingBench evaluates three complementary settings: Expert, fixed expert-guided inverse reconstruction; Planner, planner-guided inverse reconstruction; and Forward, forward-system simulation for consistency checking. We benchmark leading proprietary and open-source image-centric multimodal systems, including Gemini, GPT, and Qwen, and compare them with representative task-specific non-agentic baselines. Across tasks, agentic models remain consistently weaker than specialized methods, especially on computational sensing problems such as lensless imaging, event-based reconstruction, time-of-flight imaging, and holography. Planner guidance provides only modest and inconsistent gains over the fixed-prompt Expert baseline. Although the models often generate visually plausible outputs, their reference-based fidelity remains poor, revealing a substantial gap between semantic visual competence and physically grounded imaging performance. ImagingBench provides a unified testbed for measuring this gap and tracking progress in agentic AI for computational imaging.

Ethan Chung, Chuanjun Zheng, Jasper Tan +3

University of Hawaii at Manoa · Glass Imaging

VLM Evaluation AI Agent Benchmarks Computational Imaging Inverse Problems

Ask PDF