Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designs

Zixin Wen; Yifu Cai; Kyle Lee; Sam Estep; Josh Sunshine; Aarti Singh; Yuejie Chi; Wode Ni

cs.LGMar 13, 2026

Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designs

Authors: Zixin Wen, Yifu Cai, Kyle Lee, Sam Estep, Josh Sunshine, Aarti Singh, Yuejie Chi, Wode Ni

Organizations: Carnegie Mellon University Pittsburgh, PA 15213, USA

Abstract

Visual design is an essential application of state-of-the-art multi-modal AI systems. Improving these systems requires high-quality vision-language data at scale. Despite the abundance of internet image and text data, knowledge-rich and well-aligned image-text pairs are rare. In this paper, we present a scalable diagram generation pipeline built with our agent, Feynman. To create diagrams, Feynman first enumerates domain-specific knowledge components (''ideas'') and performs code planning based on the ideas. Given the plan, Feynman translates ideas into simple declarative programs and iterates to receives feedback and visually refine diagrams. Finally, the declarative programs are rendered by the Penrose diagramming system. The optimization-based rendering of Penrose preserves the visual semantics while injecting fresh randomness into the layout, thereby producing diagrams with visual consistency and diversity. As a result, Feynman can author diagrams along with grounded captions with very little cost and time. Using Feynman, we synthesized a dataset with more than 100k well-aligned diagram-caption pairs. We also curate a visual-language benchmark, Diagramma, from freshly generated data. Diagramma can be used for evaluating the visual reasoning capabilities of vision-language models. We plan to release the dataset, benchmark, and the full agent pipeline as an open-source project.

Figures & tables

Figure 3: Iterate Step : At each step, Feynman attempts to write Penrose program to create a diagram. The generated program is then compiled into images and sent to a panel of visual judges (MLLMs) for critical feedback. We term this algorithm Iterative Visual-Refine ( Algorithm 1 ).

Figure 4: Examples of conceptual diagrams and their Substance notations : a graph where node connections form a cube (left) and the Lewis structure of the formaldehyde molecule ( $\mathrm{CH_{2}O}$ ).

Figure 5: Feynman generates programs that Penrose compiles to generate an layout optimization problem. The Penrose layout engine then solves the optimization problem.

Figure 6: Diverse visual layouts of Penrose diagram variations: using the same Substance , Penrose can produce diagram variations while preserving the semantics, by sampling random initial values for shapes, colors, and other numerical quantities in the diagram. We show 4 random seed for 4 Substance programs for (A) ray-tracing diagrams, (B) Cayley graphs, (C) Chaos game as a Sierpinski triangle, and (D) Euler diagrams for sets.

Figure 7: Comparison of different diagramming approaches : We select the best image out of three attempts using Flux-Pro and AutomaTikZ , compared with those generated by our agent. We provide them with the same prompt for each column. Flux-Pro produces visually diverse diagrams, however none of them contains legible text that matches the intent. AutomaTikZ, on the other hand, only successfully compiled for 3 out of 5 of all prompts.

Source metadata
Subdomain	108
Substance	1058
Subjects
Math	401 (37.9%)
CS	342 (32.3%)
Science	241 (22.8%)

Table 1: Metadata of Diagramma .

Model Name	All	Math	CS	Science	Chart	Commonsense	Statistics
Claude-3.5-Sonnet	59.64	64.59	42.98	74.69	53.33	77.27	54.55
GPT-4o ( OpenAI (2024a) )	57.28	63.09	50.58	60.17	53.33	50.00	36.36
Claude3-Opus ( Anthropic (2024) )	49.15	54.11	40.35	55.60	33.33	45.45	50.00
Gemini-1.5-Flash ( Reid et al. (2024) )	47.54	55.36	40.06	45.64	50.00	54.55	31.82
Claude3-Sonnet ( Anthropic (2024) )	47.54	50.62	38.01	58.92	46.67	36.36	27.27
GPT-4o-mini ( OpenAI (2024a) )	44.42	47.63	36.55	53.53	26.67	45.45	31.82

Table 2: Accuracy results of Diagramma on state-of-the-art MLLMs on 1058 samples. Models marked with (*) are evaluated through the OpenRouter API. Claude-3.5-Sonnet achieved the highest overall accuracy, with notable performance on science and commonsense diagrams.

Figure 8: Scaling behavior of input/output tokens versus generated image samples across four subdomains. The figure illustrates two distinct trends: linear scaling in knowledge-dense domains, and decaying trends in knowledge-sparse domains. These trends highlight the impact of domain knowledge diversity and scalability on the performance of Feynman .

Ablations	Avg	PR	LT	SP	NR	CR	Compile %	Yield %	Rounds
KP + CP + S	65.4	69.7	30.9	60.4	91.8	74.2	82.5	82.5	2.63
CP + S	65.0	65.9	32.1	59.9	95.0	72.2	97.0	87.5	2.44
KP + S	62.8	63.9	35.6	56.8	91.3	66.2	87.0	86.5	6.69
S	63.2	64.3	31.5	57.6	90.6	71.8	96.5	84.0	6.29
KP + CP	61.5	67.9	25.9	52.1	89.7	71.9	81.5	81.5	8.00
CP	61.8	69.3	23.6	52.5	92.3	71.1	96.5	92.5	8.00

Table 3: The table shows a break down of critic scores in five categories (see Section D.1 , PR : proper element relationship; LT : legible text; NR : non-redundancy; CR : correct representation; SP : simplicity), diagram compilation rate ( Compile % ), final yield rate ( Yield % ), and total number of rounds in the iterate step ( Rounds ). The combination of explicit knowledge planning ( KP ), code planning ( CP ), and early stop based on scoring results ( S ) received the best judge critic score.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Setup	Score
Image	Same Substance	0.9595
	Varied Substance	0.8710
	Varied Domain	0.6227
Code	Same Domain	0.0763
Code	Varied Domain	0.0304

Appendix

Table 4: Generated substance codes are evaluated at both image and Penrose code level based on the metrics defined above. For images, the lower the score ( CLIP score ), the more diverse they are. For codes, the lower the score ( CrystalBLEU ), the more diverse they are. We provide details for both metrics in Section D.2

Figure 9: Examples of Feynman -generated conceptual diagrams (Part 1 of 2).

Figure 10: Examples of Feynman -generated conceptual diagrams (Part 2 of 2).

Figure 11: Ti k Z generation using GPT-4o-mini and o1-preview

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 623 papers

From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
Sep 1, 2026Raul Ortega, José Manuel Gómez-PérezVisual Question Answering Scientific QA
Language Technology Research Laboratory Expert.ai 17 Henri Dunant, 28036 Madrid, Spain
DiagramNet: An End-to-End Recognition Framework and Dataset for Non-Standard System-Level Diagrams
May 2, 2026Jincheng Lou, Ruohan Xu, Jiapeng Li +6Multi-Agent LLM Systems Electronic Design Automation
School of IC, Peking University, Beijing, China · College of Artificial Intelligence, Xi’an Jiaotong University, Xi’an, China · Department of Precision Instruments, Tsinghua University, Beijing, China +4
OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward
Apr 7, 2026Haoyue Yang, Xuanle Zhao, Xuexin Liu +2Code Generation RL for Code Generation
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhejiang University

Sep 1, 2026J/K move · Enter open · S save

From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding

Raul Ortega, José Manuel Gómez-Pérez

Language Technology Research Laboratory Expert.ai 17 Henri Dunant, 28036 Madrid, Spain

Visual Question Answering Scientific QA Synthetic Data Generation for Language Models Visual Instruction Tuning Multimodal Instruction Tuning Scientific Figure Understanding

Open paper AskPDF arXiv

Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images. However, they continue to struggle with scientific diagrams, which are designed to convey functional or relational meaning rather than literal scenes. We therefore introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula. Our approach systematically extracts domain concepts, synthesizes atomic facts, retrieves relevant diagrams from the web, and generates multimodal supervision in the form of diagram captions and multiple-choice questions. Using this pipeline, we construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences. Despite relying on noisy web data and synthetic annotations, models fine-tuned on SciGram achieve substantial improvements on diagram-centric benchmarks, including TQA, ScienceQA, and AI2D, outperforming or matching state-of-the-art VLMs while using fewer training instances. Furthermore, augmenting existing models such as LLaVA OneVision with SciGram establishes new state-of-the-art performance on diagram question answering. Our results highlight the effectiveness of terminology-grounded instruction generation as a general strategy for improving vision-language reasoning in scientific domains. To support future research in scientific diagram understanding, we release both the SciGram dataset and models.

Abstract

Synthetic Data Generation

Explore similar work

From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding

DiagramNet: An End-to-End Recognition Framework and Dataset for Non-Standard System-Level Diagrams

OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward