Paper ID: 2409.19242 • Published Sep 28, 2024
SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement
Ishani Mondal, Zongxia Li, Yufang Hou, Anandhavelu Natarajan, Aparna Garimella, Jordan Boyd-Graber
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
Automating the creation of scientific diagrams from academic papers can
significantly streamline the development of tutorials, presentations, and
posters, thereby saving time and accelerating the process. Current
text-to-image models struggle with generating accurate and visually appealing
diagrams from long-context inputs. We propose SciDoc2Diagram, a task that
extracts relevant information from scientific papers and generates diagrams,
along with a benchmarking dataset, SciDoc2DiagramBench. We develop a multi-step
pipeline SciDoc2Diagrammer that generates diagrams based on user intentions
using intermediate code generation. We observed that initial diagram drafts
were often incomplete or unfaithful to the source, leading us to develop
SciDoc2Diagrammer-Multi-Aspect-Feedback (MAF), a refinement strategy that
significantly enhances factual correctness and visual appeal and outperforms
existing models on both automatic and human judgement.