PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing

Yiwen Song; Yale Song; Tomas Pfister; Jinsung Yoon

cs.AIApr 6, 2026

PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing

Authors: Yiwen Song, Yale Song, Tomas Pfister, Jinsung Yoon

Organizations: Google

Abstract

Synthesizing unstructured research materials into manuscripts is an essential yet under-explored challenge in AI-driven scientific discovery. Existing autonomous writers are rigidly coupled to specific experimental pipelines, and produce superficial literature reviews. We introduce PaperOrchestra, a multi-agent framework for automated AI research paper writing. It flexibly transforms unconstrained pre-writing materials into submission-ready LaTeX manuscripts, including comprehensive literature synthesis and generated visuals, such as plots and conceptual diagrams. To evaluate performance, we present PaperWritingBench, the first standardized benchmark of reverse-engineered raw materials from 200 top-tier AI conference papers, alongside a comprehensive suite of automated evaluators. In side-by-side human evaluations, PaperOrchestra significantly outperforms autonomous baselines, achieving an absolute win rate margin of 50%-68% in literature review quality, and 14%-38% in overall manuscript quality.

Figures & tables

Method	E2E L a T E X Manuscript Gen.	Decoupled Standalone Writer	Robust to Unstructured Input	Targeted Lit Review Gen.	Conceptual Diagram Gen.
PaperRobot ( Wang et al., 2019 )	✗	✗	✗	✗	✗
data-to-paper ( Technion-Kishony Lab, 2024 )	✗	✓	✓	✗	✗
AI-Researcher ( Tang et al., 2025a )	✓	✗	✗	✓	✗
Cycle Researcher ( Weng et al., 2024 )	✓	✗	✗	✗	✗
AI Scientist-v2 ( Yamada et al., 2025 )	✓	✗	✓	✓	✗
PaperOrchestra (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of PaperOrchestra against existing AI writing systems.

Figure 1: Overview of the PaperOrchestra framework. Specialized agents systematically parse unstructured inputs, synthesize plots and literature, compile a full draft, and iteratively refine the manuscript into a submission-ready PDF. (This figure was generated using PaperBanana ( Zhu et al., 2026 ) .)

Figure 2: Automated SxS evaluation of PaperOrchestra against baselines on the CVPR and ICLR 2025 datasets under the sparse idea setting. The Human (GT) baseline serves as an upper-bound reference. For both (a) literature review and (b) overall paper quality, PaperOrchestra significantly outperforms the AI baselines (Single Agent and AI Scientist-v2) across both judge models.

AI Scientist-v2 Reviewer ( Yamada et al., 2025 )
Venue	Pipeline	Originality (1-4)	Quality (1-4)	Clarity (1-4)	Significance (1-4)	Soundness (1-4)	Presentation (1-4)	Contribution (1-4)	Overall (1-10)	Acceptance Rate (%)
CVPR 2025	Original (GT)	2.96	2.81	3.44	3.06	2.74	3.26	2.91	5.95	71.00
	SingleAgent	3.01	2.27	2.78	3.03	2.26	2.79	2.74	4.60	33.00
	AI Scientist-v2	2.93	2.11	2.80	2.85	2.15	2.82	2.61	4.22	22.00
	PaperOrchestra (PlotOff)	2.95	2.43	3.40	2.97	2.42	3.22	2.74	5.12	48.00
ICLR 2025	Original (GT)	2.99	2.79	3.41	3.03	2.76	3.30	2.94	5.81	63.00

Table 2: Holistic technical quality evaluation via the AI Scientist-v2 ( Yamada et al., 2025 ) and ScholarPeer ( Goyal et al., 2026 ) frameworks (Sparse Idea Setting). Higher scores indicate better manuscript quality and higher simulated acceptance rates. Bold indicates best scores; while green highlights the highest score achieved by an AI pipeline.

Venue	Pipeline	GT Avg # Cites	Avg # Cites	Gemini-3.1-Pro			GPT5
Venue	Pipeline	GT Avg # Cites	Avg # Cites	Overall F1	P0 Recall	P1 Recall	Overall F1	P0 Recall	P1 Recall
CVPR 2025	SingleAgent	58.52	11.46	28.03	60.84	2.77	28.16	63.04	3.27
	AI Scientist-v2		14.18	17.15	36.12	3.26	17.26	37.46	3.30
	PaperOrchestra (PlotOff)		47.98	29.65	63.58	15.85	29.92	65.17	16.43
ICLR 2025	SingleAgent	59.18	9.75	23.16	48.41	3.09	23.29	50.75	3.23
	AI Scientist-v2		13.71	15.80	33.66	3.07	15.94	34.73	3.36

Table 3: Citation F1 evaluation (Sparse Idea Setting). P0 and P1 denote must-cite and good-to-cite references, respectively. F1 and Recall are percentages.

Judge Model: Gemini-3.1-Pro
Venue	Pipeline	Citation	Coverage &	Critical Analysis	Organization	Positioning	Relevance	Overall
Venue	Pipeline	Practices	Completeness	& Synthesis	& Writing	& Novelty	& Focus	Score
CVPR 2025	Human (GT)	69.88	71.55	65.80	74.31	75.55	78.25	71.08
	SingleAgent	36.83	30.10	52.93	77.03	66.45	71.40	43.04
	AI Scientist-v2	39.24	34.25	54.83	77.21	65.16	69.12	45.43
	PaperOrchestra (PlotOff)	75.36	70.15	79.01	83.74	82.68	81.15	78.30

Table 4: Literature review quality assessed by LLM-as-a-Judge (0–100 scale, Sparse Idea Setting). All numbers are percentages.

Panel of Figure 3 — Figure 3: Human & Autorater Evaluation (Sparse Idea Setting). (a) Human preferences for PaperOrchestra vs. baselines. (b) GPT-5 strongly correlates with human scores on overall quality, alongside high inter-autorater consistency between distinct LLM judges.

Venue	Metric	Gemini-3.1-Pro			GPT-5
Venue	Metric	Sparse Win	Tie	Dense Win	Sparse Win	Tie	Dense Win
CVPR 2025	Paper Quality	18%	26%	56%	24%	32%	44%
CVPR 2025	Lit Review	37%	24%	39%	40%	22%	38%
ICLR 2025	Paper Quality	24%	31%	45%	20%	37%	43%
ICLR 2025	Lit Review	33%	33%	34%	32%	40%	28%

Table 5: Automated SxS evaluation of Sparse vs. Dense idea settings.

Venue	Gemini-3.1-Pro			GPT-5
Venue	PlotOff Win	Tie	PlotOn Win	PlotOff Win	Tie	PlotOn Win
CVPR 2025	34%	32%	34%	49%	19%	32%
ICLR 2025	35%	37%	28%	36%	38%	26%

Table 6: Automated SxS Paper Quality evaluating the Plotting Agent. Despite lacking access to data represented by GT visuals, PlotOn achieves highly competitive performance against PlotOff that use human-authored figures.

Figure 4: Impact of the Content Refinement Agent. (a) In automated SxS paper quality evaluation, refined manuscripts dominates. (b) This improvement is reflected through absolute gains in acceptance rates simulated by AgentReview ( Jin et al., 2024 ) .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method	#LLM Calls	Latency (Mean)	System Description
Single Agent	1	3.3 mins	Single monolithic prompt and generation.
AI Scientist-v2	$\sim$ 40–45	35.1 mins	Multi-step pipeline (20-round citation scan, drafting, $3\times$ reflections, and vision critique).
PaperOrchestra	$\sim$ 60–70	39.6 mins	Multi-agent orchestration (outline generation, parallel literature search, sequential citation verification, writing, plotting with critique, and a $3\times$ content refinement loop).

Appendix

Table 7: Comparison of computational cost and single-paper latency. This table outlines the estimated number of LLM API calls, mean execution time, and core architectural components for each pipeline.

Panel of Figure 5 — Figure 5: CVPR 2025 Dataset Statistics

Panel of Figure 6 — Figure 6: ICLR 2025 Dataset Statistics

Metric	CVPR 2025	ICLR 2025
Visual Elements
Number of Figures	5.20 $\pm$ 1.73	9.19 $\pm$ 5.39
Number of Tables	4.20 $\pm$ 1.65	8.13 $\pm$ 5.19
Citations
Total Citations	58.52 $\pm$ 17.55	59.18 $\pm$ 20.01
P0 Citations (must-cite)	14.86 $\pm$ 6.72	13.65 $\pm$ 5.57

Appendix

Table 8: Detailed statistical breakdown of the PaperWritingBench dataset across CVPR 2025 and ICLR 2025. Values are reported as mean

\pm

standard deviation.

Panel of Figure 7 — Figure 7: Human Side-by-Side (SxS) Evaluation Results. Heatmaps display the win rate margin (%) of PaperOrchestra against baselines across various evaluation dimensions. Positive values (green) favor our approach. PaperOrchestra consistently outperforms both AI baselines (Single Agent and AI Scientist-v2), though a quality gap remains compared to the human-written ground truth (GT).

Panel of Figure 8 — Figure 8: Streamlit-based Human Evaluation Interface. The custom UI displays two manuscripts side-by-side (a) and provides a fine-grained questionnaire alongside a final preference selection (b).

Figure 9: CVPR Sample (Single Agent). Manuscript generated by the Single Agent baseline from raw materials under the sparse idea setting.

Figure 10: CVPR Sample (AI Scientist-v2). Manuscript generated by the AI Scientist-v2 baseline from raw materials under the sparse idea setting.

Figure 11: CVPR Sample ( PaperOrchestra - PlotOff). Manuscript generated by PaperOrchestra (PlotOff) from raw materials under the sparse idea setting.

Figure 12: CVPR Sample ( PaperOrchestra - PlotOn). Manuscript generated by PaperOrchestra (PlotOn) from raw materials under the sparse idea setting.

Figure 13: ICLR Sample (Single Agent). Manuscript generated by the Single Agent baseline from raw materials under the sparse idea setting.

Figure 14: ICLR Sample (AI Scientist-v2). Manuscript generated by the AI Scientist-v2 baseline from raw materials under the sparse idea setting.

Figure 15: ICLR Sample ( PaperOrchestra - PlotOff). Manuscript generated by PaperOrchestra (PlotOff) from raw materials under the sparse idea setting.

Figure 16: ICLR Sample ( PaperOrchestra - PlotOn). Manuscript generated by PaperOrchestra (PlotOn) from raw materials under the sparse idea setting.

Explore similar work

Prompt-to-Paper: Agentic AI System for Bioinformatics
Jul 5, 2026Ramsha Kamran, Maheera Amjad, Zartasha Mustansar +3Long-Form Document Generation AI-Assisted Scientific Research
School of Interdisciplinary Engineering and Sciences (SINES), National University of Sciences and Technology (NUST), Islamabad, Pakistan · College of Electrical and Mechanical Engineering (CEME), NUST, Rawalpindi, Pakistan
How Far Are We From True Auto-Research?
May 18, 2026Zhengxin Zhang, Ning Wang, Sainyam Galhotra +1LLM Agent Evaluation AI Agent Benchmarks
Cornell University
Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Papers
Apr 1, 2026Atsuyuki Miyai, Mashiro Toyooka, Zaiying Zhao +3Generative AI Evaluation Language Model Generation Evaluation
The University of Tokyo

Jul 5, 2026J/K move · Enter open · S save

Prompt-to-Paper: Agentic AI System for Bioinformatics

Ramsha Kamran, Maheera Amjad, Zartasha Mustansar, Arsalan Shaukat+2

School of Interdisciplinary Engineering and Sciences (SINES), National University of Sciences and Technology (NUST), Islamabad, Pakistan · College of Electrical and Mechanical Engineering (CEME), NUST, Rawalpindi, Pakistan

Long-Form Document Generation AI-Assisted Scientific Research Agentic RAG Automated Evaluation Autonomous Research Agents

Open paper AskPDF arXiv

While recent advances in large language models have enabled end-to-end automated manuscript generation, existing systems suffer from three critical deficiencies: (i) generated claims are not deterministically grounded in verifiable literature, (ii) experimental results are frequently fabricated rather than executed, and (iii) there exists no standardized, multi-dimensional framework to assess whether AI-generated manuscripts meet the quality and rigor required for real-world publication. We present Prompt-to-Paper, a multi-agent framework that directly addresses this evaluation gap through three integrated innovations. First, a deterministic retrieval-augmented generation pipeline with section-aware relevance scoring and snowball citation expansion grounds every claim in a verifiable corpus of 60--100 papers. Second, an autonomous coding agent executes real computational biology experiments replacing synthetic outputs with genuine numerical results. Third, an eight-dimensional automated quality scorer, benchmarked with approximate reference statistics from published papers and augmented with explicit hallucination penalties, provides standardized, reproducible quality assessments. The quality-driven improvement loop uses a context-rich reviser that routes each iteration to one of three researcher actions and fires a deep research cycle every ten iterations to re-run experiments and re-manuscript from stronger outputs. We validate the system on five bioinformatics case studies; all five cases compiled submission-formatted PDFs with zero out-of-range citations. The improvement loop raises manuscript quality by an average of +17.96 points on a 0--100 scale (maximum +26.04. As partial external checks, a human reviewer scored the five manuscripts at an average of 7.0 out of 10. Complete manuscripts are produced at approximately 0.31 USD per paper.

Abstract

Explore similar work

Prompt-to-Paper: Agentic AI System for Bioinformatics

How Far Are We From True Auto-Research?

Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Papers