Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework

Komal Kumar; Aman Chadha; Salman Khan; Fahad Shahbaz Khan; Hisham Cholakkal

cs.CLApr 7, 2026

Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework

Authors: Komal Kumar, Aman Chadha, Salman Khan, Fahad Shahbaz Khan, Hisham Cholakkal

Organizations: Mohamed bin Zayed University of Artificial Intelligence · AWS Generative AI Innovation Center, Amazon Web Services

Abstract

The rapid growth of scientific literature has made it increasingly difficult for researchers to efficiently discover, evaluate, and synthesize relevant work. Recent advances in multi-agent large language models (LLMs) have demonstrated strong potential for understanding user intent and are being trained to utilize various tools. In this paper, we introduce Paper Circle, a multi-agent research discovery and analysis system designed to reduce the effort required to find, assess, organize, and understand academic literature. The system comprises two complementary pipelines: (1) a Discovery Pipeline that integrates offline and online retrieval from multiple sources, multi-criteria scoring, diversity-aware ranking, and structured outputs; and (2) an Analysis Pipeline that transforms individual papers into structured knowledge graphs with typed nodes such as concepts, methods, experiments, and figures, enabling graph-aware question answering and coverage verification. Both pipelines are implemented within a coder LLM-based multi-agent orchestration framework and produce fully reproducible, synchronized outputs including JSON, CSV, BibTeX, Markdown, and HTML at each agent step. This paper describes the system architecture, agent roles, retrieval and scoring methods, knowledge graph schema, and evaluation interfaces that together form the Paper Circle research workflow. We benchmark Paper Circle on both paper retrieval and paper review generation, reporting hit rate, MRR, and Recall at K. Results show consistent improvements with stronger agent models. We have publicly released the website at https://papercircle.vercel.app/ and the code at https://github.com/MAXNORM8650/papercircle.

Figures & tables

Panel of Table 1 — Table 1: Comparison of Paper Circle against prior literature systems. Green indicates supported, orange indicates partial support, and red indicates unsupported.

Figure 2: The main iterative diagram for the paper discovery framework. The system maintains an explicit, evolving discovery state (papers, links, statistics, and summaries) that is iteratively updated through agentic steps. Starting from an empty draft, the orchestrator agent alternates between noising and denoising operations over multiple steps, progressively refining the draft into a final result. When necessary, a web search agent is invoked for clarification or recent information.

Figure 3: A paper analysis orchestrator agents for concepts, methods, experiments, and cross-entity linkages. The pipeline consists of four main stages: ingestion, which parses PDFs into structured elements (sections, figures, tables, equations); semantic chunking, which produces structure-aware text units; graph construction, which builds a typed knowledge graph of concepts, methods, experiments, and their relations with full traceability to source text; and a Q&A layer that enables graph-aware retrieval, verification, and export.

Figure 4: Multi-agent paper analysis and review architecture. Given a paper specified by a PDF or URL, an orchestrator agent coordinates PDF processing and maintains shared paper metadata and agent context. Specialized agents operate in parallel to perform deep technical analysis, contribution extraction, critical review, literature linking, reproducibility checking, summarization, and knowledge graph construction. External tools such as arXiv search, Semantic Scholar, and targeted text localization are invoked as needed. The orchestrator aggregates agent outputs into a unified, structured final report, enabling comprehensive, reviewer-style analysis with modular extensibility.

Conference	ICLR	NeurIPS	ICML	CVPR	IROS	ICRA	AAAI	ACL	ICCV	EMNLP	Other
Count	12	39	13	13	25	25	5	5	7	4	144

Table 2: The Database corpus across major conferences. The “Other” category includes venues such as AISTATS, RSS, SIGGRAPH, and WACV. Count indicates the number of the most recent conference venue included.

Model/Method	Type	Success	Hit Rate	MRR	R@1	R@5	R@10	R@20	R@50	Time (s)	Steps
Qwen3C-30B-Inst-Q3_K_M	Agent	100%	0.80	0.627	0.58	0.66	0.74	0.78	0.80	22.2	1.42
qwen3-coder:30b Team (2025)	Agent	100%	0.80	0.518	0.46	0.52	0.72	0.76	0.80	21.1	1.34
BM25 Chen and Wiseman (2023)	Baseline	100%	0.78	0.541	0.48	0.60	0.66	0.78	0.78	–	–
microcoder-deepseekr1-14.8	Agent	52%	0.73	0.453	0.38	0.46	0.65	0.69	0.73	107.4	4.15
deepseek-coder-v3:16b Zhu et al. (2024a)	Agent	100%	0.66	0.396	0.32	0.46	0.52	0.60	0.66	47.9	1.54
qwen2.5-coder:3b ( Hui et al., 2024 )	Agent	94%	0.60	0.366	0.28	0.45	0.53	0.55	0.57	210.4	1.51

Table 3: Combined benchmark results for agent-based models and retrieval baselines. Best results are shown in bold . All the results are calculated using semantic benchmarks. Only the last ( blue ) is evaluated on 500 RAbench queries, which shows syntetically written query is easier to retrieve compared to the random template following.

Figure 5: The main outputs of the analysis agent for a representative paper. (A) Interactive concept graph constructed from the paper, where nodes correspond to extracted concepts and edges denote semantic relationships. (B) Automatically generated concept explanations, each linked to the originating paper sections and pages. (C) Graph-aware question answering interface, providing answers grounded in extracted content along with supporting figures and references. (D) Structured Markdown exports summarizing all extracted concepts and methods for downstream use. (E) Flowchart view illustrating the high-level organization and relationships among concepts, methods, and experimental components of the paper.

Figure 6: Paper review results analysis. This study was conducted on 50 randomly selected ICLR 2024 reviews.

Metric	arXiv	Semantic Scholar	Google Scholar	PaperCircle
Sources queried per run	1	1	1	8.7 avg.
Papers not retrievable $\dagger$	70.9%	80.4%	36.9% $\ddagger$	9.0%
PDF availability	~90%	~60%	Variable	62.5%
Supported export formats	0	1–2	1	5
Bulk export support	✗	✗	✗	✓
Process-level logs	✗	✗	✗	✓

Table 4: Comparison of source coverage and export-related functionality across literature discovery systems. Percentages are computed with respect to the PaperCircle paper set.

\dagger

Fraction of PaperCircle’s 21,115 papers not retrievable from that single source alone.

\ddagger

Estimated based on the natural query.

Metric	Value	Interpretation
Sessions	81	Observed user sessions
Papers	21,115	Total papers processed
arXiv miss	70.9%	Fraction not retrievable from arXiv
Semantic Scholar miss	80.4%	Fraction not retrievable from Semantic Scholar alone
Duplicates removed	18,613 (43.5%)	Duplicate entries removed during processing
Median time	2.3 min	Median runtime per session

Table 5: Summary statistics of Paper Circle usage and outputs.

Configuration	Queries	Hit Rate	MRR	R@1	R@5	Time (s)
Default (Full Agent)	500	0.9818	0.8824	0.8381	0.9312	21.54
With Filters & Offline	50	0.9600	0.8485	0.7800	0.9000	22.76
Offline Only	50	0.9200	0.6476	0.5600	0.7400	41.45
No Mentions	50	0.6400	0.4316	0.3600	0.5200	38.35
Online/Offline Mix	50	0.6200	0.4595	0.4200	0.5000	38.50

Table 6: Extended benchmark results for the Qooba agent ( qwen3-coder-30b ) across different configurations.

Configuration	Baseline	Structure	Hit Rate	MRR	R@1	R@5	Time (s)
BM25 Full	bm25	full	0.9600	0.8629	0.8000	0.9200	33.75
BM25 Search Sort	bm25	search_sort	0.9600	0.8620	0.8000	0.9200	33.95
BM25 No Intent	bm25	no_intent	0.9600	0.8554	0.8000	0.9200	31.47
BM25 Search Analysis	bm25	search_analysis	0.9600	0.8437	0.7800	0.9200	32.81
BM25 Minimal	bm25	minimal	0.9600	0.8420	0.7800	0.9200	33.34
Hybrid Full	hybrid	full	0.9600	0.8620	0.8000	0.9200	31.65

Table 7: Ablation study results comparing retrieval baselines and pipeline structures using qwen3-coder-30b . Full represents the full pipeline structure, minimal represents

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Category	MSE	MAE	RMSE	Pearson	Spearman	Acc. $\pm 0.5$	Acc. $\pm 1.0$	Acc. $\pm 1.5$	Mean Err.	Std Err.	N
oss-120B	RATING	4.6934	1.6844	2.1664	-0.0407	0.0571	25.00%	43.75%	58.33%	0.2177	2.1555	48
oss-120B	SOUNDNESS	0.7316	0.6351	0.8554	-0.0054	0.0474	58.33%	85.42%	87.50%	-0.0816	0.8515	48
oss-120B	PRESENTATION	0.6564	0.6038	0.8102	0.0701	0.1259	60.42%	83.33%	91.67%	-0.0920	0.8049	48
oss-120B	CONTRIBUTION	0.6349	0.6240	0.7968	0.0717	0.0734	56.25%	85.42%	91.67%	0.0087	0.7967	48
oss-20	RATING	4.7607	1.7647	2.1819	0.0989	0.1869	21.43%	40.48%	52.38%	1.5980	1.4856	42
oss-20	SOUNDNESS	0.4241	0.5190	0.6512	-0.0106	-0.0226	59.52%	92.86%	97.62%	0.3294	0.5618	42

Appendix

Table 8: Paper review score prediction on ICLR 2024. We compare four LLMs on predicting human review scores across rating, soundness, presentation, and contribution. We report error metrics (MSE/MAE/RMSE), correlation (Pearson/Spearman), and thresholded accuracy (within

\pm 0.5

\pm 1.0

\pm 1.5

of the human score).

N

denotes the number of papers evaluated for each model after preprocessing.

Figure 7: Paper analysis and database management for fast inference.

AI-Assisted Scientific Research

AI-assisted scientific research uses machine learning and AI agents to support discovery, analysis, experimentation, and scientific communication.

See all 253 papers

Jul 1, 2026

Multi-Turn Agentic Scientific Literature Search via Workflow Induction

Scientific literature search often requires more than retrieving papers from a single query: users' intents are underspecified, preference-dependent, and evolve through interaction. Existing search agents typically rely on fixed pipelines or implicit language-only reasoning, making their search strategies difficult to control, inspect, and refine. We introduce PaperPilot, a multi-turn literature search agent that frames scientific search as workflow induction. Given an anchor paper and a user query, PaperPilot constructs an executable DAG of paper-search operators, including keyword search, citation expansion, filtering, scoring, reranking, and evidence extraction. User feedback is then used to refine both the query and the workflow itself. We train PaperPilot with supervised workflow imitation and preference optimization over controlled workflow corruptions. Experiments show that PaperPilot-9B improves over the base Qwen3.5-9B toolset agent under multi-turn interaction, increasing Hit@5 from 58.0 to 77.0, MRR from 47.5 to 59.4, and nDCG@10 from 26.8 to 32.5, while reducing workflow execution errors from 9.5% to 0%. These results show that explicit, editable search workflows provide an effective and controllable interface for aligning literature search agents with complex scientific intent.

Jisen Li, Bingxuan Li, Nanyi Jiang +10

University of Illinois Urbana-Champaign · Together AI · University of Pennsylvania +1

Agentic Search Agentic Workflows Information Retrieval LLM Agent Training

Ask PDF

Jun 11, 2026

Agents-K1: Towards Agent-native Knowledge Orchestration

Current LLM-based research agents have advanced through agent orchestration, yet largely overlook scientific knowledge orchestration. Existing works often reduce papers to abstracts, surface mentions, and flat cites edges, omitting key entities, claims, evidence, mechanisms, and method lineages essential for scientific reasoning. To this end, we introduce Agents-K1, an end-to-end knowledge orchestration pipeline that converts raw documents into agent-native scientific knowledge graphs. Agents-K1 integrates three components under a unifying theoretical foundation: a multimodal parser whose five-module schema captures entities, multimodal evidence, citations, and typed inter-entity relations across the full paper rather than abstracts alone; a 4B information-extraction backbone trained with GRPO under a rule-based reward; and a graphanything CLI, a tri-source agent interface that unifies web search, multimodal graph retrieval, and cross-document traversal. On top of this, we process 2.46 million scientific papers across six subjects to produce Scholar-KG, of which we release a one-million-paper subset, and the full Scholar-KG is accessible via the SCP link below. The same pipeline can be extended to general-domain corpora and to schema-conformant data synthesis. Extensive experiments demonstrate that Agents-K1 achieves superior performance in scientific information extraction, knowledge graph construction, and multi-hop scientific reasoning.

Zongsheng Cao, Bihao Zhan, Jinxin Shi +23

Shanghai Artificial Intelligence Laboratory · East China Normal University · Fudan University

KG Construction AI for Science Document Information Extraction Knowledge Representation LLM Agents

Ask PDF

Jun 18, 2026

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks offer limited coverage of research intents and insufficient support for scalable answer construction and reproducible multi-turn evaluation. We propose ScholarQuest, a taxonomy-guided benchmark constructed from over 1,000 computer science topics and organized around four research intents. To construct answer sets at scale, we combine multi-source retrieval, citation expansion, and multi-stage relevance adjudication. We further build ScholarBase, a million-scale literature environment with unified retrieval and citation-expansion interfaces, together with a shared multi-turn search harness. We benchmark complete literature search systems and compare language models within this harness, evaluating retrieval coverage, search efficiency, and robustness across research intents. Results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing search system achieves only 0.312 Recall@100 and 0.352 Recall@All. Comparisons under the shared harness reveal intent-dependent model strengths, with higher Planner token consumption not consistently yielding higher recall. We release the benchmark, retrieval backend, and evaluation harness for reproducible research

Tingyue Pan, Mingyue Cheng, Daoyu Wang +4

State Key Lab of Cognitive Intelligence, University of Science and Technology of China

Agentic Search Information Retrieval IR Evaluation Agent Benchmarks

Ask PDF

Abstract

AI-Assisted Scientific Research

Explore similar work

Multi-Turn Agentic Scientific Literature Search via Workflow Induction

Agents-K1: Towards Agent-native Knowledge Orchestration

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments