Large language models (LLMs) are increasingly used to support research idea generation and scientific discovery. Multi-agent debate offers a collaborative approach to this task by bringing together diverse perspectives to develop research ideas. However, a key challenge is how to equip agents with distinct and meaningful expertise to guide their contributions. Inspired by how researchers draw on accumulated research experience during collaboration, we introduce InsightAgents, a framework that constructs literature-grounded agent expertise by distilling insights from graphs of related papers. Guided by insights that trace evolving research approaches, reveal underlying assumptions, and identify unresolved challenges, within a line of research, agents jointly identify a research problem and propose, challenge, and refine potential approaches through debate to develop a research proposal. We evaluate InsightAgents against three research idea generation frameworks on 100 NLP research topics through pairwise comparisons by two LLM judges. In both online and offline knowledge settings, InsightAgents achieves the highest average score across the five evaluation dimensions under both judges.
Figures & tables
Figure 1: Overview of InsightAgents . Insights distilled from graph-structured literature define agent expertise, with each insight linked to its supporting evidence. The agents use these perspectives to develop a research proposal through multi-agent debate.
Availability & input
Method features
Framework
Open source
Topic control
Lit. retrieval
Lit. graph
Insight extraction
Multi-agent debate
Agent expertise source
SciMON ( 2024 )
✓
∼
✓
✓
—
—
—
ResearchAgent ( 2025 )
✓
∼
✓
✓
—
—
—
AI Scientist-v2 ( 2025 )
✓
✓
✓
—
—
—
—
Chain of Ideas ( 2025 )
✓
✓
✓
✓
✓
—
—
SciAgents ( 2025 )
✓
∼
✓
✓
—
✓
Functional roles
Table 1: Comparison with research idea generation frameworks. Shaded framework names indicate the three baselines used in our experiments. ✓ : documented support; —: not documented or no official code located. Open source denotes public official code. For topic control, ✓ denotes direct topic-to-idea input; ∼ denotes indirect or constrained control. Literature graphs include chains and citation or knowledge graphs. Insight extraction denotes intermediate trends, patterns, gaps, or transferable mechanisms. Debate requires agents to exchange and respond to arguments, beyond independent reviews. Agent expertise source describes what defines the expertise of debating agents; — marks frameworks without debate.
Aspect
Online
Offline
Source
Semantic Scholar API
Contribution graph
Construction
On demand
From prebuilt graph
Nodes
Papers
Papers
Edges
Recommendations
Prerequisite relations
Direction
Older → newer
Prerequisite → dependent
Table 2: Comparison of the two knowledge settings used by InsightAgents . In the offline setting, contribution nodes are mapped to their source papers.
Figure 2: Elo profiles on 100 topics per setting: (a–b) online and (c–d) offline, each evaluated by GPT-5.6 Sol and Claude Opus 4.8. Ours denotes InsightAgents ; Avg., Signif., Feasib., and Eff. denote average Elo, significance, feasibility, and expected effectiveness. All panels use the same radial scale (250–1,650; higher is better).
Figure 3: Human Elo ratings on 10 online topics, derived from rankings across all four methods. Ours denotes InsightAgents . Axes follow Figure 2 ; the radial scale is 800–1,250.
Figure 4: Agreement on InsightAgents –baseline pairs over 150 topic–dimension–pair instances from 10 online topics. Each bar splits one comparison into exact agreement (same winner or both ties), tie vs. preference (one judge returns a tie and the other does not), and opposite preference. Non-contradiction is the sum of the two blue segments. Sol and Opus denote GPT-5.6 Sol and Claude Opus 4.8.
Variant
Nov.
Sig.
Feas.
Clar.
Eff.
Avg.
InsightAgents
50.0
50.0
50.0
50.0
50.0
50.0
w/o insight
41.7
43.3
55.0
51.7
45.0
47.3
w/o retrieval
43.3
48.3
51.7
53.3
50.8
49.5
w/o debate
19.2
40.0
51.7
42.5
11.7
33.0
Table 3: Ablation results against InsightAgents , which scores 50 by construction. Scores below 50 indicate that the removed step benefits the full framework.
Figure 5: LLM judges’ preferences by proposal property, based on their stated reasoning. Each point shows the percentage of explanations favoring InsightAgents over a baseline among those discussing that property. Results pool both LLM judges, GPT-5.6 Sol and Claude Opus 4.8 (3,000 explanations each), in the online setting. Values above 50% (dashed line) favor InsightAgents . See Appendix E.3 for details.
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Research area
Topics
Resources and Evaluation
12
Ethics, Bias, and Fairness
10
Multimodality and Language Grounding to Vision, Robotics and Beyond
10
Efficient/Low-Resource Methods for NLP
9
Information Retrieval and Text Mining
8
Question Answering
8
Appendix
Table 4: Research areas of the 100 evaluation topics. Each source paper is assigned one ACL Rolling Review area by GPT-5.6 Luna when its topic is derived.
Stage
Calls
Cost (¢)
Time (s)
Share
Seed selection
0.9
0.04
6.0
0.4
Graph construction
18.6
2.61
111.0
25.2
Insight extraction
1.0
0.83
32.4
8.0
Problem selection
8.0
0.92
57.4
8.8
Proposal dev.
13.0
5.27
202.4
50.9
Final synthesis
1.0
0.68
18.6
6.6
Appendix
Table 5: Generation cost of InsightAgents per topic by stage, averaged over the 100 online topics. Cost is in US cents. Time sums the recorded stage durations and includes Semantic Scholar API latency. Share is the stage’s percentage of total cost; Proposal dev. denotes proposal development.
Method
Cost (¢)
Input (k)
Output (k)
AI Scientist-v2
0.32
6.7
1.8
Chain of Ideas
6.48
121.7
34.4
ResearchAgent
10.88
237.3
57.0
InsightAgents
10.36
321.1
35.9
Appendix
Table 6: Generation cost per topic in the online setting, averaged over 100 topics and taken from each system’s own usage log. All methods use GPT-5.6 Luna at the same token prices. Tokens are in thousands; input includes cached input where logged. AI Scientist-v2 runs only its idea generation stage; Chain of Ideas includes embedding calls.
Baseline
Input
Literature
Generation and refinement
Evaluated output
AI Scientist-v2 (ideation only)
Topic used as workshop description
Semantic Scholar search; 10 results per query; 1.7 searches (1–4)
1 idea; up to 5 rounds of searching or finalizing; 2.7 rounds (2–5), ending upon finalization
Generated idea
Chain of Ideas
Topic
Semantic Scholar citations and references; 1 chain of up to 5 papers; 3.8 papers (1–5)
Trend summarization; repeated idea generation until passing the novelty check, 1.3 rounds (1–6); experiment design with 1 refinement round
Final idea
ResearchAgent
Topic and InsightAgents -selected seed paper
Up to 10 seed-paper references and an entity store from the original release; 7.5 references
Problem, method, and experiment design; each generated and refined twice using feedback from 5 reviewing agents
Highest-rated problem, method, and experiment design
Appendix
Table 7: Baseline configurations using official implementations and GPT-5.6 Luna. Adaptations are detailed in Appendix D . Executed counts are averaged over 100 online evaluation topics, with ranges in parentheses where available.
Setup
Method
Novelty
Significance
Feasibility
Clarity
Eff.
Avg.
Online ( n=100 ) GPT-5.6 Sol
AI Scientist-v2
1021.0
976.7
1248.1
1111.0
1001.4
1071.6
Chain of Ideas
789.3
850.7
855.8
692.2
576.2
752.8
ResearchAgent
775.9
983.9
791.3
861.3
1027.7
888.0
InsightAgents (ours)
1413.8
1188.6
1104.8
1335.5
1394.6
1287.5
Online ( n=100 ) Claude Opus 4.8
AI Scientist-v2
1027.5
974.2
1267.2
1258.5
1080.7
1121.6
Chain of Ideas
701.0
789.9
882.5
741.8
556.4
734.3
Appendix
Table 8: Pairwise Elo ratings ( ↑ ). Setup lists the knowledge setting, topic count n , and judge. Eff. denotes expected effectiveness; Avg. is the mean of the five dimensions. Shading highlights InsightAgents ; bold numbers mark the highest rating within each setup.
Opponent
Novelty
Significance
Feasibility
Clarity
Eff.
Avg. [95% CI]
Online, GPT-5.6 Sol
AI Scientist-v2
89.0
74.0
26.0
77.0
88.0
70.8 [67.6, 73.8]
Chain of Ideas
97.5
89.0
80.0
97.5
100.0
92.8 [90.7, 94.8]
ResearchAgent
98.5
75.5
90.0
94.0
89.5
89.5 [87.0, 91.9]
Online, Claude Opus 4.8
AI Scientist-v2
93.0
80.8
16.0
34.0
83.8
61.5 [58.0, 64.9]
Appendix
Table 9: Direct comparison scores (%, ↑ ) of InsightAgents against each baseline, with half credit for ties. Each dimension uses 200 online or 200 offline judgments. A score above 50 means InsightAgents is preferred more often; Eff. denotes expected effectiveness and Avg. equally weights all five dimensions. Brackets give 95% percentile bootstrap intervals for Avg., resampling topics 10,000 times while retaining both orders and all dimensions together.
Sol
Opus
n=127
n=143
Text describing InsightAgents mentions
Human annotation or studies
44.9
49.0
Multiple components
38.6
49.7
Validation protocol
28.3
48.3
New or counterfactual data
11.0
20.3
Appendix
Table 10: What feasibility rationales mention when AI Scientist-v2 wins (% of rationales). n counts rationales that refer to both proposals by their anonymous labels. Categories are keyword matches on the text describing each proposal, so a mention is not a criticism; a rationale can match several categories.
Property
Favors a proposal when the judge says it
Disfavors a proposal when the judge says it
Originality
reframes the problem, the unit of analysis, or the object of measurement, or introduces a distinctive mechanism
assembles familiar components, follows an established experimental paradigm, or is incremental to existing work
Breadth
would generalize across models, tasks, domains, languages, or architectures
is restricted to one dataset, language pair, model family, or domain
Focus
centers on one decisive question whose outcome is informative either way
is diffuse, spread across many goals or loosely coupled components
Payoff
delivers a kind of result the judge values, such as a usable method or a validity check on existing evaluation
delivers a kind of result the judge discounts, such as one that is merely diagnostic
Precision
defines estimands, formulas, thresholds, and success or refutation criteria that tie the claim to its test
leaves the central mechanism, scoring rule, or construct undefined, or has parts that do not fit together
Evidence
uses matched baselines, counterfactual pairs, oracle conditions, or controls that rule out alternative explanations
relies on circular evaluation, confounded attribution, few units, or a measure that may not capture the construct
Appendix
Table 11: Proposal properties used to label judge explanations. For each explanation and property, the labeler records which proposal the judge favors, or that the property is not discussed. A property can be cited under any evaluation dimension.
Figure 6: LLM judges’ preferences by proposal property, based on their stated reasoning. Bars show the percentage of explanations favoring InsightAgents (right) or the baseline (left), among those discussing each property. Numbers beside the bars give the total count of explanations discussing that property. Results pool both judges in the online setting.
Dimension
Order agreement
Judge agreement
Sol
Opus
Online
Novelty
89.5
88.2
86.8
Significance
83.2
73.3
71.2
Feasibility
92.2
82.3
83.4
Clarity
91.2
74.2
68.9
Appendix
Table 12: Evaluation agreement (%) on 100 topics per setting. Order agreement measures whether swapping presentation order preserves the winner or tie ( n=600 pairs per dimension and setting). Judge agreement compares the two judges on identical pairs and orders ( n=1,200 ).
Figure 8: Agreement by dimension on the same InsightAgents –baseline comparisons as Figure 4 . Each dimension contains 30 topic–pair instances, and Overall pools all 150. Error bars give 95% topic-bootstrap intervals. Sol and Opus denote GPT-5.6 Sol and Claude Opus 4.8.
Figure 9: Outcomes of InsightAgents –baseline comparisons for each judge and dimension. Each bar gives the share of 30 topic–pair instances in which the judge prefers InsightAgents , returns a tie, or prefers the baseline. Labels show shares of at least 15%. Ours denotes InsightAgents .
Online
Offline
Sol
Opus
Sol
Opus
Longer proposal wins (%)
Novelty
30.4
29.7
31.4
29.5
Significance
43.8
41.5
46.5
41.3
Feasibility
21.1
22.5
21.8
22.3
Clarity
31.4
18.1
29.2
18.6
Appendix
Table 13: Proposal length and judgments. Top: share (%) of decided judgments won by the longer proposal, pooled over all six method pairs and both orders. Bottom: Pearson r over 100 topics between the word-count difference ( InsightAgents minus AI Scientist-v2) and InsightAgents ’s average score across the five dimensions.
1. What you are asked to do
Each of the 10 topics in this study shows 4 research ideas side by side, labelled A, B, C, D. Which system wrote which idea is hidden from you, and the order is randomised per annotator.
Read them, then rank them from best to worst separately for each of the five criteria: one text box per criterion, > = strictly better, = = equally good, every label used exactly once. For example:
B > A > C B best, then A, then C
B > A = C B best, A and C equally good
A = B = C all indistinguishable on this criterion
A topic counts as done only once all five criteria are ranked.
Appendix
Table 14: Instructions shown to human annotators. The bracketed block stands for the criteria text, which is identical to the LLM judge prompt in Table 39 and is not repeated here. Instructions about saving are omitted.
Generated output: Expert A insight — Memory Utilization Evaluation
Core insight. This line keeps finding that retrieving a remembered fact is easier than recognizing when it matters and integrating it naturally into an answer. Evaluation has shifted from direct question answering toward implicit, composed, guidance, and evidence-synthesis tasks. It assumes useful memory must be judged in context, not as isolated retrieval. The field remains stuck at the boundary between recovered evidence and coherent, appropriately personalized reasoning.
Historical tension. Broader, more realistic evaluation exposes genuine failures but sacrifices the clean attribution and comparability of pointwise recall benchmarks.
Grounding summary. This reading rests on P18, P37, P44, and P5, representing the graph’s evaluation and benchmarking line. The graph shows a clear progression from elicited recall toward natural integration, distributed evidence use, and decision-support evaluation.
Supporting findings.
[P18] In a four-month deployment with 40 users, 1,872 sessions, and seven memory conditions, Direct QA ranged from 19.7% to 70.1% without changing satisfaction; only 7.9% of recalled facts were naturally referenced.
[P37] On LOCOMO-CONV, conversational framing exposed retrieval gaps for implicit and composed queries; multi-facet rewriting helped raw-turn memory but not abstractive memory.
Appendix
Table 15: Insight distilled for Expert A (Memory Utilization Evaluation), reproduced verbatim. Shaded rows are model output; the paper key below them is added by the authors.
Generated output: Expert B insight — Evolving Memory Architecture
Core insight. This line keeps running into the fact that static summaries and independent top-k retrieval cannot represent changing truths, provenance, conflicts, and query-specific evidence needs together. Systems have progressed from profile-memory graphs to temporal state graphs, multi-round retrieval, selective memory types, and lifecycle-aware relations. They assume explicit structure can make memory auditable without retaining the full history. The unresolved cost is deciding which structure to maintain and when updates or retrieval should reshape it.
Historical tension. More explicit temporal and evidential structure improves correctness and traceability, but increases maintenance, retrieval complexity, and dependence on intermediate decisions.
Grounding summary. This reading rests on P10, P9, P24, P11, and P30, the graph’s densest line of structured, evolving, and lifecycle-aware memory systems. It shows a progression from organizing memory to explicitly governing evidence state and retrieval-driven maintenance.
Supporting findings.
[P10] PGMem linked persona nodes to event nodes through typed provenance and evidence edges, consistently outperforming summary-based, persona-aware, graph, and agentic baselines across three benchmarks.
[P9] EMIR 2 used a State-Evolving Memory Graph and intent-guided multi-round retrieval, producing relative improvements exceeding 12% in some LoCoMo and MemConflict categories.
Appendix
Table 16: Insight distilled for Expert B (Evolving Memory Architecture), reproduced verbatim. Shaded rows are model output; the paper key below them is added by the authors.
Generated output: Expert C insight — Continual Personalization Alignment
Core insight. This line keeps finding that personalization signals are latent, noisy, time-varying, and easy to over-apply. It has moved from static profiles toward uncertainty-aware belief revision, preference hypotheses, adaptive stopping, and online tracing of short- and long-term intent. It assumes alignment should reflect inferred user constraints rather than merely repeat stored preferences. It remains stuck between adapting quickly enough to matter and resisting transient evidence, preference narrowing, sycophancy, and irrelevant personalization.
Historical tension. Personalization must be responsive to emerging preferences without turning every weak signal into a persistent commitment or an excuse to flatter the user.
Grounding summary. This reading rests on P19, P16, P17, P42, P47, and P46, spanning personalization-risk evaluation, behavioral alignment, belief revision, hypothesis tracking, and context-budget control. The graph shows adaptation becoming more explicit and selective, but not yet reliably safe.
Supporting findings.
[P19] Across 13 LLMs, PRISK found personalization increased irrelevant personalization, preference narrowing, and sycophantic bias, with reported average drops of 45.9%, 41.7%, and 61.7%, respectively.
[P16] IBA-Bench showed a knowledge-to-action gap: agents must execute tasks under implicit, noisy, temporally inconsistent constraints, and IBA-Agent improved behavioral alignment across nine application domains.
Appendix
Table 17: Insight distilled for Expert C (Continual Personalization Alignment), reproduced verbatim. Shaded rows are model output; the paper key below them is added by the authors.
Generated output: Final research proposal
Title. Lifecycle-Aware Counterfactual Control of Memory Influence
Problem and motivation. Can a personalized dialogue system determine which evolving user-memory evidence is currently justified, whether it applies to the present request, and how strongly it should change the response? A memory may be current, historical, context-specific, superseded, unresolved, or irrelevant. The appropriate behavior may therefore be silent personalization, explicit grounding, confirmation, abstention, or no behavioral change. Existing evaluations often reduce personalization to retrieval accuracy, answer correctness, or explicit memory mention.
This misses useful silent grounding and cannot distinguish missing evidence from a system that retrieved the right evidence but gave it unjustified authority. The proposal studies whether lifecycle-aware evidence states and auditable influence control can produce more useful and less harmful personalization in recommendation, planning, and decision-support tasks.
Method. The system constructs a lifecycle-aware evidence unit from retrieved memories. It represents whether claims are current, historical, context-specific, superseded, unresolved, or irrelevant, together with provenance and supporting or conflicting relations. A lightweight controller receives the current request, this evidence unit, and a no-memory shadow response. It selects an action—silent use, explicit grounding, confirmation, or abstention—and one of four discrete influence levels. Level 0 prevents memory from changing ranking, membership, or plan constraints. Level 0.25 permits reranking only among options that already satisfy the explicit task constraints.
Level 0.5 permits a soft preference or constraint while retaining at least one viable alternative. Level 1 permits filtering options or materially changing a plan only when the evidence is a confirmed current constraint, such as an explicit recent statement or concordant evidence meeting a preregistered rule. If level 1 is predicted without confirmation, the system downgrades to level 0.5 or asks for confirmation. The controller applies the intervention to a common candidate pool or structured plan representation; the generator then receives only that representation, not the retrieved evidence or controller action, and verbalizes it with deterministic decoding.
Confirmation produces only a clarification question, explicit grounding must accurately mention the relevant evidence, and abstention uses the no-memory response. Influence is measured primarily through changes in option ranking, membership, retained alternatives, and plan constraints, and secondarily through textual grounding, usefulness, and naturalness. This makes memory influence an auditable intervention rather than a latent score or a proxy based on memory mention.
Appendix
Table 18: Final proposal produced by Stage 3, reproduced verbatim (all reader-facing fields; the internal expert-attribution field is omitted). Long fields are split into rows at sentence boundaries for page layout only.
Generated output: Expert proposals in the first debate turn
Expert A, turn 1
“A system receives credit when the memory-induced change is useful and warranted—even if the response never mentions the memory.”
Expert C, turn 1
“Rather than passing retrieved memories directly to a generator, IGP produces an influence budget for each hypothesis.”
Expert B, turn 1
“From the evolving-memory perspective, however, influence cannot be calibrated reliably if the input to the controller is merely a bag of retrieved memories with confidence scores. Confidence is not a substitute for lifecycle state.”
Appendix
Table 19: Excerpts from the three expert proposals in Stage 2, turn 1 (speaking order C, A, B), reproduced verbatim.
Generated output: Revision chain 1 — making the influence budget operational
Draft, turn 1
“The budget determines how strongly a memory may alter the response: a confirmed current constraint may substantially change a plan, while a weak inferred preference may only softly rerank options.”
Critic, after turn 1
“Otherwise the claimed budget may merely be a latent label and the observed behavior could be caused by generation noise or prompt wording.”
Expert C, turn 2
“The system should first generate a common candidate pool or structured plan representation. The influence gate then modifies that representation according to the budget, and the same frozen generator verbalizes the result.”
Draft, turn 2
“Level 0 excludes memory from candidate selection and planning; level 0.25 permits reranking compatible options without removing alternatives; level 0.5 permits a soft constraint while retaining alternatives; and level 1 permits filtering or materially changing a plan only when the evidence is a confirmed current constraint.”
Final proposal
“The controller applies the intervention to a common candidate pool or structured plan representation; the generator then receives only that representation, not the retrieved evidence or controller action, and verbalizes it with deterministic decoding.”
Appendix
Table 20: Excerpts tracing the influence budget from the first draft to the final proposal, reproduced verbatim.
Generated output: Revision chain 2 — aligning the claim with the evaluation scope
Draft, turn 1
“Repeated use of an uncertain hypothesis reduces its authority unless new independent evidence supports it, preventing the system’s own recommendations from becoming false confirmation.”
Critic, after turn 1
“The proposed primary benchmark appears largely case-based, while this rule requires multi-turn trajectories and outcomes; either add a fully specified longitudinal protocol and baselines or remove the rule from the primary claim.”
Expert C, turn 2
“The longitudinal authority-decay mechanism is interesting, but it currently asks for a different paper: multi-turn exposure, user feedback, and a model of independence between evidence sources.”
Final proposal
“If the primary result holds, a follow-up study can test longitudinal authority decay and recommendation-induced self-confirmation; these are not claims of the present paper.”
Appendix
Table 21: Excerpts tracing the deferred authority-decay mechanism, reproduced verbatim.
Research topic and literature → paper graph and three evidence-supported insights
§ H.2 Agent setup
Expert and moderator system messages
Insight and supporting papers → expert context; standing rules for moderation
§ H.3 Problem selection
Problem statements; anonymous shortlist; blind ranking; final selection
Expert perspectives → one shared research problem
§ H.4 Proposal development
Expert proposals; first draft; critic review; revision; final positions; draft update
Selected problem and expert knowledge → a revised proposal
§ H.5 Final synthesis
Final proposal write-up
Discussion and latest draft → one structured research idea
§ H.6 Idea evaluation
Pairwise judge, including the five evaluation criteria
Topic and two proposals → a preference and rationale for each criterion
Appendix
Table 22: Prompt directory. The first five groups generate a research idea; the last evaluates completed ideas. Group names link to the corresponding appendix subsections.
Seed-paper selection
Reading note. Used once to select a starting paper. Candidate papers contain numbered titles and abstracts.
System message
You pick the starting paper for a research graph about a topic.
You get a topic and a numbered list of recent papers that a search returned for it. Search ranks by word overlap, so the top hit is often a paper from a different research area that happens to share the topic’s words. Your job is to pick the one paper that is really about the topic.
Rules: 1. The paper must be about the topic itself, not merely share words with it. If a term in the topic means something different in that paper’s field, it is the wrong pick. 2. Prefer a paper whose main problem IS the topic over one that only touches it as a component. 3. Prefer a paper that sits in the middle of the topic’s literature over a narrow or unusual application of it — the whole graph is grown from this paper. 4. Citations, venue and date only break ties. 5. Pick exactly one, even if none is a perfect fit; say so in ‘why‘ when the fit is poor.
Return ONLY this JSON object, no other text: {"pick": <candidate number>, "why": "<at most 20 words>"}
Appendix
Table 23: Seed-paper selection
Related-paper filtering
Reading note. The first expansion level is shown with a limit of 5 papers per parent. The same system message uses 3 and 2 at the second and third levels. Source paper and candidate papers include metadata and abstracts.
System message
You filter paper recommendations for a research topic.
You get: a topic, one source paper, and a numbered list of papers that Semantic Scholar recommends for that source paper. Recommendations are noisy — many are only loosely related. Keep the ones worth adding to a research graph about the topic.
Rules: 1. Relevance to the topic comes first. A paper that is close to the source paper but off-topic is not worth keeping. 2. The paper must also be a real neighbour of the source paper’s line of work: it addresses the same problem, uses or replaces the same kind of method, or evaluates the same thing. 3. Prefer papers that add something the graph does not have yet — a different method, setting, or failure mode — over near-duplicates of the source paper or of each other. 4. Keep at most 5. Keeping fewer is correct when fewer are relevant; keeping none is allowed.
Return ONLY this JSON object, no other text: {"selected": [{"id": <candidate number>, "why": "<at most 15 words>"}]}
Appendix
Table 24: Related-paper filtering
Insight distillation
Reading note. One call reads the whole graph to produce 3 insights, each supported by at least 2 papers. Graph papers contain identifiers, metadata and abstracts; graph relations use those identifiers. The numbers and provenance paragraph are filled for the online configuration, where an edge links two papers that Semantic Scholar recommends as similar and its direction follows publication date only. The transition summary is the model’s interpretation and is not shown to the experts.
System message
You read a graph of related papers and write 3 pieces of background knowledge from it — the understanding a researcher has after living inside one line of work for a while.
The graph was grown from one recent paper: each round asked Semantic Scholar for related papers and kept the ones about the topic. Most of it sits near the seed, but the graph usually also holds papers from other kinds of work on the topic — training it in, steering it inside the model, watching it in agents, building data for it. Those matter as much as the dense centre.
FIRST, SORT THE GRAPH
Before writing anything, sort the papers into the distinct lines of work the graph contains. A line of work is a kind of research — measuring something, training for it, locating it inside models, studying it in deployed agents, building the data — not a different angle on one question. Then choose the 3 most different lines that have at least two papers each. If the graph only carries fewer than 3 kinds of work, the remaining insight may be a second line inside the largest kind; say so in grounding_summary.
Appendix
Table 25: Insight distillation
Expert setup
Reading note. Persistent system message for problem statements and proposal discussion. The blind vote uses a separate neutral voting prompt.
System message
You are one of three researchers working out a research proposal together. The goal is one idea good enough for a top ML venue: new, and worth doing.
Where you are coming from
You have spent the last while inside one line of work, and it left you with a way of seeing the field:
[Expertise block: this expert’s insight name, research perspective and historical tension; supporting findings linked to source papers; and evidence-paper identifiers, titles, authors, years, venues when available, and abstracts. The block also notes that only abstracts, not full papers, are available.]
Appendix
Table 26: Expert setup
Moderator setup
Reading note. Persistent system message for shortlisting, selection, drafting, revision and final synthesis. The moderator receives the substantive discussion records available at each step.
System message
You are running a discussion among three researchers and writing up what it produces: one research proposal.
Stage 1 settles which problem to work on; once settled it stays settled. Stage 2 develops a draft through several rounds. Stage 3 is the final write-up.
Your write-ups are built from what the researchers said — you choose, combine and sharpen, you don’t add methods or claims of your own. Within that, be ambitious on their behalf: the proposal should be the strongest paper their ideas support, not the safest.
Appendix
Table 27: Moderator setup
Stage 1: problem statements
Reading note. Each expert states an open problem using its standing system message. The instruction below is appended to the discussion history available to that expert.
Expert instruction
Stage 1 — choosing the problem. Topic: [Research topic]
From where you stand, what is the most important open problem in [Research topic] right now — the one a good paper would move? State it as a research question and say why it matters and why it is still open. Problem only; no solutions yet.
Keep it under 900 words.
Appendix
Table 28: Stage 1: problem statements
Stage 1: anonymous problem shortlist
Reading note. The moderator reads the three problem statements and produces a shortlist. The implementation shuffles the anonymous candidates before voting.
Moderator instruction
From the three statements above, write a shortlist of exactly 3 candidate problems for the discussion to work on. Stay faithful to what was raised — you may sharpen wording, and combine two problems only where a researcher explicitly connected them. For each, say why it matters: what an answer would change, and who would use it.
The shortlist is voted on blind, so write each candidate without any reference to who raised it — no "Expert A", no "one researcher’s view". Papers and methods may be named; participants may not.
Reading note. Turn 2 of problem selection. Each voter receives only the neutral system message, topic and anonymous shortlist, without its expertise prompt, discussion history or other votes.
System message
You are ranking candidate research problems on a topic. Judge them on their own terms; you are not told, and should not guess, who raised which.
User message
Stage 1, turn 2 — vote. Topic: [Research topic]
Three candidate problems, shown anonymously in arbitrary order:
Appendix
Table 30: Stage 1: blind problem ranking
Stage 1: final problem selection
Reading note. One of these instructions is sent to the moderator. Vote totals use Borda scores of 3, 2 and 1; only tied top candidates are passed to the tie branch.
Moderator instruction: unique winner
The vote is in. Winning candidate: [Winning problem]
Tally: [Vote totals]
Write the problem statement the discussion will work on: 3-5 sentences that state the question, why it matters, and what would count as answering it. Stay with what was raised — sharpen, don’t broaden or merge. Output the problem statement only.
Moderator instruction: tie
Appendix
Table 31: Stage 1: final problem selection
Shared proposal output format
Reading note. This exact JSON format is inserted into first-draft, update and final-synthesis instructions. Those tables link here to avoid repeating it. For final synthesis it is wrapped in a one-element array. The expert-attribution field is excluded from pairwise evaluation.
JSON format
{ "title": "the idea in one line", "problem_and_motivation": "the question this answers and why the field needs it answered", "method": "the approach, at the level a paper describes its contribution: what is built, studied or measured, and what makes it different from what exists", "experiment": "the primary study that decides the claim — data, models, baselines, and what result confirms or refutes it — then, briefly, what you would add if it holds. Keep the experiments simple and feasible, and don’t rely on human annotation unless the claim can’t be tested without it", "novelty": "what is new here", "builds_on": "the 2-3 specific results the method actually uses — each named by paper title and year with what it contributes. Only work the discussion drew on; not background reading",
"distinct_from": "the 1-2 closest existing works and the delta: what this proposal does that each of them does not", "potential_impact": "what changes for the field if this works, and what is learned if it does not", "trajectory_attribution": { "expert_A": "what this researcher’s thinking contributed to the final idea, in a sentence", "expert_B": "what this researcher’s thinking contributed to the final idea, in a sentence", "expert_C": "what this researcher’s thinking contributed to the final idea, in a sentence" }
}
Appendix
Table 32: Shared proposal output format
Stage 2: expert proposals
Reading note. Turn 1. The task, expert-specific perspective and speech-length rule are shown together as assembled for one expert. Insight name identifies that expert’s research line.
Expert instruction
Stage 2, turn 1 — proposals.
The problem is settled above. Propose the research you would actually do on it, seen from where you stand. One developed proposal beats several sketches: what is the idea, what is new about it, how would you test it, and what would the field learn either way.
You are the one here who has spent time inside [Insight name] . Say what that line of work sees in this problem that the other two would miss, and name the tension from it that your proposal resolves. Build on the specific findings you hold — name the results your proposal depends on. Then bring in whatever else you know.
Keep it under 900 words.
Appendix
Table 33: Stage 2: expert proposals
Stage 2: first proposal draft
Reading note. The moderator reads the three proposals and writes one draft. The cited-paper directory maps evidence identifiers to paper titles and years. Its instruction permits naming only papers actually cited in the discussion.
Moderator instruction
Write the first draft of the proposal from the three proposals above.
There are three sets of ideas on the table. Don’t pick one and reduce the others to controls and caveats, and don’t staple all three together. Look for the strongest idea the discussion supports — often that is a combination, where one researcher’s framing gives another’s method a reason to exist, or two methods turn out to be one idea seen from two sides — and build one coherent proposal around it. Where a researcher’s ideas genuinely don’t fit, leave them out. Merge the ideas, not the experiments: the proposal keeps one primary study that decides the claim, and the other researchers’ tests become extensions, not requirements.
An example of the difference. Three researchers on evaluating long-context models: one wants a benchmark of tasks whose answers require combining distant passages; one has studied how attention concentrates on a few positions; one works on synthetic data generation. Bolt-on: "A benchmark of distant-passage tasks, with an attention-concentration analysis as a diagnostic and synthetic data as an ablation." Combination: "A benchmark whose tasks are generated synthetically so that the positions of the required passages are controlled, letting it measure directly whether failure tracks the attention-concentration pattern — turning a benchmark into a test of a hypothesis about why models fail." The second is one idea; the first is three.
Return one JSON object with this shape:
Appendix
Table 34: Stage 2: first proposal draft
Stage 2: independent draft review
Reading note. The critic reviews the initial and intermediate drafts. It receives only the current proposal and the latest discussion summary when available, not the debate transcript. The same critic prompt is reused after draft updates.
System message
You are reviewing a research proposal draft cold: you have not seen the discussion that produced it, and you judge only the page.
Read it as a good reviewer would — someone who wants this paper to exist and is telling the authors what stands between the draft and that paper. Check:
(a) Contribution — is this research, or an engineering adjustment dressed as research? Would the field learn something from the result whichever way it comes out? (b) Soundness — does the experiment actually test the claim? Could the proposed evidence come out the same whether the idea is right or wrong? (c) Consistency — do the parts fit together? Does every step have the inputs it needs from an earlier step, and do the sections agree with each other? (d) Feasibility — could a few people do this in a submission cycle with the compute a university lab can get? Does it build new data or run human annotation where existing resources or automatic measurement would test the same claim? (e) Clarity — can a reader tell what is proposed, what is claimed, and how they would go about doing it?
Novelty and importance are not yours to judge here; the discussion handles those. Report at most 6 issues, the ones that most threaten the core claim first, each with the passage it concerns and a concrete suggestion. Submit them by calling submit_critic_report exactly once — with an empty list if the draft holds up.
Appendix
Table 35: Stage 2: independent draft review
Stage 2: revision and final positions
Reading note. The configured three-turn discussion uses turn 2 for revision and turn 3 for final positions. Reviewer notes are empty when no issues remain. Unanswered objections addressed to an expert are also appended when present.
Expert instruction: revision
Stage 2, turn 2 — improving the draft.
Current draft: [Current proposal] [Reviewer notes, if any] Answer anything aimed at you. Then push the draft toward a better paper: fix a weakness, strengthen the claim or the way it is tested, bring in something the draft is missing, or — if you think it is right as it stands — say so and why. Silence reads as agreement, so say what you think. Substantial changes are welcome if they make it better research.
Keep it under 900 words.
Expert instruction: final positions
Appendix
Table 36: Stage 2: revision and final positions
Stage 2: discussion summary and draft update
Reading note. After expert responses, the moderator records the discussion and revises the draft. The cited-paper directory maps evidence identifiers to paper titles and years. Its instruction permits naming only papers actually cited in the discussion.
Moderator instruction
Summarize this turn and update the draft. Return one JSON object with exactly two top-level keys:
{ "summary": { "turn_recap": "1-2 sentences", "draft_updates": [{"component": "what changed", "change": "how", "source": "who argued for it"}], "settled": ["points everyone now accepts"], "open_attacks": [{"from": "EXPERT_X", "target": "expert or draft component", "claim": "one sentence", "status": "unanswered"}], "dropped": [{"component": "what was removed", "reason": "why", "source": "who argued for it"}],
"unresolved": ["what is still contested"], "converged": false }, "updated_draft": [Shared proposal JSON: Table 32 ] }
Every change to the draft comes from something a researcher said this turn. Carry forward objections nobody answered; drop the ones that were answered. Set "converged" to true only when no researcher asked for further changes and the reviewer’s earlier blocking issues are addressed.
Appendix
Table 37: Stage 2: discussion summary and draft update
Stage 3: final proposal synthesis
Reading note. The moderator receives the substantive discussion and latest draft, and returns one final proposal. The cited-paper directory maps evidence identifiers to paper titles and years. Its instruction permits naming only papers actually cited in the discussion.
Moderator instruction
Stage 3 — final proposal. From the discussion and the latest draft, write the proposal a reader who never saw the debate would receive. Apply the changes the group agreed on. Where a reviewer’s issue was never resolved, fix it if the discussion gives you the means, otherwise state it as a limitation. Keep the problem from Stage 1. The method or experiment must use at least one specific recent result the researchers brought — used, not mentioned: some step of the proposal depends on it being true.
Return a JSON array holding exactly one proposal object:
[Shared proposal JSON: Table 32 ]
Field values are markdown written for a reader who did not see the discussion: define terms, and don’t refer to the researchers anywhere except trajectory_attribution. Papers are cited by title and year, never by [P#] id — the reader cannot resolve an id, but a named result is something they can check and build on.
Appendix
Table 38: Stage 3: final proposal synthesis
Pairwise research idea evaluation
Reading note. Proposal 0 and Proposal 1 are the final ideas in the current presentation order; they are exchanged for the reverse-order judgment. The five criteria appear directly in the prompt. Effectiveness is reported as expected effectiveness in the paper. Braced descriptions in the response format specify required outputs, not missing input values.
User message
You are a judge in a competition. You have to decide which idea is better.
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
LLM-empowered multi-agent systems offer new potential to accelerate scientific discovery by generating novel research ideas. However, existing methods typically coordinate agents through temporary texts, such as drafts or chat logs; it is difficult to pinpoint the weaknesses in the generated ideas and how the agents refine them. To this end, we introduce Evolving Idea Graphs (EIG), a graph-based multi-agent scientific ideation framework that can generate high-performance research ideas across various benchmark-native metrics, such as novelty, feasibility, and clarity. Instead of coordinating solely through texts, EIG represents a partially formed proposal as an evolving idea graph, where nodes capture scientific claims and edges encode relations (e.g., support and conflict), enabling unresolved weaknesses to remain identifiable throughout the idea evolving process. Specifically, a learned two-head controller operates over the evolving graph to guide the ideation: one head selects graph edits for agents to execute, while the other decides when the graph is ready for commit as final proposal synthesis. On AI Idea Bench 2025 and LiveIdeaBench, EIG outperforms all compared systems on both automatic benchmark scores and blind expert ratings. Ablations further show that explicit graph state provides the main performance gains, and learned edit-and-commit control adds consistent improvements.
Jiangwen Dong, Bo Li, Wanyu Lin
1The Hong Kong Polytechnic University · 2The Hong Kong University of Science and Technology
Scientific progress depends on the continual generation of innovative re-search ideas. However, the rapid growth of scientific literature has greatly increased the cost of knowledge filtering, making it harder for researchers to identify novel directions. Although existing large language model (LLM)-based methods show promise in research idea generation, the ideas they produce are often repetitive and lack depth. To address this issue, this study proposes a multi-agent iterative planning search strategy inspired by com-binatorial innovation theory. The framework combines iterative knowledge search with an LLM-based multi-agent system to generate, evaluate, and re-fine research ideas through repeated interaction, with the goal of improving idea diversity and novelty. Experiments in the natural language processing domain show that the proposed method outperforms state-of-the-art base-lines in both diversity and novelty. Further comparison with ideas derived from top-tier machine learning conference papers indicates that the quality of the generated ideas falls between that of accepted and rejected papers. These results suggest that the proposed framework is a promising approach for supporting high-quality research idea generation. The source code and dataset used in this paper are publicly available on Github repository: https://github.com/ChenShuai00/MAGenIdeas. The demo is available at https://huggingface.co/spaces/cshuai20/MAGenIdeas.
Shuai Chen, Chengzhi Zhang
Department of Information Management, Nanjing University of Science and Technology, Nanjing, 210094 China
As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience
Rongcan Pei, Fang Guo, Qinglin Qi +7
Tongji University · Westlake University · Zhejiang University +2