DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation

Rongcan Pei; Fang Guo; Qinglin Qi; Qi Zhu; Yun Luo; Jianhao Yan; Minjun Zhu; Qiujie Xie

cs.CLAug 14, 2026

DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation

Authors: Rongcan Pei, Fang Guo, Qinglin Qi, Qi Zhu, Yun Luo, Jianhao Yan, Minjun Zhu, Qiujie Xie, +2 more

Organizations: Tongji University · Westlake University · Zhejiang University · Shanghai AI Lab · Fudan University

Abstract

As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience-grounded reasoning used by human instructors. To address this, we propose DeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct-based agent to retrieve dimension-specific evidence for traceable evaluation. We further introduce DeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24.4% and 29.7%, respectively. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience

Figures & tables

Figure 1: Framework of our proposed DeepInstructor.

Figure 2: Examples of Triplets Extraction.

Model	Novelty		Significance		Feasibility Score		Feasibility Concerns
Model	Hit@ $\pm 1$	Hit@ $\pm 2$	Hit@ $\pm 1$	Hit@ $\pm 2$	Hit@ $\pm 1$	Hit@ $\pm 2$	Mean Cov.	Mean Prec.
$RAG$	0.452	0.681	0.000	0.133	0.681	0.933	0.579	0.310
DeepReview	0.504	0.741	0.044	0.207	0.667	0.895	0.007	0.000
ChatGPT-4o	0.089	0.215	0.000	0.133	0.652	0.926	0.575	0.305
ChatGPT-4o-mini	0.044	0.141	0.000	0.141	0.719	0.970	0.440	0.246
DeepSeek-V4-Pro	0.378	0.548	0.044	0.200	0.385	0.682	0.522	0.391

Table 1: Performance comparison across Novelty, Significance, and Feasibility evaluation. We report Hit@

\pm 1

and Hit@

\pm 2

for scalar scores, and mean coverage and precision for feasibility concerns.

Ours

indicates the proposed DeepInstructor Model. Best and second-best results are highlighted in bold and underlined , respectively.

Model	Novelty (%) Pair. / Lev.	Feasibility (%) Pair. / Lev.	Significance (%) Pair. / Lev.
RAG	46.15 / 36.32	24.41 / 48.43	13.33 / 20.00
4o-mini	41.03 / 23.50	22.83 / 45.67	23.33 / 25.00
DS	33.33 / 40.31	42.64 / 15.50	30.00 / 48.33
$\text{Ours}_{4o\text{-mini}}$	47.86 / 43.16	25.20 / 52.36	36.67 / 26.67
$\Delta$	+7% / +20%	+2% / +7%	+13% / +2%
$\text{Ours}_{\text{DS}}$	37.34 / 42.58	39.46 / 50.39	40.00 / 49.11

Table 2: Level Accuracy and Pairwise Comparison Accuracy on DeepInstruct. Best results are in bold , and second-best are underlined . 4o-mini = ChatGPT-4o-mini; DS = DeepSeek-V4-Pro

Figure 3: Ablation study on the effect of experience coverage using different subsets of the Experience Graph.

Ours

denotes the full-coverage setting, while

Ours_{2024}

and

Ours_{2025}

use year-specific data only.

Method	$\bar{\Delta}$
ChatGPT-4o-mini prompt feedback	+0.068
ChatGPT-4o-mini Instructor feedback	+0.121
ChatGPT-4o Instructor feedback	+0.203

Table 3: Average overall feedback score improvements (

\bar{\Delta}

) across different methods.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Pipeline of DeepInstruct Dataset Construction.

Combination	Type	Precision	Recall	F1
gpt4_turbo + gpt4o + qwen_plus	intersection	0.7824	0.3788	0.3948
gemini + gpt4_turbo + qwen_plus	intersection	0.7744	0.3809	0.3913
gpt4_turbo + qwen_plus	intersection	0.7634	0.4000	0.4140
gpt4_turbo + gpt4o_mini + qwen_plus	intersection	0.7631	0.3675	0.3819
gemini + gpt4_turbo + gpt4o	intersection	0.7554	0.3654	0.3696
gpt4_turbo + gpt4o	intersection	0.7297	0.3896	0.3981

Appendix

Table 4: Performance of Different Model Combinations (Top 6 by Precision)

Method	Score (%)
Method	6	7	8	9	10
Direct Scoring	0.0	0.0	76.3	23.7	0.0
Evaluation+Scorer	1.5	18.5	3.0	43.0	34.1

Appendix

Table 5: Comparison of Novelty score distributions. Scores from 1 to 5 are omitted (counts are zero).

Dimension	Precision	Pearson	Spearman	Kappa
Significance	$\pm 1$ : 0.7232; $\pm 2$ : 0.9345	0.8220	0.8473	0.7760
Novelty	$\pm 1$ : 0.6528; $\pm 2$ : 0.9021	0.8086	0.8128	0.7551
Feasibility_score	$\pm 1$ : 0.8980; $\pm 2$ : 0.9796	0.7714	0.7698	0.6556

Appendix

Table 6: Performance of LLM scorer across different evaluation dimensions.

Metric	Score
Problem coverage	92.8%
Method coverage	88.6%
Overall coverage	90.7%
Macro-average F1	0.9064

Appendix

Table 7: Coverage of reviewer-described ideas by the Experience Graph.

Panel of Figure 5 — Figure 5: System prompt utilized for Experience Graph Extraction.

Type	Dimension	$R_{0}$	$R_{1}$	$R_{2}$	$R_{3}$	Avg. Change
ChatGPT-4o Instructor feedback	Feasibility	6.203	6.359 (+0.156)	6.224 (+0.021)	6.218 (+0.015)	+0.064
	Novelty	5.802	5.880 (+0.078)	6.191 (+0.389)	6.202 (+0.400)	+0.289
	Significance	8.270	8.509 (+0.239)	8.355 (+0.085)	8.716 (+0.446)	+0.257
ChatGPT-4o-mini Instructor feedback	Feasibility	5.569	5.559 (-0.010)	5.582 (+0.013)	5.639 (+0.070)	+0.024
	Novelty	5.914	6.022 (+0.108)	6.125 (+0.211)	5.922 (+0.008)	+0.109
	Significance	8.612	8.752 (+0.140)	8.899 (+0.287)	8.876 (+0.264)	+0.230

Appendix

Table 8: Feedback scores and relative changes (compared to

R_{0}

) across different dimensions and iterations.

Figure 6: Case study on novelty evaluation.

Figure 7: Case study on significance evaluation.

Panel of Figure 8 — Figure 8: Case study on Novelty : DeepInstructor (DeepSeek). Blue = Thinking, Green = Retrieval, Red = Final Evaluation.

Diff.	Count	%	Cumul. (%)
0	37,993	33.92	33.92
2	51,888	46.33	80.25
$\geq 4$	22,120	19.75	100.0
Total	112,001	100.0	–

Appendix

Table 9: Distribution of Absolute Differences Between Reviewer Pairs

Range Threshold	Count	Percentage
Range $\geq 2$	18,117	93.2%
Range $\geq 4$	10,052	51.7%
Average Range	$3.16\pm 1.63$

Appendix

Table 10: Distribution of Score Range per Paper

Figure 9: Distribution of reviewer score differences. (a) Histogram of absolute differences. (b) Distribution of score range per paper. (c) Distribution of standard deviation. (d) Cumulative distribution. (e) Mean vs. std scatter plot. (f) Proportion pie chart.

Metric	Value	Interpretation
Hit@1	0.3392	33.92% of pairs differ by $\leq 1$ point
Hit@2	0.8025	80.25% of pairs differ by $\leq 2$ points
Pearson correlation	0.1796	Weak positive correlation
Spearman correlation	0.1759	Weak positive correlation

Appendix

Table 11: Consistency Metrics Across All Reviewer Pairs

Group	Count	%	Hit@1	Hit@2	Pearson	Spearman
diff = 0	37,993	33.9%	1.0000	1.0000	1.0000	1.0000
diff = 2	51,888	46.3%	0.0000	1.0000	0.3324	0.3318
diff $\geq 4$	22,120	19.7%	0.0000	0.0000	-0.5981	-0.6012

Appendix

Table 12: Consistency Metrics by Reviewer Difference Group

Demonstration	Prompt
Novelty	You are a professional evaluator focusing on the novelty of the idea. I will provide you with an academic idea. Your task is to evaluate only its level of innovation. Please focus exclusively on the novelty and originality of the idea — how new, unique, or creative it is compared to existing research or conventional approaches in the field. Your response should: Be concise and academic in tone. Avoid discussing feasibility, impact, or methodology. Provide a clear judgment on the innovation level with your serious analysis and reasoning.
Significance	You are a professional evaluator focusing on the significance of the idea. I will provide you with an academic idea. Your task is to evaluate only its level of significance. Please focus exclusively on the importance, scope, and potential impact of the idea — how meaningful, influential, or valuable it would be compared to existing research or conventional approaches in the field. Your response should: Be concise and academic in tone. Avoid discussing feasibility, novelty, or methodology. Provide a clear judgment on the significance level with your serious analysis and reasoning.
Feasibility (Output concerns)	You are a rigorous peer-reviewer. Task: Critically evaluate the given idea/proposal and GENERATE potential ’concerns’ of feasibility (feasibility, feasibility doubts, missing evaluations). Do NOT extract phrases from the text verbatim; instead, propose concerns based on your assessment. Output Policy (STRICT): - Return ONLY a JSON array of strings, starting with ’[’ and ending with ’]’. - Each item must be a single-line short sentence (no line breaks). - Do NOT include any code fences, markdown, comments, labels, or extra text. - No leading bullets, numbering, or trailing commas inside items. - Aim for 8-12 high-quality, non-duplicative items covering: feasibility, feasibility doubts, missing evaluations.
Feasibility (Output evaluative comments)	You are a rigorous peer-reviewer evaluating the feasibility of an academic research idea. Task: Critically evaluate the given idea/proposal and write a comprehensive peer review evaluation text. Your response should: - Be a continuous, natural text similar to a peer review comment (like "all comments" in academic reviews) - Discuss feasibility, implementation challenges, effectiveness, and potential issues - Be concise and academic in tone - Provide an evaluation covering both positive aspects and concerns - Include your assessment of: * How easy or difficult it is to implement the idea * Whether the experimental setup is feasible * Whether the method is likely to work effectively * Any resource requirements or challenges * Comparison with existing approaches if relevant - Write in a natural, flowing style as if you are providing feedback to the authors

Appendix

Table 13: Prompts used by GPT for evaluating each dimension.

Demonstration	Prompt
Novelty	{GPT’s_prompt_for_novelty} Below are evidence sentences retrieved from reviews based on your query: {context} Please generate a novelty evaluation based on these evidence sentences and your professional knowledge for the following idea. Output Policy (STRICT): - Return a text evaluating innovation (which should include reasonable reasons) based on the evidence sentences. - Please evaluate the innovativeness of the idea clearly and emphatically.
Significance	{GPT’s_prompt_for_significance} Below are evidence sentences retrieved from reviews based on your query: {context} Please generate a significance evaluation based on these evidence sentences and your professional knowledge for the following idea. Output Policy (STRICT): - Return a text evaluating significance (which should include reasonable reasons) based on the evidence sentences. - Please evaluate the significance of the idea clearly and emphatically.
Feasibility (Output concerns)	{GPT’s_prompt_for_feasibility_concerns} Below are evidence sentences retrieved from reviews based on your query:{context} Please generate concerns based on these evidence sentences and your professional knowledge for the following idea.
Feasibility (Output evaluative comments)	{GPT’s_prompt_for_feasibility_comments} Below are evidence sentences retrieved from reviews based on your query:{context} Please generate comments based on these evidence sentences and your professional knowledge for the following idea.

Appendix

Table 14: Prompts used by RAG-based LLM for evaluating each dimension.

Demonstration	Prompt
Novelty	{GPT’s_prompt_for_novelty} You can use tools to learn about relevant experiences related to innovation. Core argument explanation: node_query: Enter a specific knowledge entity(from problem and method of the idea) (e.g.,"LLM’s hallucination", "dialect recognition", "speech processing", "the method using XXX"). edge_query: Enter ONLY predicate phrases WITHOUT including the entity name. you can use generic relationship expressions to search for the experience of novelty: - For relationships: "is a method to solve", "is solved by", "has been researched widely", "has not been used to solve" …(Use these more) - For explict novelty comment: "lacks novelty", "is a new method", "is a new problem" …(Use these more) - For positive effects: "can improve", "helps"… - For negative effects: "has limitations", "faces challenges"… This will return you several pieces of evidence and a summary. You need to evaluate the innovativeness of this idea based on these pieces of evidence and the summary. You MUST perform at least 20 tool calls before providing the final answer. Iteration & Tool Scheduling: - Prefer making only 1 tool call per step; absolutely no more than 2 in any single step. - If multiple queries are needed, split them into multiple steps/rounds to collect evidence gradually. - After each tool result, briefly reflect and plan the next single tool call.
Significance	{GPT’s_prompt_for_novelty} You can use tools to learn about relevant experiences related to significance. Core argument explanation: node_query: Enter a specific knowledge entity(from problem and method of the idea) (e.g.,"LLM’s hallucination", "dialect recognition", "speech processing", "the method using XXX"). edge_query: Enter ONLY predicate phrases WITHOUT including the entity name. you can use generic relationship expressions to search for the experience of novelty: - For relationships: "is a method to solve", "is solved by", "has been researched widely", "has not been used to solve" …(Use these more) - For explict significance comment: "lacks significance", "is a significant method", "is a significant problem" …(Use these more) - For positive effects: "can improve", "helps"… - For negative effects: "has limitations", "faces challenges"… This will return you several pieces of evidence and a summary. You need to evaluate the significance of this idea based on these pieces of evidence and the summary. You MUST perform at least 20 tool calls before providing the final answer.

Appendix

Table 15: Prompts used by DeepInstructor for evaluating each dimension.

Demonstration	Prompt
Feasibility (Output concerns)	{GPT’s_prompt_for_feasibility_concerns} You can use tools to learn about relevant experiences related to feasibility and systematically explore different entities mentioned in the research idea and query them separately. Entity Discovery Strategy: - Extract ALL distinct knowledge entities from the research idea (e.g., for "LLM improving dialect recognition", extract "LLM", "dialect", "recognition", "speech recognition", "language processing" etc.) - Query each entity separately with different node_query values - Use multiple search rounds to cover different aspects and entities Edge Query Design Rules: node_query: Enter a specific knowledge entity (e.g., "Large Language Models", "Graph Neural Networks", "dialect recognition", "speech processing", "machine learning models"). edge_query: Enter ONLY predicate phrases WITHOUT including the entity name. - For positive effects: "can improve", "helps", "enhances", "strengthens", "boosts", "optimizes" - For negative effects: "has limitations", "faces challenges", "has shortcomings", "struggles with" - For relationships: "is related to", "depends on", "influences", "causes" Examples: - WRONG: edge_query = "LLM improves dialect recognition" (contains entity) - CORRECT: edge_query = "can improve recognition" (pure predicate) - For "LLM improving dialect recognition" idea, query: * node_query="LLM", edge_query="can improve recognition" * node_query="dialect", edge_query="difficult to recognize" * node_query="speech recognition", edge_query="faces challenges" You MUST perform at least 20 tool calls before providing the final answer.
Feasibility (Output evaluative comments)	{GPT’s_prompt_for_feasibility_comments} … (The rest is the same as the previous line.)

Appendix

Table 16: Prompts used by DeepInstructor for evaluating each dimension (Continued).

Model	Prompt
Novelty Scorer	You are a precise scorer. I will provide you with a professional evaluation of an academic idea, and you need to give a novelty score based on this evaluation. The novelty score depends on the attitude of the evaluation. If the evaluation is positive, the novelty score should be high; if the evaluation is negative, the novelty score should be low. Please note that the novelty score ranges from 1 to 10, where 1 indicates the lowest novelty and 10 indicates the highest novelty. Here are some examples(Some specific method or idea is replaced with Method A, Method B, Method C, etc.): {6 cases distributed across different scores from the novelty dataset}
Significance Scorer	You are a precise scorer. I will provide you with a professional evaluation of an academic idea, and you need to give a significance score based on this evaluation. The significance score depends on the attitude of the evaluation. If the evaluation is positive, the significance score should be high; if the evaluation is negative, the significance score should be low. Please note that the significance score ranges from 1 to 10, where 1 indicates the lowest significance and 10 indicates the highest significance. Here are some examples(Some specific method or idea is replaced with Method A, Method B, Method C, etc.): {6 cases distributed across different scores from the significance dataset}
Feasibility Scorer	You are a precise scorer. I will provide you with a professional peer review evaluation of an academic idea, and you need to give a feasibility score based on this evaluation. Feasibility means whether the idea is easy to implement and execute and whether the idea is effective. The feasibility score depends on how feasible and executable the idea is according to the evaluation. Please note that the feasibility score ranges from 1 to 10, where 1 indicates the lowest feasibility and 10 indicates the highest feasibility. Here are some examples based on real evaluation data with full review comments (all comments) and their corresponding average scores: {6 cases distributed across different scores from the feasibility dataset}
DeepSeek Reasoner to match feasibility concerns	You are a precise evaluator. You are currently dealing with the opinions of two reviewers, ’original’ (gold) and ’generated’ (model output). Your task is to explain whether each point in the gold standard has been reflected in the "generated" content. Now decide for each ORIGINAL item whether it is covered by any GENERATED item under a criterion: - Mark as covered if the overall meaning is similar, paraphrased, or broadly aligned (approximate semantic similarity), If two or more generated concerns jointly express one concern, it is also considered covered. But don’t be too loose. Thinking and reasoning Requirement (STRICT): - Process ORIGINAL concerns sequentially, one-by-one. - For each ORIGINAL item, carefully check all GENERATED items and determine matches. - Perform your reasoning internally. - Only include a concise, one-sentence justification in the "reason" field per item. Output Policy (STRICT): - Return ONLY a JSON object, starting with ” and ending with ”. - Keys must be exactly: per_item (array), summary (object). - Each per_item element: "original": string, "covered": bool, "matched_indices": [int], "reason": string. - summary: "covered_count": int, "total": int, "coverage_ratio": number. - Do NOT include code fences, markdown, comments, or extra explanatory text.

Appendix

Table 17: Prompts for Few-shot LLM Scorers and DeepSeek–Reasoner

Explore similar work

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem
Feb 16, 2026Shuofei Qiao, Yunxiang Wei, Xuehai Wang +10Automated Evaluation
Zhejiang University · University College London · The Chinese University of Hong Kong +1
Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
Aug 30, 2026Zhiyu Chen, Keyu Zhao, Jigao Fu +8AI Agents for Scientific Discovery LLM-as-a-Judge
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1
LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation
Aug 13, 2026Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6LLM Evaluation Benchmark Design
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China

Feb 16, 2026J/K move · Enter open · S save

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

Shuofei Qiao, Yunxiang Wei, Xuehai Wang, Bin Wu+9

Zhejiang University · University College London · The Chinese University of Hong Kong+1

Automated Evaluation

Open paper AskPDF arXiv

The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.

Abstract

Explore similar work

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation