MemoNoveltyAgent: A Historical Research Memory-Aware Agent Workflow for Paper Novelty Assessment
Authors: Jiajun Hou, Hexuan Deng, Wenxiang Jiao, Xuebo Liu, Xiaopeng Ke, Derek F. Wong, Min Zhang
Organizations: The Chinese University of Hong Kong, Shenzhen, China · Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Zhongguancun Academy, Beijing, China · Xiaohongshu Inc. · NLP2CT Lab, Department of Computer and Information Science, University of Macau, China
To alleviate the heavy burden of paper screening, researchers increasingly rely on existing AI agents, such as AI reviewers or DeepResearch, for paper evaluation and novelty assessment. However, lacking specialized mechanisms for processing scholarly literature, their analyses often produce superficial results with noticeable deficiencies in quality. To bridge this gap, we introduce MemoNoveltyAgent, a multi-agent system designed to generate comprehensive and faithful novelty reports. Beyond retrieving concrete prior-paper evidence via RAG, our system incorporates a high-level abstract memory constructed from large-scale scholarly corpora. This memory organizes research into hierarchical trees to distill field-specific evolutionary trajectories, thereby providing a broader historical context. Furthermore, we decompose papers into discrete novelty points for fine-grained analysis and retrieval, while employing a self-validation mechanism to improve report faithfulness. Finally, to address the evaluation challenges of such open-ended generation tasks, we propose a RAG-augmented checklist evaluation method that enables reliable and evidence-grounded assessments. Extensive experiments demonstrate that MemoNoveltyAgent outperforms GPT-5 DeepResearch by 13.69%. Code and demo are available at https://github.com/SStan1/MemoNoveltyAgent
Figures & tables
Figure 1: The MemoNoveltyAgent workflow. The system consists of four primary modules: (§3.1) constructing a citation-based literature database; (§3.2) building an offline historical research memory database from ICLR accepted papers; (§3.3) generating a point-wise report by combining local RAG and memory retrieval; and (§3.4) performing faithfulness-enhanced self-validation to output the final report.
Figure 2: A concrete example of the evolutionary trajectory of a parent node.
Models
Score (0-10) ↑
Completeness
Depth
Effectiveness
Faithfulness
Fluency
Overall
Web Research
Kimi-2
7.57
7.01
9.22
4.85
9.77
7.68
GPT-5 Thinking
7.92
7.21
9.67
4.90
9.87
7.91
Gemini-2.5-Flash DeepResearch
8.18
8.03
9.58
4.00
8.61
7.68
GPT-5 DeepResearch
8.06
7.72
9.45
6.06
9.59
8.18
Table 1: Main evaluation results on the 50-paper ICLR dataset (0–10 scale). Best results are highlighted.
Models
Proxy ( n−1 )
Proxy ( n )
MSE ↓
MAE ↓
MSE ↓
MAE ↓
Gemini-2.5-Flash-Nothinking
0.28
0.44
0.19
0.37
Gemini-2.5-Flash
0.37
0.49
0.26
0.41
GPT-4o-Mini
0.72
0.79
0.50
0.66
Claude-3-Haiku-20240307
0.93
0.77
0.64
0.64
DeepSeek-V3-250324
1.11
0.99
0.77
0.82
Table 2: LLM Consistency result on a 32-paper subset.
Models
Score (1–5) ↑
Completeness
Depth
Effectiveness
Faithfulness
Fluency
Overall
H
AI
H
AI
H
AI
H
AI
H
AI
H
AI
DeepReview
3.875
4.17
3.50
4.16
4.75
4.78
3.75
4.04
4.875
4.91
4.15
4.41
GPT-5 DeepResearch
3.50
4.15
3.25
3.99
4.50
4.67
3.00
3.15
4.50
4.78
3.75
4.15
MemoNoveltyAgent
4.25
4.53
4.50
4.34
4.88
4.89
4.25
4.32
4.50
4.87
4.48
4.59
Table 3: Human (H) and AI evaluation result on a 32-paper subset. AI scores are scaled to the 1–5 human rubric.
Models
Score (0-10) ↑
Completeness
Depth
Effectiveness
Faithfulness
Fluency
Overall
MemoNoveltyAgent
9.20
8.80
9.93
8.76
9.83
9.30
w/o Memory
8.76
8.49
9.79
8.43
9.66
9.03
Direct Generation
8.35
7.44
8.29
7.78
9.86
8.34
Table 4: Ablation results on the main 50-paper ICLR dataset. “w/o Memory” only removes the memory module. “Direct Generation” removes memory, point-wise analysis, and self-validation.
Method
Completeness
Depth
Reviewer Helpfulness
Faithfulness
Kimi-2 Online
3.06
3.14
3.02
4.74
GPT-5 Thinking
3.42
3.42
3.44
4.88
Gemini-2.5-Flash DeepResearch
3.40
3.62
3.40
4.98
GPT-5 DeepResearch
3.66
3.82
3.68
4.98
AgentReviewer
4.00
4.04
3.98
5.00
DeepReview
4.00
4.22
4.02
4.96
Table 5: Reviewer-reference evaluation against an approximate ground truth constructed from novelty-related comments in the original ICLR 2025 OpenReview reviews. Scores are reported on a 1–5 scale, and the best result in each dimension is highlighted.
Models
Score (0–10) ↑
Completeness
Depth
Effectiveness
Faithfulness
Fluency
Overall
GPT-5.4 Online
6.30
3.87
9.49
5.18
9.73
6.91
GPT-5.4 DeepResearch
7.64
7.42
9.66
6.91
9.40
8.21
DeepReview
8.42
7.72
9.79
7.30
9.54
8.55
MemoNoveltyAgent
8.82
8.50
9.94
8.57
9.49
9.06
Table 6: Evaluation result on the cross-venue dataset, including 50 papers from ACL, CVPR, ACM MM, TPAMI, and JMLR.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Case Type 1: Generic Statements vs. Fine-Grained Technical Details
Context
Evaluating the specific technical alignment and similarities with AMIE.
GPT-5 DeepResearch
AMIE also simulated patient-doctor conversations to improve diagnosis. Overly generic
MemoNoveltyAgent
Resembling the proposed approach, AMIE utilized OSCE-style simulated consultations for training and evaluation to assess history-taking, diagnostic reasoning, and communication skills.
Context
Evaluating whether the report explains the mathematical mechanism behind a proposed emergence metric.
GPT-5 DeepResearch
The paper provides a formal measure of emergence using topology and category theory. High-level restatement
MemoNoveltyAgent
The paper identifies emergence with the first derived functor of an observation functor, computes it through projective/injective resolutions in quiver representations , and reduces the result to a path-count formula over affected network edges.
Appendix
Table 7: Case studies illustrating report quality. Case Type 1 shows that point-wise analysis helps recover concrete technical details rather than generic comparisons. Case Type 2 shows that historical research memory helps calibrate novelty judgments by distinguishing genuine contributions from domain-specific integrations, targeted refinements, or supporting engineering choices.
Figure 3: An example of theme tree from the historical research memory: a clustered development tree for the search-guided planning and optimization theme. Branches are recursively refined only when the corresponding theme exceeds the refinement threshold; therefore, some branches are further divided into subthemes, while others directly list papers.
Venue
Dimension
DeepReview
MemoNoveltyAgent
GPT-5.4 DeepResearch
GPT-5.4 Online
ACL
Completeness
8.61
9.07
7.84
6.00
Depth
7.07
8.15
7.74
3.48
Effectiveness
9.77
9.94
9.82
9.59
Faithfulness
6.48
7.93
6.79
5.79
Fluency
9.86
9.00
9.50
9.79
Overall
8.36
8.82
8.34
6.93
Appendix
Table 8: Detailed venue-level breakdown for cross-venue evaluation (0–10 scale).
Dimension
Better (Ours)
Worse (Ours)
Tie
Completeness
100.0%
0.0%
0.0%
Depth
100.0%
0.0%
0.0%
Effectiveness
100.0%
0.0%
0.0%
Faithfulness
100.0%
0.0%
0.0%
Fluency
100.0%
0.0%
0.0%
Overall Win Rate
100.0%
0.0%
0.0%
Appendix
Table 9: Pairwise comparison results between our generated reports and OpenNovelty reports on 10 randomly selected ICLR 2026 submissions. “Better” indicates the percentage of cases where our report was preferred.
Figure 4: Paper content summary excerpted from the AgentClinic report.
Figure 5: Point-wise novelty analysis excerpted from the AgentClinic report.
Figure 6: Novelty summary excerpted from the AgentClinic report.
Figure 7: Human evaluation rubric, part 1.
Figure 8: Human evaluation rubric, part 2.
Figure 9: Fluency checklist questions used in checklist evaluation.
Figure 10: Effectiveness checklist questions used in checklist evaluation.
Figure 11: Completeness checklist questions used in checklist evaluation.
Figure 12: Faithfulness checklist questions used in checklist evaluation.
Figure 13: Depth checklist questions used in checklist evaluation.
Figure 14: Core prompt used by the evaluator to answer checklist questions.
Figure 15: Prompt used for point-wise novelty analysis against prior work.
Figure 16: Prompt used to generate the novelty summary.
Figure 17: Prompt used to write leaf-level development histories in the historical research memory.
School of Economics and Management, Nanjing University of Science and Technology · School of Electronic Engineering and Computer Science, Queen Mary University of London · School of Management, Anhui University