NoveltyScope: A Multi-Agent Research Assistant for Evidence-Grounded Novelty Assessment of Scientific Papers
Organizations: School of Computing Newcastle University Newcastle upon Tyne, UK
Abstract
NoveltyScope is a prototype that compares a scientific paper with retrieved prior work and produces a structured report and a five-level novelty score. The system extracts the core task and at most two author-claimed contributions, then searches a closed local corpus using BM25 and dense retrieval with reciprocal rank fusion. Contribution comparisons request paired passages for potential refutations, and quotations are checked against cached source text. Four scoring roles (Prosecutor, Defender, Judge, and Critic) use the resulting evidence, with at most one revision of the Judge's score. Evaluation uses 50 class-balanced RINoBench samples whose idea-level labels are retained for full-paper input. Mean absolute error is 1.48 and 1.26 for single-LLM and multi-agent scoring with Gemini, respectively, and 1.22 and 1.16 with Qwen3. The vector-similarity baseline has an error of 1.68. A HiCLIP case study illustrates the contribution comparisons and paired evidence passages. Overall, the results demonstrate the feasibility of evidence-grounded novelty assessment and show that the multi-agent configurations achieve lower scoring errors than their single-LLM counterparts across both evaluated models.
Figures & tables
| Role | Additional input | Responsibility and output |
|---|---|---|
| Prosecutor | None | Evidence-supported overlap; low-novelty claims and signal. |
| Defender | None | Specific central distinctions; high-novelty claims and signal. |
| Judge | Both arguments; calibration exemplars | Score 1–5, justification, evidence, and adjacent-label explanations. |
| Critic | Both arguments; Judge output; exemplars | Rubric audit; accept or revise, suggested score, and boundary-error reason. |
| Group | Scoring configuration | Backend | MAE | Reduction vs. E1 |
|---|---|---|---|---|
| E1 | Dense similarity mapping | None | 1.68 | – |
| E2 | Single-LLM scoring | Gemini | 1.48 | 11.9% |
| E3 | Multi-agent scoring | Gemini | 1.26 | 25.0% |
| E4 | Single-LLM scoring | Qwen3 | 1.22 | 27.4% |
| E5 | Multi-agent scoring | Qwen3 | 1.16 | 31.0% |
| Extracted claim | Generated comparison | Scope of the comparison |
|---|---|---|
| Hierarchy-aware CLIP | Tree Transformer is flagged for overlap in language-side hierarchical attention; paired passages are shown. | Component overlap does not alone refute the full multimodal contribution. |
| Group Transformer for vision | No refutation is recorded among ten candidates; brief technical distinctions are provided. | The absence of refutation is limited to these ten candidates. |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Temperature | Maximum tokens |
|---|---|---|
| Contribution extraction | 0.0 | 4,000 |
| Prior-work query generation | 0.0 | 3,000 |
| Query-variant generation | 0.7 | 1,000 |
| Core-task extraction | 0.0 | 2,000 |
| Taxonomy construction | 0.0 | 20,000 |
| Narrative generation | 0.0 | 5,000 |
| Score | Definition |
|---|---|
| 1 | The idea is not novel. All aspects already exist in prior work. |
| 2 | The idea is marginally novel. It represents only a minor variation of existing work. |
| 3 | The idea is somewhat novel. Aspects already exist in prior work. However, it might combine known approaches in new ways, apply them to new contexts, or propose incremental updates. |
| 4 | The idea is novel. It introduces new aspects not present in existing work. |
| 5 | The idea is highly innovative and novel. It is not present in existing work and potentially encourages new thinking or opens up new research directions. |
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 550 papers