Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as "medium novel". To mitigate this, we propose Think-Probe-Respond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent "medium novelty" bias.
Figures & tables
Figure 1: Overview of our tpr approach. The snowflake ( ) indicates frozen parameters, while the flame ( ) indicates trainable parameters.
F1
MAE
ALI
Recall
Add. Ratio
Hall. Rate
Model
Macro
1
2
3
4
5
KA
NA
KA
NA
KA
NA
Gemini 2.5 Pro
15.6
0.0
6.3
28.8
42.9
0.0
1.0
0.41
64.7
60.6
104.6
96.5
8.2
1.2
Gemini 3 Pro
15.3
9.8
30.9
35.8
0.0
9.8
1.0
0.46
64.9
67.0
101.9
81.2
13.8
3.2
Claude Sonnet 4.5
15.5
0.0
14.6
43.2
19.5
0.0
0.9
0.59
80.0
70.3
150.1
14.8
8.7
1.2
Claude Opus 4.5
17.1
0.0
12.1
41.6
31.9
0.0
0.8
0.61
80.4
68.8
143.4
106.0
8.0
1.2
GPT-5 mini
16.2
0.0
5.1
40.8
34.9
0.0
0.9
0.62
77.3
67.8
144.8
104.8
7.8
1.0
Table 1: Evaluation results of novelty judgments on the rino test set. The reported metrics include F1 macro averaged and for each rubric category (1-5), mae , Alignment (ALI) , Recall , Additional Ratio (in %), and Hallucination Rate (in %) for Known Aspects (KA) and Novelty Aspects (NA) respectively (for more details on the metrics, see Appendix B ).
Figure 2: Macro- F1 scores for different approaches and llm on the rino test set.
F1
Model
Macro
1
2
3
4
5
Non-Reas.
Gemma-3-4B
24.4
27.3
21.5
35.11
29.71
8.3
Gemma-3-12B
21.8
7.7
16.3
34.1
35.0
15.7
Gemma-3-27B
25.2
18.2
19.8
28.9
33.5
25.4
Llama-3.1-8B
23.4
14.3
26.0
36.0
31.2
9.5
Llama-3.1-70B
20.6
7.7
27.9
33.9
24.1
9.5
Table 2: F1 scores per novelty class using tpr with different llm .
Thinking Phase
Response Generation Phase
Model
t1
t25%
t50%
t75%
tn
Avg.
r1
r25%
r50%
r75%
rn
Avg.
Non-Reason.
Gemma-3-4B
22.74
16.26
15.47
16.86
24.38
19.14
20.84
18.09
19.86
17.94
19.81
19.31
Gemma-3-12B
22.34
22.12
17.60
21.12
21.74
20.98
17.87
16.23
23.73
20.53
17.20
19.11
Gemma-3-27B
18.88
21.36
18.42
21.51
25.17
21.07
21.90
16.98
15.48
19.83
14.96
17.83
Llama-3.1-8B
17.01
19.90
19.77
19.94
23.38
20.00
17.92
19.47
17.40
17.37
19.51
18.33
Llama-3.1-70B
21.11
19.12
20.16
15.42
20.61
19.28
15.82
21.56
18.49
20.78
16.50
18.63
Table 3: Macro F1 scores for research idea novelty judgments on the rino test set obtained by probing last-layer hidden states of various llm at different generation steps: when generating the first and last think tokens ( t1 , tn ), the first and last response tokens ( r1 , rn ), and intermediate steps during both the thinking and response generation phases (reported as percentages of token generation within each phase; e.g., t50% denotes probing halfway through the thinking phase, after 50% of think tokens have been generated).
# Think Tokens
# Response Tokens
Model
Min
Max
Avg.
Min
Max
Avg.
Non-Reason.
Gemma-3-4B
380
1572
741.98
21
128
80.69
Gemma-3-12B
233
1672
631.21
44
105
69.25
Gemma-3-27B
206
804
431.64
48
107
71.81
Llama-3.1-8B
17
4358
435.33
21
152
67.79
Llama-3.1-70B
18
1034
313.76
23
120
57.23
Table 4: Number of tokens generated by different llm during research idea novelty prediction using our tpr approach. Note, we use GPT-OSS-20B with reasoning level “low” , resulting in a small number of think tokens.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Score
Degree of Novelty
1
The idea is not novel. All aspects already exist in prior work.
2
The idea is marginally novel. It represents only a minor variation of existing work.
3
The idea is somewhat novel. Aspects already exist in prior work. However, it might combine known approaches in new ways, apply them to new contexts, or propose incremental updates.
4
The idea is novel. It introduces new aspects not present in existing work.
5
The idea is highly innovative and novel. It is not present in existing work and potentially encourages new thinking or opens up new research directions.
Appendix
Table 5: Novelty Judgment Rubric
Model
Zero-shot
Few-shot
CoT
Moose
Research Agent
AI Scientist
AI Researcher
FineTune
tpr
Non-Reas.
Gemma-3-4B
1.0
0.9
1.0
0.9
0.9
0.9
0.9
1.1
1.0
Gemma-3-12B
1.0
0.9
0.9
1.0
0.9
0.9
0.9
1.1
1.1
Gemma-3-27B
1.0
1.0
0.9
1.0
0.9
1.0
0.9
1.1
1.1
Llama-3.1-8B
1.0
1.0
1.0
0.9
0.9
0.9
0.9
1.1
1.1
Llama-3.1-70B
1.0
1.0
1.1
1.0
1.0
1.0
0.9
1.1
1.1
Reas.
Qwen3-4B
1.0
1.0
1.0
1.0
0.9
1.0
1.0
1.0
1.1
Appendix
Table 6: mae scores for different approaches and llm on the rino test set.
ALI
Recall
Add. Ratio
Hall. Rate
Model
KA
NA
KA
NA
KA
NA
Non-Reas.
Gemma-3-4B
0.25
39.4
35.7
40.8
51.9
38.8
44.2
Gemma-3-12B
0.39
58.2
50.4
56.7
72.4
8.6
6.8
Gemma-3-27B
0.42
62.4
57.2
60.9
80.5
6.5
2.0
Llama-3.1-8B
0.36
52.0
45.3
40.0
74.1
12.6
10.5
Llama-3.1-70B
0.30
51.0
50.0
35.1
55.6
12.0
11.1
Appendix
Table 7: Evaluation of textual justifications generated by different llm using tpr .
Gold Novelty
Predicted Novelty
Gold Justification
LLM-generated Justification
1
3↑
The idea of using neural networks to learn Green’s functions is already known and the proposed contribution is incremental, offering no new aspects beyond existing approaches.
The proposed idea combines fundamental solutions […] represents a somewhat novel synthesis , but the core components […] are well-established in the literature.
2
4↑
The approach adds […] extensions of existing communication mechanisms rather than fundamentally new concepts, resulting in only marginal novelty .
The research idea proposes […] meaningful novelty […], though it builds on existing concepts in communication […]
3
3✓
The approach primarily assembles existing components […], resulting in a somewhat novel contribution.
The research idea combines several existing concepts […] adds incremental novelty , […]
4
4✓
The approach is novel […] extending diffusion models beyond the usual focus on learning only the reverse process. […] has not been presented in prior work […]
[…] introduces a novel approach by jointly parameterizing both forward and reverse diffusion processes […] related works […] primarily focus on the reverse process […]
5
4↓
The idea is highly novel because it uncovers a previously unreported generalization phenomenon and establishes a new theoretical link […]
[…] calibration literature doesn’t explicitly connect ensemble disagreement to generalization […] builds incrementally on existing calibration and ensemble concepts.
Appendix
Table 8: Selected comparison of gold novelty judgments and LLM(Claude Opus 4.5)-generated novelty judgments. Green denotes alignment of model and gold justifications. Red highlights miscalibration, where the model’s novelty judgment diverges from the gold judgment despite exhibiting a rationale aligned with the gold justification. Symbols indicate novelty overestimation ( ↑ ), correct prediction ( ✓ ), and underestimation ( ↓ ).
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.
Tim Schopf, Michael Färber
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan
AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that seemingly small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.
Noy Sternlicht, Simra Shahid, Peter Jansen +3
Hebrew University of Jerusalem · Allen Institute for AI · Microsoft +2
LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.