Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

Tim Schopf; Tobias Schreieder; Akiko Aizawa

cs.CLAug 26, 2026

Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

Authors: Tim Schopf, Tobias Schreieder, Akiko Aizawa

Organizations: National Institute of Informatics, Tokyo, Japan · TU Dresden & ScaDS.AI Dresden/Leipzig, Germany

Abstract

Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as "medium novel". To mitigate this, we propose Think-Probe-Respond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent "medium novelty" bias.

Figures & tables

Figure 1: Overview of our tpr approach. The snowflake ( ) indicates frozen parameters, while the flame ( ) indicates trainable parameters.

	$F_{1}$						MAE	ALI	Recall		Add. Ratio		Hall. Rate
Model	Macro	1	2	3	4	5			KA	NA	KA	NA	KA	NA
Gemini 2.5 Pro	15.6	0.0	6.3	28.8	42.9	0.0	1.0	0.41	64.7	60.6	104.6	96.5	8.2	1.2
Gemini 3 Pro	15.3	9.8	30.9	35.8	0.0	9.8	1.0	0.46	64.9	67.0	101.9	81.2	13.8	3.2
Claude Sonnet 4.5	15.5	0.0	14.6	43.2	19.5	0.0	0.9	0.59	80.0	70.3	150.1	14.8	8.7	1.2
Claude Opus 4.5	17.1	0.0	12.1	41.6	31.9	0.0	0.8	0.61	80.4	68.8	143.4	106.0	8.0	1.2
GPT-5 mini	16.2	0.0	5.1	40.8	34.9	0.0	0.9	0.62	77.3	67.8	144.8	104.8	7.8	1.0

Table 1: Evaluation results of novelty judgments on the rino test set. The reported metrics include

F_{1}

macro averaged and for each rubric category (1-5), mae , Alignment (ALI) , Recall , Additional Ratio (in %), and Hallucination Rate (in %) for Known Aspects (KA) and Novelty Aspects (NA) respectively (for more details on the metrics, see Appendix B ).

Figure 2: Macro-

F_{1}

scores for different approaches and llm on the rino test set.

	Model	Macro	1	2	3	4	5
		$F_{1}$
Non-Reas.	Gemma-3-4B	24.4	27.3	21.5	35.11	29.71	8.3
	Gemma-3-12B	21.8	7.7	16.3	34.1	35.0	15.7
	Gemma-3-27B	25.2	18.2	19.8	28.9	33.5	25.4
	Llama-3.1-8B	23.4	14.3	26.0	36.0	31.2	9.5
	Llama-3.1-70B	20.6	7.7	27.9	33.9	24.1	9.5

Table 2:

F_{1}

scores per novelty class using tpr with different llm .

		Thinking Phase						Response Generation Phase
	Model	$t_{1}$	$t_{25\%}$	$t_{50\%}$	$t_{75\%}$	$t_{n}$	Avg.	$r_{1}$	$r_{25\%}$	$r_{50\%}$	$r_{75\%}$	$r_{n}$	Avg.
Non-Reason.	Gemma-3-4B	22.74	16.26	15.47	16.86	24.38	19.14	20.84	18.09	19.86	17.94	19.81	19.31
	Gemma-3-12B	22.34	22.12	17.60	21.12	21.74	20.98	17.87	16.23	23.73	20.53	17.20	19.11
	Gemma-3-27B	18.88	21.36	18.42	21.51	25.17	21.07	21.90	16.98	15.48	19.83	14.96	17.83
	Llama-3.1-8B	17.01	19.90	19.77	19.94	23.38	20.00	17.92	19.47	17.40	17.37	19.51	18.33
	Llama-3.1-70B	21.11	19.12	20.16	15.42	20.61	19.28	15.82	21.56	18.49	20.78	16.50	18.63

Table 3: Macro

F_{1}

scores for research idea novelty judgments on the rino test set obtained by probing last-layer hidden states of various llm at different generation steps: when generating the first and last think tokens (

t_{1}

t_{n}

), the first and last response tokens (

r_{1}

r_{n}

), and intermediate steps during both the thinking and response generation phases (reported as percentages of token generation within each phase; e.g.,

t_{50\%}

denotes probing halfway through the thinking phase, after 50% of think tokens have been generated).

		# Think Tokens			# Response Tokens
	Model	Min	Max	Avg.	Min	Max	Avg.
Non-Reason.	Gemma-3-4B	380	1572	741.98	21	128	80.69
	Gemma-3-12B	233	1672	631.21	44	105	69.25
	Gemma-3-27B	206	804	431.64	48	107	71.81
	Llama-3.1-8B	17	4358	435.33	21	152	67.79
	Llama-3.1-70B	18	1034	313.76	23	120	57.23

Table 4: Number of tokens generated by different llm during research idea novelty prediction using our tpr approach. Note, we use GPT-OSS-20B with reasoning level “low” , resulting in a small number of think tokens.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Score	Degree of Novelty
1	The idea is not novel. All aspects already exist in prior work.
2	The idea is marginally novel. It represents only a minor variation of existing work.
3	The idea is somewhat novel. Aspects already exist in prior work. However, it might combine known approaches in new ways, apply them to new contexts, or propose incremental updates.
4	The idea is novel. It introduces new aspects not present in existing work.
5	The idea is highly innovative and novel. It is not present in existing work and potentially encourages new thinking or opens up new research directions.

Appendix

Table 5: Novelty Judgment Rubric

	Model	Zero-shot	Few-shot	CoT	Moose	Research Agent	AI Scientist	AI Researcher	FineTune	tpr
Non-Reas.	Gemma-3-4B	1.0	0.9	1.0	0.9	0.9	0.9	0.9	1.1	1.0
	Gemma-3-12B	1.0	0.9	0.9	1.0	0.9	0.9	0.9	1.1	1.1
	Gemma-3-27B	1.0	1.0	0.9	1.0	0.9	1.0	0.9	1.1	1.1
	Llama-3.1-8B	1.0	1.0	1.0	0.9	0.9	0.9	0.9	1.1	1.1
	Llama-3.1-70B	1.0	1.0	1.1	1.0	1.0	1.0	0.9	1.1	1.1
Reas.	Qwen3-4B	1.0	1.0	1.0	1.0	0.9	1.0	1.0	1.0	1.1

Appendix

Table 6: mae scores for different approaches and llm on the rino test set.

		ALI	Recall		Add. Ratio		Hall. Rate
	Model		KA	NA	KA	NA	KA	NA
Non-Reas.	Gemma-3-4B	0.25	39.4	35.7	40.8	51.9	38.8	44.2
	Gemma-3-12B	0.39	58.2	50.4	56.7	72.4	8.6	6.8
	Gemma-3-27B	0.42	62.4	57.2	60.9	80.5	6.5	2.0
	Llama-3.1-8B	0.36	52.0	45.3	40.0	74.1	12.6	10.5
	Llama-3.1-70B	0.30	51.0	50.0	35.1	55.6	12.0	11.1

Appendix

Table 7: Evaluation of textual justifications generated by different llm using tpr .

Gold Novelty	Predicted Novelty	Gold Justification	LLM-generated Justification
$1$	$3$ $\uparrow$	The idea of using neural networks to learn Green’s functions is already known and the proposed contribution is incremental, offering no new aspects beyond existing approaches.	The proposed idea combines fundamental solutions […] represents a somewhat novel synthesis , but the core components […] are well-established in the literature.
$2$	$4$ $\uparrow$	The approach adds […] extensions of existing communication mechanisms rather than fundamentally new concepts, resulting in only marginal novelty .	The research idea proposes […] meaningful novelty […], though it builds on existing concepts in communication […]
$3$	$3$ $\checkmark$	The approach primarily assembles existing components […], resulting in a somewhat novel contribution.	The research idea combines several existing concepts […] adds incremental novelty , […]
$4$	$4$ $\checkmark$	The approach is novel […] extending diffusion models beyond the usual focus on learning only the reverse process. […] has not been presented in prior work […]	[…] introduces a novel approach by jointly parameterizing both forward and reverse diffusion processes […] related works […] primarily focus on the reverse process […]
$5$	$4$ $\downarrow$	The idea is highly novel because it uncovers a previously unreported generalization phenomenon and establishes a new theoretical link […]	[…] calibration literature doesn’t explicitly connect ensemble disagreement to generalization […] builds incrementally on existing calibration and ensemble concepts.

Appendix

Table 8: Selected comparison of gold novelty judgments and LLM(Claude Opus 4.5)-generated novelty judgments. Green denotes alignment of model and gold justifications. Red highlights miscalibration, where the model’s novelty judgment diverges from the gold judgment despite exhibiting a rationale aligned with the gold justification. Symbols indicate novelty overestimation (

\uparrow

), correct prediction (

\checkmark

), and underestimation (

\downarrow

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 529 papers

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas
Mar 11, 2026Tim Schopf, Michael FärberAutomated Evaluation LLM-as-a-Judge
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Oct 1, 2026Noy Sternlicht, Simra Shahid, Peter Jansen +3Prompt Sensitivity LLM-as-a-Judge
Hebrew University of Jerusalem · Allen Institute for AI · Microsoft +2
On the Limits of LLM-as-Judge for Scientific Novelty Assessment
Jun 10, 2026Soumitra Sinhahajari, Navonil Majumder, Soujanya PoriaAI for Science LLM-as-a-Judge
DeCLaRe Lab, Nanyang Technological University

Mar 11, 2026J/K move · Enter open · S save

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

Tim Schopf, Michael Färber

TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan

Automated Evaluation LLM-as-a-Judge LLM Evaluation

Open paper AskPDF arXiv

Judging the novelty of research ideas is crucial for advancing science, enabling the identification of unexplored directions, and ensuring contributions meaningfully extend existing knowledge rather than reiterate minor variations. However, given the exponential growth of scientific literature, manually judging the novelty of research ideas through literature reviews is labor-intensive, subjective, and infeasible at scale. Therefore, recent efforts have proposed automated approaches for research idea novelty judgment. Yet, evaluation of these approaches remains largely inconsistent and is typically based on non-standardized human evaluations, hindering large-scale, comparable evaluations. To address this, we introduce RINoBench, the first comprehensive benchmark for large-scale evaluation of research idea novelty judgments. It comprises 1,381 research ideas derived from and judged by human experts as well as nine automated evaluation metrics designed to assess both rubric-based novelty scores and textual justifications of novelty judgments. Using this benchmark, we evaluate several state-of-the-art large language models (LLMs) on their ability to judge the novelty of research ideas. Our findings reveal that while LLM-generated reasoning closely mirrors human rationales, this alignment does not reliably translate into accurate novelty judgments, which diverge significantly from human gold standard judgments - even among leading reasoning-capable models. Data and code available at: https://github.com/TimSchopf/RINoBench.

Abstract

LLM-as-a-Judge

Explore similar work

Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas

Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

On the Limits of LLM-as-Judge for Scientific Novelty Assessment