Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

Noy Sternlicht; Simra Shahid; Peter Jansen; Daniel S. Weld; Pao Siangliulue; Tom Hope

cs.CLOct 1, 2026

Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

Authors: Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue, Tom Hope

Organizations: Hebrew University of Jerusalem · Allen Institute for AI · Microsoft · University of Arizona · University of Washington

Abstract

Automated ideation systems are often evaluated on the novelty of the ideas they produce, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We first build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.

Figures & tables

Figure 1: We run a systematic controlled study of novelty judges. Seemingly small changes, such as minor prompt variations, can wildly swing performance. Judges can grow markedly more brittle once LLM-generated ideas enter the data ( Human+Generated ), a failure that does not show up on the human-authored distribution they are typically tested on ( Human-Only ).

Figure 2: Automatic data collection process. We mine novelty labels from OpenReview, keeping only submissions whose reviewers agree on their originality, and complement them with weakly labeled lower-novelty ideas from a non-SOTA LLM ideator. The result is an open-source novelty evaluation dataset with two setups, Human-Only and Human+Generated . We instantiate the pipeline on ICLR 2026, but it can be rerun on future conferences.

	Ideas		Eval. instances
	$D_{+}$	$D_{-}$	Pairwise	Pointwise
Human-Only	154	145 ICLR	154	299
Human+Generated	154	154 LLM	154	308

Table 1: Data setups statistics.

D_{+}

holds the novel ideas and

D_{-}

the lower-novelty ones. Both setups share

D_{+}

(

154

ICLR-validated novel ideas) and differ in

D_{-}

: validated lower-novelty ICLR ideas in Human-Only , and LLM-generated ideas in Human+Generated .

Figure 3: Study overview. We cross two judgment formats with six judge backbones and six evaluation dimensions. We run one configuration at a time and report the results on both data setups, Human-Only and Human+Generated .

Judge prompt (reference: requests judges to assess ideas against a novelty criterion defining what counts as novel)
		Protocol		Data setup
Change	Description	Pt	Pw	HO	HG	Fig.
P1: Criteria without guardrails	Weaker criterion, w/o guardrails (e.g., no trivial combinations)					4
P2: Criteria removed	Drop novelty criterion entirely					4
P3: “Which was judged novel?”	Prior review found exactly one idea novel; judge predicts which	–				4(b)
P4: “Which is novel?”	P3 minus the presumption of prior review	–				4(b)

Table 2: Controlled changes. Each change modifies a single design choice relative to the reference configuration, holding the rest of the novelty evaluation process fixed. Columns mark the protocols (Pt: pointwise, Pw: pairwise) and data setups (HO: Human-Only , HG: Human+Generated ) each change is run under. Beyond these single changes, we compare prompted judges to dedicated novelty judges (Fig. 6 ) and to a strong agentic RAG judge (Fig. 5 ).

Panel of Figure 4 — Figure 4: Cells report the change from the reference configuration, with the absolute value in parentheses. claude-sonnet-4-5 exposes no reasoning_effort parameter, hence the blank entry. Judges are highly unstable under even minor configuration changes, with large variance between models.

Figure 5: Stronger retrieval does not help. Pointwise macro-F1 on Human+Generated , for a strong agentic RAG judge ( gpt-5.6-sol with live, iterative web search) against judges using the standard retrieval setup (§ 4 ), which see five pre-retrieved abstracts. All judges run once per idea, without verdict aggregation. Despite a newer backbone, higher reasoning effort, and unlimited search, the agentic judge shows no significant gains.

Figure 6: Cost vs. pointwise macro-F1 for claude-opus-4-6 , comparing prompted-judge configurations against the dedicated novelty judges, in the Human-Only ( left ) and Human+Generated ( right ) settings (error bars: 95% CIs from a paired bootstrap with

10{,}000

resamples). The Pareto front (line) contains only prompted judges: the cheapest prompted configuration outscores both dedicated judges in both settings, at over

30\times

lower cost.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — List of Prompts 1 Novelty signals extraction prompt.

ICLR 2026 submission	Extracted reviewer novelty signal examples
Positive novelty signals
Image Quality Assessment for Embodied AI	“The paper identifies and clearly defines a completely new, critical, and timely research problem ”
Quotient-Space Diffusion Models	“It is the first work I have seen that attempts to constrain the diffusion process within the quotient space ”
Critical Confabulation: Can LLMs Hallucinate for Social Good?	“this paper is so unlike most of the ICLR paper I’ve seen in recent years , especially with its use of historical archival data …”
Negative novelty signals
DFCA: Decentralized Federated Clustering Algorithm	“While the theoretical contribution is not highly novel , the paper provides a useful convergence discussion…”

Appendix

Table 3: Novelty signals extracted from ICLR 2026 reviews.

Panel of Figure 11 — List of Prompts 2 Remove evaluation information prompt.

Panel of Figure 12 — List of Prompts 3 Idea generation prompt.

ICLR area: applications to computer vision, audio, language, and other modalities
0 (validated novel)	Despite advances in pretraining with extended context sizes, large language models (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimization objectives. To address these limitations, we propose a framework named Short-to-Long Preference Optimization (SoLoPO), decoupling long-context preference optimization (PO) into two components: short-context PO and short-to-long reward alignment (SoLo-RA), supported by both theoretical and empirical evidence. Specifically, short-context PO leverages preference pairs sampled from short contexts to enhance the model’s contextual knowledge utilization ability. Meanwhile, SoLo-RA explicitly encourages reward score consistency for the responses when conditioned on both short and long contexts that contain identical task-relevant information. This facilitates transferring the model’s ability to handle short contexts into long-context scenarios. SoLoPO is compatible with mainstream preference optimization algorithms, while substantially improving the efficiency of data construction and training processes.
1 (weakly labeled lower-novelty)	Multimodal understanding requires more than aligning representations across vision, audio, and language—it demands reasoning about temporal precedence and causal relationships between modalities. Existing approaches treat modalities symmetrically through attention mechanisms, failing to capture how information in one modality temporally influences or explains events in another. We introduce Causal Multimodal Transformers, a framework that explicitly models directional causal dependencies across asynchronous modality streams. Our architecture incorporates learned temporal offsets and causal masking patterns that respect the natural information flow between modalities, enabling the model to distinguish between coincidental co-occurrence and genuine causal influence. By decomposing cross-modal attention into temporally-ordered causal graphs, the framework learns which modality provides predictive information for events in others at different time scales. This causally-aware design enhances interpretability and enables counterfactual reasoning—answering questions like “what would the visual scene be if this sound had not occurred?” Applications span video understanding, audio-visual speech processing, and multimodal content generation where temporal causality is fundamental to meaning.
Expert selected: 0	The idea in 1 of causal/temporal dependence between modalities is old. Temporal offsets, causal masks with directional cross-modal attention etc. It’s all a generic shallow combination of these things. e.g., https://arxiv.org/abs/1906.00295 (>3K citations). The idea in 0 of decoupling long-context preference optimization into short and short-to-long seems clever and original, and a very specific method invention.
ICLR area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
0 (weakly labeled lower-novelty)	Representation learning paradigms—unsupervised, self-supervised, semi-supervised, and supervised—are typically treated as distinct methodologies requiring separate architectural designs and training protocols. We introduce Spectrum Learning, a unified framework that views supervision as a continuous spectrum rather than discrete categories. Our approach employs a meta-learned weighting mechanism that dynamically modulates the contribution of multiple learning objectives based on local supervision density in the feature space. By treating each data point as existing along a supervision gradient, the framework adaptively combines contrastive self-supervised losses, consistency regularization, pseudo-labeling, and supervised classification within a single coherent optimization. A novel supervision-aware attention module enables the model to identify which learning paradigm is most informative for different regions of the data manifold. This fluid integration allows seamless transitions as supervision availability changes, from fully unsupervised to fully supervised settings, without architectural modification. Spectrum Learning naturally handles mixed supervision scenarios common in practice, where different samples have varying levels of annotation quality and granularity, providing a principled approach to leveraging all available learning signals simultaneously.
1 (validated novel)	Multi-view clustering integrates the consistency and complementarity of different views to achieve unsupervised data grouping. Existing multi-view clustering methods primarily confront two challenges: i) they generally perform feature extraction in the feature domain, which is sensitive to noise and may neglect cluster-specific information that is indistinguishable in the original space; ii) current dynamic fusion methods adopt static strategies to learn weights, lacking capability to adjust strategies adaptively under complex scenarios according to variations in data distribution and view quality. To address these issues, we propose a large language model assisted dynamic agent for multi-view clustering (LLM-DAMVC), a novel framework that recasts multi-view clustering as a dynamic decision-making problem orchestrated by a large language model. Specifically, each view is equipped with complementary agents dedicated to feature extraction. A dual-domain contrastive module is introduced to optimize feature consistency and enhance cluster separability in both the feature domain and frequency domain. Additionally, an LLM-assisted view fusion mechanism provides a flexible fusion weight learning strategy that can be adaptively applied to complex scenarios and significantly different views.

Appendix

Table 4: Expert annotation examples. Representative pairs from the blind annotation study on Human+Generated , alongside the rater’s written justification.

Panel of Figure 14 — List of Prompts 4 Reference pairwise judge prompt , shown exactly as sent to the judge apart from the two IDEA placeholders, which hold the ideas under comparison.

Panel of Figure 15 — List of Prompts 5 Reference pointwise judge prompt. The pointwise counterpart of Prompt 4 .

Panel of Figure 16 — List of Prompts 6 Strong retrieval judge prompt. The judge searches the literature itself instead of being handed a fixed candidate list.

Figure 7: Cost vs. pointwise macro-F1 for gpt-5.4 , the counterpart of Figure 6 , comparing prompted-judge configurations against the dedicated novelty judges, in the Human-Only ( left ) and Human+Generated ( right ) settings. The same pattern holds: every prompted configuration outscores both dedicated judges, at roughly

6\times

lower cost (error bars: 95% CIs from a paired bootstrap with

10{,}000

resamples).

Evaluator	Judge	$k$	F 1	F ${}_{1}^{+}$	F ${}_{1}^{-}$	Cost (USD)
Human-Only
AI-Scientist	claude-opus-4-6	1	$0.652_{\pm 0.055}$	$0.743_{\pm 0.050}$	$0.561_{\pm 0.079}$	77.91
AI-Scientist	gpt-5.4	1	$0.481_{\pm 0.056}$	$0.293_{\pm 0.084}$	$0.668_{\pm 0.053}$	27.41
Idea Novelty Checker	claude-opus-4-6	1	$0.560_{\pm 0.059}$	$0.711_{\pm 0.050}$	$0.408_{\pm 0.087}$	30.11
Idea Novelty Checker	gpt-5.4	1	$0.521_{\pm 0.057}$	$0.701_{\pm 0.050}$	$0.342_{\pm 0.088}$	29.98
Prompted (no aggregation)	claude-opus-4-6	1	$0.782_{\pm 0.047}$	$0.792_{\pm 0.051}$	$0.772_{\pm 0.054}$	4.56

Appendix

Table 5: Pointwise results and cost for every judge configuration, in the Human-Only ( top ) and Human+Generated ( bottom ) settings.

k

is the number of judge calls per decision, F 1 is macro-F1, and F

{}_{1}^{+}

and F

{}_{1}^{-}

are the per-class scores on novel and non-novel ideas. Cost is the total USD spent judging the full set. Best macro-F1 per setting in bold . Subscripts are the half-width of the 95% bootstrap confidence interval over test instances, from 10,000 resamples. These numbers underlie Figures 6 and 7 .

Panel of Figure 19 — List of Prompts 7 P1: Criteria without guardrails, diff against Prompts 4 and 5 . P1 deletes the three paragraphs shown, which constrain what counts as novel.

Panel of Figure 20 — List of Prompts 8 P2: Criteria removed, pairwise.

Panel of Figure 21 — List of Prompts 9 P2: Criteria removed, pointwise.

Panel of Figure 22 — List of Prompts 10 P3: “Which was judged novel?” , adapted from Si et al. (2024) , whose original prompt asks which of two abstracts was accepted at a conference.

Panel of Figure 23 — List of Prompts 11 P4: “Which is novel?” , which drops P3’s reviewer and venue framing.

Panel of Figure 24 — List of Prompts 12 P5: “Which was judged more novel?” , which replaces P3’s binary novel/not-novel contrast with a graded one.

Panel of Figure 25 — List of Prompts 13 Research plan extraction prompt.

Panel of Figure 26 — List of Prompts 14 Retrieval contribution prompt.

Panel of Figure 27 — List of Prompts 15 Retrieval query generation prompt.

Panel of Figure 28 — List of Prompts 16 Retrieval, diff against Prompts 4 and 5 . Top : the retrieved abstracts are appended to each idea inside a delimited block, shown here for the pairwise reference prompt; the pointwise reference prompt takes a single such block, with the delimiters reading [Start Related Work] and [End Related Work] . Bottom : the criterion picks up the three highlighted insertions, so that it asks about originality relative to the supplied prior work. The elided paragraphs and the rest of the prompt are unchanged.

Figure 8: Pairwise ablation results under soft -accuracy, where ties count as half-correct. Each cell reports the change relative to the reference configuration (top row), with the resulting absolute soft-accuracy in parentheses. Asterisks mark statistically significant changes.

Figure 10: Tie rates across controlled configurations . Each point is one configuration of Figure 4(b) for a given judge (roughly

20

data points per judge), with the vertical bar marking the median per setting. Every judge ties more often in the Human+Generated setting than in the Human-Only one. Exact per-configuration values are in Table 6 .

Configuration	son-4-5	op-4-5	op-4-6	5.1	5.2	5.4	son-4-5	op-4-5	op-4-6	5.1	5.2	5.4
	Human-Only						Human+Generated
Reference	12.3	4.5	0.6	6.5	3.2	4.5	35.1	18.2	3.2	7.1	2.6	5.2
P1 $\cdot$ Criteria without guardrails	29.2	5.2	3.2	1.9	3.2	1.9	52.6	21.4	5.2	7.8	8.4	3.9
P2 $\cdot$ Criteria removed	27.3	11.0	4.5	2.6	2.6	3.9	57.8	24.7	7.1	6.5	3.9	2.6
P3 $\cdot$ “Which was judged novel?”	12.3	11.7	11.0	1.9	4.5	4.5	11.7	6.5	0.0	6.5	5.2	2.6
P4 $\cdot$ “Which is novel?”	12.3	24.7	35.7	6.5	5.8	4.5	16.2	37.0	26.6	7.1	4.5	9.7

Appendix

Table 6: Tie rates (%) across controlled configurations , the numbers behind Figure 10 . Judges are abbreviated: son-4-5 is claude-sonnet-4-5 , op-4-5 is claude-opus-4-5 , op-4-6 is claude-opus-4-6 , 5.1 is gpt-5.1 , 5.2 is gpt-5.2 , 5.4 is gpt-5.4 . Judges are substantially more uncertain on the Human+Generated set.

Figure 12: Prompt variants with ideas as research plans , pairwise strict-accuracy (ties count as failures). The first row applies the plan format alone, and rows P3–P5 combine it with the corresponding judge prompt. Scores are generally lower than with abstracts, but the effects of the prompt variants keep their direction.

Figure 13: Effect of the lower-novelty ideas source. Judge performance when the pool of high-novelty, human-authored ideas (

D_{+}

) is held fixed and the pool of lower-novelty ideas (

D_{-}

) is regenerated by each of four ideation backbones. Human-Only , where

D_{-}

holds human-authored ideas criticized for lacking novelty, is shown as a reference. Judge performance drops once the low-novelty side is LLM-generated, falling below chance level in some instances.

Figure 14: Retrieval across ideation backbones. Effect of giving the judge retrieved related work, with

D_{+}

held fixed and

D_{-}

regenerated by each of four ideation backbones (rows), for one judge per model family (columns). Retrieval yields small and inconsistent effects that do not agree in sign across the two protocols, and it leaves the hardest backbones far below chance.

Idea (

D_{+}

): “…Multi-Agent System (MAS) powered by Visual Language Models … suffers from a novel failure term, multi-agent visual hallucination snowballing, where hallucinations are seeded in a single agent and amplified by following ones… we propose ViF, a lightweight, model-agnostic mitigation paradigm…”

Reviewers : “The authors introduce a new task setting for MAS hallucination and create a multi-turn evaluation metric which could be beneficial for future researchers.”

Without retrieval : “…although several ingredients have precedents, the problem setting plus the specific diagnosis-and-mitigation package appears substantively original…”

\rightarrow

novel

Retrieved : Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models (2024)…

With retrieval : “…although the MAS-specific framing is somewhat new, the main concept, analysis direction, and mitigation ingredients are all strongly anticipated by the provided related work…”

\rightarrow

not novel

Idea (

D_{+}

): “…we revisit Sobolev integral probability metrics (IPM) on trees to obtain a practical generalization of TSW…”

Reviewers : “The idea of combining tree-sliced Wasserstein with the Sobolev IPM to handle the general

p

-Wasserstein distance is novel”

Appendix

Table 7: Examples of retrieval hurting gpt-5.4 ( Human+Generated , pointwise). In the first row, a single close precursor outweighs the contribution the reviewers valued. In the second, the judge matches each component to a different retrieved paper and dismisses the combination the reviewers called novel. In the third, the retrieved papers override the judge’s own recognition that the idea is a broad composition of known components: since no single retrieved paper covers the idea in full, the judge concludes it is novel.

Explore similar work

Jun 10, 2026cs.DL

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.

Soumitra Sinhahajari, Navonil Majumder, Soujanya Poria

DeCLaRe Lab, Nanyang Technological University

Scientific Ideation Llm-As-A-Judge Scientific Discovery Large Language Model Judges Assessment

Ask PDF

Aug 30, 2026cs.AI

Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

Evaluating research ideas generated by LLMs is difficult because their scientific value cannot be fully determined by objective criteria, and no single reference answer specifies what counts as a good idea. To address this challenge, we introduce Ideation Arena, a battle style platform that evaluates research ideas through pairwise human assessment. Ideation Arena evaluates ideas generated by 14 frontier LLMs and 5 research agent architectures built on 2 base models. To ensure a common starting point, Ideation Arena builds shared literature contexts from papers familiar to the participating researchers and provides the same contexts to all LLMs and agents. We collect over 6,000 double blind pairwise comparisons from 105 active computer science researchers and construct an Elo rating leaderboard of proposal-stage expert preferences in computer science under a shared closed-context protocol. We validate the rankings through interrater agreement and robustness analyses, showing that the leaderboard remains stable under changes in annotator composition and domain coverage. Our results show substantial variation in agent effectiveness, with some frameworks improving ideation quality over their backbones and others offering little benefit or even underperforming their base models. We further construct Ideation Arena Eval, a benchmark for assessing whether automated evaluators align with human preferences in research ideation. Experiments with current LLM judges show that they still cannot reliably reproduce expert preferences, with the best judge reaching 72.56% Soft Accuracy on Overall Quality. Our code, data, and leaderboards are available at https://github.com/foss12138/Research-Ideation-Arena.

Zhiyu Chen, Keyu Zhao, Jigao Fu +8

1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1

Scientific Ideation Assessment Human Annotators Experts

Ask PDF

Apr 22, 2026cs.HC

The Effect of Idea Elaboration on the Automatic Assessment of Idea Originality

Automatic systems are increasingly used to assess the originality of responses in creative tasks. They offer a potential solution to key limitations of human assessment (cost, fatigue, and subjectivity), but there is preliminary evidence of a self-preference bias. Accordingly, automatic systems tend to prefer outcomes that are more closely related to their style, rather than to the human one. In this paper, we investigated how Large Language Models (LLMs) align with human raters in assessing the originality of responses in a divergent thinking task. We analysed 4,813 responses to the Alternate Uses Task produced by higher and lower creative humans and ChatGPT-4o. Human raters were two university students who underwent intensive training. Machine raters were two specialised systems fine-tuned on AUT responses and corresponding human ratings (OCSAI and CLAUS) and ChatGPT-4o, which was prompted with the same instructions as human raters. Results confirmed the presence of a self-preference bias in LLMs. Automatic systems tended to privilege artificial responses. However, this self-preference bias disappeared when the analyses controlled for the idea elaboration. We discuss theoretical and methodological implications of these findings by highlighting future directions for research on creativity assessment.

Umberto Domanti, Moritz Mock, Sergio Agnoli +1

Free University of Bozen-Bolzano · University of Trieste · Marconi Institute for Creativity

Creativity Rater Code Completion Impact Analysis

Ask PDF