cs.CLOct 1, 2026
SaveOld Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Organizations: Hebrew University of Jerusalem · Allen Institute for AI · Microsoft · University of Arizona · University of Washington
Abstract
Automated ideation systems are often evaluated on the novelty of the ideas they produce, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We first build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.
Figures & tables
Figure 1: We run a systematic controlled study of novelty judges. Seemingly small changes, such as minor prompt variations, can wildly swing performance. Judges can grow markedly more brittle once LLM-generated ideas enter the data ( Human+Generated ), a failure that does not show up on the human-authored distribution they are typically tested on ( Human-Only ).
Figure 2: Automatic data collection process. We mine novelty labels from OpenReview, keeping only submissions whose reviewers agree on their originality, and complement them with weakly labeled lower-novelty ideas from a non-SOTA LLM ideator. The result is an open-source novelty evaluation dataset with two setups, Human-Only and Human+Generated . We instantiate the pipeline on ICLR 2026, but it can be rerun on future conferences.
| Ideas | Eval. instances | |||
|---|---|---|---|---|
| Pairwise | Pointwise | |||
| Human-Only | 154 | 145 ICLR | 154 | 299 |
| Human+Generated | 154 | 154 LLM | 154 | 308 |
Table 1: Data setups statistics. holds the novel ideas and the lower-novelty ones. Both setups share ( ICLR-validated novel ideas) and differ in : validated lower-novelty ICLR ideas in Human-Only , and LLM-generated ideas in Human+Generated .
Figure 3: Study overview. We cross two judgment formats with six judge backbones and six evaluation dimensions. We run one configuration at a time and report the results on both data setups, Human-Only and Human+Generated .
| Protocol | Data setup | |||||
| Change | Description | Pt | Pw | HO | HG | Fig. |
| Judge prompt (reference: requests judges to assess ideas against a novelty criterion defining what counts as novel) | ||||||
| P1: Criteria without guardrails | Weaker criterion, w/o guardrails (e.g., no trivial combinations) | 4 | ||||
| P2: Criteria removed | Drop novelty criterion entirely | 4 | ||||
| P3: “Which was judged novel?” | Prior review found exactly one idea novel; judge predicts which | – | 4(b) | |||
| P4: “Which is novel?” | P3 minus the presumption of prior review | – | 4(b) | |||
Table 2: Controlled changes. Each change modifies a single design choice relative to the reference configuration, holding the rest of the novelty evaluation process fixed. Columns mark the protocols (Pt: pointwise, Pw: pairwise) and data setups (HO: Human-Only , HG: Human+Generated ) each change is run under. Beyond these single changes, we compare prompted judges to dedicated novelty judges (Fig. 6 ) and to a strong agentic RAG judge (Fig. 5 ).
Figure 4: Cells report the change from the reference configuration, with the absolute value in parentheses. claude-sonnet-4-5 exposes no reasoning_effort parameter, hence the blank entry. Judges are highly unstable under even minor configuration changes, with large variance between models.
Figure 5: Stronger retrieval does not help. Pointwise macro-F1 on Human+Generated , for a strong agentic RAG judge ( gpt-5.6-sol with live, iterative web search) against judges using the standard retrieval setup (§ 4 ), which see five pre-retrieved abstracts. All judges run once per idea, without verdict aggregation. Despite a newer backbone, higher reasoning effort, and unlimited search, the agentic judge shows no significant gains.
Figure 6: Cost vs. pointwise macro-F1 for claude-opus-4-6 , comparing prompted-judge configurations against the dedicated novelty judges, in the Human-Only ( left ) and Human+Generated ( right ) settings (error bars: 95% CIs from a paired bootstrap with resamples). The Pareto front (line) contains only prompted judges: the cheapest prompted configuration outscores both dedicated judges in both settings, at over lower cost.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
List of Prompts 1 Novelty signals extraction prompt.
| ICLR 2026 submission | Extracted reviewer novelty signal examples |
|---|---|
| Positive novelty signals | |
| Image Quality Assessment for Embodied AI | “The paper identifies and clearly defines a completely new, critical, and timely research problem ” |
| Quotient-Space Diffusion Models | “It is the first work I have seen that attempts to constrain the diffusion process within the quotient space ” |
| Critical Confabulation: Can LLMs Hallucinate for Social Good? | “this paper is so unlike most of the ICLR paper I’ve seen in recent years , especially with its use of historical archival data …” |
| Negative novelty signals | |
| DFCA: Decentralized Federated Clustering Algorithm | “While the theoretical contribution is not highly novel , the paper provides a useful convergence discussion…” |
Table 3: Novelty signals extracted from ICLR 2026 reviews.
List of Prompts 2 Remove evaluation information prompt.
List of Prompts 3 Idea generation prompt.
| ICLR area: applications to computer vision, audio, language, and other modalities | |
| 0 (validated novel) | Despite advances in pretraining with extended context sizes, large language models (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimization objectives. To address these limitations, we propose a framework named Short-to-Long Preference Optimization (SoLoPO), decoupling long-context preference optimization (PO) into two components: short-context PO and short-to-long reward alignment (SoLo-RA), supported by both theoretical and empirical evidence. Specifically, short-context PO leverages preference pairs sampled from short contexts to enhance the model’s contextual knowledge utilization ability. Meanwhile, SoLo-RA explicitly encourages reward score consistency for the responses when conditioned on both short and long contexts that contain identical task-relevant information. This facilitates transferring the model’s ability to handle short contexts into long-context scenarios. SoLoPO is compatible with mainstream preference optimization algorithms, while substantially improving the efficiency of data construction and training processes. |
| 1 (weakly labeled lower-novelty) | Multimodal understanding requires more than aligning representations across vision, audio, and language—it demands reasoning about temporal precedence and causal relationships between modalities. Existing approaches treat modalities symmetrically through attention mechanisms, failing to capture how information in one modality temporally influences or explains events in another. We introduce Causal Multimodal Transformers, a framework that explicitly models directional causal dependencies across asynchronous modality streams. Our architecture incorporates learned temporal offsets and causal masking patterns that respect the natural information flow between modalities, enabling the model to distinguish between coincidental co-occurrence and genuine causal influence. By decomposing cross-modal attention into temporally-ordered causal graphs, the framework learns which modality provides predictive information for events in others at different time scales. This causally-aware design enhances interpretability and enables counterfactual reasoning—answering questions like “what would the visual scene be if this sound had not occurred?” Applications span video understanding, audio-visual speech processing, and multimodal content generation where temporal causality is fundamental to meaning. |
| Expert selected: 0 | The idea in 1 of causal/temporal dependence between modalities is old. Temporal offsets, causal masks with directional cross-modal attention etc. It’s all a generic shallow combination of these things. e.g., https://arxiv.org/abs/1906.00295 (>3K citations). The idea in 0 of decoupling long-context preference optimization into short and short-to-long seems clever and original, and a very specific method invention. |
| ICLR area: unsupervised, self-supervised, semi-supervised, and supervised representation learning | |
| 0 (weakly labeled lower-novelty) | Representation learning paradigms—unsupervised, self-supervised, semi-supervised, and supervised—are typically treated as distinct methodologies requiring separate architectural designs and training protocols. We introduce Spectrum Learning, a unified framework that views supervision as a continuous spectrum rather than discrete categories. Our approach employs a meta-learned weighting mechanism that dynamically modulates the contribution of multiple learning objectives based on local supervision density in the feature space. By treating each data point as existing along a supervision gradient, the framework adaptively combines contrastive self-supervised losses, consistency regularization, pseudo-labeling, and supervised classification within a single coherent optimization. A novel supervision-aware attention module enables the model to identify which learning paradigm is most informative for different regions of the data manifold. This fluid integration allows seamless transitions as supervision availability changes, from fully unsupervised to fully supervised settings, without architectural modification. Spectrum Learning naturally handles mixed supervision scenarios common in practice, where different samples have varying levels of annotation quality and granularity, providing a principled approach to leveraging all available learning signals simultaneously. |
| 1 (validated novel) | Multi-view clustering integrates the consistency and complementarity of different views to achieve unsupervised data grouping. Existing multi-view clustering methods primarily confront two challenges: i) they generally perform feature extraction in the feature domain, which is sensitive to noise and may neglect cluster-specific information that is indistinguishable in the original space; ii) current dynamic fusion methods adopt static strategies to learn weights, lacking capability to adjust strategies adaptively under complex scenarios according to variations in data distribution and view quality. To address these issues, we propose a large language model assisted dynamic agent for multi-view clustering (LLM-DAMVC), a novel framework that recasts multi-view clustering as a dynamic decision-making problem orchestrated by a large language model. Specifically, each view is equipped with complementary agents dedicated to feature extraction. A dual-domain contrastive module is introduced to optimize feature consistency and enhance cluster separability in both the feature domain and frequency domain. Additionally, an LLM-assisted view fusion mechanism provides a flexible fusion weight learning strategy that can be adaptively applied to complex scenarios and significantly different views. |
Table 4: Expert annotation examples. Representative pairs from the blind annotation study on Human+Generated , alongside the rater’s written justification.
List of Prompts 4 Reference pairwise judge prompt , shown exactly as sent to the judge apart from the two IDEA placeholders, which hold the ideas under comparison.
List of Prompts 5 Reference pointwise judge prompt. The pointwise counterpart of Prompt 4 .
List of Prompts 6 Strong retrieval judge prompt. The judge searches the literature itself instead of being handed a fixed candidate list.
Figure 7: Cost vs. pointwise macro-F1 for gpt-5.4 , the counterpart of Figure 6 , comparing prompted-judge configurations against the dedicated novelty judges, in the Human-Only ( left ) and Human+Generated ( right ) settings. The same pattern holds: every prompted configuration outscores both dedicated judges, at roughly lower cost (error bars: 95% CIs from a paired bootstrap with resamples).
| Evaluator | Judge | F 1 | F | F | Cost (USD) | |
| Human-Only | ||||||
| AI-Scientist | claude-opus-4-6 | 1 | 77.91 | |||
| AI-Scientist | gpt-5.4 | 1 | 27.41 | |||
| Idea Novelty Checker | claude-opus-4-6 | 1 | 30.11 | |||
| Idea Novelty Checker | gpt-5.4 | 1 | 29.98 | |||
| Prompted (no aggregation) | claude-opus-4-6 | 1 | 4.56 | |||
Table 5: Pointwise results and cost for every judge configuration, in the Human-Only ( top ) and Human+Generated ( bottom ) settings. is the number of judge calls per decision, F 1 is macro-F1, and F and F are the per-class scores on novel and non-novel ideas. Cost is the total USD spent judging the full set. Best macro-F1 per setting in bold . Subscripts are the half-width of the 95% bootstrap confidence interval over test instances, from 10,000 resamples. These numbers underlie Figures 6 and 7 .
List of Prompts 7 P1: Criteria without guardrails, diff against Prompts 4 and 5 . P1 deletes the three paragraphs shown, which constrain what counts as novel.
List of Prompts 8 P2: Criteria removed, pairwise.
List of Prompts 9 P2: Criteria removed, pointwise.
List of Prompts 10 P3: “Which was judged novel?” , adapted from Si et al. (2024) , whose original prompt asks which of two abstracts was accepted at a conference.
List of Prompts 11 P4: “Which is novel?” , which drops P3’s reviewer and venue framing.
List of Prompts 12 P5: “Which was judged more novel?” , which replaces P3’s binary novel/not-novel contrast with a graded one.
List of Prompts 13 Research plan extraction prompt.
List of Prompts 14 Retrieval contribution prompt.
List of Prompts 15 Retrieval query generation prompt.
List of Prompts 16 Retrieval, diff against Prompts 4 and 5 . Top : the retrieved abstracts are appended to each idea inside a delimited block, shown here for the pairwise reference prompt; the pointwise reference prompt takes a single such block, with the delimiters reading [Start Related Work] and [End Related Work] . Bottom : the criterion picks up the three highlighted insertions, so that it asks about originality relative to the supplied prior work. The elided paragraphs and the rest of the prompt are unchanged.
Figure 8: Pairwise ablation results under soft -accuracy, where ties count as half-correct. Each cell reports the change relative to the reference configuration (top row), with the resulting absolute soft-accuracy in parentheses. Asterisks mark statistically significant changes.
Figure 9: Per-class decomposition of the pointwise ablation results. Each cell reports the change relative to the reference configuration (top row), with the resulting absolute F1 in parentheses. Asterisks mark statistically significant changes.
Figure 10: Tie rates across controlled configurations . Each point is one configuration of Figure 4(b) for a given judge (roughly data points per judge), with the vertical bar marking the median per setting. Every judge ties more often in the Human+Generated setting than in the Human-Only one. Exact per-configuration values are in Table 6 .
| Human-Only | Human+Generated | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Configuration | son-4-5 | op-4-5 | op-4-6 | 5.1 | 5.2 | 5.4 | son-4-5 | op-4-5 | op-4-6 | 5.1 | 5.2 | 5.4 |
| Reference | 12.3 | 4.5 | 0.6 | 6.5 | 3.2 | 4.5 | 35.1 | 18.2 | 3.2 | 7.1 | 2.6 | 5.2 |
| P1 Criteria without guardrails | 29.2 | 5.2 | 3.2 | 1.9 | 3.2 | 1.9 | 52.6 | 21.4 | 5.2 | 7.8 | 8.4 | 3.9 |
| P2 Criteria removed | 27.3 | 11.0 | 4.5 | 2.6 | 2.6 | 3.9 | 57.8 | 24.7 | 7.1 | 6.5 | 3.9 | 2.6 |
| P3 “Which was judged novel?” | 12.3 | 11.7 | 11.0 | 1.9 | 4.5 | 4.5 | 11.7 | 6.5 | 0.0 | 6.5 | 5.2 | 2.6 |
| P4 “Which is novel?” | 12.3 | 24.7 | 35.7 | 6.5 | 5.8 | 4.5 | 16.2 | 37.0 | 26.6 | 7.1 | 4.5 | 9.7 |
Table 6: Tie rates (%) across controlled configurations , the numbers behind Figure 10 . Judges are abbreviated: son-4-5 is claude-sonnet-4-5 , op-4-5 is claude-opus-4-5 , op-4-6 is claude-opus-4-6 , 5.1 is gpt-5.1 , 5.2 is gpt-5.2 , 5.4 is gpt-5.4 . Judges are substantially more uncertain on the Human+Generated set.
Figure 11: Results without verdict aggregation . Cells report the change from the reference configuration, with the absolute value in parentheses. The reference row itself still aggregates. claude-sonnet-4-5 exposes no reasoning_effort parameter, hence the blank entry.
Figure 12: Prompt variants with ideas as research plans , pairwise strict-accuracy (ties count as failures). The first row applies the plan format alone, and rows P3–P5 combine it with the corresponding judge prompt. Scores are generally lower than with abstracts, but the effects of the prompt variants keep their direction.
Figure 13: Effect of the lower-novelty ideas source. Judge performance when the pool of high-novelty, human-authored ideas ( ) is held fixed and the pool of lower-novelty ideas ( ) is regenerated by each of four ideation backbones. Human-Only , where holds human-authored ideas criticized for lacking novelty, is shown as a reference. Judge performance drops once the low-novelty side is LLM-generated, falling below chance level in some instances.
Figure 14: Retrieval across ideation backbones. Effect of giving the judge retrieved related work, with held fixed and regenerated by each of four ideation backbones (rows), for one judge per model family (columns). Retrieval yields small and inconsistent effects that do not agree in sign across the two protocols, and it leaves the hardest backbones far below chance.
| Idea ( ): “…Multi-Agent System (MAS) powered by Visual Language Models … suffers from a novel failure term, multi-agent visual hallucination snowballing, where hallucinations are seeded in a single agent and amplified by following ones… we propose ViF, a lightweight, model-agnostic mitigation paradigm…” |
| Reviewers : “The authors introduce a new task setting for MAS hallucination and create a multi-turn evaluation metric which could be beneficial for future researchers.” |
| Without retrieval : “…although several ingredients have precedents, the problem setting plus the specific diagnosis-and-mitigation package appears substantively original…” novel |
| Retrieved : Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models (2024)… |
| With retrieval : “…although the MAS-specific framing is somewhat new, the main concept, analysis direction, and mitigation ingredients are all strongly anticipated by the provided related work…” not novel |
| Idea ( ): “…we revisit Sobolev integral probability metrics (IPM) on trees to obtain a practical generalization of TSW…” |
| Reviewers : “The idea of combining tree-sliced Wasserstein with the Sobolev IPM to handle the general -Wasserstein distance is novel” |
Table 7: Examples of retrieval hurting gpt-5.4 ( Human+Generated , pointwise). In the first row, a single close precursor outweighs the contribution the reviewers valued. In the second, the judge matches each component to a different retrieved paper and dismisses the combination the reviewers called novel. In the third, the retrieved papers override the judge’s own recognition that the idea is a broad composition of known components: since no single retrieved paper covers the idea in full, the judge concludes it is novel.