Language Models Make New Research Sound Old
Organizations: University of Miami
Abstract
Researchers are increasingly using large language models (LLMs) to validate research questions. We find that LLMs make open research questions sound partly studied and overstate what related papers establish. Models recognize most of these overstatements when asked to check the papers' full texts separately. On 40 questions drawn from newly accepted papers, GPT-6-Astra correctly judges 98% to be unstudied, but states that status plainly in only 8% of ordinary answers. All six frontier systems show reporting gaps of 30-90 percentage points. Across 22 models given identical papers, higher capability is associated with more accurate judgments and larger reporting gaps. This reporting failure is asymmetric: models correctly identify studied questions in 88-100% of ordinary answers when we supply the answering papers. To correct this, we build a fixed answer from each model's judgment and separate checks of cited papers. This raises Astra's clear reporting of open status from 8% to 95%, with improvements in all six systems. Finally, we predict reviewers' novelty grades by combining several models' written ratings with a rule learned from graded examples, reaching the performance range reported by Think-Probe-Respond. Unlike TPR, our approach needs no access to model weights or internal activations, so it also works with closed-weight frontier models.
Figures & tables
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Question type | Status | Full set | Subset |
| L0: Published question | Studied | 108 | 20 |
| L1: One part changed | Open | 107 | 20 |
| L2: Two parts changed | Open | 108 | 15 |
| L3: Distant field | Open | 108 | 15 |
| Real entity | Studied | 36 | 12 |
| Invented entity | Open | 36 | 18 |
| Group | Models |
|---|---|
| Current frontier | GPT-6-Astra, GPT-6-Sol, GPT-6-Luna, Claude Sonnet 5.5, Claude Opus 5.5, Claude Fable 5.1 |
| Current open weights | Qwen3.8 27B, Qwen3.8 Flash, Qwen3.8 2.4T, GLM 5.3 Flash, GLM 5.3, Kimi K3, Qwen3.5 122B, Gemma 4 31B, Qwen3.6 27B, Qwen3.6 35B-A3B |
| Older | Gemini 2.5 Pro, GPT-5 mini, gpt-oss-20b, Qwen3 32B, Llama 3.1 70B, Gemma 3 27B |
| Grade | Coverage of the research question |
|---|---|
| 0 | Unrelated |
| 1 | Topical |
| 2 | Differs in several core parts |
| 3 | Differs in one core part |
| 4 | Answers the question |
| Question set | System | (%) | (%) | (points) | Balanced judgment accuracy (%) |
|---|---|---|---|---|---|
| Accepted-paper questions | Astra | 8 [0, 18] | 98 | +90 [80, 98] | N/A |
| Luna | 70 [55, 82] | 100 | +30 | N/A | |
| Sol | 38 [22, 52] | 100 | +62 | N/A | |
| Sonnet | 68 [52, 82] | 98 | +30 | N/A | |
| Opus | 48 [32, 62] | 78 | +30 | N/A | |
| Fable | 35 [20, 50] | 92 | +57 | N/A |
| System | (%) | (%) | (%) | (%) | Mean (%) | (points) | Rebuilt (%) |
|---|---|---|---|---|---|---|---|
| Astra | 30 [18, 45] | 68 | 2 | 95 [88, 100] | 25 | +65 [+50, +80] | 95 |
| Sol | 57 [42, 72] | 42 | 0 | 100 [100, 100] | 8 | +43 [+28, +57] | 100 |
| Luna | 72 [57, 85] | 28 | 0 | 95 [88, 100] | 21 | +22 [+10, +38] | 95 |
| Sonnet | 75 [62, 88] | 25 | 0 | 98 [92, 100] | 15 | +22 [+10, +35] | 98 |
| Opus | 42 [28, 57] | 57 | 0 | 85 [72, 95] | 34 | +42 [+28, +57] | 85 |
| Fable | 38 [22, 52] | 62 | 0 | 82 [70, 92] | 32 | +45 [+27, +62] | 82 |
| System | Pursue, explain contribution (%) | Reader minus model probability (points) |
|---|---|---|
| Astra | 68 | +2 [ , +6] |
| Sol | 40 | +12 [+9, +15] |
| Luna | 22 | [ , +6] |
| Sonnet | 42 | +10 [+7, +13] |
| Opus | 72 | [ , +6] |
| Fable | 68 | +0 [ , +6] |
| System | Question and list | Novelty grade | ||||
|---|---|---|---|---|---|---|
| Astra | Studied, answering paper listed | 100 | 0 | 0 | 100 | 1.10 |
| Open, source paper listed | 0 | 70 | 30 | 5 | 2.55 | |
| Open, topical only | 0 | 18 | 82 | 5 | 3.27 | |
| Studied, answering paper absent | 45 | 12 | 42 | 68 | 3.42 | |
| Sonnet | Studied, answering paper listed | 98 | 2 | 0 | 98 | 1.05 |
| Open, source paper listed | 0 | 62 | 38 | 0 | 2.62 |
| (%) | Closest minus random | |||
|---|---|---|---|---|
| System | All papers | Closest removed | Random removed | removal (points) |
| Astra | 72.5 | 52.5 | 70.0 | [ , ] |
| Sol | 47.5 | 32.5 | 60.0 | [ , ] |
| Luna | 35.0 | 37.5 | 40.0 | [ , +15.0] |
| Sonnet | 32.5 | 22.5 | 35.0 | [ , +2.5] |
| Opus | 60.0 | 47.5 | 52.5 | [ , +10.0] |
| System | Paper | |||
|---|---|---|---|---|
| Astra | Supplied | 40 | 100 | 100 |
| Absent, recalled | 26 | 62 | 92 | |
| Not recalled | 14 | 14 | 21 | |
| Sonnet | Supplied | 40 | 98 | 98 |
| Absent, recalled | 28 | 89 | 68 | |
| Not recalled | 12 | 17 | 25 |
| Measure against capability | Models | Spearman correlation |
|---|---|---|
| Gap, combined open questions | 22 | +0.81 [+0.55, +0.94] |
| Gap, accepted-paper questions | 22 | +0.79 [+0.50, +0.92] |
| share, accepted-paper questions | 22 | [ , ] |
| Balanced judgment accuracy, complete set | 22 | +0.65 [+0.24, +0.91] |
| Balanced judgment accuracy, ladders | 25 | +0.68 [+0.33, +0.89] |
| Gap, open ladders | 25 | +0.73 [+0.51, +0.86] |
| Measure | System | Original | Template | Reworded | Abstracts | Full text |
|---|---|---|---|---|---|---|
| (%) | Astra | 3 | 83 | 80 | 57 | 90 [79, 100] |
| Sonnet | 47 | 93 | 90 | 50 | 87 [72, 97] | |
| True status plainly (%) | Astra | 47 [35, 59] | 88 [80, 95] | 85 | N/A | 90 [83, 97] |
| Sonnet | 68 | 95 [89, 100] | 90 | N/A | 86 | |
| Answering paper retained (%) | Astra | 100 | 93 | 100 | 73 | 100 |
| Sonnet | 97 | 97 | 100 | 100 | 100 |
| Readers | Original answer group | Normal (%) | Rebuilt (%) | Change (points) |
|---|---|---|---|---|
| Sonnet / Luna | 84 | 58 | [ , ] | |
| 23 | 40 | +17 [6, 27] | ||
| All | 52 | 49 | [ , +7] | |
| Opus / Astra | 96 | 50 | ||
| 46 | 29 | |||
| All | 70 | 39 | [ , ] |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 862 papers