Eliciting Research Taste in LLMs through Future Research Direction Choice
Organizations: Lossfunk
Abstract
Research taste is increasingly discussed in the context of AI scientists, yet remains little studied or measured in LLMs. This judgment matters especially in open-ended research, where a direction must be chosen before a verifiable objective is available. We operationalize research taste through one task of the scientific workflow: choosing which future research direction to pursue given a research goal and current results. Rather than predicting downstream scientometric metrics or treating author choices as objectively correct, we evaluate models against human-authored future-work directions and report author alignment, stable divergence, and unstable choice as behavioral measures of research-direction preference. We construct an order-controlled evaluation of 80 pairwise comparisons from 19 ML papers published between December 2025 and February 2026 across four subdomains. Each query presents a research goal, hypotheses tested with key results, and two candidate future directions: one author-recommended direction and one plausible distractor sampled from a similar paper. Across eight LLMs, author alignment ranges from 45.0% to 67.5%, while stable divergence ranges from 15.0% to 36.2%. These divergences largely persist under similarity controls, prompt ablations, and an author-preference framing check, and inter-model agreement suggests that some non-author preferences are shared across models. We characterize what models diverge toward using transition-category labels and selected reasoning examples, finding recurring movement toward scaling/generalization and trade-offs between methodological and infrastructure/data extensions. We view this as an early descriptive evaluation for eliciting research taste profiles in LLMs, and as a step toward understanding and steering their research preferences as they take on more autonomous roles in novel open-ended scientific discovery.
Figures & tables
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Pair | Embedding | TF-IDF |
|---|---|---|
| GT, Distractor | 0.41 | 0.05 |
| GT, Context | 0.48 | 0.14 |
| Distractor, Context | 0.40 | 0.08 |
| (GT, Context) (Dist, Context) | 0.08 0.09 | 0.05 0.07 |
| Family | Model | Author Alignment | Stable Divergence | Unstable Choice |
|---|---|---|---|---|
| Gemini 3 Pro | 67.5% [-2pt][56.6, 76.8] | 15.0% [-2pt][8.8, 24.4] | 17.5% [-2pt][10.7, 27.3] | |
| Gemini 3 Flash | 62.5% [-2pt][51.5, 72.3] | 23.8% [-2pt][15.8, 34.1] | 13.8% [-2pt][7.9, 23.0] | |
| Anthropic | Claude Opus 4.5 | 55.0% [-2pt][44.1, 65.4] | 26.2% [-2pt][17.9, 36.8] | 18.8% [-2pt][11.7, 28.7] |
| Claude Haiku 4.5 | 50.0% [-2pt][39.3, 60.7] | 21.2% [-2pt][13.7, 31.4] | 28.7% [-2pt][20.0, 39.5] | |
| OpenAI | GPT-5.2 Pro | 57.5% [-2pt][46.6, 67.7] | 30.0% [-2pt][21.1, 40.8] | 12.5% [-2pt][6.9, 21.5] |
| GPT-OSS-120B | 50.0% [-2pt][39.3, 60.7] | 36.2% [-2pt][26.6, 47.2] | 13.8% [-2pt][7.9, 23.0] |
| Threshold | Pairs |
|---|---|
| 1 model | 45/80 |
| 2 models | 34/80 |
| 3 models | 27/80 |
| 4 models | 24/80 |
| 5 models | 18/80 |
| 6 models | 11/80 |
| Model | Stable Div. | GT More Similar | Dist More Similar |
|---|---|---|---|
| Google Gemini 3 Pro | 12 | 8 (66.7%) | 4 (33.3%) |
| Google Gemini 3 Flash | 19 | 14 (73.7%) | 5 (26.3%) |
| Claude Opus 4.5 | 21 | 16 (76.2%) | 5 (23.8%) |
| Claude Haiku 4.5 | 17 | 14 (82.4%) | 3 (17.6%) |
| GPT-5.2 Pro | 24 | 18 (75.0%) | 6 (25.0%) |
| GPT-OSS-120B | 29 | 23 (79.3%) | 6 (20.7%) |
| Variant | Retained Divergence | Became Unstable | Reversed to Alignment |
|---|---|---|---|
| No Role | 124/169 (73.4%) | 36/169 (21.3%) | 9/169 (5.3%) |
| No Research Goal | 113/169 (66.9%) | 44/169 (26.0%) | 12/169 (7.1%) |
| Reframed Task | 136/169 (80.5%) | 30/169 (17.8%) | 3/169 (1.8%) |
| No Role | No Goal | Reframed Task | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Pairs | Ret. | Unst. | Rev. | Ret. | Unst. | Rev. | Ret. | Unst. | Rev. |
| GPT-5.2 Pro | 24 | 83.3 | 12.5 | 4.2 | 79.2 | 20.8 | 0.0 | 91.7 | 4.2 | 4.2 |
| GPT-OSS-120B | 29 | 82.8 | 10.3 | 6.9 | 75.9 | 17.2 | 6.9 | 89.7 | 10.3 | 0.0 |
| Qwen3 Max Think. | 19 | 78.9 | 10.5 | 10.5 | 73.7 | 26.3 | 0.0 | 89.5 | 5.3 | 5.3 |
| Gemini 3 Flash | 19 | 78.9 | 10.5 | 10.5 | 47.4 | 36.8 | 15.8 | 68.4 | 26.3 | 5.3 |
| Claude Haiku 4.5 | 17 | 70.6 | 29.4 | 0.0 | 64.7 | 35.3 | 0.0 | 76.5 | 23.5 | 0.0 |
| Model | Events | Retained Div. | Unstable | Reversed |
|---|---|---|---|---|
| GPT-5.2 Pro | 24 | 62.5% (15) | 16.7% (4) | 20.8% (5) |
| GPT-OSS-120B | 29 | 82.8% (24) | 13.8% (4) | 3.4% (1) |
| Qwen3 Max Think. | 19 | 84.2% (16) | 10.5% (2) | 5.3% (1) |
| Gemini 3 Flash | 19 | 68.4% (13) | 26.3% (5) | 5.3% (1) |
| Claude Haiku 4.5 | 17 | 70.6% (12) | 29.4% (5) | 0.0% (0) |
| Llama 4 Maverick | 28 | 71.4% (20) | 28.6% (8) | 0.0% (0) |
| Model | Condition | Retained/Events | Retention | Wilson 95% CI (%) |
|---|---|---|---|---|
| Gemini 3 Pro | No Role | 7/12 | 58.3% | [32.0, 80.7] |
| No Goal | 6/12 | 50.0% | [25.4, 74.6] | |
| Reframed | 7/12 | 58.3% | [32.0, 80.7] | |
| Gemini 3 Flash | No Role | 15/19 | 78.9% | [56.7, 91.5] |
| No Goal | 9/19 | 47.4% | [27.3, 68.3] | |
| Reframed | 13/19 | 68.4% | [46.0, 84.6] |
| Model Pair | Agreement | Shared Div. | Shared Unstable Choice |
| Intra-Family | |||
| GPT-5.2 Pro – GPT-OSS-120B | 80.0% | 26.2% | 6.2% |
| Gemini 3 Pro – Gemini 3 Flash | 67.5% | 11.2% | 3.8% |
| Claude Opus 4.5 – Claude Haiku 4.5 | 65.0% | 12.5% | 8.8% |
| Inter-Family | |||
| Llama 4 Maverick – GPT-OSS-120B | 71.2% | 25.0% | 8.8% |
| Pair ID | Judg. | Author | Alt. | Abst. | Author Share | Majority |
|---|---|---|---|---|---|---|
| 2512.22322_fw0_d0 | 7 | 3 | 3 | 1 | 50.0% | Tie |
| 2512.22322_fw1_d1 | 4 | 2 | 2 | 0 | 50.0% | Tie |
| 2601.02439_fw0_d0 | 6 | 4 | 1 | 1 | 80.0% | Author |
| 2601.05175_fw0_d0 | 4 | 3 | 1 | 0 | 75.0% | Author |
| 2601.05175_fw3_d1 | 4 | 4 | 0 | 0 | 100.0% | Author |
| 2601.09680_fw3_d0 | 6 | 2 | 3 | 1 | 40.0% | Alternative |
| Pooled choice | Macro choice | Pair majority | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Subset | Pairs | Judg. | Fixed | Abst. | Author | Alt. | Author | Alt. | Author | Alt. | Tie |
| All 23 pairs | 23 | 108 | 95 | 13 | 61.1% | 38.9% | 61.6% | 38.4% | 12 | 8 | 3 |
| fixed responses | 20 | 100 | 90 | 10 | 62.2% | 37.8% | 63.3% | 36.7% | 11 | 7 | 2 |
| No query-quality abstentions | 13 | 56 | 55 | 1 | 58.2% | 41.8% | 55.6% | 44.4% | 6 | 5 | 2 |
| Participant | Highest degree / role | Years ML exp. | Judg. | Fixed | Abst. |
|---|---|---|---|---|---|
| P01 | PhD | 2 | 3 | 3 | 0 |
| P02 | Bachelor’s | 3 | 10 | 10 | 0 |
| P03 | Bachelor’s | 6 | 23 | 16 | 7 |
| P04 | Master’s | 4 | 10 | 10 | 0 |
| P05 | Bachelor’s | 13 | 5 | 3 | 2 |
| P06 | Other | 1 | 19 | 18 | 1 |
| Abstention reason | Count | Share of all judgements |
|---|---|---|
| Options do not make sense | 6 | 5.6% |
| Context unclear / insufficient | 4 | 3.7% |
| Outside domain expertise | 2 | 1.9% |
| Other | 1 | 0.9% |
| Total | 13 | 12.0% |
| Category | Description |
|---|---|
| Infrastructure, Data & Benchmarking | Build datasets, benchmarks, tooling, or evaluation infrastructure. |
| Mechanistic / Interpretability / Theory | Explain why a result holds, probe internals, or formalize the phenomenon. |
| Methodological & Architectural Innovation | Modify the algorithm, training objective, architecture, or system design. |
| Scaling, Generalization & Robustness | Test the existing method across scales, domains, settings, or stress conditions. |
| Application & Downstream Integration | Apply the method to a concrete downstream workflow, real-world pipeline, domain, or agentic/use-case setting. |
| GT Category | Distractor Category | Count | % of Total | % within GT |
| Methodological (53) | Infrastructure | 25 | 14.8% | 47.2% |
| Scaling | 23 | 13.6% | 43.4% | |
| Methodological | 2 | 1.2% | 3.8% | |
| Mechanistic | 2 | 1.2% | 3.8% | |
| Application | 1 | 0.6% | 1.9% | |
| Infrastructure (75) | Methodological | 30 | 17.8% | 40.0% |
| GT Category | Distractor Category | Events | % within GT |
|---|---|---|---|
| Methodological (32) | Infrastructure | 18 | 56.2% |
| Scaling | 14 | 43.8% | |
| Infrastructure (74) | Methodological | 30 | 40.5% |
| Infrastructure | 25 | 33.8% | |
| Application (25) | Scaling | 12 | 48.0% |
| Application | 8 | 32.0% |
| Family | Model | Accuracy | Format Compl. | T=0 Det. | Choice Dist. |
|---|---|---|---|---|---|
| Anthropic | Claude Opus 4.5 | 54.3% | 100% | 74.3% | 23/36/46 |
| Claude Sonnet 4.5 | 34.3% | 100% | 54.3% | 15/30/60 | |
| Claude Haiku 4.5 | 45.7% | 100% | 85.7% | 20/41/44 | |
| Gemini 3 Pro | 60% | 99% | 91.4% | 41/30/34 | |
| Gemini 3 Flash | 57.1% | 100% | 74.3% | 38/35/32 | |
| OpenAI | GPT-5.2 Pro | 40% | 100% | 75% | 22/21/62 |
| Model | GT at A | GT at B | GT at C | Range(%) |
|---|---|---|---|---|
| Gemini 3 Flash | 65.7% | 60.0% | 68.6% | 8.6 |
| Claude Opus 4.5 | 42.9% | 60.0% | 71.4% | 28.5 |
| Qwen3 Max Think. | 25.7% | 51.4% | 68.6% | 42.9 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,527 papers