IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?
Organizations: Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence
Abstract
Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26% of the human level on citation matching and only 18% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.
Figures & tables
| Area | Corpus | Reference |
| Machine learning systems | 88,184 | 5,188 |
| Code generation | 94,674 | 3,375 |
| World models | 50,271 | 1,867 |
| Automated research | 30,264 | 1,136 |
| Machine learning systems | Code generation | World models | Automated research | |||||||||||||
| Match | Diversity | Match | Diversity | Match | Diversity | Match | Diversity | |||||||||
| Population | Cite | Idea | Cite | Idea | Cite | Idea | Cite | Idea | Cite | Idea | Cite | Idea | Cite | Idea | Cite | Idea |
| Human | 20.84 | 13.56 | 13.50 | 84.97 | 18.75 | 11.25 | 18.64 | 87.41 | 16.38 | 13.87 | 14.68 | 84.31 | 18.51 | 10.53 | 21.69 | 87.52 |
| 2025-04-05, Llama 4 Scout ( Meta AI, 2025b ) | ||||||||||||||||
| free | 3.44 | 4.32 | 3.89 | 34.91 | 2.72 | 3.41 | 5.35 | 42.70 | 2.65 | 2.74 | 2.96 | 35.32 | 6.42 | 4.79 | 3.87 | 38.58 |
| topic | 3.87 | 4.93 | 13.71 | 67.47 | 3.29 | 4.21 | 12.77 | 63.90 | 3.42 | 4.42 | 9.41 | 61.39 | 5.44 | 3.85 | 13.84 | 68.53 |
| Quality | Execution | Overall | ||||||
| Population | Soundness | Presentation | Contribution | Feasibility | Success | Rating | Confidence | Judged AI |
| Human | 49.83 | 62.50 | 53.99 | 75.39 | 70.70 | 49.48 | 56.51 | 39.06 |
| LLM agent (mean) | 38.54 | 58.16 | 45.31 | 63.80 | 50.39 | 38.54 | 59.90 | 86.46 |
| Llama 4 Scout | 13.54 | 30.21 | 17.71 | 55.47 | 35.94 | 22.50 | 60.94 | 81.25 |
| Claude Haiku 4.5 | 38.54 | 57.29 | 41.67 | 67.97 | 50.78 | 39.38 | 55.47 | 84.38 |
| Gemini 3.5 Flash-Lite | 39.58 | 56.25 | 45.83 | 64.06 | 58.59 | 35.62 | 62.50 | 90.62 |
| Machine learning systems | Code generation | World models | Automated research | |||||||||||||
| Population | Problem | Motivation | Approach | Plan | Problem | Motivation | Approach | Plan | Problem | Motivation | Approach | Plan | Problem | Motivation | Approach | Plan |
| Human | 28.32 | 42.58 | 3.99 | 7.21 | 24.74 | 33.19 | 4.21 | 3.64 | 25.12 | 29.41 | 7.12 | 6.16 | 19.10 | 38.29 | 4.49 | 2.99 |
| 2025-04-05, Llama 4 Scout ( Meta AI, 2025b ) | ||||||||||||||||
| free | 7.09 | 17.62 | 0.81 | 1.45 | 6.73 | 6.99 | 1.87 | 1.07 | 4.02 | 5.25 | 1.18 | 0.91 | 6.95 | 14.96 | 3.35 | 0.44 |
| topic | 10.00 | 20.34 | 0.87 | 0.94 | 6.13 | 10.10 | 1.63 | 2.61 | 7.02 | 10.28 | 2.09 | 1.23 | 5.28 | 14.08 | 1.58 | 0.53 |
| landscape | 8.54 | 18.04 | 0.87 | 2.35 | 2.93 | 4.30 | 1.13 | 1.01 | 1.71 | 2.25 | 0.37 | 1.45 | 5.99 | 15.32 | 2.38 | 0.62 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Area | Definition and topic search |
|---|---|
| Machine learning systems | Systems research for efficient training and inference of ML and LLM models: inference serving and scheduling, KV-cache and memory management, distributed training and parallelism, quantization, pruning and distillation for efficiency, speculative decoding, efficiency-motivated attention and architecture changes, ML compilers and kernels, hardware acceleration, and MoE systems. Not in scope: architecture work aimed at task quality, downstream applications that merely use these systems, retrieval-augmented generation, alignment, and generic high-performance computing unrelated to ML. Topic search: the OpenAlex topics Parallel Computing and Optimization Techniques and Stochastic Gradient Optimization Techniques, and title and abstract search for “llm serving”, “inference serving”, and “distributed training”. |
| Code generation | Research on automatically producing program code from natural-language or other specifications with learned models: code LLMs and pretrained code models, program synthesis, natural language to code including text-to-SQL, code benchmarks and code-specific evaluation, program repair and self-debugging, coding agents and LLM-based software engineering, and code data curation. Not in scope: generic LLM and NLP foundation work such as GPT, LLaMA, RLHF and LoRA, generic agent or reasoning work without a code focus, generic machine translation metrics, and papers that merely use code as an application domain without studying generation. Topic search: OpenAlex keywords “code-generation” and “program-synthesis”. |
| World models | Learned internal models of an environment that represent its current state and predict future states under actions, used for planning and decision-making, simulation, and data generation. The lineages are model-based reinforcement learning with latent dynamics, video-generative world simulators, JEPA-style predictive representation learning, and domain instances in robotics, autonomous driving and games. Not in scope: generic video generation or vision backbones not positioned as environment models, model-free reinforcement learning, and generic LLM and NLP work. Topic search: title and abstract search for “world model”. |
| Automated research | AI systems, especially LLM agents, that automate stages of the scientific research process: hypothesis and idea generation, experiment planning and execution including ML engineering and laboratory automation, analysis, scientific writing and peer review, up to end-to-end autonomous AI scientists. Research-agent benchmarks and equation or function discovery are included. Not in scope: philosophy of science without AI, generic agent frameworks without a research-automation focus, domain-science papers that merely use AI as a tool, and generic literature-retrieval tools. Topic search: title and abstract search for “ai scientist”, “research agent”, “automated scientific discovery”, and “autonomous research”. |
| Year | Landmark paper |
|---|---|
| Machine learning systems | |
| 2018 | TVM: An Automated End-to-End Optimizing Compiler for Deep Learning |
| 2019 | GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism |
| 2019 | Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism |
| 2019 | Triton: an intermediate language and compiler for tiled neural network computations |
| 2020 | DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters |
| Corpus ( ) | Reference | ||||||||
| Area | Total | 2009 | 2010–14 | 2015–19 | 2020–21 | 2022–23 | 2024 | 2025 | 2026 |
| Machine learning systems | 88,184 | 5,626 | 3,646 | 16,572 | 14,393 | 19,383 | 14,868 | 13,696 | 5,188 |
| Code generation | 94,674 | 6,779 | 4,282 | 14,232 | 12,285 | 20,524 | 18,288 | 18,284 | 3,375 |
| World models | 50,271 | 5,541 | 3,116 | 11,245 | 8,758 | 10,591 | 6,004 | 5,016 | 1,867 |
| Automated research | 30,264 | 3,134 | 1,505 | 4,886 | 4,213 | 7,321 | 5,362 | 3,843 | 1,136 |
| Month of 2026 | |||||||||
| Area | Total | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 |
| Machine learning systems | 5,188 | 825 | 512 | 539 | 694 | 953 | 703 | 542 | 420 |
| Code generation | 3,375 | 540 | 306 | 346 | 493 | 558 | 441 | 464 | 227 |
| World models | 1,867 | 203 | 173 | 223 | 167 | 312 | 342 | 271 | 176 |
| Automated research | 1,136 | 155 | 86 | 139 | 127 | 190 | 173 | 151 | 115 |
| Corpus ( ) | Reference (2026) | |||||
| Area | Papers | Abstract | References | Papers | Abstract | References |
| Machine learning systems | 88,184 | 96.9 | 65.2 | 5,188 | 100 | 96.2 |
| Code generation | 94,674 | 96.5 | 62.5 | 3,375 | 100 | 97.6 |
| World models | 50,271 | 96.0 | 70.5 | 1,867 | 100 | 91.4 |
| Automated research | 30,264 | 95.3 | 71.8 | 1,136 | 100 | 93.7 |
| All | 263,393 | 96.4 | 66.0 | 11,566 | 100 | 95.6 |
| Survey | arXiv | References | In area |
| Machine learning systems | 761 | 409 | |
| Towards Efficient Generative LLM Serving | 2312.15234 | 334 | 240 |
| A Survey on Efficient Inference for LLMs | 2404.14294 | 304 | 202 |
| LLM Inference Serving: Recent Advances | 2407.12391 | 63 | 54 |
| A Comprehensive Survey of Speculative Decoding | 2401.07851 | 60 | 31 |
| Code generation | 1,482 | 718 | |
| Area | Construction | References | Covered |
| Machine learning systems | Citation expansion | 409 | 88.3% |
| Code generation | Citation expansion | 718 | 92.9% |
| World models | Citation expansion | 435 | 94.0% |
| Automated research | Citation expansion | 457 | 79.0% |
| All | Citation expansion | 2,019 | 89.1% |
| Code generation | Keyword search | 718 | 17.8% |
| Area | ( Diversity-Cite ) | |||
| Machine learning systems | 20.81 | 70.68 | 19.10 | 13.50 |
| Code generation | 20.94 | 73.58 | 25.33 | 18.64 |
| World models | 26.46 | 68.05 | 21.57 | 14.68 |
| Automated research | 18.28 | 73.51 | 29.51 | 21.69 |
| Slot | Judge match | Judge–judge | Judge–human |
| Problem | 20 / 40 | 83.00 | 87.50 |
| Motivation | 20 / 40 | 77.00 | 70.00 |
| Proposed approach | 19 / 40 | 90.00 | 87.50 |
| Research plan | 18 / 40 | 79.00 | 72.50 |
| Document type | Per area | Total |
| Proposal overlapping a 2026 paper | 12 | 48 |
| 2026 paper overlapped by a proposal | 12 | 48 |
| Proposal, non-overlapping | 12 | 48 |
| 2026 paper, non-overlapping | 12 | 48 |
| Total | 192 |
| Area | # Used | # | Topics, most allocated first |
| Machine learning systems | 156 | 2,665 | Topic Modeling, Natural Language Processing Techniques, Multimodal Machine Learning Applications, Advanced Neural Network Applications, Domain Adaptation and Few-Shot Learning, Parallel Computing and Optimization Techniques, Neural Networks and Applications, Speech Recognition and Synthesis, Advanced Image and Video Retrieval Techniques, Generative Adversarial Networks and Image Synthesis, Adversarial Robustness in Machine Learning, Semantic Web and Ontologies, Speech and dialogue systems, Explainable Artificial Intelligence (XAI), Artificial Intelligence in Healthcare and Education |
| Code generation | 146 | 2,609 | Topic Modeling, Natural Language Processing Techniques, Software Engineering Research, Multimodal Machine Learning Applications, Software Testing and Debugging Techniques, Semantic Web and Ontologies, Advanced Malware Detection Techniques, Artificial Intelligence in Healthcare and Education, Adversarial Robustness in Machine Learning, Explainable Artificial Intelligence (XAI), Domain Adaptation and Few-Shot Learning, Speech and dialogue systems, Software Reliability and Analysis Research, Multi-Agent Systems and Negotiation, Software System Performance and Reliability |
| World models | 156 | 2,296 | Reinforcement Learning in Robotics, Multimodal Machine Learning Applications, Generative Adversarial Networks and Image Synthesis, Topic Modeling, Domain Adaptation and Few-Shot Learning, Natural Language Processing Techniques, Human Pose and Action Recognition, Advanced Vision and Imaging, Robot Manipulation and Learning, Advanced Neural Network Applications, Autonomous Vehicle Technology and Safety, Adversarial Robustness in Machine Learning, Computer Graphics and Visualization Techniques, Human Motion and Animation, Advanced Image and Video Retrieval Techniques |
| Automated research | 177 | 2,387 | Topic Modeling, Machine Learning in Materials Science, Natural Language Processing Techniques, Computational Drug Discovery Methods, Artificial Intelligence in Healthcare and Education, Scientific Computing and Data Management, Explainable Artificial Intelligence (XAI), Multimodal Machine Learning Applications, Semantic Web and Ontologies, Model Reduction and Neural Networks, Multi-Agent Systems and Negotiation, Neural Networks and Applications, Biomedical Text Mining and Ontologies, Evolutionary Algorithms and Applications, Ethics and Social Impacts of AI |
| Axis | Ten most common, with the share of the area’s 2025 papers carrying each |
|---|---|
| Machine learning systems , 824 task tags and 2,072 method tags | |
| Task | llm inference acceleration (7.6%), llm serving (5.6%), low-bit quantization (4.8%), mixture-of-experts (4.6%), large language model compression (4.1%), large language model training (3.7%), post-training quantization (3.7%), edge llm inference (3.6%), kv cache compression (3.5%), distributed training (3.2%) |
| Method | post-training quantization (4.3%), speculative decoding (4.1%), knowledge distillation (3.8%), sparse attention (3.4%), algorithm-hardware co-design (3.2%), token pruning (3.1%), mixture of experts (2.8%), mixed-precision quantization (2.4%), reinforcement learning (2.4%), performance modeling (2.3%) |
| Pair | low-bit quantization post-training quantization (1.8%), post-training quantization post-training quantization (1.6%), visual token compression token pruning (1.5%), kv cache compression key-value cache compression (1.2%), efficient vision-language model inference token pruning (1.2%), autoregressive decoding acceleration speculative decoding (1.2%), llm inference acceleration speculative decoding (1.0%), mixture-of-experts expert routing (1.0%), long-context inference sparse attention (0.9%), low-bit quantization mixed-precision quantization (0.9%) |
| Code generation , 618 task tags and 805 method tags | |
| Task | code generation (19.9%), text-to-sql (10.0%), code generation evaluation (8.3%), automated program repair (6.6%), benchmark construction (5.5%), code generation benchmark (3.2%), software engineering agent (3.1%), code translation (2.9%), verilog code generation (2.5%), code completion (2.3%) |
| Role | Model | Context | Max output | Reply |
| Schema extraction | DeepSeek-V4-Pro | 1M | 16,384 | JSON |
| Tag extraction | DeepSeek-V4-Flash | 1M | 16,384 | JSON |
| Area screening | DeepSeek-V4-Flash | 1M | 4,096 | JSON |
| Idea matching | GPT-5.6 Luna | 1M | 1,200 | JSON |
| LLM agent | Llama 4 Scout | 10M | 8,192 | text |
| LLM agent | Claude Haiku 4.5 | 200K | 16,384 | text |
| World models | Automated research | ||||||
| Method | Metric | Run 1 | Run 2 | Run 1 | Run 2 | ||
| Gemini 3.5 Flash-Lite ( Google DeepMind, 2026 ) | |||||||
| free | Match-Cite | 4.89 | 4.74 | 0.15 | 7.18 | 6.88 | 0.30 |
| Match-Idea | 3.38 | 3.11 | 0.27 | 2.22 | 2.42 | 0.20 | |
| Diversity-Cite | 4.76 | 4.99 | 0.23 | 5.52 | 5.71 | 0.19 | |
| Diversity-Idea | 18.76 | 18.73 | 0.03 | 12.92 | 11.04 | 1.88 | |
| Machine learning systems | Code generation | World models | Automated research | |||||
| Population | Cite | Idea | Cite | Idea | Cite | Idea | Cite | Idea |
| Full reference , January to August 2026 (5,188 / 3,375 / 1,867 / 1,136 papers) | ||||||||
| Human | 20.84 | 13.56 | 18.75 | 11.25 | 16.38 | 13.87 | 18.51 | 10.53 |
| free | 3.73 | 1.36 | 3.72 | 1.52 | 4.89 | 3.38 | 7.18 | 2.22 |
| topic | 4.59 | 2.29 | 4.23 | 2.19 | 5.33 | 3.86 | 6.58 | 2.14 |
| landscape | 3.67 | 1.38 | 3.88 | 1.47 | 4.44 | 2.31 | 7.18 | 2.14 |
| Population | Overlap | Problem | Motivation | Approach | Plan |
| Human | 10.26 | 19.10 | 37.06 | 3.43 | 2.99 |
| free | 6.34 | 12.68 | 30.99 | 0.26 | 0.09 |
| topic | 4.48 | 9.77 | 18.13 | 0.26 | 0.00 |
| landscape | 4.26 | 8.36 | 17.61 | 0.35 | 0.09 |
| frontier | 4.19 | 10.48 | 12.85 | 0.53 | 0.00 |
| Axis | Population | Tags used | Per doc | Human tags covered | Missed human mass (%) | Head of the miss (%) | Top-10 share (%) | TVD |
| Task (2025 vocabulary: 318 tags; human 2026 uses 200) | ||||||||
| Human reference | 200 | 1.71 | — | — | — | 43.5 | — | |
| Free | 36 | 2.17 | 34 / 200 | 49.7 | 40.5 | 94.4 | 0.717 | |
| Topic | 125 | 2.19 | 106 / 200 | 22.7 | 48.1 | 77.7 | 0.554 | |
| Landscape | 39 | 2.18 | 37 / 200 | 47.9 | 41.7 | 91.5 | 0.663 | |
| Frontier | 192 | 2.81 | 155 / 200 | 5.5 | 53.3 | 55.3 | 0.499 | |
| Population | Top task tags | Top method tags | Top task method pairs |
| Human | AI scientist (9%) agent benchmarking (9%) autonomous experimentation (5%) | llm agent (10%) multi-agent system (7%) benchmark construction (7%) | agent benchmarking benchmark construction (2.3%) AI scientist llm agent (1.5%) AI scientist multi-agent system (1.1%) |
| Free | agent benchmarking (24%) scientific claim verification (23%) AI scientist safety (16%) | benchmark construction (27%) execution-based evaluation (12%) provenance tracking (6%) | agent benchmarking benchmark construction (7.7%) scientific claim verification benchmark construction (6.3%) scientific claim verification execution-based evaluation (4.6%) |
| Topic | agent benchmarking (23%) experimental design (9%) scientific claim verification (8%) | benchmark construction (22%) llm agent (8%) execution-based evaluation (5%) | agent benchmarking benchmark construction (7.1%) scientific claim verification benchmark construction (2.1%) experimental design benchmark construction (1.9%) |
| Landscape | agent benchmarking (21%) scientific hypothesis evaluation (17%) experimental design (14%) | benchmark construction (24%) execution-based evaluation (8%) llm agent (7%) | agent benchmarking benchmark construction (6.8%) scientific hypothesis evaluation benchmark construction (3.6%) experimental design benchmark construction (2.9%) |
| Frontier | experimental design (9%) agent benchmarking (9%) scientific hypothesis evaluation (7%) | llm agent (13%) benchmark construction (13%) information gain (4%) | experimental design llm agent (1.9%) agent benchmarking benchmark construction (1.9%) AI scientist safety benchmark construction (1.1%) |
Research Idea Generation
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
See all 70 papers