Astro2Method: Benchmarking LLMs on Machine Learning Method Selection in Astronomy

Xingzhong Fan; Hangfeng He; Fatemeh Nargesian

astro-ph.IMOct 21, 2026

Astro2Method: Benchmarking LLMs on Machine Learning Method Selection in Astronomy

Authors: Xingzhong Fan, Hangfeng He, Fatemeh Nargesian

Organizations: Department of Computer Science, University of Rochester Rochester, NY, USA

Abstract

Machine learning (ML) is increasingly used in astronomy, but choosing a suitable method for a scientific problem often requires expertise in both astronomy and ML. Large language models (LLMs) could help astronomers make such choices, yet this ability remains underexplored in existing benchmarks. We introduce Astro2Method, a benchmark of 500 expert-reviewed binary questions from 346 astronomy papers (2024/01-2026/06), based on clear performance differences and complete (Easy) or partial (Hard) explanations from the authors. Across five LLMs, direct prompting achieves 87.50-92.55% accuracy on Easy questions but only 58.87-64.52% on Hard. Using historical papers, neither retrieval-augmented generation nor most agent methods with additional reasoning or retrieval (including ReAct, Self-Ask, and CRITIC) bring consistent improvement. Agents required to search the web improve accuracy mainly when they find the papers the questions are based on, with no consistent improvement otherwise. Providing the source paper's pipeline description with method names masked improves Hard accuracy, matching or exceeding web search. Together, these results suggest that relevant experimental information helps distinguish ML methods, but inferring it from related studies and domain knowledge remains challenging for LLMs in astronomy.

Figures & tables

Figure 1: Dataset construction pipeline. We extract comparisons from astronomy papers, screen them using performance differences and authors’ explanations, and write binary questions with expert review throughout.

Model	Overall	Easy	Hard
GPT-5.6 Luna	82.40	89.63	60.48
GPT-6 Luna	80.40	87.50	58.87
GPT-6 Astra	85.60	92.55	64.52
DeepSeek V4.1 Flash	82.20	89.10	61.29
Qwen 3.8 Flash	82.60	89.89	60.48

Table 1: Accuracy (%) with direct prompting on 500 questions (376 Easy, 124 Hard). Best results are in bold.

Text RAG ( $K=3$ )
	GPT-5.6 Luna			DeepSeek V4.1 Flash			Qwen 3.8 Flash
Method	Overall	Easy	Hard	Overall	Easy	Hard	Overall	Easy	Hard
[0pt][24pt] Direct prompting	82.40	89.63	60.48	82.20	89.10	61.29	82.60	89.89	60.48
BM25	80.20	87.50	58.06	77.60	84.84	55.65	81.60	88.83	59.68
E5	80.60	88.03	58.06	81.40	87.77	62.10	81.00	88.30	58.87
RRF	82.00	88.83	61.29	81.40	88.03	61.29	81.60	88.83	59.68

Table 2: Accuracy (%) on 500 questions (376 Easy, 124 Hard). Bold indicates the highest accuracy for each model and subset; the shaded row indicates the direct-prompting baseline. A leading + indicates a modification to the no-skill OpenClaw agent. † Web use is optional; ‡ Web search is required.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Journal	Downloaded papers	Source papers	Questions	Historical retrieval
Monthly Notices of the Royal Astronomical Society (MNRAS)	171	48	70	413
The Astrophysical Journal (ApJ)	247	72	105	241
Astronomy & Astrophysics (A&A)	222	70	102	176
Physical Review D (PhRvD)	61	14	17	89
The Astrophysical Journal Supplement Series (ApJS)	148	67	94	64
The Astronomical Journal (AJ)	74	31	51	65

Appendix

Table 3: Journal distribution of downloaded papers, dataset source papers, questions, and historical retrieval corpus (2014–2023).

Figure 2: Publication-year distribution of the historical retrieval corpus (2014–2023), stacked by journal. Numbers above the bars show annual paper counts.

Figure 3: Dataset distribution by astronomy topic (left) and ML method group (right). Circle sizes and numbers show question counts; blue and hatched orange indicate Easy and Hard questions. Each question has one astronomy label, while ML groups cover both answer options and may overlap. Other methods combines groups with fewer than 10 questions, with duplicates removed. AEs: autoencoders; RNNs: recurrent neural networks; SVMs: support vector machines; kNN: nearest-neighbor methods; SOM: self-organizing maps. High-energy and time-domain includes compact objects, transients, gravitational waves, and particle observations.

Figure 4: RAG

K

ablation for GPT-5.6 Luna, with Text RAG on top and Structured RAG on the bottom. Dash-dotted lines show the baseline for direct prompting.

Model Selection

Model selection chooses models, features, or configurations that balance predictive fit, generalization, complexity, and deployment constraints.

See all 96 papers

AstroRAG -- A Pagerank-Based Retrieval-Augmented Generation Pipeline for Question Answering in Astronomy
May 24, 2026Zhifeng Wang, Jason Jingshi Li, Kaihao Zhang +1Retrieval-Augmented Generation Scientific QA
Australian National University, Canberra, ACT, Australia · Learning Machines Pty Ltd, Canberra, ACT, Australia
AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification
May 7, 2026Claire Chen, Jiabao Sean Xiao, Shuze Daniel Liu +7Multimodal Reasoning Multimodal Model Evaluation
1California Institute of Technology · 2Massachusetts Institute of Technology · 3Purdue University
Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot
Aug 17, 2026Hui MaoLLM-as-a-Judge
Independent Researcher

May 24, 2026J/K move · Enter open · S save

AstroRAG -- A Pagerank-Based Retrieval-Augmented Generation Pipeline for Question Answering in Astronomy

Zhifeng Wang, Jason Jingshi Li, Kaihao Zhang, Ramesh Sankaranarayana

Australian National University, Canberra, ACT, Australia · Learning Machines Pty Ltd, Canberra, ACT, Australia

Retrieval-Augmented Generation Scientific QA Graph-Based Retrieval

Open paper AskPDF arXiv

Large language models (LLMs) demonstrate strong performance in natural language processing but often generate factual errors when relying solely on parametric knowledge. Retrieval-Augmented Generation (RAG) mitigates these errors by grounding responses in external evidence, yet conventional retrieve-and-dump approaches frequently introduce irrelevant context that degrades answer quality. In this work, we present AstroRAG -- a PageRank-based retrieval-augmented generation (RAG) pipeline adapted for question answering in astronomy. The system performs token-aware chunking and per-instance, ephemeral indexing in Elasticsearch, then executes a two-stage retrieval: (i) Maximal Marginal Relevance (MMR) to obtain a small, diverse candidate set and (ii) a reader-driven PageRank (PR) re-ranking on a similarity graph to identify a compact, mutually supportive context under a strict token budget. Our design is training-free, privacy-preserving, and reproducible, as each instance is processed through transient indexing to prevent cross-task leakage. We evaluate the pipeline on the AstroQA benchmark for astronomy QA, and demonstrate competitive performance across all difficulty levels. In particular, the RAG-enhanced Mistral-7B achieves 79.49% accuracy and 79.49% F1-score, nearly doubling the performance of its non-RAG counterpart. These results highlight the effectiveness of disciplined retrieval and refinement in boosting domain-specific reasoning, establishing a robust foundation for extending RAG to other scientific fields.

Abstract

Model Selection

Explore similar work

AstroRAG -- A Pagerank-Based Retrieval-Augmented Generation Pipeline for Question Answering in Astronomy

AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification

Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot