cs.CLApr 2, 2026

AstroConcepts: A Large-Scale Multi-Label Classification Corpus for Astrophysics

Authors: Atilla Kaan Alkan, Felix Grezes, Sergi Blanco-Cuaresma, Jennifer Lynn Bartlett, Daniel Chivvis, Anna Kelbert, Kelly Lockhart, Alberto Accomazzi

Organizations: Harvard-Smithsonian Center for Astrophysics, Cambridge, MA, USA · Faculty of Psychology, UniDistance Suisse, Brig, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Scientific multi-label text classification suffers from extreme class imbalance, where specialized terminology exhibits severe power-law distributions that challenge standard classification approaches. Existing scientific corpora lack comprehensive controlled vocabularies, focusing instead on broad categories and limiting systematic study of extreme imbalance. We introduce AstroConcepts, a corpus of English abstracts from 21,702 published astrophysics papers, labeled with 2,367 concepts from the Unified Astronomy Thesaurus. The corpus exhibits severe label imbalance, with 76% of concepts having fewer than 50 training examples. By releasing this resource, we enable systematic study of extreme class imbalance in scientific domains and establish strong baselines across traditional, neural, and vocabulary-constrained LLM methods. Our evaluation reveals three key patterns that provide new insights into scientific text classification. First, vocabulary-constrained LLMs achieve competitive performance relative to domain-adapted models in astrophysics classification, suggesting a potential for parameter-efficient approaches. Second, domain adaptation yields relatively larger improvements for rare, specialized terminology, although absolute performance remains limited across all methods. Third, we propose frequency-stratified evaluation to reveal performance patterns that are hidden by aggregate scores, thereby making robustness assessment central to scientific multi-label evaluation. These results offer actionable insights for scientific NLP and establish benchmarks for research on extreme imbalance.

Figures & tables

CorpusDocsLabelsHierarchySupervisionVocabularyDomain
General Domain
Loza Mencía and Fürnkranz (2010)19K7.2K2-levelHierarchicalEuroVocLegal
Lewis (1987)21K90NoneFlatAd-hocNews
RCV1800K1034-levelFlatAd-hocNews
Scientific Domain
Giles et al. (1998)3K6NoneFlatAd-hocCS
Table 1: Multi-label classification benchmarks sorted by corpus size. AstroConcepts provides tractable scale (21K documents) with deep hierarchical structure (11 levels) and domain-specific controlled vocabulary (UAT), enabling comprehensive experimentation on astrophysics literature classification.
StatisticValue
Documents
Total abstracts21,702
Labels
Total assignments93,547
Assigned unique labels1,864
Per abstract (mean/median)4.31 / 4
Table 2: Overall statistics for AstroConcepts .
Figure 1: Label frequency distribution (log-log scale) and fitted power-law function with exponent α=1.50\alpha=1.50 ( R2=0.825R^{2}=0.825 ). The long tail contains 76% of labels with fewer than 50 occurrences each, creating a severe class imbalance characteristic of scientific multi-label classification.
Bin# Labels%# Assign.%
Head ( >> 500)170.912,28813.1
Torso (50–500)42923.062,80867.1
Tail ( << 50)1,41876.118,45119.7
Total1,864100.093,547100.0
Table 3: Label frequency distribution across bins. The long tail contains 76% of labels but only 20% of assignments, creating a severe class imbalance characteristic of scientific multi-label classification.
DepthCount%
11064.5
21064.5
340517.1
468028.7
555423.4
634114.4
Table 4: Distribution of UAT concept depths in AstroConcepts showing annotation preference patterns across taxonomic levels.
 OverallRanking PrecisionRanking Recall
MethodPrecisionRecallF 1P@1P@3P@5R@1R@3R@5
Non-parametric  
Rule-based w/o var0.22900.21300.1950‡‡‡‡‡‡
Rule-based w var0.21700.27300.2140‡‡‡‡‡‡
kk -NN0.11650.75090.20170.61250.46070.36980.13980.31550.4221
Supervised neural models  
Table 5: Overall performance on AstroConcepts test set. Best results shown in bold . ‡ Not applicable for rule-based methods.
 F 1HeadTorsoTail 
MethodHeadTorsoTailP@3R@3P@3R@3P@3R@3Δ∗\Delta^{*}
BERT0.1800.1670.0210.0840.2050.1680.1830.0120.0240.159
SciBERT0.1930.1970.0230.0840.2110.1990.2120.0150.0260.170
astroBERT0.2160.3120.0810.0920.2300.3110.3330.0460.0880.135
DeepSeek0.2430.3760.1980.1070.2660.4170.4430.1350.2670.045
Table 6: performance across frequency bins. Head/Torso/Tail thresholds: >500>500 / 5050 – 500500 / <50<50 training examples. ∗ Δ\Delta = Head F 1 - Tail F 1 (lower is better). Best results in bold .
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Fine-tuned astroBERT Label Coverage
More in this topic

Astrophysics

See all 29 papers

Explore similar work

CardsList
May 7, 2026

AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification

Modern astronomical observatories generate a massive volume of multimodal data, creating a critical bottleneck for expert human review. While multimodal large language models (LLMs) have shown promise in interpreting complex visual and textual inputs, their ability to perform specialized scientific classification while providing interpretable reasoning remains understudied. We introduce AstroAlertBench, a comprehensive multimodal benchmark designed to evaluate LLM performance in astronomical event review along a three-stage logical chain: metadata grounding, scientific reasoning, and hierarchical classification over five categories. We use a pilot sample of 1,500 real-world alerts from the Zwicky Transient Facility (ZTF), a wide-field survey that scans the northern sky to detect transient astronomical events. On this dataset, we benchmark 13 frontier closed-source and open-weight LLMs that support visual input. Our results reveal that high accuracy does not always align with model ``honesty,'' defined as the ability to self-evaluate its reasoning, which affects its reliability as a real-world assistant. We further initialize a human-in-the-loop evaluation protocol as a precursor to future community-scale participation. Together, AstroAlertBench provides a framework for developing calibrated and interpretable astronomical assistants.

Claire Chen, Jiabao Sean Xiao, Shuze Daniel Liu +7

1California Institute of Technology · 2Massachusetts Institute of Technology · 3Purdue University

Multimodal ReasoningMultimodal Model EvaluationLLM Honesty
AskPDF
Aug 29, 2026

Efficient Context-Limited Telescope Bibliography Classification for the WASP-2025 Shared Task Using SciBERT

The creation of telescope bibliographies is a crucial part of assessing the scientific impact of observatories and ensuring reproducibility in astronomy. This task involves identifying, categorizing, and linking scientific publications that reference or use specific telescopes. However, this process remains largely manual and resource intensive. In this work, we present an efficient SciBERT-based approach for automatic classification of scientific papers into four categories - science, instrumentation, mention, and not telescope. Despite strict context-length constraints (maximum 512 tokens) and limited compute resources, our approach achieved a macro F1 score of 0.89, ranking at the top of the WASP-2025 leaderboard. We analyze the effect of truncation and show that even with half the samples exceeding the token limit, SciBERT's domain alignment enables robust classification. We discuss trade-offs between truncation, chunking, and long-context models, providing insights into the efficiency frontier for scientific text curation.

Madhusudhana Naidu

Topic pending
AskPDF
Sep 9, 2026

More than half of recent astronomy papers are written with language-model assistance

Language models leave a distinctive vocabulary in the prose they help write, and we measure how much of the astronomy literature now carries it. From the full text of 207,111 astro-ph papers spanning 2015 to mid-2026, we count those words in each paper and model the counts, in proportion to paper length, as a mixture of assisted and unassisted writing in a hierarchical Bayesian model. Papers from before 2020 calibrate the unassisted rate, and the 392 papers that disclose model use calibrate the assisted one. Our answer depends on how often these words would appear today if nobody used a model, a rate that must be modeled rather than observed, so we extend it past 2020 under three assumptions and report all three. For 2025 that gives 54−8+8 (stat, 95%) −0+26 (sys, background)54^{+8}_{-8}\,(\mathrm{stat},\,95\%)\,^{+26}_{-0}\,(\mathrm{sys,\ background})% of papers, the second error being the spread across the three. The estimate stays at or above 36% when we vary that choice, the calibration, and the requirement that adoption only rises. A word list built from the astro-ph corpus, keeping only words that rose across every subfield, leaves 2025 in the same range. Assisted writing is also getting harder to see, since authors adapt to the words that reveal it and the marker excess more than halves between 2023 and 2026. Our model allows for that fading, so it can separate a fainter trace from reduced use. More than half of recent astro-ph papers therefore carry a language-model trace, while only 0.81% of 2025 papers disclose it, one declaration for every ∼\sim66 papers with a trace.

Serat M. Saad, Yuan-Sen Ting

Department of Astronomy, The Ohio State University, 140 West 18th Avenue, Columbus, OH 43210, USA · Center for Cosmology and AstroParticle Physics (CCAPP), The Ohio State University, Columbus, OH 43210, USA · Max-Planck-Institut für Astronomie, Königstuhl 17, D-69117 Heidelberg, Germany

Language Model Auditing
AskPDF