Leveraging Multimodality for Real-Time Classification of Transients and Variables found by the Zwicky Transient Facility
Authors: Ved G. Shah, Nabeel Rehemtulla, Adam A. Miller, Sushant Sharma Chaudhary, Michael W. Coughlin, Antoine Le Calloch, Matthew J. Graham, Joahan Castaneda Jaimes, +8 more
Organizations: Department of Physics and Astronomy, Northwestern University, 2145 Sheridan Road, Evanston, IL 60208, USA · Center for Interdisciplinary Exploration and Research in Astrophysics (CIERA), 1800 Sherman Ave, Evanston, IL 60201, USA · NSF-Simons AI Institute for the Sky (SkAI), 172 E. Chestnut St., Chicago, IL 60611, USA · School of Physics and Astronomy, University of Minnesota, Minneapolis, Minnesota 55455, USA · Division of Physics, Mathematics, and Astronomy, California Institute of Technology, Pasadena, CA 91125, USA · Center for Data Driven Discovery, California Institute of Technology, Pasadena, CA 91125, USA · California Institute of Technology, Pasadena, CA 91125, USA · IPAC, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA 91125, USA · Caltech Optical Observatories, California Institute of Technology, Pasadena, CA 91125, USA · The Oskar Klein Centre, Department of Astronomy, AlbaNova, SE-106 91 Stockholm, Sweden · Department of Astronomy, University of California, University Drive, Berkeley, CA 94720, USA
Modern time-domain surveys such as the Zwicky Transient Facility (ZTF) generate hundreds of thousands of alerts each night, making real-time decisions for follow-up observations a central challenge in time-domain astronomy. Robust early classification is crucial for making informed decisions, but is hindered by sparse light curves and degeneracies between classes. In this work, we leverage multimodality to substantially improve real-time classification and demonstrate the practicality of our approach by deploying our model on the ZTF alert stream. Building on the Online Ranked Astrophysical CLass Estimator (ORACLE), we introduce the ORACLE-2 models, which combine light curves, metadata, and images for real-time hierarchical classification. Using both real and simulated datasets, we show that incorporating additional modalities consistently improves classification performance. On observations from ZTF's Bright Transient Survey, our best-performing model, ORACLE-2 Omni, achieves a macro F1 score of 0.73 -- an improvement of up to 11% over models using light curves and metadata alone, and up to 40% over light-curve-only models, with the strongest gains realized at early times. To demonstrate applicability to the Legacy Survey of Space and Time, which will increase alert volume by more than an order of magnitude, we train a light curve + metadata variant on the simulated ELAsTiCC dataset. This model achieves a macro F1 score of 0.88, an improvement of up to 13% over the light-curve-only variant, matching the performance of other state-of-the-art models. Finally, we quantify the trade-offs between performance and throughput, identifying regimes where multimodal approaches offer the greatest benefit. These results show that combining multiple modalities improves early-time classification, enabling more effective triage of high-volume alert streams for current and future time-domain surveys.
Modern astronomical observatories generate a massive volume of multimodal data, creating a critical bottleneck for expert human review. While multimodal large language models (LLMs) have shown promise in interpreting complex visual and textual inputs, their ability to perform specialized scientific classification while providing interpretable reasoning remains understudied. We introduce AstroAlertBench, a comprehensive multimodal benchmark designed to evaluate LLM performance in astronomical event review along a three-stage logical chain: metadata grounding, scientific reasoning, and hierarchical classification over five categories. We use a pilot sample of 1,500 real-world alerts from the Zwicky Transient Facility (ZTF), a wide-field survey that scans the northern sky to detect transient astronomical events. On this dataset, we benchmark 13 frontier closed-source and open-weight LLMs that support visual input. Our results reveal that high accuracy does not always align with model ``honesty,'' defined as the ability to self-evaluate its reasoning, which affects its reliability as a real-world assistant. We further initialize a human-in-the-loop evaluation protocol as a precursor to future community-scale participation. Together, AstroAlertBench provides a framework for developing calibrated and interpretable astronomical assistants.
Claire Chen, Jiabao Sean Xiao, Shuze Daniel Liu +7
The Nancy Grace Roman Space Telescope (Roman), set for launch as early as September 2026, will conduct wide-field infrared imaging surveys with unprecedented spatial resolution and cadence, enabling the discovery of millions of astronomical transients. Hence, it is necessary to have automated pipelines for generating alerts in place so that the telescope can begin discovering reliable transients and variable objects soon after it is launched. However, no real Roman data currently exist, making the development of such pipelines difficult. In this work, we present a machine learning model RuBR and a general methodology for distinguishing genuine transient and variable detections from spurious (bogus) detections within the RAPID pipeline. In particular, we present three models using this methodology: RuBRcomb trained and tested on combined locally injected and OpenUniverse2024 transients, RuBRloc trained on locally injected transients and tested on OpenUniverse2024 transients, and RuBRDA that combines locally injected transients with a fraction of OpenUniverse2024 transients in domain-adaptation mode for training. This paves the way for strategies to adapt the RuBRcomb model to real observations in the absence of any ground-truth labels during the early phases of the Roman mission. While the image differencing pipeline continues to be improved, our experimental results demonstrate the effectiveness of the proposed approach and its promise for robust real-bogus classification in the Roman era.
Karan Gandhi, Ashish A. Mahabal, Jacob E. Jencson +6
Time-domain surveys generate many transient candidates, making Real-Bogus classification a critical step in automated discovery pipelines. Reliable labels are costly, while community labels can be noisy and survey-dependent. We aim to develop a Real-Bogus classification framework that can be trained without human-labeled data using injected transients and bogus-dominated survey data, remains robust under strong class contamination, and provides calibrated uncertainty quantification. We combine simulated transient injections with a contaminated survey class and train a dual-network model using asymmetric co-teaching for classes with different label-noise levels. We evaluate performance on a benchmark subset and analyze the learned representation with latent-space visualization tools. For uncertainty quantification (UQ), we compare MC dropout and deep ensembles and propose a low-cost hybrid strategy that exploits the dual-network setting to improve calibration. We extend the evaluation to the light-curve domain to assess recovery of light-curve classes. The method achieves strong Real-Bogus performance on the labeled subset and remains stable under severe class contamination. It recovers transient light-curve classes with high fidelity, while single-source identification is limited by ambiguity in light-curve-derived labels. Our hybrid UQ approach achieves competitive calibration relative to more expensive ensemble baselines. Latent-space analyses indicate that uncertainty aligns with the decision boundary and reveal subclasses within the bogus population. Our results show that injection-driven, weakly supervised training can enable scalable and consistent Real-Bogus classification without human-labeled training data while providing calibrated uncertainties. The method is suited for transfer to forthcoming surveys by re-running the injection-based training pipeline.
Raphaël Bonnet-Guerrini, Bruno Sanchez, Dominique Fouchez +5