Leveraging Multimodality for Real-Time Classification of Transients and Variables found by the Zwicky Transient Facility
Organizations: Department of Physics and Astronomy, Northwestern University, 2145 Sheridan Road, Evanston, IL 60208, USA · Center for Interdisciplinary Exploration and Research in Astrophysics (CIERA), 1800 Sherman Ave, Evanston, IL 60201, USA · NSF-Simons AI Institute for the Sky (SkAI), 172 E. Chestnut St., Chicago, IL 60611, USA · School of Physics and Astronomy, University of Minnesota, Minneapolis, Minnesota 55455, USA · Division of Physics, Mathematics, and Astronomy, California Institute of Technology, Pasadena, CA 91125, USA · Center for Data Driven Discovery, California Institute of Technology, Pasadena, CA 91125, USA · California Institute of Technology, Pasadena, CA 91125, USA · IPAC, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA 91125, USA · Caltech Optical Observatories, California Institute of Technology, Pasadena, CA 91125, USA · The Oskar Klein Centre, Department of Astronomy, AlbaNova, SE-106 91 Stockholm, Sweden · Department of Astronomy, University of California, University Drive, Berkeley, CA 94720, USA
Abstract
Modern time-domain surveys such as the Zwicky Transient Facility (ZTF) generate hundreds of thousands of alerts each night, making real-time decisions for follow-up observations a central challenge in time-domain astronomy. Robust early classification is crucial for making informed decisions, but is hindered by sparse light curves and degeneracies between classes. In this work, we leverage multimodality to substantially improve real-time classification and demonstrate the practicality of our approach by deploying our model on the ZTF alert stream. Building on the Online Ranked Astrophysical CLass Estimator (ORACLE), we introduce the ORACLE-2 models, which combine light curves, metadata, and images for real-time hierarchical classification. Using both real and simulated datasets, we show that incorporating additional modalities consistently improves classification performance. On observations from ZTF's Bright Transient Survey, our best-performing model, ORACLE-2 Omni, achieves a macro F1 score of 0.73 -- an improvement of up to 11% over models using light curves and metadata alone, and up to 40% over light-curve-only models, with the strongest gains realized at early times. To demonstrate applicability to the Legacy Survey of Space and Time, which will increase alert volume by more than an order of magnitude, we train a light curve + metadata variant on the simulated ELAsTiCC dataset. This model achieves a macro F1 score of 0.88, an improvement of up to 13% over the light-curve-only variant, matching the performance of other state-of-the-art models. Finally, we quantify the trade-offs between performance and throughput, identifying regimes where multimodal approaches offer the greatest benefit. These results show that combining multiple modalities improves early-time classification, enabling more effective triage of high-volume alert streams for current and future time-domain surveys.