Missed Detections and False Alarms in a CNN Technosignature Detector on Breakthrough Listen Data

Omer Imran

astro-ph.IMOct 15, 2026

Missed Detections and False Alarms in a CNN Technosignature Detector on Breakthrough Listen Data

Authors: Omer Imran

Organizations: Independent researcher

Abstract

Radio technosignature searches rely on machine learning detectors to shrink the candidate pool before a human looks at it. They are validated on held-out data from the training distribution, which excludes the cases where they are most likely to fail: a technosignature has unknown morphology, and terrestrial interference is unbounded in variety. I trained a standard competition detector once, froze it, and probed it with 15,000 injections across ten morphologies and five amplitudes, placed into real Green Bank backgrounds. Two directions were tested: genuine signals with morphologies other than a straight narrowband line, and interference resembling the training data. Recall did not follow the predicted ordering, and it fell as signals brightened, even for a needle of the kind the detector was trained on (from 0.590 to 0.090). One interference morphology reached a false positive rate of 0.627, comparable to the detector's own baseline recall of 0.5525 on genuine signals. Gradient-weighted Class Activation Mapping (Grad-CAM) showed attention along the injected path in one missed family by a pre-specified test, and visually in a second. A pre-specified, time-averaged ON-OFF filter removed false alarms and genuine detections at the same rate. The detector reads only the ON panels. Since the only evidence distinguishing a genuine candidate from terrestrial interference lies in the discarded OFF panels, it cannot tell if a signal originates from the target star; it can only tell if it resembles the training data. This explains the false alarm directly; the missed signals reflect what the detector learned to recognise, which reading the OFF panels would not by itself change.

Figures & tables

Figure 1: An example cadence from the dataset. Panels 0, 2 and 4 are ON (telescope pointed at the target); panels 1, 3 and 5 are OFF. The bright vertical line is present in every panel and is therefore interference.

Direction 1 — genuine signal morphologies
ID	Name	Description	Parameters
D1-1	extreme drift	a straight line with a steep tilt	linear drift of 60 ch across the panel
D1-2	quadratic	a line that curves smoothly in one direction	quadratic path, 60 ch of curvature across the panel
D1-3	S-shaped	a line that oscillates from side to side	sinusoid of $\pm$ 25 ch, one cycle per panel
D1-4	pulsed	a line broken into separate segments	drift of 5 ch; on for 35% of each 55-sample period
D1-5	dispersed burst	a line that sweeps and curves sharply	broadband; arrival at sample $30+180(f/256)^{2}$ , $\sigma=6$ samples

Table 1: The ten injected signal families. Lines are Gaussian in frequency with

\sigma=1.2

channels (ch); a panel is 256 channels by 273 time samples.

Figure 2: Examples of the five Direction 1 morphologies, each beside the clean background.

Figure 3: Examples of the five Direction 2 morphologies, each beside the clean background.

Figure 4: Average of the ON panels minus average of the OFF panels for the cadence in Figure 1 .

Figure 5: Recall at the fixed threshold for each Direction 1 family and amplitude. The dashed line marks baseline recall (0.5525); the dotted line marks the clean control rate (0.0133).

Figure 6: Detection rate against injected amplitude. Left: recall for Direction 1. Right: false positive rate for Direction 2.

Figure 7: False positive rate at the fixed threshold for each Direction 2 family and amplitude. The figure’s label ‘real needles’ refers to the competition’s labelled needles, which are synthetic.

Panel of Figure 8 — Figure 8: Grad-CAM autopsy at an amplitude of 4.0. (a) Q2_bright_hit: D1-3 S-shaped, detected. (b) Q2c_bright_mis: D1-1 extreme drift, missed. Columns: input, Grad-CAM map, and CAM difference (injected minus clean). Inputs are shown as the detector sees them: the three ON panels side by side, with frequency running vertically; white lines mark panel boundaries and the green line the injection’s starting channel.

Panel of Figure 9 — Figure 9: Grad-CAM autopsy at an amplitude of 4.0. (a) Q3_pulsed: D1-4. (b) Q3_dispersed: D1-5. Both missed. Layout as in Figure 8 .

Panel of Figure 10 — Figure 10: Grad-CAM autopsy at an amplitude of 0.5 and matched scores. (a) Q1_fool: D2-5 high drift, a false alarm. (b) Q1c_true_det: D1-2 quadratic, a true detection. Layout as in Figure 8 .

Figure 11: Distribution of the ON-OFF contrast statistic S for 300 clean backgrounds and 1,775 flagged genuine signals. The dashed line is the veto threshold, t = 1.0266. Genuine signals are pooled across families; D1-3 contributes 57% of them.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Family	0.25	0.5	1.0	2.0	4.0
D1-1 extreme drift	0.183 (0.144–0.231)	0.410 (0.356–0.466)	0.273 (0.226–0.326)	0.077 (0.052–0.112)	0.020 (0.009–0.043)
D1-2 quadratic	0.453 (0.398–0.510)	0.533 (0.477–0.589)	0.340 (0.289–0.395)	0.127 (0.094–0.169)	0.017 (0.007–0.038)
D1-3 S-shaped	0.253 (0.207–0.305)	0.570 (0.513–0.625)	0.750 (0.698–0.796)	0.880 (0.838–0.912)	0.900 (0.861–0.929)
D1-4 pulsed	0.023 (0.011–0.047)	0.017 (0.007–0.038)	0.017 (0.007–0.038)	0.003 (0.001–0.019)	0.010 (0.003–0.029)
D1-5 dispersed burst	0.013 (0.005–0.034)	0.017 (0.007–0.038)	0.010 (0.003–0.029)	0.010 (0.003–0.029)	0.010 (0.003–0.029)
D2-1 zero drift	0.013 (0.005–0.034)	0.017 (0.007–0.038)	0.010 (0.003–0.029)	0.013 (0.005–0.034)	0.010 (0.003–0.029)

Appendix

Table 2: Rate above the fixed threshold for every family and amplitude: recall for Direction 1, false positive rate for Direction 2. Each cell is computed on 300 injections; 95% Wilson confidence intervals in brackets.

Radio Astronomy

See all 8 papers

Sep 16, 2026

Deep Learning Detection of Beyond-General-Relativity Deviations in Gravitational-Wave Signals: A Detection-Threshold Study with Real LIGO Noise

We study machine-learning detection of controlled beyond-General-Relativity (beyond-GR) deviations in gravitational-wave signals, using both synthetic aLIGO-PSD noise and real LIGO H1 detector strain. Three deviation families are applied to General-Relativistic inspiral-merger-ringdown waveforms: amplitude modulation, phase modulation, and frequency modulation, each parameterized by a dimensionless strength coefficient

β

. A hybrid classifier combining a one-dimensional convolutional neural network with ten hand-crafted waveform statistics is trained on GR and modified waveforms and tested on a deviation type excluded from training. The central result is a quantitative detectability curve as a function of

β

. Using the real GW150914 strain as a template and real H1 detector noise, we find a detection threshold at

β\approx 0.25

, with accuracy rising smoothly from chance at

β\leq 0.2

to perfect classification at

β\geq 0.5

. The threshold value is specific to the quadratic-in-time modulation form adopted here and should not be interpreted as a generic constraint on beyond-GR parameters. We nevertheless argue that the negative result at small

β

is informative: it establishes a quantitative limit on machine-learning-only beyond-GR searches in real detector noise, in the absence of matched-filter signal extraction.

Muhammad Adnan Shahzad

Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, QC, Canada

Gravitational-Wave Astronomy

Ask PDF

Sep 30, 2026

MADGRAV: a multilevel anomaly-detection pipeline for gravitational-wave searches applied to LIGO data

We present the results of MADGRAV, a deep-learning-based search for high-mass compact binary coalescences, applied to the data collected by the LIGO interferometers during the third observing run and during the first and second part of the fourth observing run. The MADGRAV pipeline consists of a series of sequential convolutional neural networks that perform anomaly detection, glitch classification, coherence testing, and signal ranking. Data from the Hanford and Livingston LIGO detectors are studied (both individually and in coherence) by way of 1 second Q-transform windows. Of the candidates that survive every stage of the pipeline, 48 reach the significance threshold, and we report 47 gravitational wave detections characterised by a false alarm rate below

1\,{\rm yr}^{-1}

with a probability of astrophysical origin

p_{\rm astro}>0.9

. Of the 47 detections, 44 are shared with the minimally modelled coherent WaveBurst search. The observed total source-frame masses, extracted from official gravitational wave transient catalogues, are in the

14-236 M_{\odot}

range with a median of

69 M_{\odot}

, and a median SNR of 16. We note that the recovered fraction of confident detections rises with mass: for LIGO detectors network SNR

>10

the pipeline recovers

8.1\%

of confident catalog events below

30 M_{\odot}

39.8\%

between

30

and

100 M_{\odot}

, and

53.3\%

above

100 M_{\odot}

, corresponding to

33.3\%

45.5\%

and

53.3\%

of the events detected by coherent WaveBurst in the same bins. These results suggest that anomaly detection pipelines can serve as an independent detection channel complementary to matched filtering in the high-mass high-SNR regime.

Gianluca Inguglia, Huw Haigh, Ulyana Dupletsa +1

Marietta Blau Institute for Particle Physics, Austrian Academy of Sciences, Vienna, Austria · Universit`a degli Studi di Urbino ‘Carlo Bo’, I-61029 Urbino, Italy · INFN, Sezione di Firenze, I-50019 Sesto Fiorentino, Firenze, Italy

Gravitational-Wave Astronomy Anomaly Detection

Ask PDF

Apr 18, 2026

AstroSURE: Learning to Remove Noise from Astronomical Images Without Ground Truth Data

In astronomical imaging, the low photon count of exposures necessitates extensive post-processing steps, including contamination removal and denoising. This paper evaluates deep-learning denoising methods that can be trained without clean ground-truth images and assesses their utility for detection11 oriented analysis of astronomical data. We adapt and compare Noise2Noise, Stein's Unbiased Risk Estimator, and blind-spot-based methods using synthetic data and real observations from the Hubble Space Telescope (HST) and the Canada-France-Hawaii Telescope (CFHT). Performance is evaluated using object-detection metrics, including correct detection rate and false alarm rate, together with image-based metrics and pixel-distribution diagnostics. The results show that these methods can improve faint-source detectability relative to the original noisy images, with encouraging gains on HST data after domain-consistent initialization, while transfer to CFHT data is more limited, highlighting the importance of instrument/domain similarity for unsupervised adaptation.

Omid Vaheb, Sebastien Fabbro, Stark Draper

Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, 10 King’s College Road, Toronto, ON M5S 3G4, Canada · National Research Council Herzberg Astronomy an Astrophysics Research Centre, 5071 West Saanich Road, Victoria, BC V9E 2E7, Canada · Department of Computer Science, University of British Columbia, Vancouver, BC, V6T 1Z4, Canada

Self-Supervised Image Denoising Noise Robustness Image Denoising Astrophysics

Ask PDF

Abstract

Radio Astronomy

Explore similar work

Deep Learning Detection of Beyond-General-Relativity Deviations in Gravitational-Wave Signals: A Detection-Threshold Study with Real LIGO Noise

MADGRAV: a multilevel anomaly-detection pipeline for gravitational-wave searches applied to LIGO data

AstroSURE: Learning to Remove Noise from Astronomical Images Without Ground Truth Data