cs.SDOct 9, 2026
SaveSpeech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription
Abstract
Automatic speech recognition produces words but drops the non-verbal content of a recording: the breaths, coughs, sighs and laughter of the speaker, and the horns, music or barking around them. We describe Speech Event Detector, a 5.8-million-parameter convolutional-Transformer model that outputs probabilities every 40 ms for 54 event classes (20 speaker vocalizations and 34 background sounds), together with post-processing that converts these probabilities into timestamped events and a procedure for merging the events into a Whisper transcript. The model is trained on material drawn from seven public datasets whose labels come in incompatible forms: single detected regions, free-text tags with timestamps, temporally strong ontology labels, symbols embedded in transcripts, isolated recordings and environmental recordings. We unify them using rule-based label mapping, recover timestamps for transcript-embedded labels through CTC forced alignment, locating 6,360 of 8,620 sounds, and combine about 168 hours of real annotated training recordings with synthetic mixtures and augmentation using real background noise. Per-class thresholds for two operating profiles are tuned on real held-out recordings, and performance is estimated using two-fold cross-validation. On whole held-out recordings from four datasets, considering the 48 classes with at least ten held-out events, the sensitive profile detects an average of 49 percent of a class's events, covering 67 percent of all 9,367 events, at a macro precision of 0.30. The balanced profile achieves a macro recall of 0.37 at a precision of 0.36, with macro event F1 scores of 0.32 and 0.33, respectively. The detector runs about 39 times faster than real time on eight CPU threads.
Figures & tables
Figure 1: System overview. (a) Training: real annotated windows, synthetic mixtures and augmented copies (thumbnails are log-mel spectrograms of actual training examples) become 8 s windows with 40 ms frame targets; the network is trained with a frame-level loss plus a clip-level term. (b) Inference: audio of any length is processed in 8 s windows (thumbnails show a held-out clip, its spectrogram and the resulting class probabilities); per-class thresholds and post-processing give timestamped events, which are merged with the word timestamps of a Whisper transcript. (c) The two repeated building blocks. marks trained parts, fixed ones. Section references: data § 4 , model § 5 , training § 6 , inference § 7 .
| Source | Size | Labels and use | Held-out part | License |
|---|---|---|---|---|
| NonVerbalSpeech-38K [ 39 ] | 38,718 clips, 130.6 h; 36,925 Mandarin, 1,793 English | one non-verbal event per clip with a detected region; event crops, speech beds, real windows | 10 % by stable hash (3,828 clips) | CC BY-NC 4.0 |
| Vaani noise events [ 1 , 32 , 29 ] | 90,637 clips, 154.6 h; 58 Indian languages + English, 162 districts | 319 free-text noise tags with start/end (11,111 verified, 61,642 unverified, 17,884 without timestamps); crops, real windows | 14 districts, verified clips only | CC BY 4.0 |
| NonverbalTTS [ 3 ] | 6,256 clips, 17.6 h, English (VoxCeleb, Expresso) | 8,620 sounds marked inside transcripts, no timestamps; forced alignment ( Section 4.2 ); real windows, crops, speech beds | official dev + test (405 clips) | annotations CC BY-NC-SA 4.0; audio per source corpora |
| AudioSet-Strong [ 15 , 8 ] | 93,609 10 s clips (79,564 train, 14,045 eval), 260 h | 455 event names with start/end (220 mapped); streamed subset; real windows, crops | official evaluation split | not stated on the mirror |
| OpenSLR 99 [ 7 ] | 727 clips, 410 speakers, 16 classes | isolated phone-recorded vocalizations; trimmed crops | 15 % of speakers | CC BY-NC-ND 4.0 |
| DNC [ 40 ] | 4,377 home recordings | noise type; ambient crops (TV/radio, music, cars, appliances) and a background-noise pool | 10 % of sessions | MIT |
Table 1: Training and evaluation sources. Sizes are those of the full releases as indexed for this work. Licenses are quoted from each source’s own page; see Section 9 for their consequences.
| Audio bank | Items | Hours |
| real windows: NVS-38K, Vaani | 37,434 | 80.6 |
| real windows: AudioSet-Strong | 33,750 | 92.5 |
| real windows: NonverbalTTS | 4,549 | 11.8 |
| event crops: NVS-38K, Vaani | 52,654 | 25.2 |
| event crops: AudioSet-Strong | 75,343 | 25.9 |
| event crops: NonverbalTTS | 6,360 | 1.0 |
Table 2: Audio banks (16 kHz int16, memory-mapped) and the cached feature pools of 8 s examples used for training and model selection.
Figure 2: Event crops per class in the audio banks (training and validation), by source. AudioSet-Strong supplies most background classes and most crops of the rarer speaker classes; NVS-38K and NonverbalTTS dominate breath , cough , laugh , sigh , sniff and throat_clear ; Vaani dominates lip_smack and nose_blow ; DNC is the main source of tv_speaker . Exact counts are in the released data/event_crops_by_source.csv .
Figure 3: Architecture, with tensor shapes measured on the released model. Top: the waveform becomes a 64-band log-mel spectrogram, which five convolutional blocks turn into 201 time steps of 768 features. The blocks are drawn to scale (height: frequency bins; width: time steps; depth: square root of the channels), with shapes channels frequency time below each and strides under the arrows; Figure 1 c shows the content of a block. Bottom: the features are projected to 256 dimensions, given sinusoidal positions and passed through six pre-norm Transformer encoder layers; a linear layer with a sigmoid gives one probability per class every 40 ms.
| Layer | Operation | Output shape | Parameters |
| input | waveform, 16 kHz, 8 s | – | |
| log-mel | STFT 512 / win 400 / hop 160, 64 mel bands, log, scale | – | |
| conv block 1 | 2 (conv 3 3, BN, ReLU); 1 48 ch; stride (f,t)=(1, 1) | 21,456 | |
| conv block 2 | 2 (conv 3 3, BN, ReLU); 48 48 ch; stride (f,t)=(2, 1) | 41,760 | |
| conv block 3 | 2 (conv 3 3, BN, ReLU); 48 96 ch; stride (f,t)=(2, 1) | 124,992 | |
| conv block 4 | 2 (conv 3 3, BN, ReLU); 96 96 ch; stride (f,t)=(2, 2) | 166,464 |
Table 3: Layer-by-layer view of the network for one 8 s window (batch dimension omitted). Shapes are channels frequency time for the convolutional part and time features afterwards; they were measured with forward hooks on the released model. Strides are (frequency, time). Parameter counts are exact.
Figure 4: Parameters per layer, coloured by component (5,802,790 in total).
Figure 5: A held-out clip passing through the released model; all panels share the time axis. Convolutional panels show the mean absolute activation over channels (blue scale); vector-valued panels show each feature standardized over time (blue below, red above its mean). The bottom panel shows the output probabilities of the annotated classes and the five most active classes (white 0, dark blue 1), with the annotated events as orange lines. The dark column at the right edge of block 1 is the zero padding of this 7.9 s clip to the 8 s window.
| Augmentation | Prob. | Parameters |
|---|---|---|
| silence inserted outside events, labels shifted | 0.5 | 1–2 pauses of 0.2–1.5 s |
| reverberation (exponentially decaying noise response) | 0.25 | RT 60 0.15–0.8 s; direct path 1–4 the tail peak |
| telephone band | 0.2 | Butterworth band-pass 300–3400 Hz (order 4 per edge); 8-bit -law with probability 0.5 |
| background noise | 0.85 | real noise 40 % (SNR 3–25 dB), babble 20 % (12–30 dB), hum 10 % (10–35 dB), white/pink/brown 30 % (8–35 dB) |
| gain | 1 | to dB |
Table 4: Augmentations applied to each stored augmented copy; each is drawn independently.
| Setting | Value |
|---|---|
| Input | 16 kHz mono, 8 s windows; 64-band log-mel, 25 ms window, 10 ms hop, 50–8000 Hz |
| Output | 54 sigmoid outputs every 40 ms (201 frames per window) |
| Parameters | 5,802,790 trainable (plus a fixed sinusoidal position table) |
| Optimizer | AdamW, weight decay 0.01, gradient-norm clip 5 |
| Schedule | OneCycle, peak LR 0.0003, 10 % warm-up, 10 epochs |
| Batch | 32 windows; 5,343 steps per epoch |
Table 5: Training configuration of the released model.
Figure 6: Training loss (a) and validation macro frame AP per held-out set (b). The selection score is the mean over the four real sets and the noisy set; the synthetic set is shown but not used for selection. Values are in data/training_log.csv .
Figure 7: Frame-level average precision (bars) and two-fold cross-validated F1 of the balanced profile (dots; thresholds tuned on the other half of the windows) per class, on real held-out windows. marks weak classes ( Section 6.3 ).
| Profile | Precision | Recall | Objective |
|---|---|---|---|
| balanced | 0.352 | 0.414 | F1 = 0.351 |
| sensitive | 0.267 | 0.521 | F2 = 0.400 |
Table 6: Cross-validated frame-level operating points of the two threshold profiles, averaged over the 43 reliably measured classes.
Figure 8: Cross-validated frame precision and recall of each reliable class under the balanced (F1) and sensitive (F2) thresholds; grey lines connect the two points of a class. Diamonds are the class means.
| Recall | Precision | F1 | Micro recall | |||||||
| Classes | Events | S | B | S | B | S | B | S | B | |
| speaker sounds | 16 | 3,271 | 0.455 | 0.327 | 0.383 | 0.395 | 0.378 | 0.338 | 0.703 | 0.542 |
| background sounds | 32 | 6,096 | 0.510 | 0.390 | 0.261 | 0.345 | 0.294 | 0.325 | 0.656 | 0.528 |
| all | 48 | 9,367 | 0.491 | 0.369 | 0.302 | 0.361 | 0.322 | 0.329 | 0.672 | 0.533 |
Table 7: Event-level results on whole held-out real clips, for the sensitive (S) and balanced (B) threshold profiles. Macro averages over classes with at least 10 held-out events; the last two columns give the event-weighted (micro) recall. A true event is found if a same-class prediction overlaps it; a prediction is correct if it overlaps a same-class true event. Thresholds were tuned on held-out data that overlaps these clips, so absolute values are optimistic ( Section 8.1 ).
Figure 9: Event-level recall (a) and precision (b) per class on whole held-out clips with the balanced and sensitive profiles; held-out event counts in parentheses. Classes with at least 10 events, sorted by sensitive recall. Values are in data/event_eval.csv .
Figure 10: Self-attention of encoder layers 1 (a) and 6 (b), averaged over the eight heads, for the clip of Figure 12 ; coloured bars on top mark the annotated events. (c) Mean attended distance per layer, ms averaged over queries .
Figure 11: Worked examples: eight held-out clips, two per source, chosen by a fixed rule (text). Each panel shows the log-mel spectrogram, the annotated events (orange, upper bar of each row) and the events detected with the sensitive profile (blue, lower bar). “found” counts annotated events overlapped by a same-class detection.
| Clip | Dur. | Annotated events | Detected (sensitive) | Found S | Found B |
|---|---|---|---|---|---|
| nv-1 | 6.9 | laugh 6.1–6.9 | cough 5.8–7.0, laugh 5.8–7.0 | 1/1 | 0/1 |
| nv-2 | 7.7 | cough 6.4–6.6 | throat_clear 5.9–7.3 | 0/1 | 0/1 |
| vaani-1 | 11.1 | breath 2.0–2.3, horn 3.8–8.2, breath 6.1–6.4, breath 8.6–8.9 | breath 0.0–2.5, horn 0.0–10.8, vehicle 0.0–10.8, breath 5.2–7.0, breath 7.8–9.5 | 4/4 | 4/4 |
| vaani-2 | 10.6 | dog_bark 0.9–1.5, breath 9.1–9.4 | fan 0.0–10.6, breath 4.8–6.3 | 0/2 | 0/2 |
| nvtts-1 | 7.5 | breath 0.1–0.3 | – | 0/1 | 0/1 |
| nvtts-2 | 4.2 | laugh 0.8–1.2 | laugh 0.5–1.8, breath 0.7–1.8 | 1/1 | 0/1 |
Table 8: Worked examples on held-out clips (selection rule in Section 8.7 ). Times are in seconds. “Found” counts annotated events overlapped by a same-class detection, for the sensitive (S) and balanced (B) profiles. Detected classes that the clip’s source does not annotate cannot be judged.
Figure 12: A held-out Vaani clip (Deoria district; selection rule in the text): (a) log-mel spectrogram; (b) frame probabilities of the annotated and detected classes, with their sensitive thresholds as thin lines; (c) annotated events (solid) and detected events (hatched). Music is detected; the sneeze and the breath are missed; tv_speaker is a false alarm.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| balanced | sensitive | event level | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Class | Type | Eval | Pos. | AP | CV F1 | CV F2 | Events | R-S | P-S | R-B | P-B | ||
| breath | S | NVTA | 16,762 | 0.499 | 0.850 | 0.499 | 0.725 | 0.616 | 1,116 | 0.80 | 0.49 | 0.67 | 0.64 |
| burp | S | A | 136 | 0.218 | 0.650 | 0.285 | 0.375 | 0.361 | 7 | 0.57 | 0.16 | 0.29 | 0.10 |
| chew † | S | A | 242 | 0.039 | 0.900 | 0.056 | 0.600 | 0.108 | 29 | 0.10 | 0.08 | 0.00 | 0.00 |
| cough | S | NVTA | 2,033 | 0.313 | 0.875 | 0.400 | 0.750 | 0.493 | 329 | 0.56 | 0.44 | 0.39 | 0.59 |
| cry † | S | NA | 963 | 0.075 | 0.900 | 0.048 | 0.600 | 0.110 | 61 | 0.43 | 0.11 | 0.08 | 0.09 |
Table 9: Per-class results. Type: S = speaker sound, B = background. Eval: validation sources that label the class (N = NVS-38K, V = Vaani, T = NonverbalTTS, A = AudioSet-Strong). Pos. = positive validation frames (40 ms). AP = frame-level average precision. = threshold; CV = 2-fold cross-validated frame F1 (balanced) or F2 (sensitive). Event level: held-out events, and event recall (R) and precision (P) with the sensitive (S) and balanced (B) profiles. † weak class.
| Class | Type | NVS-38K | Vaani | NV-TTS | AudioSet | SLR99 | DNC | DEMAND | Total |
|---|---|---|---|---|---|---|---|---|---|
| breath | S | 2,162 | 35,663 | 6,121 | 27,232 | 44 | 0 | 0 | 71,222 |
| music | B | 0 | 6,872 | 0 | 51,454 | 0 | 229 | 0 | 58,555 |
| bird | B | 0 | 17,694 | 0 | 37,443 | 0 | 0 | 0 | 55,137 |
| vehicle | B | 0 | 4,854 | 0 | 25,450 | 0 | 692 | 4 | 31,000 |
| wind | B | 0 | 0 | 0 | 25,079 | 0 | 0 | 0 | 25,079 |
| laugh | S | 6,912 | 99 | 1,278 | 14,489 | 25 | 0 | 0 | 22,803 |
Table 10: Annotated events per class in each source’s full index, before any sampling caps. DEMAND counts environments.