Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription

Ayush Kumar Bar; Promila Ghosh

cs.SDOct 9, 2026

Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription

Authors: Ayush Kumar Bar, Promila Ghosh

Abstract

Automatic speech recognition produces words but drops the non-verbal content of a recording: the breaths, coughs, sighs and laughter of the speaker, and the horns, music or barking around them. We describe Speech Event Detector, a 5.8-million-parameter convolutional-Transformer model that outputs probabilities every 40 ms for 54 event classes (20 speaker vocalizations and 34 background sounds), together with post-processing that converts these probabilities into timestamped events and a procedure for merging the events into a Whisper transcript. The model is trained on material drawn from seven public datasets whose labels come in incompatible forms: single detected regions, free-text tags with timestamps, temporally strong ontology labels, symbols embedded in transcripts, isolated recordings and environmental recordings. We unify them using rule-based label mapping, recover timestamps for transcript-embedded labels through CTC forced alignment, locating 6,360 of 8,620 sounds, and combine about 168 hours of real annotated training recordings with synthetic mixtures and augmentation using real background noise. Per-class thresholds for two operating profiles are tuned on real held-out recordings, and performance is estimated using two-fold cross-validation. On whole held-out recordings from four datasets, considering the 48 classes with at least ten held-out events, the sensitive profile detects an average of 49 percent of a class's events, covering 67 percent of all 9,367 events, at a macro precision of 0.30. The balanced profile achieves a macro recall of 0.37 at a precision of 0.36, with macro event F1 scores of 0.32 and 0.33, respectively. The detector runs about 39 times faster than real time on eight CPU threads.

Figures & tables

Panel of Figure 1 — Figure 1: System overview. (a) Training: real annotated windows, synthetic mixtures and augmented copies (thumbnails are log-mel spectrograms of actual training examples) become 8 s windows with 40 ms frame targets; the network is trained with a frame-level loss plus a clip-level term. (b) Inference: audio of any length is processed in 8 s windows (thumbnails show a held-out clip, its spectrogram and the resulting class probabilities); per-class thresholds and post-processing give timestamped events, which are merged with the word timestamps of a Whisper transcript. (c) The two repeated building blocks. marks trained parts, fixed ones. Section references: data § 4 , model § 5 , training § 6 , inference § 7 .

Source	Size	Labels and use	Held-out part	License
NonVerbalSpeech-38K [ 39 ]	38,718 clips, 130.6 h; 36,925 Mandarin, 1,793 English	one non-verbal event per clip with a detected region; event crops, speech beds, real windows	10 % by stable hash (3,828 clips)	CC BY-NC 4.0
Vaani noise events [ 1 , 32 , 29 ]	90,637 clips, 154.6 h; 58 Indian languages + English, 162 districts	319 free-text noise tags with start/end (11,111 verified, 61,642 unverified, 17,884 without timestamps); crops, real windows	14 districts, verified clips only	CC BY 4.0
NonverbalTTS [ 3 ]	6,256 clips, 17.6 h, English (VoxCeleb, Expresso)	8,620 sounds marked inside transcripts, no timestamps; forced alignment ( Section 4.2 ); real windows, crops, speech beds	official dev + test (405 clips)	annotations CC BY-NC-SA 4.0; audio per source corpora
AudioSet-Strong [ 15 , 8 ]	93,609 10 s clips (79,564 train, 14,045 eval), 260 h	455 event names with start/end (220 mapped); streamed subset; real windows, crops	official evaluation split	not stated on the mirror
OpenSLR 99 [ 7 ]	727 clips, 410 speakers, 16 classes	isolated phone-recorded vocalizations; trimmed crops	15 % of speakers	CC BY-NC-ND 4.0
DNC [ 40 ]	4,377 home recordings	noise type; ambient crops (TV/radio, music, cars, appliances) and a background-noise pool	10 % of sessions	MIT

Table 1: Training and evaluation sources. Sizes are those of the full releases as indexed for this work. Licenses are quoted from each source’s own page; see Section 9 for their consequences.

Audio bank	Items	Hours
real windows: NVS-38K, Vaani	37,434	80.6
real windows: AudioSet-Strong	33,750	92.5
real windows: NonverbalTTS	4,549	11.8
event crops: NVS-38K, Vaani	52,654	25.2
event crops: AudioSet-Strong	75,343	25.9
event crops: NonverbalTTS	6,360	1.0

Table 2: Audio banks (16 kHz int16, memory-mapped) and the cached feature pools of 8 s examples used for training and model selection.

Figure 2: Event crops per class in the audio banks (training and validation), by source. AudioSet-Strong supplies most background classes and most crops of the rarer speaker classes; NVS-38K and NonverbalTTS dominate breath , cough , laugh , sigh , sniff and throat_clear ; Vaani dominates lip_smack and nose_blow ; DNC is the main source of tv_speaker . Exact counts are in the released data/event_crops_by_source.csv .

Panel of Figure 3 — Figure 3: Architecture, with tensor shapes measured on the released model. Top: the waveform becomes a 64-band log-mel spectrogram, which five convolutional blocks turn into 201 time steps of 768 features. The blocks are drawn to scale (height: frequency bins; width: time steps; depth: square root of the channels), with shapes channels $\times$ frequency $\times$ time below each and strides under the arrows; Figure 1 c shows the content of a block. Bottom: the features are projected to 256 dimensions, given sinusoidal positions and passed through six pre-norm Transformer encoder layers; a linear layer with a sigmoid gives one probability per class every 40 ms.

Layer	Operation	Output shape	Parameters
input	waveform, 16 kHz, 8 s	$128000$	–
log-mel	STFT 512 / win 400 / hop 160, 64 mel bands, log, scale	$64\times 801$	–
conv block 1	2 $\times$ (conv 3 $\times$ 3, BN, ReLU); 1 $\rightarrow$ 48 ch; stride (f,t)=(1, 1)	$48\times 64\times 801$	21,456
conv block 2	2 $\times$ (conv 3 $\times$ 3, BN, ReLU); 48 $\rightarrow$ 48 ch; stride (f,t)=(2, 1)	$48\times 32\times 801$	41,760
conv block 3	2 $\times$ (conv 3 $\times$ 3, BN, ReLU); 48 $\rightarrow$ 96 ch; stride (f,t)=(2, 1)	$96\times 16\times 801$	124,992
conv block 4	2 $\times$ (conv 3 $\times$ 3, BN, ReLU); 96 $\rightarrow$ 96 ch; stride (f,t)=(2, 2)	$96\times 8\times 401$	166,464

Table 3: Layer-by-layer view of the network for one 8 s window (batch dimension omitted). Shapes are channels

\times

frequency

\times

time for the convolutional part and time

\times

features afterwards; they were measured with forward hooks on the released model. Strides are (frequency, time). Parameter counts are exact.

Figure 4: Parameters per layer, coloured by component (5,802,790 in total).

Figure 5: A held-out clip passing through the released model; all panels share the time axis. Convolutional panels show the mean absolute activation over channels (blue scale); vector-valued panels show each feature standardized over time (blue below, red above its mean). The bottom panel shows the output probabilities of the annotated classes and the five most active classes (white 0, dark blue 1), with the annotated events as orange lines. The dark column at the right edge of block 1 is the zero padding of this 7.9 s clip to the 8 s window.

Augmentation	Prob.	Parameters
silence inserted outside events, labels shifted	0.5	1–2 pauses of 0.2–1.5 s
reverberation (exponentially decaying noise response)	0.25	RT 60 0.15–0.8 s; direct path 1–4 $\times$ the tail peak
telephone band	0.2	Butterworth band-pass 300–3400 Hz (order 4 per edge); 8-bit $\mu$ -law with probability 0.5
background noise	0.85	real noise 40 % (SNR 3–25 dB), babble 20 % (12–30 dB), hum 10 % (10–35 dB), white/pink/brown 30 % (8–35 dB)
gain	1	$-12$ to $+6$ dB

Table 4: Augmentations applied to each stored augmented copy; each is drawn independently.

Setting	Value
Input	16 kHz mono, 8 s windows; 64-band log-mel, 25 ms window, 10 ms hop, 50–8000 Hz
Output	54 sigmoid outputs every 40 ms (201 frames per window)
Parameters	5,802,790 trainable (plus a fixed sinusoidal position table)
Optimizer	AdamW, weight decay 0.01, gradient-norm clip 5
Schedule	OneCycle, peak LR 0.0003, 10 % warm-up, 10 epochs
Batch	32 windows; 5,343 steps per epoch

Table 5: Training configuration of the released model.

Figure 6: Training loss (a) and validation macro frame AP per held-out set (b). The selection score is the mean over the four real sets and the noisy set; the synthetic set is shown but not used for selection. Values are in data/training_log.csv .

Figure 7: Frame-level average precision (bars) and two-fold cross-validated F1 of the balanced profile (dots; thresholds tuned on the other half of the windows) per class, on real held-out windows.

\dagger

marks weak classes ( Section 6.3 ).

Profile	Precision	Recall	Objective
balanced	0.352	0.414	F1 = 0.351
sensitive	0.267	0.521	F2 = 0.400

Table 6: Cross-validated frame-level operating points of the two threshold profiles, averaged over the 43 reliably measured classes.

Figure 8: Cross-validated frame precision and recall of each reliable class under the balanced (F1) and sensitive (F2) thresholds; grey lines connect the two points of a class. Diamonds are the class means.

			Recall		Precision		F1		Micro recall
Classes	$n$	Events	S	B	S	B	S	B	S	B
speaker sounds	16	3,271	0.455	0.327	0.383	0.395	0.378	0.338	0.703	0.542
background sounds	32	6,096	0.510	0.390	0.261	0.345	0.294	0.325	0.656	0.528
all	48	9,367	0.491	0.369	0.302	0.361	0.322	0.329	0.672	0.533

Table 7: Event-level results on whole held-out real clips, for the sensitive (S) and balanced (B) threshold profiles. Macro averages over classes with at least 10 held-out events; the last two columns give the event-weighted (micro) recall. A true event is found if a same-class prediction overlaps it; a prediction is correct if it overlaps a same-class true event. Thresholds were tuned on held-out data that overlaps these clips, so absolute values are optimistic ( Section 8.1 ).

Figure 9: Event-level recall (a) and precision (b) per class on whole held-out clips with the balanced and sensitive profiles; held-out event counts in parentheses. Classes with at least 10 events, sorted by sensitive recall. Values are in data/event_eval.csv .

Figure 10: Self-attention of encoder layers 1 (a) and 6 (b), averaged over the eight heads, for the clip of Figure 12 ; coloured bars on top mark the annotated events. (c) Mean attended distance per layer, $\sum_{j}a_{ij}\,|i-j|\cdot 40$ ms averaged over queries $i$ .

Figure 11: Worked examples: eight held-out clips, two per source, chosen by a fixed rule (text). Each panel shows the log-mel spectrogram, the annotated events (orange, upper bar of each row) and the events detected with the sensitive profile (blue, lower bar). “found” counts annotated events overlapped by a same-class detection.

Clip	Dur.	Annotated events	Detected (sensitive)	Found S	Found B
nv-1	6.9	laugh 6.1–6.9	cough 5.8–7.0, laugh 5.8–7.0	1/1	0/1
nv-2	7.7	cough 6.4–6.6	throat_clear 5.9–7.3	0/1	0/1
vaani-1	11.1	breath 2.0–2.3, horn 3.8–8.2, breath 6.1–6.4, breath 8.6–8.9	breath 0.0–2.5, horn 0.0–10.8, vehicle 0.0–10.8, breath 5.2–7.0, breath 7.8–9.5	4/4	4/4
vaani-2	10.6	dog_bark 0.9–1.5, breath 9.1–9.4	fan 0.0–10.6, breath 4.8–6.3	0/2	0/2
nvtts-1	7.5	breath 0.1–0.3	–	0/1	0/1
nvtts-2	4.2	laugh 0.8–1.2	laugh 0.5–1.8, breath 0.7–1.8	1/1	0/1

Table 8: Worked examples on held-out clips (selection rule in Section 8.7 ). Times are in seconds. “Found” counts annotated events overlapped by a same-class detection, for the sensitive (S) and balanced (B) profiles. Detected classes that the clip’s source does not annotate cannot be judged.

Figure 12: A held-out Vaani clip (Deoria district; selection rule in the text): (a) log-mel spectrogram; (b) frame probabilities of the annotated and detected classes, with their sensitive thresholds as thin lines; (c) annotated events (solid) and detected events (hatched). Music is detected; the sneeze and the breath are missed; tv_speaker is a false alarm.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

					balanced		sensitive		event level
Class	Type	Eval	Pos.	AP	$\theta$	CV F1	$\theta$	CV F2	Events	R-S	P-S	R-B	P-B
breath	S	NVTA	16,762	0.499	0.850	0.499	0.725	0.616	1,116	0.80	0.49	0.67	0.64
burp	S	A	136	0.218	0.650	0.285	0.375	0.361	7	0.57	0.16	0.29	0.10
chew †	S	A	242	0.039	0.900	0.056	0.600	0.108	29	0.10	0.08	0.00	0.00
cough	S	NVTA	2,033	0.313	0.875	0.400	0.750	0.493	329	0.56	0.44	0.39	0.59
cry †	S	NA	963	0.075	0.900	0.048	0.600	0.110	61	0.43	0.11	0.08	0.09

Appendix

Table 9: Per-class results. Type: S = speaker sound, B = background. Eval: validation sources that label the class (N = NVS-38K, V = Vaani, T = NonverbalTTS, A = AudioSet-Strong). Pos. = positive validation frames (40 ms). AP = frame-level average precision.

\theta

= threshold; CV = 2-fold cross-validated frame F1 (balanced) or F2 (sensitive). Event level: held-out events, and event recall (R) and precision (P) with the sensitive (S) and balanced (B) profiles. † weak class.

Class	Type	NVS-38K	Vaani	NV-TTS	AudioSet	SLR99	DNC	DEMAND	Total
breath	S	2,162	35,663	6,121	27,232	44	0	0	71,222
music	B	0	6,872	0	51,454	0	229	0	58,555
bird	B	0	17,694	0	37,443	0	0	0	55,137
vehicle	B	0	4,854	0	25,450	0	692	4	31,000
wind	B	0	0	0	25,079	0	0	0	25,079
laugh	S	6,912	99	1,278	14,489	25	0	0	22,803

Appendix

Table 10: Annotated events per class in each source’s full index, before any sampling caps. DEMAND counts environments.

Explore similar work

Sep 9, 2026

NVV-Locator: From Transcript Tags to Acoustic Boundaries for Fine-Grained Nonverbal Vocalization Grounding

Human speech includes nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, which convey affective and interactional information. Existing approaches typically represent NVVs as transcript-level tags, providing limited supervision for their waveform-time boundaries. We present NVV-Locator for fine-grained NVV temporal grounding. We first unify 26 NVV categories across public resources and construct large-scale timestamp-supervised training data through dual-LLM verification, transcript-guided forced alignment, and energy-based boundary refinement. We further introduce NVV-TimeBench, an expert-refined benchmark with 667 utterances and 1,094 events. NVV-Locator uses a non-autoregressive slot-filling architecture to jointly predict lexical timestamps, NVV categories, and event boundaries. On NVV-TimeBench, it achieves 71.0% Micro F1, 70.2% Macro F1, 80.4% Macro mIoU, and 59.6 ms Macro mMAE, outperforming the evaluated large audio model counterparts. Evaluation on an external corpus further demonstrates the cross-corpus generalization of NVV-Locator.

Yuang Cao, Bingshen Mu, Zhennan Lin +7

School of Intelligence Science and Technology, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China

Audio Understanding Sound Event Detection Temporal Audio Grounding

Ask PDF

Sep 25, 2026

Tracing and Relearning Detection Evidence in Text-to-Speech Systems

Recent audio deepfake detectors separate bona fide speech from synthetic speech, yet it remains unclear which stage of a text-to-speech system supplies the detection evidence. We address this with controlled resynthesis and detector adaptation in an F5-TTS-BigVGAN pipeline. Since vocoder reconstruction of a real mel can itself be separable from the source utterance, we fix the vocoder and trace the larger change in detector separation to acoustic generation. Adversarially fine-tuning the acoustic model, with no detector in its objective, raises EER against fixed detectors at comparable quality. However, adapting a detector only on the tuned model's VCTK outputs lowers its LibriSpeech EER from 19.42% to 7.46% and improves detection of unseen base F5-TTS outputs. These results suggest that acoustic-model updates can reduce the detection evidence available to fixed detectors, while detector adaptation keeps the updated outputs detectable in this pipeline.

Eunji Shin, Kyudan Jung, Jihwan Kim +2

Ewha W. University · KAIST AI

Audio Deepfake Detection Speech Processing TTS Synthesis

Ask PDF

Feb 26, 2026

Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper

Deepfake speech utterances can be forged by replacing one or more words in a bona fide utterance with semantically different words synthesized with speech-generative models. While a dedicated synthetic word detector could be developed, we developed a cost-effective method that fine-tunes a pre-trained Whisper model to detect synthetic words while transcribing the input utterance via next-token prediction. We further investigate using partially vocoded utterances as the fine-tuning data, thus reducing the cost of data collection. Our experiments demonstrate that, on in-domain test data, the fine-tuned Whisper yields low synthetic-word detection error rates and transcription error rates. On out-of-domain test data with synthetic words produced with unseen speech-generative models, the fine-tuned Whisper remains on par with a dedicated ResNet-based detection model; however, the overall performance degradation calls for strategies to improve its generalization capability.

Hoan My Tran, Xin Wang, Wanying Ge +2

Université de Rennes, France · National Institute of Informatics, Japan

Audio Deepfake Detection Speech Foundation Models

Ask PDF