Organizations: Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, USA · Thomas Lord Department of Computer Science, University of Southern California, USA · Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California, USA
Audio preprocessing can affect how well a system detects stuttering. We study a simulated chain of denoising,loudness normalisation, Opus coding, and voice activity detection on SEP-28k. We use frozen WavLM Base+ features and report pointwise confidence intervals from episode-level bootstrap resampling. At a fixed threshold of 0.5, the chain reduces block F1 from 0.638 to 0.465, with smaller decreases for the other four classes. ROC-AUC decreases for all five classes. Blocks show the largest F1 and ROC-AUC losses, while sound repetitions show the largest average precision loss. Tuning the threshold on processed validation audio raises block F1 to 0.630. Retraining on processed audio with threshold tuning gives 0.628. Threshold adjustment therefore accounts for most of the observed block F1 recovery. It does not change ROC-AUC, which retraining raises only from 0.620 to 0.633, compared with 0.724 on clean audio.
Figures & tables
Figure 1: Changes in pooled out-of-fold F1 relative to clean audio, in percentage points. All conditions use clean-trained classifiers with a decision threshold of 0.5 . Positive values indicate improvement. Component effects need not sum to the full-chain effect. Colors and annotations indicate effect sizes, not statistical significance.
Class
clean
ΔF1 [95% CI]
Δ AUC
Δ AP
Block
0.638
−17.3[−19.5,−15.2]
−10.4
−8.9
SoundRep
0.659
−8.9[−10.6,−7.3]
−8.3
−10.5
WordRep
0.645
−7.2[−9.0,−5.5]
−2.8
−1.8
Prolong.
0.624
−3.0[−4.3,−1.7]
−3.2
−4.1
Interj.
0.756
−2.3[−3.1,−1.3]
−1.9
−2.1
Table 1: Clean audio versus the full chain, per class, at threshold 0.5 . Changes are in percentage points, with pointwise 95% bootstrap intervals on ΔF1 . AP is average precision.
Condition
ΔF1 fix [95% CI]
ΔF1 tun
Δ AUC
Opus 16k
−0.53[−1.05,−0.05]
+0.08
−0.17
Opus 16k VoIP
−0.22[−0.67,+0.20]
+0.07
−0.14
Opus 8k
−2.73[−3.59,−1.90]
−0.82
−1.00
Denoise
−10.59[−12.59,−8.82]
−1.64
−2.78
VAD zero
−11.73[−13.51,−10.09]
−3.73
−7.46
VAD delete
−11.98[−13.70,−10.31]
−3.58
−7.16
Table 2: Block detection by condition, with clean-trained classifiers, in percentage points. The fix column uses threshold 0.5 and carries pointwise 95% bootstrap intervals; tun tunes the threshold as in Section 3.5 and has no interval.
F1
AUC
Class
clean-t
thr
mat
clean
proc
mat
Block
0.668
0.630
0.628
0.724
0.620
0.633
SoundRep
0.657
0.579
0.595
0.844
0.761
0.789
WordRep
0.643
0.607
0.636
0.851
0.823
0.841
Prolong.
0.618
0.604
0.594
0.789
0.757
0.755
Interj.
0.757
0.734
0.740
0.866
0.846
0.849
Table 3: Threshold adaptation against matched retraining, full chain only. F1 : clean-t is clean audio at a clean-tuned threshold, thr is processed audio with the clean-trained model at a processed-tuned threshold, and mat adds matched retraining. AUC columns are clean, processed clean-trained and processed matched-trained; AUC does not depend on the threshold. Point estimates, no intervals.
Audio-based stuttering systems to date have been trained for detection -- what disfluency is present now -- leaving prediction, the capability needed for closed-loop intervention, unstudied at deployable scale. We train a 616K-parameter CNN on SEP-28k (Apple, 20,131 three-second clips) to predict whether the next contiguous clip contains any disfluency. (1) Severity-selective precursor signal: on the episode-grouped test set, aggregate preblock AUC is modest (0.581 [0.542, 0.619]), but stratifying by upcoming event type reveals concentration on clinically severe events -- blocks 0.601 [0.554, 0.651] and sound repetitions 0.617 [0.567, 0.667] both exclude chance, while fillers (0.45) and word repetitions (0.49) are at chance. The aggregate objective converges to a severity-selective predictor because severe events carry prosodic precursors; fillers do not. (2) Cross-population transfer: without fine-tuning, the same checkpoint applied to 1,024 pediatric Children-Who-Stutter utterances (FluencyBank Teaching) attains AUC 0.674 detection and 0.655 prediction; DisfluencySpeech and LibriStutter reach 0.58-0.60 AUC. (3) Deployable on-device: lossless export to CoreML (1.19 MB), ONNX (40 KB), TFLite. Neural-Engine latency per 3 s window: 0.25 ms (iPhone 17 Pro Max, A19 Pro) to 0.55 ms (iPhone SE 3rd-gen and M1 Max). A 4 Hz streaming simulation uses 0.54% of the real-time budget. Platt-calibrated outputs (test ECE 0.010, from 0.177 raw). Five negative ablations -- output-level Future-Guided Learning, multi-clip GRU, time-axis concatenation, asymmetric focal loss, direct block-targeted training -- none improved over the vanilla baseline.
Nazar Kozak
Kozak Technologies Inc, Los Angeles, CA 90036, USA
Automatic stuttering systems primarily detect dysfluencies after they have occurred, limiting their use for anticipatory intervention. We address this gap by formulating stuttering prediction as continuous causal time-to-event forecasting, where a model estimates the risk of an upcoming stuttering onset every 100 ms using only preceding speech. To support this task, we augment FluencyBank Timestamped for 36 speakers with manually reviewed repetitions, prolongations, and blocks, yielding 2,872 forecasting events. We then model the temporal evolution of frozen Whisper-large-v3 speech representations using a compact causal Conformer with 3.67M trainable parameters. The proposed approach achieves an AUROC of 0.63. Temporal analysis further shows that explicitly modeling preceding speech states outperforms simply increasing the static acoustic window, while a discrete-time survival formulation provides coherent forecasts from 300 ms to 1.5 s using a single model. These results provide initial evidence that stuttering onset is preceded by measurable temporal information in speech and establish a framework for future anticipatory stuttering systems.
Tampu Ravi Kumar, Ayoub Daliri, Vivek Gupta
School of Computing and Augmented Intelligence, Arizona State University, USA · College of Health Solutions, Arizona State University, USA
We introduce Balalaika, an open-source, data-centric pipeline for processing audio and producing prosody-aware annotations. It combines semantic VAD for context-preserving segmentation, multi-ASR ensembling with ROVER consensus decoding, while retaining optional word-level timestamps, followed by automatic quality and speaker-purity filtering. The text is further enriched with punctuation restoration, lexical stress and "\textipa{e}/\textipa{He}" normalization, and IPA phonemes. Using Balalaika, we build a 5.1k-hour multi-source Russian corpus with rich annotations, and show consistent gains under equalized training budgets for both speech denoising and TTS; ablations confirm complementary benefits of stress and punctuation and improved synthesis with stricter MOS filtering. The datasets are publicly available at \href{https://huggingface.co/collections/lab260/balalaika-dataset}{**HuggingFace**}