SpecAugment is the dominant speech augmentation policy, but researchers have studied it primarily for automatic speech recognition (ASR), and it operates on spectrograms, limiting its applicability to other speech tasks and waveform models. We propose Wav2Aug, a drop-in waveform augmentation policy that randomly samples two augmentations from a calibrated pool of waveform transformations. We calibrate the augmentation hyperparameters once and reuse them across tasks and architectures, eliminating task-specific tuning. Experiments on ten speech benchmarks show that Wav2Aug achieves ASR performance comparable to SpecAugment while improving over it on average across non-ASR tasks, yielding an average relative error reduction of 12.8% over the strongest compatible SpecAugment baseline. We further show that multi-view training provides additional gains, with two augmented views offering the best tradeoff between performance and computational cost.
Figures & tables
Fig. 1: Overview of Wav2Aug. We sample two augmentations uniformly per batch and apply them sequentially to each utterance, with hyperparameters drawn from globally fixed ranges.
Fig. 2: Waveform and log-mel spectrogram before and after noise addition followed by chunk swapping. Colored lines mark chunk boundaries. Matching letters track swapped chunks.
Rank
Augmentation
Hyperparameter
Range / Value
1
Noise Add.
SNR (dB)
[0,10]
2
Freq. Dropout
Drop bounds
[0,1]
Band count
[1,8]
Band width
0.1
3
Time Dropout
Chunk count
[1,8]
Chunk length (ms)
[0,250]
TABLE I: Wav2Aug augmentations and fixed ranges at 16 kHz. Rows are ordered by individual calibration effectiveness, the error reduction obtained when each operation is evaluated alone.
Fig. 3: Pool-size study on Google Speech Commands (top) and CommonLang (bottom), using the first k ranked operations. Orange marks the lowest mean error in each panel. Bars show means and error bars show one sample standard deviation over five runs.
This paper proposes SpecAugment-Patch Merging, a simple yet effective method to accelerate Audio Spectrogram Transformer (AST) training. We first apply SpecAugment to mask input spectrograms at the patch level, and after positional embeddings are added, the method selects r pairs of masked patches and merges them, reducing the number of tokens processed by the Transformer. Increasing the number of merged pairs r from 0 to 100 keeps mAP on AudioSet nearly unchanged (34.07 to 34.08) while throughput increases from 43.3 to 49.3 samples/sec, which is a relatively 13.9% improvement. Similar patterns appear on ESC-50 and Speech Commands V2, where throughput steadily improves with only minor accuracy changes, demonstrating that this merging approach provides faster training with minimal performance loss.
Data augmentation is standard practice for low-resource automatic speech recognition (ASR), but its benefits are almost always reported as aggregate error rates, leaving open why and when a method helps. On four low-resource languages and English at five data sizes, we cross two augmentations that modify existing audio (speed perturbation and SpecAugment) with two that modify text transcriptions and synthesize audio from it. We analyze errors at the phoneme level and find that they help for complementary reasons. SpecAugment has the most reliable aggregate improvements, but increases tone and palatalization errors, whereas text-based methods introduce consonant insertions; in combination, each reduces the errors the other introduces. This complementarity is predictable. Tone and palatalization are each signaled by a single acoustic property and occur least frequently in training (lowest training exposure), and text-based methods that flatten word frequency raise their frequency by 37--38%, reversing those errors while further improving aggregate error. The improvement from combining the methods exceeds the sum of their individual improvements; the superadditivity vanishes by 100 hours, limiting the complementarity to the low-resource regime. Where specific error types matter, we recommend selecting augmentation by training exposure and acoustic properties, both measurable before any training.
Katsumi Ibaraki, David Chiang
University of Notre Dame Dept. of Computer Science and Engineering
We propose AugCodec, a low-bitrate disentangled neural speech codec that leverages data augmentation to decompose speech into three distinct components: semantic, speaker, and prosody tokens. Specifically, we employ tailored augmenta tion strategies to transform speech into distinct variants, each serving as input for extracting tokens that preserve the target attribute while suppressing others. This disentanglement strategy enables substantial reduction in token rate. Further more, we introduce an augmentation loss that aligns semantic encoder outputs between source and voice-converted speech, encouraging speaker-agnostic embeddings while mitigating the acoustic mismatch induced by voice conversion. Experiments on LibriSpeech test-clean demonstrate that AugCodec significantly outperforms state-of-the-art methods in both reconstruction quality and disentanglement, while operating at only 12.5Hz with three token streams.