cs.CLSep 17, 2026

Scaling Forced Alignment to End-User Devices

Authors: Lawry Sorenson, Michael Crandall, Eric K. Ringger, Stephen D. Richardson

Organizations: Brigham Young University Computer Science Department Provo, UT, United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The Viterbi algorithm has been previously used to perform forced alignment of audio to text to mine training data from online resources. However, many existing implementations have quadratic time and space complexity, scaling poorly to long input sequences. We propose two optimizations to address this issue. First, we apply the Hirschberg algorithm to perform the alignment in place using linear memory. Second, we model the alignment between speech and text as a constrained random walk, allowing us to prune the search space with arbitrary confidence while accounting for transcription errors. The Hirschberg optimization reduces memory usage from 140 GB to 5 MB for three-hour inputs while producing identical alignments in one-third the time of torchaudio when both run on a CPU. We achieve an additional 2x speedup with pruning on inputs longer than 20 minutes while preserving alignment accuracy in more than 98% of tested cases.

Figures & tables

Figure 1: Logarithmic heatmaps showing a random sample of alignments from the CJCLDS-GC dataset (left), and the alignment prior that we use to predict FA pruning bounds (right). This prior is created from the assumption that phoneme durations are randomly distributed and stationary.
 BuckeyeCJCLDS-GCEuroSpeechOverall
MethodExact<250<250 msExact<250<250 msExact<250<250 msFailureTimeMax TimeMax Mem.
Unpruned HV100%100%100%100%100%100%0%59:36:392:50:4422.4 MB
UncutHV + VAD Hint100% †‡\dagger\ddagger100% †‡\dagger\ddagger98.2% †‡\dagger\ddagger98.3% ‡\ddagger100% †‡\dagger\ddagger100% ‡\ddagger0%32:38:361:22:2641.5 MB
Kaldi67.5% ‡\ddagger75.0% ‡\ddagger93.8% ‡\ddagger99.5% ∗‡\ast\ddagger69.6% ‡\ddagger94.2% ‡\ddagger0.03%287:58:12>> 3 days21.5 GB
ctc-seg0%52.5%0%70.0%0%18.8%6.0%10:15:451:04:20>> 80 GB
VADHV0%77.5% ‡\ddagger1.97% †‡\dagger\ddagger93.4% ‡\ddagger0%82.0% ‡\ddagger0%11:40:3927:231.9 GB
Table 1: Pruning performance for different methods with their default parameters. We include metrics when VAD is used to cut periods of silence before processing. Runtime and memory usage is aggregated across all datasets. The longest input across the test sets is 16 hours. Systems marked with ∗\ast , †\dagger , and ‡\ddagger significantly outperformed our system, Kaldi, and ctc-seg, respectively, with p-values under 0.050.05 .
Figure 2: Runtime and memory usage of various implementations of the Viterbi algorithm. Note that torchaudio and NeMo have the same memory usage and overlap in the graph. The GPU used has only 80 GB of VRAM, so that line for NeMo ends early once it no longer has enough VRAM to run.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Visualization of the Hirschberg algorithm applied to forced alignment. The algorithm works from both sides to compute an optimal pivot point in the middle, then recurses to the lower left and upper right areas in the figure.
Figure 4: In this specific alignment, which is for the Bulgarian interpretation of Russell Nelson’s discourse ”Think Celestial!” from 2023, the interpreter did not read a list of cities at the end of the discourse, instead using the original English audio. The city names are in the transcription, but there is no Bulgarian audio corresponding to them. The list of city names occupies 3.7% of the transcription, and adjusting the pruning bound for 96.3% transcription precision produces the outer pruning bounds, which fully contain the alignment.
Figure 5: The alignments in the EuroSpeech dataset after removing audio between labeled sentences. We color the heatmap using the CER of the transcription. Hue depicts CER, while color intensity indicates logarithmic frequency in the dataset. We note that alignments further from their expected position tend to have higher CER.
Figure 6: After removing labeled periods of silence from the Buckeye dataset, all transcription alignments were contained within their predicted pruning space. The lines on the right depict the pruning space for s2002a, one of the shorter recordings with a wider relative pruning space. Note that these depict the segmented interview labels available with dataset, not our reconstruction of the 40 interviews.
DatasetLanguagesFilesTotal Duration (h)Minimum LengthMaximum Length
Buckeye1403839:251:17:54
CGCLDS-GC85639611250900:0441:28
— Our Test Split735766112801:0332:16
EuroSpeech22276837703100:1017:42:45
— Test Split2114079400:2216:20:36
Appendix
Table 2: The size of each of the datasets used in our study. Note that this details our reconstructions of the unsegmented audio files for each dataset. We use the full Buckeye dataset in our tests, both only evaluate our pruning on the test sets for CGCLDS-GC and EuroSpeech. The CGCLDS-GC dataset contains the data through 2023, when we took a snapshot of the data at the time. There is one fewer language in the EuroSpeech test set, becuase EuroSpeech on HuggingFace is missing a test set for Italian. There are 12 fewer languages in our test set from the CGCLDS-GC dataset, because we limit our experiments to languages supported by MMS.
More in this topic

Forced Alignment

See all 9 papers

Explore similar work

CardsList

  1. FuseAlign: Forced Alignment in the Wild

    Sep 27, 2026Mithilesh Vaidya, Stephen Bailey, Sumukh Badam +2Forced AlignmentAudio-Text Alignment

    Descript, Inc.

  2. Where Words Begin and End

    Oct 13, 2026Husein ZolkepliForced AlignmentAudio-Text Alignment

    Scicom (MSC) Berhad, Malaysia

  3. Adaptive Duration Model for Text Speech Alignment

    Jul 30, 2025Junjie CaoAudio-Text AlignmentTTS Synthesis

    Tsinghua Shenzhen International Graduate School, Tsinghua University