Organizations: Olewave, USA · Stanford University · Cartesia · Microsoft · Northwestern Polytechnical University · Tsinghua University · University of California, Los Angeles · Carnegie Mellon University
Forced alignment estimates the timestamps of each word, phone or character in speech given its transcript. Published comparisons normalize transcripts, split the data and match boundaries differently, so their numbers cannot be read together. We present FA-Bench, an open framework that fixes those choices once and releases the code, splits, phone mapping, text normalization and scoring script, with results published periodically. Track 1 gives every aligner the reference transcript and Track 2 gives it a recognizer's output, on the same audio, clean and degraded four ways, with 21 open models and 9 commercial APIs under a unified protocol. We score every boundary of an utterance and check the two labels beside it, so a word the recognizer missed or invented is charged. Using a tolerance-based F1 as our primary metric eliminates the 9% to 14% score inflation that standard MAE causes on recognition-dependent systems in conversational speech. We then group boundaries by position and how many adjacent words were recognized correctly, which shows where a system lost the score. We discovered systematic bias in how current systems time words, with Whisper about 150 ms early and several commercial ASR APIs over 50 ms late. Code and results are at https://github.com/olewave/fa-bench
Figures & tables
Figure 1: Mean signed word-boundary error on TIMIT test, hypothesis minus reference, over the words the label alignment matched. Track 2 only. Every two-step point aligns Qwen3-ASR’s transcript, so the recognizer is fixed and only the aligner varies. Both axes are warped by asinh(v/4) . The distance from the dashed line y=x is the mean error in word duration.
Family
TIMIT test
Buckeye test
System
clean
noisy
clean
noisy
Track 1 , given the reference transcript
MAE
F120
F150
MAE
F120
F150
MAE
F120
F150
MAE
F120
F150
Open
NeMo-FA [ 14 ]
0 47.4
.17
0 .41
0 47.1
.18
0 .43
0 62.8
.21
0 .49
0 65.1
.21
0 .50
stable-ts [ 29 ]
0 96.8
.18
0 .39
0 94.5
.19
0 .41
0 71.2
.29
0 .55
0 75.2
.30
0 .55
WhisperX [ 10 ]
0 37.2
.18
0 .43
0 39.5
.17
0 .42
0 41.7
.16
0 .49
0 52.0
.17
0 .48
Table 1: Word-tier boundary error on the test split of each corpus, on clean audio and under degradation, noisy being the mean of the four. Every number here comes from our own runs. MAE is over word boundaries in ms. F1 is over every word boundary, the two utterance edges included. A boundary is a hit only when the words on each side of it match the reference and the time falls within the tolerance of the reference boundary. The superscript on F1 is that tolerance in ms. The step size of a word boundary of each system is shown in Table 3 . Open means the weights are published, and NeMo-FA, WhisperX, TorchAudio, BFA, MMS-FA, UnitY2, Charsiu, MFA and MAPS also publish the code that trained them. The NeuFA model is trained with its open-source code on GitHub and is not released. W is Whisper large-v3, P and Parakeet are Parakeet-TDT, Q and Qwen3 are Qwen3-ASR, and G is Google Chirp 2. Whisper-ts is Whisper-timestamped and NeMo-FA uses its Conformer checkpoint. The APIs are Speechmatics Enhanced, Deepgram Nova-3, IBM Large US English, Azure en-US, AssemblyAI Universal 3.5 Pro, Amazon Transcribe en-US, ElevenLabs Scribe v2, Google Chirp 2 and Olign 0.9. MFA is Montreal Forced Aligner 3.4. MAPS and NeuFA trained on 7 of 8 speakers in both Buckeye splits.
Family
TIMIT test
Buckeye test
System
clean
noisy
clean
noisy
Track 1 , reference words, except FALCON †
MAE
F120
PER
MAE
F120
PER
MAE
F120
PER
MAE
F120
PER
Open
BFA [ 7 ]
45.1
.18
36.2
48.9
.18
36.2
50.2
.25
32.1
0 54.8
.25
32.2
TorchAudio [ 30 ]
33.8
.21
35.9
33.3
.21
35.9
34.5
.27
28.5
0 41.3
.26
28.5
NeuFA [ 8 ]
20.5
.33
36.0
56.1
.28
39.1
22.6
.46
28.0
0 46.5
.40
30.7
Table 2: Phone-tier boundary error on the test split of each corpus, on clean audio and under degradation, noisy being the mean of the four, laid out as Table 1 . Every Track 1 aligner is given the reference words and derives its own phones, and PER is the phone error rate of what it derived against the gold phones, per split and condition. FALCON, daggered, is handed the gold phones, and overlaps our Buckeye split in its training data. We use the FALCON released model and did not retrain it. No one-step ASR in Table 1 emits phone timestamps, so Track 2 has only the two-step rows whose aligner does, and their PER carries the recognizer’s word errors too.
Grid
Buckeye test clean
Buckeye test noisy
(ms)
M2I
M1I
M1B
M1E
M0
M2I
M1I
M1B
M1E
M0
% of boundaries
69.2
10.4
7.2
7.6
5.5
64.4
12.2
6.7
6.9
9.7
Open
Whisper-ts
20
.17
.08
.58
.03
.14
.17
.09
.57
.02
.14
Whisper
20
.12
.06
.57
.05
.13
.12
.07
.57
.04
.14
Parakeet
80
.23
.15
.22
.04
.08
.23
.16
.23
.03
.09
W → WhisperX
20
.13
.07
.10
.28
.06
.13
.08
.08
.25
.08
Table 3: The F1 s at 20 ms of different types of word-tier boundaries of Track 2 on Buckeye test, on clean audio and under degradation, noisy being the mean of the four. A boundary inside ( I ) an utterance has two adjacent words. We use M2I to denote both words are matched and M1I to denote one label is matched, where M denotes matched. M1B and M1E denote the beginning ( B ) and the ending ( E ) boundary of an utterance, each with only one adjacent word. M0 denotes the remaining boundaries. The first row is the share of reference boundaries in each category, averaged over the rows.
Forced aligners locate each word of a known transcript in the audio, and many speech pipelines depend on them. In our own data, a widely used CTC aligner inserts a pause at nearly every word boundary, and we found almost no aligner evaluated on telephone audio. We compare nine aligner configurations against human word boundaries on read, conversational and telephone English. The Montreal Forced Aligner (MFA) is the most precise, while CTC aligners consistently start words late, because most frames inside a word go to the blank token. We then fine-tune Whisper with LoRA to predict each word's start and end as timestamp tokens, training only on boundaries placed by people: 47 hours in 87 languages. The resulting aligner is more precise than MFA on every English test condition, and than MFA's own Korean and Dutch models. With FA-Bench's protocol and code on TIMIT, its word-boundary error is 11.6 ms, against 18.2 ms for the best published system. Two choices matter most on telephone conversations: placing the timestamp slots after the word they time, and training on noisy transcripts, which also makes the aligner nearly insensitive to errors in the transcript. The released model reaches 16.0 ms on real landline calls, against 36.5 ms for MFA. The models are available at https://huggingface.co/collections/Scicom-intl/force-alignment
Word-level forced alignment estimates when each transcript word occurs in an audio recording. It underpins text-based media editing, subtitling, speech-data curation, and phonetic analysis. Existing evaluations understate the difficulty of forced alignment by relying on short, clean speech, perfect transcripts, and metrics that obscure consequential alignment errors. In contrast, real-world media and data-processing pipelines operate on long and diverse recordings. Additionally, forced aligners often operate on error-prone automatic speech recognition (ASR) output. We address these gaps with improved evaluation metrics, a scoring protocol for real ASR transcripts, and AlignBench, a benchmark spanning diverse speaker, acoustic, and text conditions. We further introduce FuseAlign, a transformer-based aligner trained on large-scale pseudo-labeled speech with online label correction. FuseAlign performs joint contextualization of audio and text for the localization of coarse words. The model then refines boundaries at millisecond resolution and detects missing transcript words in the audio without lexicon-based or Viterbi decoding. On AlignBench, FuseAlign substantially outperforms all baselines and remains robust under real ASR transcripts. Ablations show that convolutional upsampling and EMA-snapshot label correction matter more than model properties such as parameter count.
Forced alignment evaluation typically requires manually annotated timestamps, limiting large-scale and multilingual analysis. We introduce two corpus-level metrics based on self-supervised (SSL) speech representations for reference-free forced alignment evaluation: Phoneme-Cluster Mutual Information (PCMI) and Word Acoustic Consistency Score (WACS). PCMI measures agreement between aligned phoneme labels and clusters induced from SSL-speech representations, while WACS measures consistency of repeated word realizations using dynamic time warping similarity between word representation sequences. Using both random and systematic perturbations, we show that PCMI and WACS degrade consistently under alignment perturbations. We further analyze the metrics across multiple alignment systems on 85 languages from FLEURS, validate them against manually annotated alignments from 45 languages in DoReCo, and evaluate them on two phonologically complex low-resource languages. The metrics effectively separate high- and low-quality alignments and correlate strongly with timestamp-based alignment quality measures. Our results demonstrate that SSL-speech representations enable scalable, reference-free forced alignment evaluation. The metrics are available as an open-source Python package at https://github.com/mahesh-ak/forced-aligner-metrics.
V. S. D. S. Mahesh Akavarapu, Michael Daniel, Gerhard Jäger