Revisiting Accent Classification with DistilHuBERT: Speaker-Disjoint Evaluation and Cross-Corpus Generalization
Authors: Xu He, Homayoon Beigi
Organizations: Department of Electrical Engineering Columbia University New York, NY, USA · Mechanical Engineering Dept. and Electrical Engineering Dept. Columbia University, NYC, NY, USA · Recognition Technologies, Inc. South Salem, NY, USA
Recent accent classification systems can achieve high accuracy in standard in-corpus evaluations. However, when speakers overlap between the training and test sets, such results may overestimate model robustness compared with real-world deployment scenarios. This study revisits the robustness of DistilHuBERT-based accent classification by using L2-ARCTIC and multiple external English accent datasets. The comparison covers file-level and speaker-disjoint splits, along with balanced external-data training and teacher-student pseudo-labeling. The large performance gap between in-corpus and external evaluations suggests that file-level evaluation substantially overestimates performance, while speaker-disjoint evaluation can provide a more realistic estimate of the generalization to unseen speakers. In this study, models trained on external data alone do not reliably generalize to L2-ARCTIC. Among the tested configurations, balanced external-data training offers the most promising supervised setting. The pseudo-labeling experiments further indicate that confidence-based filtering alone is insufficient under domain mismatch. Overall, this study suggests that robust accent classification evaluation should simultaneously include speaker-disjoint evaluation and cross-corpus testing.
Figures & tables
Figure 1: Overview of the speaker-disjoint and cross-corpus evaluation framework. Solid arrows indicate supervised training and evaluation, while dashed arrows indicate the use of external data for balanced training, pseudo-labeling, and held-out external evaluation.
Setting
Speaker overlap
External use
Classes
File-level baseline
Yes
No
6
Speaker-disjoint baseline
No
No
6
Balanced external training
No
Yes
8
Semi-supervised training
No
Pseudo-labels
6/8
Second-level external test
N/A
Held out
8
Table 1: Evaluation and training protocols used in this study.
Model
Internal L2 Acc. (%)
External Acc. (%)
Baseline model
99.76
44.52
Table 2: Internal and external evaluation of the reproduced file-level baseline model.
Model
Labels
Training data
L2 Acc. (%)
Ext. Acc. (%)
Hugging Face baseline
6
L2-ARCTIC
91.65
58.96
File-level baseline
6
L2-ARCTIC
100.00
53.76
Speaker-disjoint baseline
6
L2-ARCTIC
81.40±3.62
61.27±15.49
L2 + external
6
L2-ARCTIC + external
81.09±4.10
100.00±0.00
External only
6
external only
16.35±0.00
100.00±0.00
L2 + external
expanded
L2-ARCTIC + external
83.01±2.80
98.55±1.00
Table 3: Round 1 evaluation results on official L2-ARCTIC and external comparable samples. Fold-based settings are reported as mean ± standard deviation across four folds.
Fold
L2-only (%)
Expanded L2+external (%)
0
86.80
87.10
1
80.15
82.50
2
79.25
80.95
3
79.40
81.50
Mean
81.40
83.01
Table 4: Fold-level official L2 accuracy for the two most relevant Round 1 supervised settings.
Model
Fold
L2 Acc. (%)
Ext. Acc. (%)
Comb. Acc. (%)
L2-Ext. Acc. (%)
Round 2 L2-only
0
86.20
71.05
85.13
42.00
Round 2 L2-only
1
74.60
47.37
72.68
–
Round 2 L2-only
2
80.10
63.16
78.90
–
Round 2 L2-only
3
80.30
67.11
79.37
–
Round 2 L2 + British/Singaporean
0
87.90
–
81.97
68.00
Round 2 L2 + British/Singaporean
1
76.20
–
70.82
–
Table 5: Round 2 evaluation results on L2-ARCTIC, external comparable samples, and the second-level external evaluation set. Dashes indicate settings where the external comparable accuracy is not directly comparable because the label space or test composition differs.