Can LLM Agents Identify Spoken Dialects like a Linguist?
Authors: Tobias Bystrich, Lukas Hamm, Maria Hassan, Lea Fischbach, Lucie Flek, Akbar Karimi
Organizations: Department of Computer Science, University of Bonn, Germany · Research Center Deutscher Sprachatlas, Marburg University, Germany · Bonn-Aachen International Center for Information Technology, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Due to the scarcity of labeled dialectal speech, audio dialect classification is a challenging task for most languages, including Swiss German. In this work, we explore the ability of large language models (LLMs) as agents in understanding the dialects and whether they can show comparable performance to models such as HuBERT in dialect classification. In addition, we provide an LLM baseline and a human linguist one. Our approach uses phonetic transcriptions produced by ASR systems and combines them with linguistic resources such as dialect feature maps, vowel history, and rules. Our findings indicate that, when linguistic information is provided, the LLM predictions improve. The human baseline shows that automatically generated transcriptions can be beneficial for such classifications, but also presents opportunities for improvement.
Figures & tables
Figure 1: Graphic for SwissDial classes and Innerschweiz made with REDE-SprachGIS and dialect region background Besch et al. (2008) .
# Train
Accuracy
# Class predictions
Macro-F1
Accuracy per class
High
Highest
High
Highest
400
66.25%
13
67
61.91%
32.5%
100%
4000
66.25%
39
41
66.25%
65%
67.5%
Table 1: HuBERT baseline for the binary task with best hyperparameters on test data from SwissDial
# Train
Accuracy
Most / least class predictions
Macro-F1
800
11.25%
SG (35)
BE (0)
8.67%
8000
26.25%
BS & GR (20)
LU (2)
22.85%
Table 2: HuBERT baseline for the 8-class task with best hyperparameters on test data from SwissDial. Abbreviations for dialects: St. Gallen (SG), Bern (BE), Basel (BS), Grisons/Graubünden (GR), Lucerne/Luzern (LU).
Setting
AG
BE
BS
GR
LU
SG
VS
ZH
8-class (800)
40%
0%
10%
0%
0%
30%
10%
0%
8-class (8000)
0%
0%
70%
50%
10%
40%
20%
20%
Table 3: Accuracy per class for HuBERT (8-class) with best hyperparameter configuration on test data from SwissDial. Abbreviations for dialects: Aargau (AG), Bern (BE), Basel (BS), Grisons/Graubünden (GR), Lucerne/Luzern (LU), St. Gallen (SG), Valais/Wallis (VS), and Zürich (ZH).
Figure 2: The agentic framework for dialect analysis
Figure 3: Sample transcription with Highest Alemannic features. Orthographic transcription: Wir haben Yvonne Beutler an ihrem letzten Arbeitstag besucht und auf ihre politische Laufbahn zurückgeschaut [ -gelugt ]
Figure 4: HuBERT baseline architecture
lr
Batch
Accuracy
# Class predictions
Macro-F1
Accuracy per class
# High
# Highest
High
Highest
10−3
8
52.5%
61
19
50.99%
70%
35%
16
46.25%
61
19
42.27%
72.5%
20%
32
56.25%
25
55
54.66%
37.5%
75%
64
62.5%
22
58
60.5%
40%
85%
10−4
8
50%
76
4
37.3%
95%
5%
Table 4: Results of hyperparameter search for the HuBERT baseline on test data from SwissDial.
Experiment Configuration
Accuracy (%)
Base Prompt Only (2 runs average)
47.8
Specialized Features Analysis Node
47.5
Vowel & Consonant Analysis Node
55.0
Vowel & Consonant Node + Specialized Features Node (2 runs average, no IPA explanations from GPT)
58.0
Table 5: Accuracy of different LangGraph experiment configurations. We use the best results for comparison with other methods.
Model
Accuracy
# Class predictions
Macro-F1
Accuracy per class
High
Highest
High
Highest
HuBERT
66.3%
39
41
66.3%
65%
67.5%
LLM
47.8%
30
50
47%
35.6%
60%
Human
72.5%
33
46
72.3%
80%
65%
Agentic
58%
23
57
56.3%
37.5%
78.8%
Table 6: Results for the 2-class problem
Figure 5: Alignments between human baseline and GPT models