Language models represent more about emphasis than they can tell you
Authors: Aarne Talman, Stergios Chatzikyriakidis
Organizations: University of Helsinki, Finland · University of Turku, Finland · Institute of Language and Speech Processing, Athena Research Centre, Greece
Prosodic prominence can largely be predicted from the words of a sentence, but part of it depends on how the speaker reads it. We study how much of prominence language models recover from text, whether they can report it when asked, and what the speech signal adds. On read English speech labelled for word-level prominence, we compare frozen and fine-tuned text models, prompted large language models (LLMs) and speech models on the same words. All text models end within about one point of each other, at best 68.8% on the 3-way task, and largely make the same errors. With the prompts we tested, LLMs do no better than a word-majority baseline, even with 500 examples. Linear probes find more in their representations. Speech models reach 84.7%. We argue that text determines only part of prominence, and that the rest has to be heard. LLMs encode what the text determines, but prompting does not bring it out: they represent more about where the emphasis falls than they tell you.
Figures & tables
Labels (%)
Split
LibriTTS subset
Utterances
Words
Speakers
0
1
2
Used for
train_360
train-clean-360
116,263
2,076,289
904
48.3
27.4
24.2
training (text)
train_100
train-clean-100
33,042
570,592
245
48.1
27.3
24.6
training (speech; matched text)
dev
dev-clean
5,727
99,200
40
47.9
27.7
24.4
model, layer and prompt selection
test
test-clean
4,822
90,063
39
48.0
27.3
24.7
evaluation
Table 1: The data splits. Words: words with a prominence label (punctuation excluded). Labels: share of words in each 3-way class.
Model
Hugging Face identifier
Params
Revision
Encoders
BERT-base
google-bert/bert-base-uncased
110M
86b5e093
BERT-base cased
google-bert/bert-base-cased
109M
cd5ef92a
BERT-large
google-bert/bert-large-uncased
336M
6da4b6a2
ModernBERT-base
answerdotai/ModernBERT-base
150M
8949b909
ModernBERT-large
answerdotai/ModernBERT-large
396M
45bb4654
Table 2: Models, with their Hugging Face identifiers, number of parameters (Whisper and Qwen2-Audio: the whole model) and the revisions used.
System
2-way
3-way
Baselines
t
Majority class
52.0
48.0
t
Word majority (2019)
80.7
62.0
Fine-tuned
t
BERT-base (2019 recipe)
83.0 ± 0.1
68.5 ± 0.1
t
BERT-large
83.1 ± 0.1
68.5 ± 0.1
Table 3: Test accuracy (%). Mean ± standard deviation over seeds where there are several. t : text input, a : audio input. Best layers and models are chosen on the development set. Bold: best text and best speech system in each column (baselines and references excluded). † 200 test utterances.
Figure 1: 3-way test accuracy of BERT-base with growing training data (subsets of train_360 , train_100 and train_360 ).
Figure 2: How often two systems give a word the same 3-way label (%), on the 89,935 test words used for the text and speech comparison. Text and speech models are BiLSTM classifiers; white lines separate the gold labels, speech models, text models and prompted LLMs.
Figure 3: 3-way test accuracy of a linear probe on each layer, by relative depth (0: embeddings, 1: last layer).
Model
Access
2-way
3-way
Qwen3.8 27B ( t )
linear probe
82.3
66.7
BiLSTM classifier
82.9 ± 0.1
67.8 ± 0.1
prompted, no examples
78.8
58.6
prompted, 500 examples
78.3
62.0
Qwen2-Audio 7B ( a )
linear probe (LM)
86.4
74.4
BiLSTM classifier (LM)
87.3 ± 0.1
76.5 ± 0.1
Table 4: The same model probed and prompted (test accuracy, %). t : text input, a : audio input. Bold: best access for each model.
Figure 4: Prompting Qwen3.8 27B with more examples (500 development utterances).
Recall (%)
Labelled 2 (%)
System
Macro-F1
0
1
2
content
function
last
Gold labels
–
–
–
–
42.2
9.2
31.9
t
ModernBERT-large, BiLSTM classifier
63.9
83.5
45.2
63.1
46.7
6.7
35.4
t
Qwen3.8 27B, BiLSTM classifier
63.8
83.2
48.4
59.2
43.1
6.0
31.7
p
Qwen3.8 27B, prompted (500 ex.)
55.9
84.2
35.4
48.3
44.2
3.4
45.5
p
DeepSeek-V4-Pro, prompted (5 ex.)
54.7
80.0
31.1
55.1
56.8
4.8
77.1
Table 5: How each system labels the test set (3-way): macro-F1 and recall of each class, and the share of content words, function words and utterance-final words given the highest level (2). t : text classifier, p : prompted, a : speech classifier.
Figure 5: Confusion matrices (3-way, test set), normalised by row: the share of the words with each gold label that receive each predicted label (%).
Right where it departs
System
Same as lookup
Departs
system
lookup
t
ModernBERT-large, BiLSTM classifier
71.7
28.3
51.8
30.7
t
Qwen3.8 27B, BiLSTM classifier
72.4
27.6
51.4
30.8
p
Qwen3.8 27B, prompted (500 ex.)
69.3
30.7
39.5
39.8
p
DeepSeek-V4-Pro, prompted (5 ex.)
68.8
31.2
37.1
42.1
p
GPT-4o, prompted (5 ex.)
68.0
32.0
36.0
44.2
Table 6: How often each system gives a word the label that the word most often has in the training data (the word-majority lookup), and, on the words where it does not, how often the system and the lookup are right (%, 3-way, test set).
Speaker A
Hilda’s face quivered , but she whispered Yes , I think it must have been .
Speech model, hearing A
Hilda’s face quivered , but she whispered Yes , I think it must have been .
Speaker B
Hilda’s face quivered , but she whispered Yes , I think it must have been .
Text model
Hilda’s face quivered , but she whispered Yes , I think it must have been .
Table 7: One test sentence read by two speakers, typeset by prominence: speaker A (test set), the best speech model hearing speaker A, speaker B (development set) and the best text model, whose prediction is the same for both readings. Grey: not prominent (0); bold: prominent (1); large bold: highly prominent (2).
System
2-way
3-way
95% CI (3-way)
a
Labelling procedure
95.0
91.5
91.3–91.7
a
Whisper large-v3 (encoder), BiLSTM classifier
90.9
84.6
84.3–84.9
a
Whisper large-v3 (encoder), linear probe
88.3
78.8
78.5–79.1
a
Qwen2-Audio 7B (LM), BiLSTM classifier
87.4
76.7
76.4–77.0
a
Duration only
83.2 ∘
65.0
64.7–65.4
t
Qwen3.5 0.8B, echo, BiLSTM classifier
82.9 ∘
68.1
67.7–68.4
Table 8: Accuracy (%) on the same 89935 test words, best systems of each kind. All differences from the best text model are significant (McNemar, p<0.05 ) unless marked ∘ . Bold: best text and best speech system (references excluded).
Words
n
Text
Audio
Δ
content words
42,281
56.1
79.0
+22.9
function words
47,654
78.7
89.6
+10.8
first word
4,740
67.0
83.4
+16.4
medial
80,394
69.1
85.2
+16.1
last word
4,801
51.7
75.1
+23.4
new (content)
29,629
56.4
79.1
+22.7
Table 9: 3-way accuracy (%) of the best text and the best speech model on the same words, by word type and by whether the text models agree. Δ : speech minus text.
Reader
Oh , yes , I go over to Paris often .
Text
Oh , yes , I go over to Paris often .
Speech
Oh , yes , I go over to Paris often .
Table 10: One test sentence as labelled by the reader and as predicted by the best text model (ModernBERT-large) and the best speech model (Whisper large-v3), both with BiLSTM classifiers. Grey: not prominent (0); bold: prominent (1); large bold: highly prominent (2).
System
Own reader
Other reader
Δ
a
Whisper large-v3
80.3
65.8
+14.5
a
WavLM-large
81.3
65.5
+15.8
t
ModernBERT-large
67.4
70.6
-3.2
t
Qwen3.8 27B
65.7
69.9
-4.2
p
Qwen3.8 27B, 500 examples
60.8
62.0
-1.2
p
DeepSeek-V4-Pro, 5 examples
59.1
58.7
+0.4
Table 11: 3-way agreement (%) of each system’s predictions with the reader of the test utterance (own) and with another speaker reading the same text in the development set (other), on 1,906 words in 143 test utterances. Classifiers are BiLSTMs. a : speech, t : text, p : prompted. Δ : own minus other.