Language models represent more about emphasis than they can tell you
Authors: Aarne Talman, Stergios Chatzikyriakidis
Organizations: University of Helsinki, Finland · University of Turku, Finland · Institute of Language and Speech Processing, Athena Research Centre, Greece
Prosodic prominence can largely be predicted from the words of a sentence, but part of it depends on how the speaker reads it. We study how much of prominence language models recover from text, whether they can report it when asked, and what the speech signal adds. On read English speech labelled for word-level prominence, we compare frozen and fine-tuned text models, prompted large language models (LLMs) and speech models on the same words. All text models end within about one point of each other, at best 68.8% on the 3-way task, and largely make the same errors. With the prompts we tested, LLMs do no better than a word-majority baseline, even with 500 examples. Linear probes find more in their representations. Speech models reach 84.7%. We argue that text determines only part of prominence, and that the rest has to be heard. LLMs encode what the text determines, but prompting does not bring it out: they represent more about where the emphasis falls than they tell you.
Figures & tables
Labels (%)
Split
LibriTTS subset
Utterances
Words
Speakers
0
1
2
Used for
train_360
train-clean-360
116,263
2,076,289
904
48.3
27.4
24.2
training (text)
train_100
train-clean-100
33,042
570,592
245
48.1
27.3
24.6
training (speech; matched text)
dev
dev-clean
5,727
99,200
40
47.9
27.7
24.4
model, layer and prompt selection
test
test-clean
4,822
90,063
39
48.0
27.3
24.7
evaluation
Table 1: The data splits. Words: words with a prominence label (punctuation excluded). Labels: share of words in each 3-way class.
Model
Hugging Face identifier
Params
Revision
Encoders
BERT-base
google-bert/bert-base-uncased
110M
86b5e093
BERT-base cased
google-bert/bert-base-cased
109M
cd5ef92a
BERT-large
google-bert/bert-large-uncased
336M
6da4b6a2
ModernBERT-base
answerdotai/ModernBERT-base
150M
8949b909
ModernBERT-large
answerdotai/ModernBERT-large
396M
45bb4654
Table 2: Models, with their Hugging Face identifiers, number of parameters (Whisper and Qwen2-Audio: the whole model) and the revisions used.
System
2-way
3-way
Baselines
t
Majority class
52.0
48.0
t
Word majority (2019)
80.7
62.0
Fine-tuned
t
BERT-base (2019 recipe)
83.0 ± 0.1
68.5 ± 0.1
t
BERT-large
83.1 ± 0.1
68.5 ± 0.1
Table 3: Test accuracy (%). Mean ± standard deviation over seeds where there are several. t : text input, a : audio input. Best layers and models are chosen on the development set. Bold: best text and best speech system in each column (baselines and references excluded). † 200 test utterances.
Figure 1: 3-way test accuracy of BERT-base with growing training data (subsets of train_360 , train_100 and train_360 ).
Figure 2: How often two systems give a word the same 3-way label (%), on the 89,935 test words used for the text and speech comparison. Text and speech models are BiLSTM classifiers; white lines separate the gold labels, speech models, text models and prompted LLMs.
Figure 3: 3-way test accuracy of a linear probe on each layer, by relative depth (0: embeddings, 1: last layer).
Model
Access
2-way
3-way
Qwen3.8 27B ( t )
linear probe
82.3
66.7
BiLSTM classifier
82.9 ± 0.1
67.8 ± 0.1
prompted, no examples
78.8
58.6
prompted, 500 examples
78.3
62.0
Qwen2-Audio 7B ( a )
linear probe (LM)
86.4
74.4
BiLSTM classifier (LM)
87.3 ± 0.1
76.5 ± 0.1
Table 4: The same model probed and prompted (test accuracy, %). t : text input, a : audio input. Bold: best access for each model.
Figure 4: Prompting Qwen3.8 27B with more examples (500 development utterances).
Recall (%)
Labelled 2 (%)
System
Macro-F1
0
1
2
content
function
last
Gold labels
–
–
–
–
42.2
9.2
31.9
t
ModernBERT-large, BiLSTM classifier
63.9
83.5
45.2
63.1
46.7
6.7
35.4
t
Qwen3.8 27B, BiLSTM classifier
63.8
83.2
48.4
59.2
43.1
6.0
31.7
p
Qwen3.8 27B, prompted (500 ex.)
55.9
84.2
35.4
48.3
44.2
3.4
45.5
p
DeepSeek-V4-Pro, prompted (5 ex.)
54.7
80.0
31.1
55.1
56.8
4.8
77.1
Table 5: How each system labels the test set (3-way): macro-F1 and recall of each class, and the share of content words, function words and utterance-final words given the highest level (2). t : text classifier, p : prompted, a : speech classifier.
Figure 5: Confusion matrices (3-way, test set), normalised by row: the share of the words with each gold label that receive each predicted label (%).
Right where it departs
System
Same as lookup
Departs
system
lookup
t
ModernBERT-large, BiLSTM classifier
71.7
28.3
51.8
30.7
t
Qwen3.8 27B, BiLSTM classifier
72.4
27.6
51.4
30.8
p
Qwen3.8 27B, prompted (500 ex.)
69.3
30.7
39.5
39.8
p
DeepSeek-V4-Pro, prompted (5 ex.)
68.8
31.2
37.1
42.1
p
GPT-4o, prompted (5 ex.)
68.0
32.0
36.0
44.2
Table 6: How often each system gives a word the label that the word most often has in the training data (the word-majority lookup), and, on the words where it does not, how often the system and the lookup are right (%, 3-way, test set).
Speaker A
Hilda’s face quivered , but she whispered Yes , I think it must have been .
Speech model, hearing A
Hilda’s face quivered , but she whispered Yes , I think it must have been .
Speaker B
Hilda’s face quivered , but she whispered Yes , I think it must have been .
Text model
Hilda’s face quivered , but she whispered Yes , I think it must have been .
Table 7: One test sentence read by two speakers, typeset by prominence: speaker A (test set), the best speech model hearing speaker A, speaker B (development set) and the best text model, whose prediction is the same for both readings. Grey: not prominent (0); bold: prominent (1); large bold: highly prominent (2).
System
2-way
3-way
95% CI (3-way)
a
Labelling procedure
95.0
91.5
91.3–91.7
a
Whisper large-v3 (encoder), BiLSTM classifier
90.9
84.6
84.3–84.9
a
Whisper large-v3 (encoder), linear probe
88.3
78.8
78.5–79.1
a
Qwen2-Audio 7B (LM), BiLSTM classifier
87.4
76.7
76.4–77.0
a
Duration only
83.2 ∘
65.0
64.7–65.4
t
Qwen3.5 0.8B, echo, BiLSTM classifier
82.9 ∘
68.1
67.7–68.4
Table 8: Accuracy (%) on the same 89935 test words, best systems of each kind. All differences from the best text model are significant (McNemar, p<0.05 ) unless marked ∘ . Bold: best text and best speech system (references excluded).
Words
n
Text
Audio
Δ
content words
42,281
56.1
79.0
+22.9
function words
47,654
78.7
89.6
+10.8
first word
4,740
67.0
83.4
+16.4
medial
80,394
69.1
85.2
+16.1
last word
4,801
51.7
75.1
+23.4
new (content)
29,629
56.4
79.1
+22.7
Table 9: 3-way accuracy (%) of the best text and the best speech model on the same words, by word type and by whether the text models agree. Δ : speech minus text.
Reader
Oh , yes , I go over to Paris often .
Text
Oh , yes , I go over to Paris often .
Speech
Oh , yes , I go over to Paris often .
Table 10: One test sentence as labelled by the reader and as predicted by the best text model (ModernBERT-large) and the best speech model (Whisper large-v3), both with BiLSTM classifiers. Grey: not prominent (0); bold: prominent (1); large bold: highly prominent (2).
System
Own reader
Other reader
Δ
a
Whisper large-v3
80.3
65.8
+14.5
a
WavLM-large
81.3
65.5
+15.8
t
ModernBERT-large
67.4
70.6
-3.2
t
Qwen3.8 27B
65.7
69.9
-4.2
p
Qwen3.8 27B, 500 examples
60.8
62.0
-1.2
p
DeepSeek-V4-Pro, 5 examples
59.1
58.7
+0.4
Table 11: 3-way agreement (%) of each system’s predictions with the reader of the test utterance (own) and with another speaker reading the same text in the development set (other), on 1,906 words in 143 test utterances. Classifiers are BiLSTMs. a : speech, t : text, p : prompted. Δ : own minus other.
As language models (LMs) have improved, an increasingly common means of evaluating their linguistic competence is through explicit prompting (e.g., "Which sentence is more acceptable?"). Prior work indicates that such prompting underperforms compared to standard token-probability-based methods in smaller models, but improves in larger models. We test this in a new domain: semantic focus-sensitivity, responsible for contrasts like "Lucy only SWAM yesterday" vs. "Lucy only swam YESTERDAY". Testing models of various sizes using text-based focus markers like boldface and italics, we find universal above-chance performance for token probabilities. Prompting results mostly align with previous findings, but in some larger models they outperform probabilities by a wide margin. We propose that this is because LMs are not prone to produce out-of-the-blue focus marking, which muddies results from token probabilities in a way that does not affect prompting. We test this by replacing standard focus markers with randomly generated ones; token probabilities still beat chance but show significantly greater performance decrease compared to prompting. These results suggest that metalinguistic prompting, while non-trivial for LMs, can also be more robust when testing stylistically atypical data. We argue that this has important consequences for the assessment of LMs' language competence, even if LMs do not perform metalinguistic judgments through human-like introspection.
While predicting prosody from text is an established task in the field, the opposite direction, predicting text that fits a given prosodic pattern, remains largely overlooked. We find this unfortunate, because this opposite direction could lead to some very interesting use cases. Therefore, in this paper, we make the first steps in the prosody-to-text direction by inves- tigating how much of the original sentence can be recovered from its prosodic pattern. To this end, we fine-tune the Whis- per model using only the 12 lowest Mel bins (low-pass filter with approximately 450Hz cutoff), and obtain surprisingly accurate results (WER 36%), with 10% of utterances be- ing recovered perfectly, and 40% of utterances having Word Error Rate at or below 25%. We also find that, given the correct prefix, the next token was predicted correctly in 79% of cases. Our results suggest that the relationship between low-frequency speech features and lexical content is much stronger than previously thought, and we believe that direct- ing more attention to this topic might open the door to new applications, such as using prosody to guide text generation of modern LLMs
David Porteš, Aleš Horák
Natural Language Processing Centre Faculty of Informatics, Masaryk University Botanicka 68a, 602 00 Brno, Czech Republic
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient. Code is available at https://github.com/dreamtheater123/TextPro-SLM
Wenqian Cui, Xiao-Hui Li, Daxin Tan +2
The Chinese University of Hong Kong · Huawei Technologies