While predicting prosody from text is an established task in the field, the opposite direction, predicting text that fits a given prosodic pattern, remains largely overlooked. We find this unfortunate, because this opposite direction could lead to some very interesting use cases. Therefore, in this paper, we make the first steps in the prosody-to-text direction by inves- tigating how much of the original sentence can be recovered from its prosodic pattern. To this end, we fine-tune the Whis- per model using only the 12 lowest Mel bins (low-pass filter with approximately 450Hz cutoff), and obtain surprisingly accurate results (WER 36%), with 10% of utterances be- ing recovered perfectly, and 40% of utterances having Word Error Rate at or below 25%. We also find that, given the correct prefix, the next token was predicted correctly in 79% of cases. Our results suggest that the relationship between low-frequency speech features and lexical content is much stronger than previously thought, and we believe that direct- ing more attention to this topic might open the door to new applications, such as using prosody to guide text generation of modern LLMs
Figures & tables
Figure 1: The global oracle WER@k curves.
Figure 2: Cumulative distribution over WER.
Figure 3: Percentage of times the correct token was present in top k predictions given correct prefix.
Figure 4: Sample reconstruction of the lowest 12 bins of the Whisper Mel spectrogram by our Conformer model.
Unfortunately the patients knew which group they were in because the other one I didn’t block the rebirth, which gave it away, but she can’t discount the placebo effect.
P:
You can now come in and edit the video and there is hidden in the settings.
R:
Unfortunately, the patients knew which group they were in because they were evidently getting broccoli burps, which gave it away, so you can’t discount the placebo effect.
R:
You can now send in your audition videos on the link given in the comment box.
Animevox WER: 60.0 ID: 67
Animevox WER: 82.4 ID: 161
P:
Clearly a lawyer can handle a search in much longer.
P:
Their marriage is quite powerful, such as they don’t want to share with one another’s child.
R:
Surely a warrior can handle a fall from this height.
R:
Though a mage is quite powerful, she could be doubly so with a warrior by her side.
Figure 5: Sample sentences with WER near the mean for each model. P is prediction, R is reference. Red indicates errors.