Speech Translation Evaluation

Speech translation evaluation studies how well systems transfer meaning between spoken languages, including in speech-to-text, speech-to-speech, and simultaneous settings. Methods range from text-based metrics such as BLEU and COMET to speech-aware quality estimators, human ratings, error annotation, and benchmarks that measure prosody, speaker identity, latency, and usability. Evaluations increasingly use real-time and long-form interactions, multilingual and dialectal data, and separate scores for distinct dimensions of performance. A central challenge is that strong text-translation scores do not guarantee natural, faithful, or usable speech, while automatic metrics still struggle to capture speech-specific features consistently.