Speech Generation Evaluation

Overview

Speech generation evaluation asks whether synthesized speech says the right words, sounds natural, preserves speaker identity, and follows instructions about prosody, emotion, or conversational style. Approaches include ASR-based WER and CER, acoustic and MOS-prediction metrics, speaker and emotion embeddings, human listening tests, and pairwise preference models or large audio-language model judges. Benchmarks increasingly test multilingual and domain-specific TTS, fine-grained paralinguistic control, nonverbal vocalizations, and timing and turn-taking in spoken dialogue. A central tension is that automatic scores can miss prosodic, linguistic, or contextual errors that people notice, while gains in intelligibility or semantic alignment may come at the expense of voice identity, expressiveness, or conversational naturalness.

Latest papers

62 papers6 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

Mar 8, 2026

StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control

Speech language models (SLMs) have significantly extended the interactive capability of text-based Large Language Models (LLMs) by incorporating paralinguistic information. For more realistic interactive experience with customized styles, current SLMs have managed to interpret and control speaking style intensity from user prompts during the dialogue process. However, there remains a lack of systematic benchmarks that quantifies and evaluates the style intensity control ability in conversations. In this paper, we propose StyleBench, a multi-turn dialogue benchmark for comprehensively evaluating the style intensity control ability across four dimensions: emotion, speed, volume, and pitch. Our results reveal the performance gaps between leading SLMs and omni language models (OLMs), suggesting the underlying reasons and promising approaches for future exploration.
Feb 26, 2026

Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis

Taiwanese Hokkien (Taigi) presents unique opportunities for advancing speech technology methodologies that can generalize to diverse linguistic contexts. We introduce Breeze Taigi, a comprehensive framework centered on standardized benchmarks for evaluating Taigi speech recognition and synthesis systems. Our primary contribution is a reproducible evaluation methodology that leverages parallel Taiwanese Mandarin resources. We provide 30 carefully curated Mandarin-Taigi audio pairs from Taiwan's Executive Yuan public service announcements with normalized ground truth transcriptions. We establish Character Error Rate (CER) as the standard metric and implement normalization procedures to enable fair cross-system comparisons. To demonstrate the benchmark's utility and provide reference implementations, we develop speech recognition and synthesis models through a methodology that leverages existing Taiwanese Mandarin resources and large-scale synthetic data generation. In particular, we fine-tune a Whisper model on approximately 10,000 hours of Taigi synthetic speech data. Our ASR model achieves 30.13% average CER on the benchmark, outperforming existing commercial and research systems. By providing standardized evaluation protocols, diverse training datasets, and open baseline models, we offer a replicable framework with methodologies applicable to various linguistic contexts.