eess.ASSep 27, 2026

Is Word Error Rate Enough? Rethinking Privacy Evaluation in Speech with Entity-Aware Metrics

Authors: Anjana Rajasekhar, Jule Pohlhausen, Nayana Jacob Alappattu, Anna Leschanowsky

Abstract

As the use of smart devices continues to increase, their potential to capture sensitive speech content raises growing privacy concerns. It is therefore critical to develop techniques that prevent information leakage while preserving the utility of the audio, and evaluation metrics that accurately quantify the level of privacy without overestimating it. In this work, we evaluate the effectiveness of two obfuscation techniques in protecting speech content, with particular emphasis on named entities, by adapting entity-aware privacy metrics from the Natural Language Processing field to the speech privacy domain. Further, we investigate several attack scenarios and show that fine-tuning on entity-rich data improves attack performance for some entity categories but not others. Finally, we provide guidance on metric selection based on whether the obfuscation method preserves temporal alignment.

Explore similar work

CardsList
  1. Voice Privacy from an Attribute-based Perspective

    Mar 19, 2026Mehtab Ur Rahman, Martha Larson, Cristian Tejedor-GarciaPrivacy AuditingSpeech Processing

  2. CLEAR: Online Speech Content Leakage Estimation through Cross-ASR Disagreement

    Sep 24, 2026Bhawana Chhaglani, Tanvi Kandepuneni, Jeremy Gummeson +1

  3. A Large-Scale Per-Speaker Analysis of Re-identification Risk in Speech Anonymization

    Jun 5, 2026Orane Dufour, Paul Magron, Mickael Rouvier +1Privacy AuditingSpeech Processing