cs.CYApr 15, 2026

Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking

Authors: Alexander Nemecek, Osama Zafar, Yuqiao Xu, Wenbiao Li, Erman Ayday

Organizations: Case Western Reserve University

Abstract

Watermarking is becoming the default mechanism for AI content authentication, with governance policies and frameworks referencing it as infrastructure for content provenance. Yet across text, image, and audio modalities, watermark signal strength, detectability, and robustness depend on statistical properties of the content itself, properties that vary systematically across languages, cultural visual traditions, and demographic groups. We examine how this content dependence creates modality-specific pathways to bias. Reviewing the major watermarking benchmarks across modalities, we find that, with one exception, none report performance across languages, cultural content types, or population groups. To address this, we propose three concrete evaluation dimensions for pluralistic watermark benchmarking: cross-lingual detection parity, culturally diverse content coverage, and demographic disaggregation of detection metrics. We argue that watermarking is part of the pluralistic alignment pipeline and should be held to the same evaluation standards. We connect this to governance frameworks currently mandating watermarking deployment without requiring fairness evaluation. Our position is that evaluation must precede deployment, and that the same bias auditing requirements applied to AI models should extend to the verification layer.

Figures & tables

Explore similar work

CardsList
  1. Auditing Cross-Lingual Fairness in Language Model Watermarking

    Aug 20, 2026Alexander Nemecek, Osama Zafar, Debargha Ganguly +3LLM WatermarkingLLM Auditing

  2. AI Watermark Evidence Fails Forensic Readiness: An Empirical Evaluation

    Jul 17, 2026Saifur Rahman Tamim, Amir Labib KhanWatermark RemovalText Watermarking

  3. Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs

    May 28, 2026Zhihao Wu, Gracia Gong, Qinglin Zhu +2Watermark RemovalLLM Watermarking