cs.CLMay 6, 2025

Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework

Authors: Cléa Chataigner, Rebecca Ma, Prakhar Ganesh, Yuhao Chen, Afaf Taïk, Elliot Creager, Golnoosh Farnadi

Organizations: Mila - Quebec AI Institute · McGill University · University of Waterloo · Vector Institute · 

Abstract

Large language models (LLMs) are highly sensitive to subtle changes in prompt phrasing, posing challenges for reliable auditing. Prior methods often apply unconstrained prompt paraphrasing, which risk missing linguistic and demographic factors that shape authentic user interactions. We introduce AUGMENT (Automated User-Grounded Modeling and Evaluation of Natural Language Transformations), a framework for generating controlled paraphrases, grounded in user behaviors. AUGMENT leverages linguistically informed rules and enforces quality through checks on instruction adherence, semantic similarity, and realism, ensuring paraphrases are both reliable and meaningful for auditing. Through case studies on the BBQ and MMLU datasets, we show that controlled paraphrases uncover systematic weaknesses that remain obscured under unconstrained variation. These results highlight the value of the AUGMENT framework for reliable auditing.

Figures & tables

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

    Jun 18, 2026Zilong Zhang, Yi-Ting Hung, Weiyi He +3LLM-as-a-JudgeLLM Auditing

  2. Neither Over- nor Under-Constrained: Evaluating LLM Personalization under a Constraint Qualification Audit

    Oct 19, 2026Xuran Li, Guanqin Zhang, Imran Razzak +4LLM EvaluationLLM Personalization