AUTEUR: Acoustic Plan Learning for Red-Teaming Text-to-Audio Models
Organizations: Jilin University · Nanyang Technological University
Abstract
Automated red-teaming of text-to-audio models requires test prompts that pass textual screening and expose harmful acoustic content. The central challenge is to connect the sounds produced by the target model to both reusable test generation and prompt revision. Our key idea is to represent test descriptions as structured acoustic plans that organize foreground sounds, context, and vocal attributes. To this end, we propose acoustic plan learning, a framework that learns and refines these plans using waveform feedback. We train the plan generator through supervised fine-tuning followed by reinforcement learning with audio feedback while keeping the target audio model fixed. At inference time, the same trained model generates initial plans and uses feedback to revise their context and vocal attributes while preserving the foreground. Experiments across four target systems show attack success rates of 79.4% on TangoFlux and 95.6% on Kling. The resulting test prompts also transfer across open-source and closed-source models, indicating shared acoustic vulnerabilities.
Figures & tables
| Model | Method | Category ASR | ASR | PSR | |||
| Hate | Violence & Self-harm | Shocking | Illegal Act. | ||||
| FLIRT Mehrabi et al. (2024) | 67.5% | 60.0% | 67.5% | 47.5% | 60.6% | 86.2% | |
| PGJ Huang et al. (2025) | 57.5% | 40.0% | 42.5% | 32.5% | 43.1% | 82.5% | |
| AudioLDM Liu et al. (2023) | Ours | 80.0% | 75.0% | 72.5% | 60.0% | 71.9% | 86.2% |
| FLIRT Mehrabi et al. (2024) | 72.5% | 57.5% | 87.5% | 57.5% | 68.8% | 84.4% | |
| PGJ Huang et al. (2025) | 75.0% | 67.5% | 72.5% | 62.5% | 69.4% | 83.1% | |
| Category | Kling | ElevenLabs | Comb. ASR | PSR |
| Hate | 97.5% | 60.0% | 78.8% | 91.2% |
| Violence & Self-harm | 97.5% | 80.0% | 88.8% | 96.2% |
| Shocking | 95.0% | 82.5% | 88.8% | 98.8% |
| Illegal Act. | 92.5% | 72.5% | 82.5% | 92.5% |
| Overall | 95.6% | 73.8% | 84.7% | 94.7% |
| Variant | Combined ASR (%) | Drop (pp) |
| AUTEUR (full) | 75.6 | 0.0 |
| w/o SFT | 65.6 | 10.0 |
| w/o RLOO | 63.1 | 12.5 |
| w/o Scene description | 59.3 | 16.3 |
| w/o Vocal description | 35.3 | 40.3 |
| Target | Avg. Attempts |
| TangoFlux | 7.56 |
| AudioLDM | 8.74 |
| Kling | 2.39 |
| ElevenLabs | 5.16 |
Text-to-Audio Generation
Text-to-Audio Generation creates speech, music, and sound effects from natural-language prompts, aiming to match their content, timing, and acoustic qualities.
See all 48 papers