MUTE: Multi-Level Alignment Uncoupling Against Talking-Head Exploitation for Voice Protection
Organizations: Soongsil University · Chung-Ang University · AIM Intelligence
Abstract
Talking-head generation models, which synthesize realistic facial animations from audio, are increasingly vulnerable to misuse in multimodal deepfake scenarios. Protecting such systems remains challenging, as talking-head models fundamentally rely on precise audio-visual alignment, while effective audio protection methods remain largely underexplored. In this work, we propose MUTE, an audio protection framework that uncouples the underlying audio-visual alignment through a multi-level strategy. Specifically, MUTE combines (1) Alignment-level Disruption, which directly perturbs cross-modal attention to disrupt structural alignment, and (2) Representation-level Degradation, which perturbs temporal audio embeddings to degrade their representations. To improve robustness, perturbations are constrained in the STFT domain to high-energy regions, making them resistant to post-processing, denoising, and adaptive attacks. An optional speaker-level objective further mitigates potential bypass via TTS-based resynthesis. Extensive experiments demonstrate that MUTE consistently degrades lip synchronization across both white-box and black-box talking-head models while preserving perceptual audio quality. The proposed method remains effective under real-world transformations and can be combined with image-based approaches to provide stronger multimodal protection. The source code and project page are available at https://github.com/jindl465/MUTE.
Figures & tables
| Method | TalkingHead-1KH [ 32 ] | CelebV-HQ [ 33 ] | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Audio Quality | A-V Align. | Visual Quality | Audio Quality | A-V Align. | Visual Quality | |||||||
| WER | STOI/PESQ | Sync | M-LMD | FID/FVD | E-FID | WER | STOI/PESQ | Sync | M-LMD | FID/FVD | E-FID | |
| C&W [ 20 ] | 13.397 0.488 | 0.922 0.000 / 2.322 0.002 | 3.740 0.016 | 1.096 0.013 | 1.855 0.083 / 1.442 0.024 | 3.646 0.111 | 13.057 0.210 | 0.948 0.001 / 2.702 0.004 | 4.039 0.032 | 1.108 0.006 | 1.662 0.151 / 1.247 0.030 | 3.508 0.123 |
| Psychoacoustic [ 21 ] | 12.140 0.543 | 0.922 0.001 / 2.333 0.003 | 3.787 0.042 | 1.097 0.005 | 1.914 0.046 / 1.427 0.013 | 3.655 0.110 | 11.643 0.098 | 0.949 0.001 / 2.740 0.002 | 3.926 0.019 | 1.126 0.011 | 1.676 0.202 / 1.276 0.055 | 3.457 0.191 |
| Qin et al. [ 22 ] | 15.649 0.652 | 0.925 0.000 / 2.301 0.002 | 4.468 0.031 | 0.983 0.003 | 1.825 0.144 / 1.329 0.039 | 3.443 0.076 | 13.870 0.079 | 0.942 0.000 / 2.544 0.005 | 4.543 0.025 | 1.054 0.005 | 1.479 0.018 / 1.182 0.016 | 3.331 0.037 |
| ARO [ 23 ] | 15.673 0.139 | 0.903 0.000 / 2.518 0.044 | 3.518 0.044 | 1.155 0.004 | 1.983 0.065 / 1.510 0.011 | 3.662 0.061 | 13.870 0.495 | 0.931 0.000 / 2.689 0.002 | 3.818 0.044 | 1.176 0.006 | 1.574 0.109 / 1.260 0.007 | 3.457 0.083 |
| Image Protection | Voice Protection | Sync | M-LMD | FID | FVD | E-FID |
|---|---|---|---|---|---|---|
| MIS [ 19 ] | - | |||||
| Silencer [ 18 ] | - | |||||
| MIS [ 19 ] | CAF [ 19 ] | |||||
| Silencer [ 18 ] | CAF [ 19 ] | |||||
| MIS [ 19 ] | MUTE | |||||
| Silencer [ 18 ] | MUTE |
| Method | Hallo2 [ 6 ] | Audio2Head [ 4 ] | SadTalker [ 1 ] | Ditto [ 9 ] | ||||
|---|---|---|---|---|---|---|---|---|
| Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | |
| C&W [ 20 ] | 3.773 0.053 | 1.144 0.018 | 4.269 0.014 | 1.222 0.044 | 5.540 0.013 | 0.629 0.006 | 4.024 0.019 | 0.697 0.004 |
| Psychoacoustic [ 21 ] | 3.747 0.034 | 1.137 0.011 | 4.241 0.018 | 1.188 0.016 | 5.503 0.056 | 0.630 0.004 | 4.002 0.008 | 0.697 0.001 |
| Qin et al. [ 22 ] | 4.535 0.017 | 1.027 0.018 | 4.510 0.029 | 1.104 0.013 | 5.702 0.067 | 0.583 0.002 | 4.182 0.038 | 0.678 0.004 |
| ARO [ 23 ] | 3.546 0.047 | 1.192 0.011 | 4.182 0.025 | 1.203 0.005 | 5.297 0.027 | 0.658 0.002 | 3.803 0.039 | 0.781 0.005 |
| SAGO [ 24 ] | 3.278 0.010 | 1.219 0.014 | 4.335 0.022 | 1.133 0.004 | 5.418 0.018 | 0.563 0.006 | 3.959 0.009 | 0.704 0.006 |
| Method | Down/Upsampling [ 41 ] | Spectral Filtering [ 41 ] | Demucs [ 42 ] | WavePurifier [ 43 ] | Adaptive Attack [ 44 ] | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | |
| C&W [ 20 ] | 3.791 0.052 | 0.971 0.002 | 3.884 0.026 | 0.987 0.010 | 3.968 0.033 | 1.031 0.009 | 4.071 0.016 | 0.940 0.013 | 4.878 0.021 | 0.557 0.016 |
| Psychoacoustic [ 21 ] | 3.825 0.028 | 0.979 0.019 | 3.878 0.013 | 1.004 0.003 | 3.953 0.055 | 1.042 0.001 | 4.059 0.009 | 0.935 0.001 | 4.840 0.034 | 0.555 0.015 |
| Qin et al. [ 22 ] | 4.032 0.044 | 0.955 0.007 | 4.346 0.052 | 0.950 0.022 | 4.569 0.041 | 0.978 0.011 | 3.974 0.123 | 0.952 0.024 | 4.926 0.040 | 0.564 0.009 |
| ARO [ 23 ] | 3.607 0.033 | 1.069 0.008 | 3.647 0.022 | 1.074 0.009 | 3.612 0.040 | 1.132 0.007 | 4.059 0.030 | 0.985 0.011 | 4.778 0.032 | 0.555 0.004 |
| SAGO [ 24 ] | 3.805 0.088 | 1.017 0.007 | 3.801 0.045 | 1.087 0.041 | 3.407 0.029 | 1.141 0.018 | 4.101 0.023 | 0.922 0.009 | 4.912 0.022 | 0.562 0.006 |
| Reference Condition | CosyVoice [ 46 ] | XTTS-v2 [ 47 ] | Chatterbox [ 48 ] | Sync | M-LMD | |||
| Sim | DSR | Sim | DSR | Sim | DSR | |||
| Unprotected | 5.656 | 0.000 | ||||||
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Hallo [ 5 ] | Audio2Head [ 4 ] | SadTalker [ 1 ] | ||||
|---|---|---|---|---|---|---|
| Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | |
| 1.485 | 1.511 | 3.963 | 1.217 | 5.083 | 0.756 | |
| 1.407 | 1.571 | 3.922 | 1.231 | 5.043 | 0.761 | |
| 1.383 | 1.608 | 3.813 | 1.255 | 4.957 | 0.795 | |
| 1.332 | 1.627 | 3.627 | 1.299 | 4.703 | 0.864 | |
| Sync | M-LMD | |
|---|---|---|
| Global | 2.484 | 1.317 |
| Frame-wise | 1.332 | 1.627 |
| Level 1 | Level 2 | Time | Hallo [ 5 ] | Audio2Head [ 4 ] | SadTalker [ 1 ] | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Loss Red. | Loss Red. | Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | |||
| 1.0 | -44.34% | 78.354 | 3.920 | 0.997 | 3.958 | 1.163 | 5.048 | 0.616 | ||
| -96.43% | ||||||||||
| 0.2 | -95.13% | -82.56% | 20.087 | 1.419 | 1.578 | 3.798 | 1.241 | 4.898 | 0.698 | |
| -36.63% | -95.45% | |||||||||
| -95.45% | -84.34% | |||||||||
| Purification Band | Down/Up [ 41 ] | Filtering [ 41 ] | Demucs [ 42 ] | WavePurifier [ 43 ] | ||||
|---|---|---|---|---|---|---|---|---|
| Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | Sync | M-LMD | |
| Speech-band (80–7600 Hz) | 2.873 | 1.268 | 2.306 | 1.434 | 1.532 | 1.584 | 3.444 | 1.222 |
| Full-band | 2.914 | 1.254 | 2.251 | 1.410 | 1.534 | 1.593 | 3.494 | 1.217 |
| Purification Method | Method | Audio2Head [ 4 ] | SadTalker [ 1 ] | ||
|---|---|---|---|---|---|
| Sync | M-LMD | Sync | M-LMD | ||
| Down/Up [ 41 ] | C&W [ 20 ] | 3.409 | 1.384 | 4.572 | 0.670 |
| Qin et al. [ 22 ] | 3.645 | 1.388 | 4.783 | 0.691 | |
| Psychoacoustic [ 21 ] | 3.450 | 1.382 | 4.592 | 0.672 | |
| ARO [ 23 ] | 3.514 | 1.337 | 4.581 | 0.716 | |
| SAGO [ 24 ] | 3.580 | 1.297 | 4.628 | 0.613 | |