MixedTruth: Component-Level Detection of Synthetic Speech and Synthetic Background in Mixed Audio
Organizations: Vanderbilt University
Abstract
Modern generative audio can fabricate not only voices but also the acoustic environments that provide evidence of when and where speech occurred. This capability undermines a longstanding forensic assumption that background sound provides reliable evidence of a recording's provenance. Yet existing detectors produce only a single real-or-synthetic verdict, leaving analysts unable to determine whether the voice, the environment, or both were manipulated. In this paper, we propose a component-level forensic detector, namely TruMix, which classifies recordings into eight categories spanning the presence and authenticity of speech and background. TruMix has a unique multi-path architecture that jointly analyzes speech-oriented and background-oriented spectral evidence, contextual information from the mixed waveform, and low-level synthesis artifacts. This design avoids the artifact propagation inherent in conventional separate-then-detect pipelines while enabling generalization across independent speech and environmental-sound generators. We also curated MixedTruth, a 4,917-hour benchmark covering all eight categories under diverse mixing conditions. Extensive experiments show that TruMix achieves 98.51% AUC for component-level attribution, significantly outperforms ten state-of-the-art synthetic-speech detectors in conventional binary classification, and exhibits strong robustness across four attack tiers.
Figures & tables
| Detector (Year) | Main Architecture | Parameters | FP32 Size |
|---|---|---|---|
| TrueEar [ 2 ] (2025) | SincConv + temporal encoder + VLAD | 0.048M | 0.192 MB |
| AASIST-L [ 11 ] (2022) | lightweight raw-waveform encoder + spectro-temporal graph attention | 0.085M | 0.340 MB |
| AASIST [ 11 ] (2022) | Raw-waveform encoder + spectro-temporal graph attention | 0.298M | 1.192 MB |
| RawNet2 [ 20 ] (2021) | SincConv + residual blocks + GRU | 17.621M | 70.484 MB |
| ALDEN [ 22 ] (2025) | XLS-R (24 blocks) + dual-level disentanglement + meta-learning | 316.274M | 1.265 GB |
| Trident of Poseidon [ 7 ] (2024) | XLS-R (24 blocks) + VIB head | 317.572M | 1.270 GB |
| WavLM Base+ with 8-class head | TruMix | |||||
| Category | Class accuracy | One-vs-Rest AUC | One-vs-Rest EER | Class accuracy | One-vs-Rest AUC | One-vs-Rest EER |
| RSRB | 53.627% | 91.357% | 16.037% | 78.813% | 97.047% | 8.893% |
| RSSB | 42.336% | 90.878% | 16.402% | 82.960% | 97.811% | 7.337% |
| SSRB | 51.336% | 90.849% | 16.517% | 78.984% | 97.346% | 8.311% |
| SSSB | 46.189% | 90.370% | 16.923% | 83.476% | 97.856% | 7.182% |
| RS | 88.636% | 98.989% | 4.377% | 94.116% | 99.521% | 2.811% |
| Detector | RS/SS AUC | RS/SS EER | +BG AUC | +BG EER |
|---|---|---|---|---|
| TrueEar | 73.094% | 32.316% | 57.847% | 44.513% |
| AASIST | 79.703% | 28.533% | 64.394% | 39.267% |
| AASIST-L | 79.889% | 28.229% | 59.502% | 43.762% |
| RawNet2 | 76.017% | 31.373% | 60.578% | 42.236% |
| AASIST2 | 90.361% | 20.773% | 74.341% | 32.142% |
| AASIST3 | 76.409% | 31.444% | 61.680% | 41.704% |
| Detector | Original AUC | Original EER | Sep. AUC | Sep. EER |
|---|---|---|---|---|
| TrueEar | 57.97% | 45.59% | 65.57% | 39.03% |
| AASIST | 70.22% | 33.19% | 69.72% | 32.97% |
| AASIST-L | 67.37% | 36.44% | 66.05% | 37.41% |
| RawNet2 | 64.64% | 38.94% | 68.73% | 34.34% |
| AASIST2 | 87.71% | 20.62% | 79.78% | 28.93% |
| AASIST3 | 67.42% | 37.53% | 59.23% | 43.07% |
| Configuration | Accuracy | Macro F1 | Voice EER | BG EER |
|---|---|---|---|---|
| (1) WavLM Base+ with 8-class multi-head | 63.390% | 63.096% | 12.904% | 38.581% |
| (2) + separated voice/background dual paths | 85.827% (+22.436) | 85.812% (+22.716) | 9.325% (-3.579) | 7.407% (-31.173) |
| (3) + waveform-artifact Path D | 85.872% (+22.482) | 85.855% (+22.759) | 9.206% (-3.699) | 7.420% (-31.161) |
| (4) + structured multi-head fusion | 85.783% (+22.392) | 85.761% (+22.665) | 8.953% (-3.951) | 7.441% (-31.139) |
| (5) Complete model | 86.933% (+23.543) | 86.917% (+23.821) | 7.419% (-5.485) | 7.174% (-31.407) |
| Transformation | TMR (TruMix) | TMR (Nes2Net-X) |
|---|---|---|
| Original | 4.70% | 30.30% |
| 0% background | 0.20% | 27.70% |
| 10% background | 4.80% | 28.90% |
| 20% background | 7.30% | 31.20% |
| 60% background | 8.30% | 33.50% |
| 80% background | 1.60% | 59.20% |
| Category | Queries | TMR | CPR | SPR | SNR (dB) | VER |
|---|---|---|---|---|---|---|
| RSSB | 100 | 100.00% | 7.28% | 66.06% | -0.87 | 0.50% |
| SSRB | 100 | 100.00% | 0.85% | 62.54% | -1.76 | 0.00% |
| SSSB | 100 | 100.00% | 0.90% | 65.19% | -1.65 | 0.00% |
| SS | 100 | 100.00% | 0.69% | 61.76% | -0.40 | 0.00% |
| Overall | 100 | 100.00% | 2.43% | 63.89% | -1.17 | 0.13% |
| RSSB | 1,000 | 100.00% | 35.77% | 84.56% | 10.82 | 18.00% |
| Category | TMR | CPR | SPR | SNR (dB) | VER |
|---|---|---|---|---|---|
| RSSB | 0.90% | 72.00% | 99.65% | 33.96 | 0.75% |
| SSRB | 25.40% | 46.65% | 99.14% | 35.42 | 1.75% |
| SSSB | 0.90% | 0.00% | 99.18% | 35.06 | 0.00% |
| SS | 23.10% | 56.66% | 96.97% | 35.67 | 2.00% |
| Overall | 12.58% | 51.72% | 98.74% | 35.50 | 1.13% |
| Attack | Budget | TMR | CPR | SPR | SNR (dB) | VER |
|---|---|---|---|---|---|---|
| TruMix | ||||||
| PGD-20 | 0.0005 | 75.48% | 61.36% | 99.42% | 45.19 | 22.88% |
| PGD-20 | 0.0010 | 90.30% | 59.68% | 99.20% | 40.25 | 26.75% |
| PGD-20 | 0.0020 | 95.38% | 57.78% | 98.87% | 34.94 | 26.63% |
| PGD-20 | 0.0040 | 97.08% | 54.99% | 98.18% | 29.27 | 22.31% |
| AutoAttack | 0.0020 | 99.18% | 56.65% | 98.17% | 33.27 | 26.44% |