Clinical speech reveals identity through both spoken identifiers and voice, yet its pauses, delivery, and speaker roles can also carry useful clinical evidence. Protecting that speech therefore requires more than preserving its words. We present AegisVox, a factorized framework that assigns consistent surrogate voices and edits identifier-bearing turns using document-consistent replacements. Across 390 public simulated or non-patient conversations totaling 63.6 hours, a matched comparison of two speech renderers separates replacement realization from retained identifiers, speaker linkage, and clinical preservation. Text-to-speech rendering realizes 535/601 planned replacements versus 453/601 for the codec-based speech editor. However, similar aggregate clinical preservation conceals more model-labeled contradictions, and conversation-wide speaker scores conceal greater edited-speech linkage under a score-direction sensitivity analysis. Fresh transcription of original audio provides a reference for interpreting the clinical measurement chain; an independent judge tests sensitivity to model-based assessment. Better replacement realization can therefore coexist with worse clinical or speaker-linkage measurements. AegisVox supports use-specific evaluation of those tradeoffs, not a guarantee of identifier removal or clinical safety.
Figures & tables
Figure 1: AegisVox separates voice identity from identifying words. SeedVC converts every turn using one consistent surrogate voice per source speaker. The lexical renderer changes only converted turns with planned identifier replacements; the remaining converted turns pass directly to reconstruction. F5-TTS is the illustrated renderer, and VoiceCraft is the matched alternative. PHI denotes protected health information.
Dataset
Language
Conversations
Hours
Median [IQR] (s)
Setting
PriMock57-mixed
EN
57
8.6
556 [456–617]
Doctor–patient consultations
Fareez
EN
272
51.8
672 [572–778]
Respiratory interviews
SIMSAMU
FR
61
3.2
187 [152–234]
Simulated emergency calls
Total
EN/FR
390
63.6
609 [471–736]
Public clinical audio
Table 1: Public evaluation population and comparison roles. The original-audio control changes no speech; all transformation conditions share the source recordings. EN and FR denote English and French.
Figure 3
Figure 2: Clinical controls and matched renderer effects on 25,778 source CAPs from 390 conversations. (a) Fresh original-audio ASR supplies the reference, not clinical ground truth. (b) F5-TTS minus VoiceCraft effects in percentage points (pp), with 95% conversation-cluster intervals. Similar strict preservation coexists with more model-labeled contradictions.
Outcome
Denominator
VoiceCraft
F5-TTS
Effect (pp)
95% CI (pp)
Replacement realized
601
453 (75.37%)
535 (89.02%)
+13.64
[+9.69,+17.59]
Original PHI retained
601
21 (3.49%)
28 (4.66%)
+1.16
[−0.62,+2.88]
CAP preserved
25,778
19,954 (77.41%)
19,929 (77.31%)
−0.10
[−0.61,+0.42]
CAP contradicted
25,778
391 (1.52%)
479 (1.86%)
+0.34
[+0.15,+0.54]
Table 2: Matched outcomes within AegisVox. Counts and item-weighted rates are shown for each renderer. Effects are F5-TTS minus VoiceCraft, with 95% conversation-cluster intervals. The lexical rows cover 601 edit intentions from 208 conversations; clinical rows cover 25,778 CAPs from 390 conversations. Realization is conditional on a planned edit, not complete PHI removal.
Figure 3: Final-waveform linkage under evaluation-label-selected score direction. F5-TTS minus VoiceCraft effects use all-turn samples (809 profiles, 390 conversations) or planned-edit regions (291 profiles, 204 conversations). Bars are 95% clustered CIs. Lower EER means easier source linkage.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Governed integration and threat boundary. AegisVox transforms speaker identity and identifying language before controlled release, while access, provenance, residual-risk review, and monitoring remain outside the model and continue to constrain use.
Figure 5: Public evaluation population by dataset, language, detected speaker count, and planned edit count. Duration strata are specified in the text.
Setting
(Dsrc,Dtgt)
Assigned
Median Dsrc
Median Dtgt
ECAPA EER
x-vector EER
Frontend WER
CAP
CLAP / HeAR
Utility-oriented
(0.60,0.55)
809/809
1.1600
0.8741
0.3115
0.3426
0.2314
0.7446
0.7296 / 0.9448
Balanced
(0.70,0.65)
809/809
1.1590
0.8741
0.3103
0.3436
0.2311
0.7453
0.7297 / 0.9448
Privacy-oriented
(0.75,0.70)
809/809
1.1585
0.8773
0.3140
0.3461
0.2310
0.7453
0.7303 / 0.9448
Appendix
Table 3: Retrieval-threshold sensitivity over separately rendered conditions. CAP is the recording-level mean strict proposition-preservation rate. ASV uses the 809-positive and 653,672-impostor trial population per backend; ECAPA is orientation-normalized so EER is at most 0.5. Absolute values belong to this sensitivity family and are not substituted for the primary-condition results.
Figure 6: Controlled qualification suite for source-anchored clinical adjudication. Each corruption family tests whether the judge distinguishes preserved content from a clinically consequential change under protocol variants.
Condition
Covered
Qwen preserved / 600
Weighted Qwen
Agreement
κ
SeedVC only
522
493 (82.2%)
90.9%
413/522 (79.1%)
0.253
AegisVox (VoiceCraft)
512
466 (77.7%)
88.8%
405/512 (79.1%)
0.314
AegisVox (F5-TTS)
511
462 (77.0%)
85.7%
405/511 (79.3%)
0.340
Appendix
Table 4: Independent Qwen sensitivity on 600 shared CAPs per condition. Covered cases have valid agreeing Qwen labels under both orders. Strict raw counts retain all 600 intentions. Weighted rates use inverse stratum probabilities. Agreement and κ are unweighted over covered cases and retain GPT uncertainty as a third category; they are not binary accuracy.
Figure 7: Independent-judge sensitivity. Left: inverse-stratum-weighted strict preservation under each judge, with conversation-cluster intervals. Center: unweighted uncertainty-retaining Cohen’s κ on covered cases. Right: realized replacements under each judge, with numerators over 601 intentions. Weighting, label agreement, and lexical ordering answer different questions.
Scoring contract
Gold
TP
FP
FN
Precision [95% CI]
Recall [95% CI]
F1 [95% CI]
Contract intersection, exact typed
2,147
2,043
137
104
0.937 [0.926, 0.947]
0.952 [0.942, 0.960]
0.944 [0.935, 0.953]
Contract intersection, boundary tolerant
2,147
2,124
56
23
0.974 [0.968, 0.981]
0.989 [0.985, 0.993]
0.982 [0.977, 0.986]
Full ASQ ontology, exact typed
2,973
2,150
259
823
0.892 [0.880, 0.905]
0.723 [0.714, 0.732]
0.799 [0.789, 0.808]
Full ASQ ontology, boundary tolerant
2,973
2,348
61
625
0.975 [0.968, 0.981]
0.790 [0.781, 0.799]
0.873 [0.866, 0.879]
Appendix
Table 5: Independent ASQ-PHI detector evaluation. Confidence intervals use 2,000 query-level bootstrap resamples. The full ontology includes the prespecified geography-contract mismatch.
Category
Gold
Exact precision
Exact recall
Exact F1
Boundary F1
Name
814
0.937
0.932
0.935
0.995
Date
806
0.944
0.989
0.966
0.967
Medical record number
305
0.909
0.918
0.914
0.995
Health-plan beneficiary number
91
0.956
0.945
0.950
0.972
Phone number
45
1.000
1.000
1.000
1.000
Social Security number
33
0.943
1.000
0.971
0.971
Appendix
Table 6: Primary category results on the contract-intersection gold set. Low-support residual categories are grouped; unique identifiers are shown separately because their performance differs.
Renderer
Planned edits
Replacement realized
Original PHI retained
Uncertain
VoiceCraft
601
453 (75.4%)
21 (3.5%)
110 (18.3%)
F5-TTS
601
535 (89.0%)
28 (4.7%)
46 (7.7%)
Appendix
Table 7: Conditional lexical-renderer outcomes over the same 601 planned edit intentions, assessed from generated-turn transcripts. Uncertainty refers to realization; outcomes may overlap. Detection is reported in Table 5 .
Condition
Preserved
Partial
Missing
Contradicted
Uncertain
Fresh original audio
24,998 (97.0%)
556 (2.2%)
128 (0.5%)
95 (0.4%)
1 (0.0%)
Speaker path only
20,089 (77.9%)
2,304 (8.9%)
2,630 (10.2%)
364 (1.4%)
391 (1.5%)
AegisVox (VoiceCraft)
19,954 (77.4%)
2,347 (9.1%)
2,663 (10.3%)
391 (1.5%)
423 (1.6%)
AegisVox (F5-TTS)
19,929 (77.3%)
2,367 (9.2%)
2,724 (10.6%)
479 (1.9%)
279 (1.1%)
Localized redaction
20,016 (77.6%)
2,225 (8.6%)
2,891 (11.2%)
387 (1.5%)
259 (1.0%)
Appendix
Table 8: Source-anchored CAP outcome decomposition. Each row contains the same 25,778 propositions; percentages are row percentages.
Condition
Critical ( n=1,269 )
Major ( n=13,486 )
Minor ( n=11,023 )
Speaker path only
976 (76.9%)
11,044 (81.9%)
8,069 (73.2%)
AegisVox (VoiceCraft)
977 (77.0%)
10,951 (81.2%)
8,026 (72.8%)
AegisVox (F5-TTS)
959 (75.6%)
10,957 (81.2%)
8,013 (72.7%)
Localized redaction
989 (77.9%)
10,985 (81.5%)
8,042 (73.0%)
Appendix
Table 9: Strict CAP preservation by source-assigned clinical importance. Numerators are preserved propositions; denominators are fixed across conditions.
Figure 8: Complete condition-level clinical and lexical profile. Clinical endpoints use the 390-conversation population; editor endpoints use the matched 601-intention population.
Figure 9: Acoustic representation, event, and timing preservation. CLAP and HeAR are embedding similarities, events are detector-defined, and timing is measured with the matched ASR protocol.
Condition
Valid notes
Exhausted
Judgments (V / Q / B)
CAPs
Original
362
28
Not applicable
Not applicable
SeedVC only
359
31
331 / 2 / 57
21,515
AegisVox (VoiceCraft)
355
35
329 / 1 / 60
21,485
AegisVox (F5-TTS)
364
26
339 / 1 / 50
22,063
Appendix
Table 10: Transcript-only note control. Each note condition schedules 390 conversations. Target judgments compare a valid transformed note with a valid source note. V, Q, and B denote valid, quarantined, and blocked judgments; invalid and missing outcomes remain unobserved.
Condition
High-cosine EER
Low-cosine EER
Low-cosine 95% CI
SeedVC only
0.6310
0.3690
[0.3483, 0.3878]
AegisVox (VoiceCraft)
0.6354
0.3646
[0.3458, 0.3861]
AegisVox (F5-TTS)
0.6332
0.3668
[0.3469, 0.3856]
Appendix
Table 11: Final-waveform ECAPA scores on all-turn samples: 809 positive and 916 within-conversation negative trials per condition. Lower-direction EER is selected using evaluation labels. The interval describes resampled conversations, not uncertainty about unseen attacker strategies.
Backend
Source profiles
Positive trials
Impostor trials
EER (95% CI)
Attacker orientation
SpeechBrain ECAPA
809
809
653,672
0.304 (0.289–0.324)
Lower score means same source
SpeechBrain x-vector
809
809
653,672
0.345 (0.331–0.361)
Higher score means same source
Appendix
Table 12: Historical shared-stage speaker-verification protocol and results. Higher EER means harder source linkage only under the declared enrollment, trials, and attack backend.
Figure 10: Historical shared-SeedVC-stage verification under two embedding backends, not final-renderer scores. Bars are 95% source-bootstrap intervals.
Figure 11: Language-stratified clinical outcomes with exact denominators and 95% intervals. Dataset and interaction setting are confounded with language.
Figure 12: Propagation of ASR and diarization choices into matched content, clinical, action, and lexical measurements. Canary lexical cells retain upstream turn/timing-linkage failures in their intention-to-transform denominators. W denotes Whisper; PyA denotes pyannote.
Figure 13: Paired chronology effects under acoustic and semantic stresses. Intervals are 95% confidence intervals; multiplicity is controlled within the prespecified robustness family.
Figure 14: Distinct privacy endpoints versus strict CAP preservation. Points are descriptive condition estimates under different endpoint populations, not a joint confidence region or a dominance test. Left and center: generated- turn lexical judgments, with uncertainty retained in their denominators. Right: final-waveform all-turn ECAPA EER under evaluation-label-selected score direction. No final-waveform x-vector estimate is implied.
Figure 15: Outcome-linked failure taxonomy. Rates with different statistical units are shown separately and must not be summed into a composite failure score.
Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER ≈ 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.
Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak +1
Centre for Artificial Intelligence ZHAW School of Engineering
End-to-end full-duplex speech models feed user audio through an always-on LLM backbone, yet the speaker privacy implications of their hidden representations remain unexamined. Following the VoicePrivacy 2024 protocol with a lazy-informed attacker, we show that the hidden states of SALM-Duplex and Moshi leak substantial speaker identity across all transformer layers. Layer-wise and turn-wise analyses reveal that leakage persists across all layers, with SALM-Duplex showing stronger leakage in early layers while Moshi leaks uniformly, and that Linkability rises sharply within the first few turns. We propose two streaming anonymization setups using Stream-Voice-Anon: a waveform-level front-end (Anon-W2W) and a feature-domain replacement (Anon-W2F). Anon-W2F raises EER by over 3.5x relative to the discrete encoder baseline (11.2% to 41.0%), approaching the 50% random-chance ceiling, while Anon-W2W retains 78-93% of baseline sBERT across setups with sub-second response latency (FRL under 0.8 s).
Nikita Kuzmin, Tao Zhong, Jiajun Deng +6
Nanyang Technological University, Singapore · A*STAR, Singapore · Huawei Leibniz Research Center +2
Speaker anonymization (SA) systems modify timbre while leaving regional or non-native accent cues intact, which is problematic because such cues can reveal a speaker's first-language or geographic background and narrow the anonymity set. To address this issue, we present PHONOS, a streaming module for real-time SA that performs accent neutralization in a privacy sense: reducing accent-origin cues by converting non-native segmental realizations toward a chosen target accent domain. Our approach pre-generates golden speaker utterances that preserve source timbre and rhythm but replace foreign segmentals with native ones using silence-aware DTW alignment and zero-shot voice conversion. These utterances supervise a causal accent translator that maps non-native content tokens to native equivalents with at most 40ms look-ahead, trained using joint cross-entropy and CTC losses. Our evaluations show an 81% reduction in non-native accent confidence, with listening-test accentedness ratings consistent with this shift. PHONOS also moves outputs away from the original speaker in embedding space, suggesting lower linkability under an embedding-based proxy, while running with ≤241ms end-to-end latency on a single GPU.
Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah +1
Department of Computer Science & Engineering, Texas A&M University, College Station, US