cs.AIOct 10, 2026
SaveAegisVox: Factorized Privacy and Clinical Preservation for Long-Form Speech
Organizations: Oracle Health AI
Abstract
Clinical speech reveals identity through both spoken identifiers and voice, yet its pauses, delivery, and speaker roles can also carry useful clinical evidence. Protecting that speech therefore requires more than preserving its words. We present AegisVox, a factorized framework that assigns consistent surrogate voices and edits identifier-bearing turns using document-consistent replacements. Across 390 public simulated or non-patient conversations totaling 63.6 hours, a matched comparison of two speech renderers separates replacement realization from retained identifiers, speaker linkage, and clinical preservation. Text-to-speech rendering realizes 535/601 planned replacements versus 453/601 for the codec-based speech editor. However, similar aggregate clinical preservation conceals more model-labeled contradictions, and conversation-wide speaker scores conceal greater edited-speech linkage under a score-direction sensitivity analysis. Fresh transcription of original audio provides a reference for interpreting the clinical measurement chain; an independent judge tests sensitivity to model-based assessment. Better replacement realization can therefore coexist with worse clinical or speaker-linkage measurements. AegisVox supports use-specific evaluation of those tradeoffs, not a guarantee of identifier removal or clinical safety.
Figures & tables
Figure 1: AegisVox separates voice identity from identifying words. SeedVC converts every turn using one consistent surrogate voice per source speaker. The lexical renderer changes only converted turns with planned identifier replacements; the remaining converted turns pass directly to reconstruction. F5-TTS is the illustrated renderer, and VoiceCraft is the matched alternative. PHI denotes protected health information.
| Dataset | Language | Conversations | Hours | Median [IQR] (s) | Setting |
|---|---|---|---|---|---|
| PriMock57-mixed | EN | 57 | 8.6 | 556 [456–617] | Doctor–patient consultations |
| Fareez | EN | 272 | 51.8 | 672 [572–778] | Respiratory interviews |
| SIMSAMU | FR | 61 | 3.2 | 187 [152–234] | Simulated emergency calls |
| Total | EN/FR | 390 | 63.6 | 609 [471–736] | Public clinical audio |
Table 1: Public evaluation population and comparison roles. The original-audio control changes no speech; all transformation conditions share the source recordings. EN and FR denote English and French.
Figure 2: Clinical controls and matched renderer effects on 25,778 source CAPs from 390 conversations. (a) Fresh original-audio ASR supplies the reference, not clinical ground truth. (b) F5-TTS minus VoiceCraft effects in percentage points (pp), with 95% conversation-cluster intervals. Similar strict preservation coexists with more model-labeled contradictions.
| Outcome | Denominator | VoiceCraft | F5-TTS | Effect (pp) | 95% CI (pp) |
|---|---|---|---|---|---|
| Replacement realized | 601 | 453 (75.37%) | 535 (89.02%) | ||
| Original PHI retained | 601 | 21 (3.49%) | 28 (4.66%) | ||
| CAP preserved | 25,778 | 19,954 (77.41%) | 19,929 (77.31%) | ||
| CAP contradicted | 25,778 | 391 (1.52%) | 479 (1.86%) |
Table 2: Matched outcomes within AegisVox. Counts and item-weighted rates are shown for each renderer. Effects are F5-TTS minus VoiceCraft, with 95% conversation-cluster intervals. The lexical rows cover 601 edit intentions from 208 conversations; clinical rows cover 25,778 CAPs from 390 conversations. Realization is conditional on a planned edit, not complete PHI removal.
Figure 3: Final-waveform linkage under evaluation-label-selected score direction. F5-TTS minus VoiceCraft effects use all-turn samples (809 profiles, 390 conversations) or planned-edit regions (291 profiles, 204 conversations). Bars are 95% clustered CIs. Lower EER means easier source linkage.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Governed integration and threat boundary. AegisVox transforms speaker identity and identifying language before controlled release, while access, provenance, residual-risk review, and monitoring remain outside the model and continue to constrain use.
Figure 5: Public evaluation population by dataset, language, detected speaker count, and planned edit count. Duration strata are specified in the text.
| Setting | Assigned | Median | Median | ECAPA EER | x-vector EER | Frontend WER | CAP | CLAP / HeAR | |
|---|---|---|---|---|---|---|---|---|---|
| Utility-oriented | 809/809 | 1.1600 | 0.8741 | 0.3115 | 0.3426 | 0.2314 | 0.7446 | 0.7296 / 0.9448 | |
| Balanced | 809/809 | 1.1590 | 0.8741 | 0.3103 | 0.3436 | 0.2311 | 0.7453 | 0.7297 / 0.9448 | |
| Privacy-oriented | 809/809 | 1.1585 | 0.8773 | 0.3140 | 0.3461 | 0.2310 | 0.7453 | 0.7303 / 0.9448 |
Table 3: Retrieval-threshold sensitivity over separately rendered conditions. CAP is the recording-level mean strict proposition-preservation rate. ASV uses the 809-positive and 653,672-impostor trial population per backend; ECAPA is orientation-normalized so EER is at most 0.5. Absolute values belong to this sensitivity family and are not substituted for the primary-condition results.
Figure 6: Controlled qualification suite for source-anchored clinical adjudication. Each corruption family tests whether the judge distinguishes preserved content from a clinically consequential change under protocol variants.
| Condition | Covered | Qwen preserved / 600 | Weighted Qwen | Agreement | |
|---|---|---|---|---|---|
| SeedVC only | 522 | 493 (82.2%) | 90.9% | 413/522 (79.1%) | 0.253 |
| AegisVox (VoiceCraft) | 512 | 466 (77.7%) | 88.8% | 405/512 (79.1%) | 0.314 |
| AegisVox (F5-TTS) | 511 | 462 (77.0%) | 85.7% | 405/511 (79.3%) | 0.340 |
Table 4: Independent Qwen sensitivity on 600 shared CAPs per condition. Covered cases have valid agreeing Qwen labels under both orders. Strict raw counts retain all 600 intentions. Weighted rates use inverse stratum probabilities. Agreement and are unweighted over covered cases and retain GPT uncertainty as a third category; they are not binary accuracy.
Figure 7: Independent-judge sensitivity. Left: inverse-stratum-weighted strict preservation under each judge, with conversation-cluster intervals. Center: unweighted uncertainty-retaining Cohen’s on covered cases. Right: realized replacements under each judge, with numerators over 601 intentions. Weighting, label agreement, and lexical ordering answer different questions.
| Scoring contract | Gold | TP | FP | FN | Precision [95% CI] | Recall [95% CI] | F1 [95% CI] |
|---|---|---|---|---|---|---|---|
| Contract intersection, exact typed | 2,147 | 2,043 | 137 | 104 | 0.937 [0.926, 0.947] | 0.952 [0.942, 0.960] | 0.944 [0.935, 0.953] |
| Contract intersection, boundary tolerant | 2,147 | 2,124 | 56 | 23 | 0.974 [0.968, 0.981] | 0.989 [0.985, 0.993] | 0.982 [0.977, 0.986] |
| Full ASQ ontology, exact typed | 2,973 | 2,150 | 259 | 823 | 0.892 [0.880, 0.905] | 0.723 [0.714, 0.732] | 0.799 [0.789, 0.808] |
| Full ASQ ontology, boundary tolerant | 2,973 | 2,348 | 61 | 625 | 0.975 [0.968, 0.981] | 0.790 [0.781, 0.799] | 0.873 [0.866, 0.879] |
Table 5: Independent ASQ-PHI detector evaluation. Confidence intervals use 2,000 query-level bootstrap resamples. The full ontology includes the prespecified geography-contract mismatch.
| Category | Gold | Exact precision | Exact recall | Exact F1 | Boundary F1 |
|---|---|---|---|---|---|
| Name | 814 | 0.937 | 0.932 | 0.935 | 0.995 |
| Date | 806 | 0.944 | 0.989 | 0.966 | 0.967 |
| Medical record number | 305 | 0.909 | 0.918 | 0.914 | 0.995 |
| Health-plan beneficiary number | 91 | 0.956 | 0.945 | 0.950 | 0.972 |
| Phone number | 45 | 1.000 | 1.000 | 1.000 | 1.000 |
| Social Security number | 33 | 0.943 | 1.000 | 0.971 | 0.971 |
Table 6: Primary category results on the contract-intersection gold set. Low-support residual categories are grouped; unique identifiers are shown separately because their performance differs.
| Renderer | Planned edits | Replacement realized | Original PHI retained | Uncertain |
|---|---|---|---|---|
| VoiceCraft | 601 | 453 (75.4%) | 21 (3.5%) | 110 (18.3%) |
| F5-TTS | 601 | 535 (89.0%) | 28 (4.7%) | 46 (7.7%) |
Table 7: Conditional lexical-renderer outcomes over the same 601 planned edit intentions, assessed from generated-turn transcripts. Uncertainty refers to realization; outcomes may overlap. Detection is reported in Table 5 .
| Condition | Preserved | Partial | Missing | Contradicted | Uncertain |
|---|---|---|---|---|---|
| Fresh original audio | 24,998 (97.0%) | 556 (2.2%) | 128 (0.5%) | 95 (0.4%) | 1 (0.0%) |
| Speaker path only | 20,089 (77.9%) | 2,304 (8.9%) | 2,630 (10.2%) | 364 (1.4%) | 391 (1.5%) |
| AegisVox (VoiceCraft) | 19,954 (77.4%) | 2,347 (9.1%) | 2,663 (10.3%) | 391 (1.5%) | 423 (1.6%) |
| AegisVox (F5-TTS) | 19,929 (77.3%) | 2,367 (9.2%) | 2,724 (10.6%) | 479 (1.9%) | 279 (1.1%) |
| Localized redaction | 20,016 (77.6%) | 2,225 (8.6%) | 2,891 (11.2%) | 387 (1.5%) | 259 (1.0%) |
Table 8: Source-anchored CAP outcome decomposition. Each row contains the same 25,778 propositions; percentages are row percentages.
| Condition | Critical ( ) | Major ( ) | Minor ( ) |
|---|---|---|---|
| Speaker path only | 976 (76.9%) | 11,044 (81.9%) | 8,069 (73.2%) |
| AegisVox (VoiceCraft) | 977 (77.0%) | 10,951 (81.2%) | 8,026 (72.8%) |
| AegisVox (F5-TTS) | 959 (75.6%) | 10,957 (81.2%) | 8,013 (72.7%) |
| Localized redaction | 989 (77.9%) | 10,985 (81.5%) | 8,042 (73.0%) |
Table 9: Strict CAP preservation by source-assigned clinical importance. Numerators are preserved propositions; denominators are fixed across conditions.
Figure 8: Complete condition-level clinical and lexical profile. Clinical endpoints use the 390-conversation population; editor endpoints use the matched 601-intention population.
Figure 9: Acoustic representation, event, and timing preservation. CLAP and HeAR are embedding similarities, events are detector-defined, and timing is measured with the matched ASR protocol.
| Condition | Valid notes | Exhausted | Judgments (V / Q / B) | CAPs |
|---|---|---|---|---|
| Original | 362 | 28 | Not applicable | Not applicable |
| SeedVC only | 359 | 31 | 331 / 2 / 57 | 21,515 |
| AegisVox (VoiceCraft) | 355 | 35 | 329 / 1 / 60 | 21,485 |
| AegisVox (F5-TTS) | 364 | 26 | 339 / 1 / 50 | 22,063 |
Table 10: Transcript-only note control. Each note condition schedules 390 conversations. Target judgments compare a valid transformed note with a valid source note. V, Q, and B denote valid, quarantined, and blocked judgments; invalid and missing outcomes remain unobserved.
| Condition | High-cosine EER | Low-cosine EER | Low-cosine 95% CI |
|---|---|---|---|
| SeedVC only | 0.6310 | 0.3690 | [0.3483, 0.3878] |
| AegisVox (VoiceCraft) | 0.6354 | 0.3646 | [0.3458, 0.3861] |
| AegisVox (F5-TTS) | 0.6332 | 0.3668 | [0.3469, 0.3856] |
Table 11: Final-waveform ECAPA scores on all-turn samples: 809 positive and 916 within-conversation negative trials per condition. Lower-direction EER is selected using evaluation labels. The interval describes resampled conversations, not uncertainty about unseen attacker strategies.
| Backend | Source profiles | Positive trials | Impostor trials | EER (95% CI) | Attacker orientation |
|---|---|---|---|---|---|
| SpeechBrain ECAPA | 809 | 809 | 653,672 | 0.304 (0.289–0.324) | Lower score means same source |
| SpeechBrain x-vector | 809 | 809 | 653,672 | 0.345 (0.331–0.361) | Higher score means same source |
Table 12: Historical shared-stage speaker-verification protocol and results. Higher EER means harder source linkage only under the declared enrollment, trials, and attack backend.
Figure 10: Historical shared-SeedVC-stage verification under two embedding backends, not final-renderer scores. Bars are 95% source-bootstrap intervals.
Figure 11: Language-stratified clinical outcomes with exact denominators and 95% intervals. Dataset and interaction setting are confounded with language.
Figure 12: Propagation of ASR and diarization choices into matched content, clinical, action, and lexical measurements. Canary lexical cells retain upstream turn/timing-linkage failures in their intention-to-transform denominators. W denotes Whisper; PyA denotes pyannote.
Figure 13: Paired chronology effects under acoustic and semantic stresses. Intervals are 95% confidence intervals; multiplicity is controlled within the prespecified robustness family.
Figure 14: Distinct privacy endpoints versus strict CAP preservation. Points are descriptive condition estimates under different endpoint populations, not a joint confidence region or a dominance test. Left and center: generated- turn lexical judgments, with uncertainty retained in their denominators. Right: final-waveform all-turn ECAPA EER under evaluation-label-selected score direction. No final-waveform x-vector estimate is implied.
Figure 15: Outcome-linked failure taxonomy. Rates with different statistical units are shown separately and must not be summed into a composite failure score.
More in this topic
Speaker Anonymization
Speaker anonymization changes identifying voice characteristics while preserving speech content and useful traits such as emotion and prosody.
See all 22 papers