cs.CVOct 5, 2026
SaveBabelFake: A Multilingual Audio-Visual DeepFake Benchmark
Organizations: TU Darmstadt & Hessian.AI, Germany
Abstract
Reliable and practical audio-visual DeepFake detection requires benchmarks that reflect diverse linguistic contexts and modern data synthesis pipelines for visual as well as audio manipulations. However, existing datasets predominantly contain footage of English-speakers, often include outdated manipulation types, or overlook the audio modality. Further, many datasets feature individuals who did not consent to be used in DeepFake creation. We introduce BabelFake, a multilingual audio-visual DeepFake benchmark recorded with consenting participants. BabelFake contains 399k clips (1,323 hours) from 496 individuals spanning five languages (English, German, Italian, French, Spanish). Our modular data generation pipeline pairs 11 modern video manipulation methods with 4 voice cloning engines, distinguishing visual-only (face swapping) and joint audio-visual manipulations (lip synchronization and portrait animation). By benchmarking state-of-the-art detectors, we show that detection difficulty depends on the audio-visual generation pairing, with substantial performance degradation when authentic audio is preserved. Cross-language/demographic evaluation reveals sensitivity varying across detector architectures and training data, while human evaluation reveals that perceived realism and machine-detection difficulty do not necessarily align.
Figures & tables
| Dataset | Year | # Lang. | English % | Consent/ | # Manipulations | Video Manipulations | Audio | Samples | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ethics Approval | Faceswap | Lipsync | Portrait Animation | Real Samples | Fake Samples | ||||||
| PolyGlotFake [ 24 ] | 2024 | 7 | 16.7% | N/N | 7 | – | ✓ | – | ✓ | 766 | 14,472 |
| ILLUSION - Set D [ 52 ] | 2025 | 26 | UnK | N/✓ | UnK | UnK | UnK | UnK | UnK | 100 | 125 |
| MAVOS-DD [ 16 ] | 2025 | 8 | 22.6% | N/N | 7 | ✓ | – | ✓ | – | 25,195 | 35,169 |
| BabelFake (Ours) | 2026 | 5 | 20.2% | ✓/✓ | 15 | ✓ | ✓ | ✓ | ✓ | 20,117 | 379,582 |
Table 1 : Comparison between BabelFake and other recent multilingual DeepFake datasets. For ILLUSION, only set D (video subset) is included in the comparison, as it is the only one considering the multilingual scenario, UnK means unknown.
Figure 1 : Geographic diversity of BabelFake. Each marker corresponds to one or more participants, with marker size proportional to the number of subjects collected at that location. Colors indicate the language in which the participant was recorded. The figure illustrates the worldwide geographic diversity of the benchmark and shows that each language subset contains participants from multiple countries rather than a single geographic region.
| Language | Participants | Real Clips | Fake Clips | Total Clips | Real Hours | Fake Hours | Total Hours |
|---|---|---|---|---|---|---|---|
| English | 100 | 4,097 | 70,465 | 74,562 | 18.59 | 250.38 | 268.97 |
| German | 98 | 4,016 | 81,211 | 85,227 | 16.57 | 264.75 | 281.32 |
| Italian | 100 | 4,047 | 84,587 | 88,634 | 15.69 | 252.28 | 267.96 |
| French | 98 | 3,899 | 70,430 | 74,329 | 16.30 | 234.92 | 251.22 |
| Spanish | 100 | 4,058 | 72,889 | 76,947 | 16.76 | 237.63 | 254.38 |
| Total | 496 | 20,117 | 379,582 | 399,699 | 83.90 | 1,239.95 | 1,323.85 |
Table 2 : BabelFake composition by language.
Figure 2 : Composition of the generated BabelFake benchmark. (a) Distribution of fake samples across the five supported languages. Gray bars correspond to videos with authentic audio, while hatched bars denote videos with synthesized speech. (b) Distribution of generated samples across the three video manipulation categories. Colors indicate the speech synthesis model paired with each manipulation method (PL [ 32 ] , AF [ 28 ] , HP [ 56 ] , UG [ 57 ] , KS [ 13 ] , MT [ 58 ] , LS [ 30 ] , CS [ 37 ] , DLC [ 19 ] , HF [ 55 ] , IS [ 54 ] ). Face-swapping methods preserve the authentic audio, whereas portrait animation and lip synchronization are available with both authentic and synthesized speech.
Figure 3 : Distribution of clip durations in BabelFake. The shaded area represents the real recordings; the colored curves correspond to manipulated videos (by language). The similarity between the real and fake distributions shows that the DeepFakes largely preserve the temporal characteristics of the source recordings.
| Model | Training Data | AUC | PersonaLive | AvatarForcing | Hunyuan Portrait | UniAV Gen | KeySync | MuseTalk | LatentSync | Canon Swap | DeepLive Cam | HiFi Face | In Swapper | |||||
| Fake Audio | Real Audio | Fake Audio | Real Audio | Real Audio | Real Audio | Fake Audio | Real Audio | Fake Audio | Real Audio | Fake Audio | Real Audio | Real Audio | Real Audio | Real Audio | Real Audio | |||
| (B)RAVEn-A | DeepSpeak | 60.09 | 68.49 | 51.70 | 67.22 | 49.79 | 53.58 | 38.05 | 68.75 | 50.51 | 64.52 | 50.62 | 69.45 | 50.70 | 49.79 | 49.46 | 49.79 | 49.75 |
| (B)RAVEn-V | DeepSpeak | 70.31 | 67.16 | 67.96 | 76.83 | 63.73 | 66.51 | 54.10 | 64.97 | 54.57 | 87.79 | 81.44 | 75.33 | 59.92 | 72.07 | 67.92 | 60.45 | 67.11 |
| (B)RAVEn-AV | DeepSpeak | 7 2.05 | 81.65 | 72.11 | 75.41 | 61.51 | 66.84 | 52.13 | 66.27 | 53.42 | 83.18 | 73.87 | 76.26 | 57.57 | 69.49 | 65.96 | 58.92 | 65.81 |
| (B)RAVEn-A | MAVOS-DD | 43.93 | 38.14 | 48.32 | 39.69 | 51.89 | 48.97 | 65.72 | 41.54 | 46.10 | 44.21 | 47.57 | 42.74 | 47.96 | 51.38 | 50.65 | 48.78 | 48.72 |
| (B)RAVEn-V | MAVOS-DD | 72.56 | 71.93 | 72.06 | 75.11 | 73.03 | 66.88 | 69.90 | 59.89 | 56.44 | 77.90 | 77.41 | 74.15 | 60.75 | 79.92 | 76.60 | 69.82 | 74.67 |
Table 3 : Performance of different DeepFake detectors on the BabelFake dataset. Models ((B)RAVEn [ 22 ] , AVH-Align [ 48 ] , SpeechForensics [ 33 ] ) are evaluated using AUC on the test set, both overall and for each manipulation method (in order, PL [ 32 ] , AF [ 28 ] , HP [ 56 ] , UG [ 57 ] , KS [ 13 ] , MT [ 58 ] , LS [ 30 ] , CS [ 37 ] , DLC [ 19 ] , HF [ 55 ] , IS [ 54 ] ). Bold values indicate the best AUC for each manipulation. Avg. Audio and Avg. Video report the average AUC across all audio-only and video-only models, respectively, while Avg. AV reports the average AUC across the best-performing audio-visual variants of each detector (underscored).
| Model | Training data | EN | DE | IT | FR | ES |
|---|---|---|---|---|---|---|
| (B)RAVEn-A | DeepSpeak | 56.79 | 62.46 | 60.66 | 59.83 | 64.41 |
| (B)RAVEn-V | DeepSpeak | 74.84 | 71.38 | 69.99 | 68.16 | 70.22 |
| (B)RAVEn-AV | DeepSpeak | 78.73 | 74.73 | 71.37 | 70.35 | 73.49 |
| (B)RAVEn-A | MAVOS-DD | 43.30 | 45.22 | 44.58 | 45.46 | 41.69 |
| (B)RAVEn-V | MAVOS-DD | 71.50 | 70.63 | 75.83 | 74.64 | 71.09 |
| (B)RAVEn-AV | MAVOS-DD | 73.09 | 72.38 | 77.41 | 75.57 | 72.05 |
Table 4 : Per-language AUC on the BabelFake benchmark. Results are reported separately for English (EN), German (DE), Italian (IT), French (FR), and Spanish (ES). Bold values indicate the best performance for each language.
Figure 4 : Demographic bias analysis across sex (Female vs. Male) and ethnicity (White vs. Non-white) subgroups on the BabelFake test set. Points report the AUC per subgroup, with connecting bars indicating the performance gap across detector backbones and training datasets.
| Engine | # Rated Samples | Mean Score | Deception Rate (%) | AUC (%) |
|---|---|---|---|---|
| Real (Baseline) | 210 | 1.59 | 87.60 | - |
| InSwapper | 20 | 2.45 | 50.0 | 72.42 |
| CanonSwap | 18 | 2.50 | 50.0 | 76.59 |
| DeepLiveCam | 19 | 2.58 | 42.10 | 72.44 |
| AvatarForcing | 141 | 2.62 | 47.52 | - |
| Real Audio | 23 | 2.57 | 43.48 | 74.02 |
Table 5: Human evaluation results on a subset of 1,036 videos from the BabelFake benchmark, annotated by 51 participants. Ratings are collected using a four-point scale (1: Real, 2: Probably Real, 3: Probably Fake, 4: Fake). The Deception Rate denotes the percentage of manipulated videos rated as Real or Probably Real (scores ). For real videos, the same quantity indicates the percentage correctly perceived as genuine. The AUC column reports the average AUC of the multimodal detectors (Avg. AV in Tab. 3 ).
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5 : Overview of the custom web application used to collect BabelFake. Participants first review the study information and provide informed consent (a), after which, they can begin the survey (b). Before recording, the platform verifies the recording environment by checking the camera, microphone, framing, and lighting (c). Finally, participants complete the recording tasks through a browser-based interface that presents the prompts and captures synchronized audio and video (d).
| # | Sentence | # | Sentence |
|---|---|---|---|
| 1 | The birch canoe slid on the smooth planks. | 2 | Glue the sheet to the dark blue background. |
| 3 | It’s easy to tell the depth of a well. | 4 | These days, a chicken leg is a rare dish. |
| 5 | Rice is often served in round bowls. | 6 | The juice of lemons makes fine punch. |
| 7 | The box was thrown beside the parked truck. | 8 | The hogs were fed chopped corn and garbage. |
| 9 | Four hours of steady work faced us. | 10 | A large size in stockings is hard to sell. |
Table 6: Standardized elicitation sentences used for controlled reading.
| # | Question | # | Question |
|---|---|---|---|
| 1 | What is your favorite color? | 2 | What is your favorite song or type of music? |
| 3 | What did you do first thing this morning? | 4 | If you could live anywhere in the world, where would it be? |
| 5 | What is something you are really good at? |
Table 7: Quick-fire questions used for short, spontaneous responses.
| Type | Question |
|---|---|
| Procedural | Please describe the steps of your morning routine. |
| Personal narrative | Tell me about a time you helped someone, or someone helped you |
| Emotional reflection | What is something that always makes you feel happy? |
| Opinion | What do you think makes a good friend? |
Table 8: Open-ended unscripted questions used for longer spontaneous responses.
| Excerpt |
|---|
| but it was such an unheard-of thing for him to be sitting in that room in the middle of the day, reading or making pretence to read, that she had never thought of his remaining. |
| Lady Cumnor and her daughters received all the school visitors at the Towers, the great family mansion standing in aristocratic seclusion in the center of the large park, |
| But every time that Missis Gibson was struck by Cynthia’s beauty, she thought it more and more advisable that Mister Osborne Hamley should be cheered up by a quiet little dinner-party. |
| But they had understood that he was not coming to the Hall until the following week, and therefore they had felt themselves at full liberty this afternoon to follow their own devices. |
| Then the carriage came round, and after numberless last words from the earl - who appeared to have put off every possible direction to the moment when he stood, |
| Molly made a point of turning the conversation from all personal subjects after this, and kept the Squire talking about the progress of his drainage during the rest of lunch. |
Table 9 : Public-domain passages from Wives and Daughters [ 20 ] by Elizabeth Cleghorn Gaskell used for extended controlled reading.
| ID | Instruction |
|---|---|
| Lean forward and emphasize point | [LEAN SLIGHTLY FORWARD TOWARDS THE CAMERA] while saying: “Let me emphasize this, this is really important:” [Wait approx. 1 second] then emphasize: “Pineapple on pizza is good!” [RETURN TO THE STARTING POSITION]. |
| Look down to the right | [LOOK DOWN TO THE RIGHT] as if you were checking something there, and say: “If I look here, I can see what’s happening on my right side.” |
| Look down to the middle and focus | [LOOK SLIGHTLY DOWN TO THE MIDDLE] Pretend you are checking a document. Name any date and briefly say that the document looks correct. |
| Look down to the left | [LOOK DOWN TO THE LEFT], as if you were checking something there, and say: “If I look here, I can see what’s happening on my left side.” |
| Look up to the left and recall | [LOOK UP TO THE LEFT], as if you were remembering something, and say: “Looking up here reminds me of the pattern on the ceiling at home.” |
| Look up to the middle and examine | [LIFT YOUR GAZE UPWARDS], as if you were examining something above you, and say: “I’m just looking up to see how everything fits together.” |
Table 10: Action prompts used during data collection.
| Type | Title | Description / Messages |
|---|---|---|
| Unscripted Answers | Please answer the question in detail (target: approx. 30 seconds). | |
| Voice Cloning Prompts | Reading a short list of sentences | Please read all displayed sentences aloud and clearly. Speak in your normal, natural tone of voice. |
| Voice Cloning Prompts | Reading a sample text | Read the entire text aloud. Make sure to speak calmly and clearly – just as you would in everyday life. |
| Q&A Task | Question and answer interview | You will see a list of questions. Read each question aloud, then answer it immediately in the same recording. |
| Interleave Reading Task | Mixed Reading and Action Tasks | You have successfully completed the introductory tasks. Now we will move on to the main block of data entry. In this section, different types of tasks will alternate: • Sometimes you will read sentences in a neutral tone. • Sometimes you will read sentences with a given emotion. • Some tasks combine speaking with small movements or facial expressions. |
| Standardized Reading Task | Reading standardized sentences | Please read the displayed sentence aloud. Try to speak naturally. |
Table 11: Task structure and interface messages used during data collection.
| Model | AUC |
|---|---|
| (B)RAVEn-A | 80.03 |
| (B)RAVEn-V | 93.47 |
| (B)RAVEn-AV | 95.52 |
| AVH-Align | 97.37 |
Table 12 : AUC results when training and evaluating on BabelFake (test set). Best performance is highlighted in bold.
| Model | Training Data | AUC | AvatarForcing | PersonaLive | Hunyuan Portrait | UniAV Gen | ||||||||
| ChatterBox | FishAudio2 | OmniVoice | Qwen3-TTS | Real Audio | ChatterBox | FishAudio2 | OmniVoice | Qwen3-TTS | Real Audio | Real Audio | Real Audio | |||
| (B)RAVEn-A | DeepSpeak | 60.09 | 65.25 | 68.44 | 67.27 | 67.91 | 49.79 | 65.92 | 68.99 | 70.13 | 68.90 | 51.70 | 53.58 | 38.05 |
| (B)RAVEn-V | DeepSpeak | 70.31 | 75.62 | 79.77 | 74.50 | 77.44 | 63.73 | 66.95 | 67.58 | 67.01 | 67.10 | 67.96 | 66.51 | 54.10 |
| (B)RAVEn-AV | DeepSpeak | 7 2.05 | 74.10 | 77.77 | 74.17 | 75.59 | 61.51 | 81.91 | 81.62 | 81.83 | 81.22 | 72.11 | 66.84 | 52.13 |
| (B)RAVEn-A | MAVOS-DD | 43.93 | 34.22 | 38.58 | 45.78 | 40.17 | 51.89 | 33.01 | 37.39 | 43.95 | 38.21 | 48.32 | 48.97 | 65.72 |
| (B)RAVEn-V | MAVOS-DD | 72.56 | 72.61 | 75.84 | 76.57 | 75.40 | 73.03 | 71.82 | 72.45 | 71.70 | 71.74 | 72.06 | 66.88 | 69.90 |
Table 13: Fine-grained benchmark evaluation across talking-head and audio-visual generation methods decomposed by driving audio condition (Part 1). Results are reported in AUC (%).
| Model | Training Data | AUC | KeySync | MuseTalk | LatentSync | Canon Swap | HiFiFace | In Swapper | DeepLive Cam | ||||||||||||
| ChatterBox | FishAudio2 | OmniVoice | Qwen3-TTS | Real Audio | ChatterBox | FishAudio2 | OmniVoice | Qwen3-TTS | Real Audio | ChatterBox | FishAudio2 | OmniVoice | Qwen3-TTS | Real Audio | Real Audio | Real Audio | Real Audio | Real Audio | |||
| (B)RAVEn-A | DeepSpeak | 60.09 | 64.61 | 66.43 | 72.09 | 71.88 | 50.51 | 57.15 | 60.5 | 77.33 | 63.09 | 50.62 | 62.82 | 76.45 | 71.08 | 67.46 | 50.7 | 49.79 | 49.79 | 49.75 | 49.46 |
| (B)RAVEn-V | DeepSpeak | 70.31 | 63.24 | 66.93 | 63.74 | 65.96 | 54.57 | 85.41 | 88.5 | 90.45 | 86.8 | 81.44 | 74.71 | 81.46 | 72.64 | 72.5 | 59.92 | 72.07 | 60.45 | 67.11 | 67.92 |
| (B)RAVEn-AV | DeepSpeak | 7 2.05 | 62.98 | 67.24 | 67.8 | 67.05 | 53.42 | 79.80 | 81.75 | 89.81 | 81.35 | 73.87 | 74.51 | 81.84 | 76.15 | 72.54 | 57.57 | 69.49 | 58.92 | 65.81 | 65.96 |
| (B)RAVEn-A | MAVOS-DD | 43.93 | 40.38 | 38.37 | 48.53 | 38.87 | 46.1 | 40.92 | 42.19 | 51.51 | 42.22 | 47.57 | 40.4 | 36.84 | 50.08 | 43.65 | 47.96 | 51.38 | 48.78 | 48.72 | 50.65 |
| (B)RAVEn-V | MAVOS-DD | 72.56 | 58.43 | 59.32 | 62.44 | 59.37 | 56.44 | 76.62 | 76.88 | 80.73 | 77.38 | 77.41 | 71.47 | 73.29 | 75.11 | 76.71 | 60.75 | 79.92 | 69.82 | 74.67 | 76.6 |
Table 14 : Fine-grained benchmark evaluation across lip-synchronization and face-swapping manipulation methods decomposed by driving audio condition (Part 2). Results are reported in AUC (%).
| Engine | # Rated Samples | Mean Score | Deception Rate (%) | AUC (%) |
|---|---|---|---|---|
| Real (Baseline) | 210 | 1.59 | 87.60 | - |
| Inswapper | 20 | 2.45 | 50.00 | 72.42 |
| CanonSwap | 18 | 2.50 | 50.00 | 76.59 |
| DeepLiveCam | 19 | 2.58 | 42.10 | 72.44 |
| AvatarForcing | 141 | 2.62 | 47.52 | - |
| Real Audio | 23 | 2.57 | 43.48 | 74.02 |
Table 15 : Human evaluation results across manipulation methods, with audio manipulations breakdown. Mean Score denotes the average human rating, while Deception Rate reports the percentage of manipulated samples perceived as real (for real samples, the percentage correctly perceived as real). AUC reports the average performance of the best-performing audio-visual DeepFake detectors evaluated on the corresponding manipulation.
Figure 6 : Qualitative examples of portrait animation manipulations in BabelFake. Each row shows consecutive frames from a manipulated video generated with a different method. From top to bottom: PersonaLive [ 32 ] , AvatarForcing [ 28 ] , HunyuanPortrait [ 56 ] , and UniAVGen [ 57 ] .
Figure 7 : Qualitative examples of lip-synchronization manipulations in BabelFake. Each row shows consecutive frames from a manipulated video generated with a different method. From top to bottom: KeySync [ 13 ] , MuseTalk [ 58 ] , and LatentSync [ 30 ] .
Figure 8 : Qualitative examples of face-swapping manipulations in BabelFake. Each row shows consecutive frames from a manipulated video generated with a different method. From top to bottom: CanonSwap [ 37 ] , DeepLiveCam [ 19 ] , HiFiFace [ 55 ] , and InSwapper [ 54 ] .