Recent advances in end-to-end trained omni-models have substantially improved audio capabilities by strengthening text-audio modality alignment. However, whether such alignment inadvertently facilitates the transfer of safety vulnerabilities across modalities remains underexplored. This question is critical as text-based jailbreak attacks are considerably more mature than audio-based ones; if they transfer systematically, current audio safety evaluations may underestimate risks originating from the text modality. In this paper, we introduce the Alignment Curse, a formally characterized and empirically validated principle showing that stronger modality alignment enables more effective transfer of attacks from text to audio, revealing a fundamental tension between capability and safety. Motivated by this principle, we conduct a comprehensive black-box evaluation of three attack categories on recent omni-models (e.g., Qwen2.5-Omni, Qwen3-Omni): text attacks, text-transferred audio attacks, and audio attacks. We find that text-transferred audio attacks perform comparably to, and often better than, audio-based attacks, exhibiting a clear advantage under audio-only access. This suggests that text-based vulnerabilities play a pivotal role in shaping audio safety risks. Finally, we empirically analyze the relationship between modality alignment and transfer effectiveness across attack methods and models, observing consistent support for the Alignment Curse: tighter modality alignment leads to more effective cross-modality attack transfer.
Figures & tables
Figure 1: Tight cross-modality alignment inadvertently propagates textual vulnerabilities to the audio modality, which we term the Alignment Curse .
Attack
Qwen2.5-Omni-3B
Qwen2.5-Omni-7B
Qwen3-Omni
InteractiveOmni
GPT-4o-audio
Avg
KW
SR
KW
SR
KW
SR
KW
SR
KW
SR
KW
SR
Text Attacks
Naive (T)
0.08
0.02
0.02
0.02
0.05
0.03
0.26
0.07
0.10
0.06
0.10
0.04
ReNeLLM (T)
0.99
0.57
0.98
0.62
0.99
0.88
1.00
0.68
0.98
0.80
0.99
0.71
AutoDAN-T (T)
1.00
0.88
0.99
0.88
1.00
0.90
0.99
0.94
0.89
0.75
0.97
0.87
PAP (T)
0.98
0.79
0.97
0.84
0.94
0.88
1.00
0.83
0.99
0.75
0.98
0.82
Table 1: Attack success rates on JailbreakBench [ 47 ] . AutoDAN-T refers to AutoDAN-Turbo, VJ refers to VoiceJailbreak, MAJ refers to Multi-AudioJail. With respect to the SR metric, blue denotes the most successful text attack on each model, and yellow denotes the most successful audio attack.
Table 3: Cross-model transferability evaluation on JailbreakBench using Qwen3-Omni as surrogate model. * denotes direct attacks on the target model without transfer.
Model
PAP
ReNeLLM
AutoDAN-T
Qwen2.5-Omni-7B
0.02
3.33
0.22
InteractiveOmni
2.93
3.28
3.13
Table 4: Estimated KL divergence between text- and audio representations on Qwen2.5-Omni-7B and InteractiveOmni.
Figure 2: Layer-wise KL divergence between text- and audio-induced representations under different inputs. From left to right: different jailbreak methods; different voice tones (PAP prompts rendered with gpt-4o-mini-tts ); different speaking speeds (PAP prompts rendered with gpt-4o-mini-tts ); and different text-to-speech (TTS) models (using PAP prompts). The dashed horizontal line denotes the KL=2.0 threshold (i.e., the curse line ). ‘(x)’ denotes the transfer score (Audio SR / Text SR), where higher values indicate stronger transfer.
Figure 3: Estimated KL shows a negative correlation with transfer score across attack methods (left, middle) and models (right), validating that lower KL is associated with more effective transfer.
Figure 4: Controlled adapter intervention on Qwen2.5-Omni using PAP (A). Markers follow increasing perturbation levels p=0,0.1,0.2,0.3,0.4 . Higher KL corresponds to lower transfer score.
Probe
Modality
Qwen2.5-Omni-7B
Qwen2.5-Omni-3B
Qwen3-Omni
InteractiveOmni
Linear Probe
Text
0.79
0.75
0.78
0.80
Audio
0.77
0.65
0.70
0.62
MLP Probe
Text
0.85
0.84
0.86
0.85
Audio
0.81
0.71
0.73
0.69
Table 5: F1 scores across modalities. Probes are trained on text modality and tested on text and audio.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: t-SNE visualization of last token’s hidden states of the last layer.
ASCII (Full)
Base64 (Full)
ReNeLLM (Semi)
PAP (Non)
KW
SR
KW
SR
KW
SR
KW
SR
Text Attack
0.75
0.58
0.44
0.37
0.99
0.88
0.94
0.88
Audio Attack
0.84
0.09
0.99
0.02
0.97
0.74
0.85
0.83
Appendix
Table 6: Performance comparison across obfuscated attacks.
PCA
10
15
20
25
None
Estimation
0.01
0.006
0.01
0.01
0.85
Appendix
Table 7: PCA=15 yields the closest estimate to the true KL divergence (KL = 0) for two identical distributions.
PCA
13
14
15
16
17
KL
0.03
0.03
0.03
0.02
0.07
Appendix
Table 8: Sensitivity of KL estimation to nearby PCA dimensions.
Classifier
Logistic Regression
Support Vector Machine
Random Forest
KL
0.03
0.02
0.02
Appendix
Table 9: Sensitivity of KL estimation to different classifier families.
Figure 6: Relationship between representation-level KL divergence and transfer effectiveness. Clean samples (without perturbation) already exhibit a negative trend, while noise-perturbed samples provide additional coverage of intermediate KL regions.
Audio Variation
Attack Success Rate (SR)
Voices
Alloy
0.35
Echo
0.32
Nova
0.39
Sage
0.27
Verse
0.30
Appendix
Table 10: Attack success rates of PAP (A) under audio variations on InteractiveOmni-8B (SD=0.05).
Audio Variation
Attack Success Rate (SR)
Voices
Alloy
0.83
Echo
0.85
Nova
0.86
Sage
0.86
Verse
0.88
Appendix
Table 11: Attack success rates of PAP (A) under audio variations on Qwen2.5-Omni-3B (SD=0.02).
Audio Variation
Attack Success Rate (SR)
Voices
Alloy
0.84
Echo
0.88
Nova
0.88
Sage
0.87
Verse
0.86
Appendix
Table 12: Attack success rates of PAP (A) under audio variations on Qwen2.5-Omni-7B (SD=0.02).
Audio Variation
Attack Success Rate (SR)
Voices
Alloy
0.83
Echo
0.87
Nova
0.83
Sage
0.85
Verse
0.79
Appendix
Table 13: Attack success rates of PAP (A) under audio variations on Qwen3-Omni-30B (SD=0.03).
“I’m sorry"
“Sorry"
“I am sorry"
“I apologize"
“As an AI"
“As a language model"
“As an Assistant"
Appendix
Table 14: The refusal dictionary of the keyword-based ASR metric.
Evaluation
Condition
Qwen2.5-Omni-3B
Qwen2.5-Omni-7B
Human voice
Before
0.88
0.81
After
0.89
0.79
Physical playback
Before
0.88
0.81
After
0.84
0.77
Appendix
Table 15: Attack success rates (SR) on 20-prompt subsets of PAP prompts from JailbreakBench under human speech and physical playback. “Before” denotes the synthesized-audio baseline for the corresponding evaluation.