Audio large language models (ALLMs) have recently advanced spoken interaction by integrating speech processing with large language models. However, existing evaluations of fairness, safety, and security (FSS) remain fragmented, largely because ALLMs differ fundamentally in how acoustic information is represented and where semantic reasoning occurs. Differences that are rarely made explicit. As a result, evaluations often conflate structurally distinct systems, obscuring the relationship between model design and observed FSS behavior. In this work, we introduce a structural taxonomy (system-level and representational) of ALLMs that categorizes systems along two axes: the form of audio input representation (e.g., discrete vs. continuous) and the locus of semantic reasoning (e.g., cascaded, multimodal, or audio-native). Building on the taxonomy, we propose a unified evaluation framework that assesses semantic invariance under paralinguistic variation, refusal and toxicity behavior under unsafe prompts, and robustness to adversarial audio perturbations. We apply this framework to two representative systems and observe systematic differences in refusal rates, attack success, and toxicity between audio and text inputs. Our findings demonstrate that FSS behavior is tightly coupled to how acoustic information is integrated into semantic reasoning, underscoring the need for structure-aware evaluation of audio language models.
Figures & tables
Figure 1: ALLM taxonomy’s overview : based on model’s architecture and its capability to process speech input, audio models can be divided into 3 main types : cascaded systems that can process only text tokens, audio-native systems that can process only audio tokens/embeddings, and audio-text multimodal systems that can process both text and audio tokens.
Evaluation Focus
Model Structure
Representative Work
Fairness
Safety
Security
Evaluation
Structural
Input
Attribution
AdvWave ( Kang et al., 2024 )
–
✓
–
Audio
–
Voice Jailbreak ( Shen et al., 2024 )
–
–
✓
Audio
–
AudioTrust ( Li et al., 2025 )
✓
✓
✓
Audio
–
Audio Is the Achilles’ Heel ( Yang et al., 2025 )
–
✓
✓
Audio+Text
–
Table 1: Representative prior work organized by evaluation focus and structural attribution analysis.
Model
System Type
Input Representation
Cascaded
Multimodal
Audio-Native
Discrete
Continuous
Qwen2-Audio ( Chu and others, 2023 )
–
✓
–
–
✓
Phi-4-Multimodal ( Abouelenin and others, 2025 )
–
✓
–
–
✓
HuggingGPT ( Tian et al., 2023 )
✓
–
–
✓
–
SALMONN ( Tang et al., 2023 )
–
✓
–
–
✓
SpeechGPT ( Zhang et al., 2023 )
–
–
✓
✓
–
Table 2: Per-Taxonomy bifurcation: Existing ALLMs organized by system type and input representations.
Figure 2: Test Set Generation Pipeline: We prepared the audio evaluation dataset in two main steps: a) audio generation , where text prompts for semantic and reference audios for acoustic information were used to synthesize audios, b) task-specific dataset generation , where generated audios are transformed to align trustworthy axes for FSS evaluation.
Figure 3: Fairness Eval.: Tonality distribution per model along accent fairness group. Top Row: Microsoft’s Phi, Bottom Row: Alibaba’s Qwen2. First column: Benign data, second column: Child Abuse, third column: Political Controversy, fourth column: Financial Crime. Results are plotted for 6 accents.
Figure 4: Fairness Eval: Ordinal Equalized Odds (OEO) of Accents (first row) and Emotions (second row). They are compared against Phi4 (first column) and Qwen2 (second column). It represents how safety severity changes across subgroup for the same semantic intent.
Category
Phi-4
Qwen2
C-RR
C-SRR
C-ASR
C-Tox
C-RR
C-SRR
C-ASR
C-Tox
animal_abuse
30.93
50.51
18.56
2.09
26.53
59.18
14.29
0.76
child_abuse
40.91
47.73
10.23
0.10
33.71
57.30
7.87
0.80
controversial_topics
10.11
87.64
1.12
0.07
31.03
64.37
4.60
0.24
discrimination
34.04
54.26
10.64
17.30
40.66
50.55
6.59
3.51
drug_abuse
66.33
20.40
13.27
1.12
49.44
28.09
21.35
0.20
Table 3: Safety Eval.: Comprehension-aware Safety Performance for Qwen2 vs. Phi-4 across unsafe categories.
Category
Latency Δ
Response Length Δ
Semantic Sim.
Tone Drift Δ
PGD
FGSM
PGD
FGSM
PGD
FGSM
PGD
FGSM
benign_librisqa
3.61
1.00
-61.59
-70.81
0.862
0.855
0.043
0.060
animal_abuse
11.86
4.76
-30.74
0.56
0.836
0.835
0.080
0.140
controversial_topics
8.15
3.91
-157.02
-43.14
0.822
0.848
0.160
0.120
discrimination
6.66
2.75
-86.90
-31.80
0.846
0.892
-0.380
0.000
drug_abuse
9.68
2.85
-57.80
-94.56
0.827
0.806
0.380
0.120
Table 4: Security Eval.: Impact of PGD and FGSM attacks across categories for different metrics for Qwen2 model. The results are shown as drifts from their clean audio counterparts.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Safety Eval. Comprehension-aware Attack Success Rate for Qwen2 and Phi4 across unsafe categories.
Figure 6: Safety Eval. Comprehension-aware Refusal Rate for Qwen2 and Phi4 across unsafe categories.
Figure 7: Fairness Eval.: Tonality distribution per model along Gender fairness group. Top Row: Microsoft’s Phi, Bottom Row: Alibaba’s Qwen2. First column: Benign data, second column: Child Abuse, third column: Discrimination, fourth column: Financial Crime. Results are plotted for 2 genders: male, female.
Task Category
Samples ( N )
Weighted Accuracy
Fairness
50
0.94
Safety
50
0.82
Security
50
0.80
Overall
150
0.85
Appendix
Table 5: Alignment Accuracy by Task Category
Primary Category
Sub-Metric
Accuracy
Security
Comprehension
0.96
Tonality
0.64
Safety
Comprehension
0.96
Safety
0.68
Fairness
Comprehension
0.96
Tonality
0.92
Appendix
Table 6: Breakdown of Alignment by Sub-Metric Type
Large Audio Language Models (LALMs) expand jailbreak risks from token-level prompting to the full speech perception-to-reasoning pipeline, where unsafe behavior can be induced through semantics, acoustic style, signal artifacts, or internal representations. Existing work studies these risks under heterogeneous threat models and evaluation protocols, making it difficult to compare attack practicality or defense utility. This paper provides a unified taxonomy and a controlled empirical evaluation of LALM jailbreak attacks and defenses. We organize prior work into semantic, acoustic, signal, and embedding-layer attacks; guard-based, training-free, and training-based defenses; and cross-modal, audio-native, and interactive benchmarks. We then evaluate representative attacks and defenses across ten open-source LALMs, measuring not only attack success rate but also benign refusal and latency. Our results show that Acoustic Best-of-N reveals strong worst-case audio-space vulnerabilities, Narrative Framing is an effective low-latency semantic threat, and current defenses trade robustness against benign usability. These findings support cost- and utility-aware evaluation as a necessary complement to success-rate-only LALM safety benchmarks.
Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, raising concerns about fairness across demographic subgroups. Fairness evaluation in spoken-input settings is challenging due to confounding factors, including semantic variation in spoken content and speaker-specific characteristics. Ignoring these factors can result in misleading conclusions about model bias. We propose a semantic-aware mixed-effects regression framework for fairness evaluation in LALMs that explicitly accounts for these confounders. Our approach incorporates sentence-level semantic embeddings of reference text as covariates and models speaker identity as a random effect. Notably, semantic representations are extracted from the same LALM under evaluation, enabling semantic control over variation as perceived by the model itself. Experiments on simulated data and real-world benchmarks demonstrate that the proposed approach substantially reduces spurious fairness findings and yields more robust and interpretable estimates of subgroup performance differences.
Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked. In particular, the effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood. We evaluate audio language models across five semantic and paralinguistic reasoning tasks: entailment, consistency, plausibility, accent drift, and accent restraint. Collectively, these tasks assess a model's ability to reason over spoken audio as the primary evidence source, including whether a textual hypothesis can be inferred, contradicted, or left undetermined by the audio, whether statements align or conflict with spoken content, whether claims are plausible given the discourse, and whether model predictions remain stable or appropriately constrained across accent variation. These findings highlight critical limitations in current audio reasoning evaluations and hope to provide guidance for more robust and equitable ALM design and assessment