Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models
Authors: Yuchen Su, Shaoxin Zhong, Yonghua Zhu, Ruofan Wang, Zijian Huang, Qiqi Wang, Na Zhao, Diana Benavides-Prado, Michael Witbrock
Organizations: School of Computer Science, University of Auckland, New Zealand · Singapore University of Technology and Design, Singapore · School of Statistics and Data Science, Nankai University, China · School of Electronic Engineering and Computer Science, Queen Mary University of London
Puns represent a typical linguistic phenomenon that exploits polysemy and phonetic ambiguity to generate humour, posing unique challenges for natural language understanding. Within pun research, audio plays a central role in human communication except text and images, while datasets and systematic resources for spoken puns remain scarce, leaving this crucial modality largely underexplored. In this paper, we present APUN-Bench, the first benchmark dedicated to evaluating large audio language models (LALMs) on audio pun understanding. Our benchmark contains 4,434 audio samples annotated across three stages: pun recognition, pun word location and pun meaning inference. We conduct a deep analysis of APUN-Bench by systematically evaluating 10 state-of-the-art LALMs, uncovering substantial performance gaps in recognizing, localizing, and interpreting audio puns. This analysis reveals key challenges, such as positional biases in audio pun location and error cases in meaning inference, offering actionable insights for advancing humour-aware audio intelligence.
Figures & tables
Figure 1: An example of the pun understanding in recognition, location and inference.
Figure 2: The Overview of APUN-Benchmark, covering data construction, human annotation and three evaluation stages: (S1) Pun Recognition, detecting the presence of an audio pun; (S2) Pun Location, identifying the specific pun word in the audio; and (S3) Pun Inference, inferring its dual meanings.
Dataset
Heterographic
Homophonic
Homographic
Negative
Overall
Synthetic Dataset
794
304
1298
1,152
3,541
- Segment Clip (s)
4.18
4.36
3.88
3.47
3.97
- Sentence Length
12.21
13.89
11.68
8.50
11.57
Real-World Dataset
223
72
287
311
582
- Segment Clip (s)
6.36
6.16
5.34
6.73
6.14
- Sentence Length
11.12
11.40
11.55
10.88
11.23
Table 1: APUN-Bench data statistics, including average audio clip duration and sentence length.
Model
Size
Pun Recognition
Pun Location
Pun Inference
Acc. ( ↑ )
Pre ( ↑ )
Rec ( ↑ )
F1 ( ↑ )
Pun ( % )
Coa ( ↑ )
Fin ( ↑ )
Heg ( ↑ )
Hog ( ↑ )
Hop ( ↑ )
Synthetic Dataset
Qwen2-Audio-Instruct
7B
45.78
71.70
39.45
50.90
38.06
50.81
31.77
13.47
17.91
26.97
SALMONN
13B
51.91
72.51
52.60
60.97
51.81
34.41
22.35
15.23
24.63
25.32
Audio-Reasoner
7B
55.27
72.62
59.81
65.59
54.47
50.39
32.31
23.05
27.25
37.50
Audio-Flamingo-3
7B
37.51
59.25
40.02
47.78
48.24
36.37
27.83
33.25
29.35
56.91
Table 2: Evaluation Results of the Synthetic and Real-World Datasets on APUN-Bench in various LALMs and cascaded systems. ✯ represents the proprietary LALMs and the symbol ’–’ indicates that the model size is not publicly disclosed. Coa and Fin repesent the accuracy of coarse-grained and fine-grained locations. Hog, hop and heg denote the accuracy of homographic puns, homophonic puns and heterographic puns.
Figure 3: Statistical comparison of predicted in different LALMs and ground-truth pun positions in APUN-Bench, showing the distribution of pun words across different sentence intervals (beginning, middle, end of the sentence).
Figure 4: The distribution of error types in pun inference for Omni-R1 and GPT4o-Audio across heterographic, homographic, and homophonic puns.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Error Type
Definition
Example (Audio Transcription)
Prediction
Ground Truth
Heterographic Puns & Homophonic Puns
Position Error (PE)
The model fails to provide the correct word corresponding to the given pun word and may instead return a word from another position in the sentence.
When asked to picture the perfect modern defensive weapon the Claymore springs to mine .
Claymore
mind
Pun Word Repeat (PWR)
The model outputs the given pun word itself rather than its intended meaning.
Did you hear about the nervous preacher ? He had sweaty psalms .
psalms
palm
Word Inference Error (WIE) (Similar)
The model infer an incorrect word, often manifested as another word with a similar pronunciation.
One ear of corn said to the other’You’re getting husky ’.
hoarse
husk
Word Inference Error (WIE) (No Similar)
The model infer an incorrect word, often manifested as another word with a completely different pronunciation.
George Westinghouse was a refrigerator magnate .
entrepreneur
magnet
Irrelevant Answer (IA)
The model’s answer is unrelated to the pun inference for the pun word.
She was only a Butcher’s daughter, but there wasn’t much more she could loin .
line human given an audio …
learn
Appendix
Table 3: Error types of pun inference in heterographic, homophonic and homographic puns
Laura
Gary
Jon
Lea
Karen
Rick
Brenda
David
Eileen
Jordan
Mike
Yann
Joy
James
Eric
Lauren
Rose
Will
Jason
Aaron
Naomie
Alisa
Patrick
Jerry
Tina
Jenna
Bill
Tom
Appendix
Table 4: List of the 34 speakers used in Parler-TTS.
Model
Heterographic
Homophonic
Homographic
Coa ( ↑ )
Fin ( ↑ )
Coa ( ↑ )
Fin ( ↑ )
Coa ( ↑ )
Fin ( ↑ )
SALMONN
32.37
17.12
35.86
8.55
35.32
28.81
Qwen2-Audio-Instruct
52.14
29.47
44.41
13.81
51.51
37.41
Audio-Reasoner
50.50
29.34
39.47
11.18
52.90
39.11
Qwen-2.5-omni
53.53
36.39
56.91
21.38
66.54
61.42
MiniCPM
56.30
34.13
62.50
20.06
61.58
56.85
Appendix
Table 5: Evaluation results of the Synthetic Dataset on APUN-Bench. We evaluate various LALMs on our proposed APUN-Bench. ✯ represents the proprietary LALMs.
Model
NOUN
ADV
VERB
ADJ
PROPN
Synthetic Dataset
954
312
246
202
146
Real-world Dataset
208
36
131
81
93
Appendix
Table 6: The part-of-speech distributions between Synthetic and Real-world Datasets.
ASR Model
Synthetic
Real-world
Heg ( ↓ )
Hog ( ↓ )
Hop ( ↓ )
Heg ( ↓ )
Hog ( ↓ )
Hop ( ↓ )
SenseVoiceSmall
0.144
0.126
0.151
0.177
0.134
0.269
Whisper-Large
0.111
0.097
0.123
0.154
0.093
0.221
Appendix
Table 7: Word Error Rate (WER) of SenseVoiceSmall and Whisper-Large on synthetic and real-world datasets.
Model
Architecture
ASR (s)
Reasoning (s)
Total (s)
MiniCPM
End-to-End
-
0.61
0.61
Qwen-2.5-omni
End-to-End
-
0.91
0.91
SenseVoiceSmall+MiniCPM
Cascaded
0.42
0.54
0.74
Whisper-Large+MiniCPM
Cascaded
1.31
0.54
1.85
SenseVoiceSmall+Qwen-2.5-Instruct
Cascaded
0.42
0.34
0.76
Whisper-Large+Qwen-2.5-Instruct
Cascaded
1.31
0.34
1.65
Appendix
Table 8: Time consumption analysis in real-world dataset between open-source LALMs and cascaded systems.
Model
Size
Pun Recognition
Pun Location
Pun Inference
Acc. ( ↑ )
Pre ( ↑ )
Rec ( ↑ )
F1 ( ↑ )
Pun(%)
Heg ( ↑ )
Hog ( ↑ )
Hop ( ↑ )
Audio
MiniCPM
8.7B
74.35
98.08
61.62
75.68
40.80
38.03
43.69
34.86
54.79
Qwen-2.5-omni
7B
71.68
99.09
56.79
72.21
37.12
42.85
50.00
35.78
57.53
Gemini 2.0 Flash ✯
-
87.51
84.82
98.51
91.15
60.98
51.03
62.78
47.36
82.85
GPT4o-Audio ✯
-
85.36
87.02
92.56
89.71
66.89
50.08
60.98
50.70
76.71
Appendix
Table 9: Comparison between text-only and audio models in real-world dataset. ✯ represents the proprietary LALMs and the symbol ’–’ indicates that the model size is not publicly disclosed.
Model
Pun Recognition
Pun Location
Pun Inference
Acc. ( ↑ )
Pun ( % )
Coa ( ↑ )
Fin ( ↑ )
Heg ( ↑ )
Hog ( ↑ )
Hop ( ↑ )
SALMONN
63.33
50.00
43.33
23.33
18.33
28.33
18.33
Qwen2-Audio-Instruct
51.66
43.33
58.33
33.33
30.00
23.63
28.33
Audio-Reasoner
58.33
50.00
63.33
40.00
31.67
34.61
35.42
Qwen-2.5-omni
78.33
58.33
60.00
36.67
43.33
31.66
65.00
MiniCPM
53.33
33.33
53.33
30.00
46.67
44.06
58.33
Appendix
Table 10: Comparison with Human Evaluation and LALMs. ✯ represents the proprietary LALMs.
Figure 5: Prompts used to guide Claude-Opus-4-1 in pun location for auxiliary annotation.
Figure 6: Prompts used to guide Claude-Opus-4-1 in pun inference for auxiliary annotation (for heterographic and homophonic puns).
Figure 7: Prompts used to guide Claude-Opus-4-1 in pun inference for auxiliary annotation (for homographic puns).
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
Large Audio Language Models (LALMs) are increasingly capable of reasoning over audio, yet existing benchmarks offer limited coverage of reasoning in polyphonic audio, where multiple sound events co-occur and induce compositional structure. To address this gap, we introduce PolyBench, a benchmark designed to evaluate compositional reasoning in polyphonic audio, comprising five evaluation subsets that cover counting, classification, detection, concurrency, and duration estimation, all of which require reasoning over multiple concurrent events and their relations. Our evaluation of state-of-the-art LALMs reveals consistent performance degradation in polyphonic settings, indicating a fundamental bottleneck in current LALMs.
Yuanjian Chen, Yang Xiao, Han Yin +3
Harbin University of Science and Technology, China · The University of Melbourne, Australia · KAIST, South Korea +1
Large Audio-Language Models show consistent performance gains across speech and audio benchmarks, yet high scores may not reflect true auditory perception. If a model can answer questions without processing the acoustic signal, the benchmark fails as a measure of auditory understanding. We present a diagnostic framework using two axes: text prior, which measures answerability from text and general knowledge alone, and audio reliance, which assesses actual dependency on the acoustic signal. Evaluating eight LALMs across three benchmarks, we find that models retain 60-72% of their full audio scores even without any audio input. Moreover, among items that require audio, only 3.0-4.2% need the complete audio clip; the majority can be resolved using localized fragments. These findings challenge the assumption that benchmark performance equals robust audio understanding, and we conclude with practical guidelines for improving evaluation reliability and benchmark design.
Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.
Jisu Jeon, Seungyeon Jwa, Joosung Lee +6
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1