Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models

Yuchen Su; Shaoxin Zhong; Yonghua Zhu; Ruofan Wang; Zijian Huang; Qiqi Wang; Na Zhao; Diana Benavides-Prado

cs.SDMar 19, 2026

Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models

Authors: Yuchen Su, Shaoxin Zhong, Yonghua Zhu, Ruofan Wang, Zijian Huang, Qiqi Wang, Na Zhao, Diana Benavides-Prado,

Organizations: School of Computer Science, University of Auckland, New Zealand · Singapore University of Technology and Design, Singapore · School of Statistics and Data Science, Nankai University, China · School of Electronic Engineering and Computer Science, Queen Mary University of London

Abstract

Puns represent a typical linguistic phenomenon that exploits polysemy and phonetic ambiguity to generate humour, posing unique challenges for natural language understanding. Within pun research, audio plays a central role in human communication except text and images, while datasets and systematic resources for spoken puns remain scarce, leaving this crucial modality largely underexplored. In this paper, we present APUN-Bench, the first benchmark dedicated to evaluating large audio language models (LALMs) on audio pun understanding. Our benchmark contains 4,434 audio samples annotated across three stages: pun recognition, pun word location and pun meaning inference. We conduct a deep analysis of APUN-Bench by systematically evaluating 10 state-of-the-art LALMs, uncovering substantial performance gaps in recognizing, localizing, and interpreting audio puns. This analysis reveals key challenges, such as positional biases in audio pun location and error cases in meaning inference, offering actionable insights for advancing humour-aware audio intelligence.

Figures & tables

Figure 1: An example of the pun understanding in recognition, location and inference.

Figure 2: The Overview of APUN-Benchmark, covering data construction, human annotation and three evaluation stages: (S1) Pun Recognition, detecting the presence of an audio pun; (S2) Pun Location, identifying the specific pun word in the audio; and (S3) Pun Inference, inferring its dual meanings.

Dataset	Heterographic	Homophonic	Homographic	Negative	Overall
Synthetic Dataset	794	304	1298	1,152	3,541
- Segment Clip (s)	4.18	4.36	3.88	3.47	3.97
- Sentence Length	12.21	13.89	11.68	8.50	11.57
Real-World Dataset	223	72	287	311	582
- Segment Clip (s)	6.36	6.16	5.34	6.73	6.14
- Sentence Length	11.12	11.40	11.55	10.88	11.23

Table 1: APUN-Bench data statistics, including average audio clip duration and sentence length.

Synthetic Dataset
Model	Size	Pun Recognition					Pun Location		Pun Inference
Model	Size	Acc. ( $\uparrow$ )	Pre ( $\uparrow$ )	Rec ( $\uparrow$ )	F1 ( $\uparrow$ )	Pun ( $\%$ )	Coa ( $\uparrow$ )	Fin ( $\uparrow$ )	Heg ( $\uparrow$ )	Hog ( $\uparrow$ )	Hop ( $\uparrow$ )
Qwen2-Audio-Instruct	7B	45.78	71.70	39.45	50.90	38.06	50.81	31.77	13.47	17.91	26.97
SALMONN	13B	51.91	72.51	52.60	60.97	51.81	34.41	22.35	15.23	24.63	25.32
Audio-Reasoner	7B	55.27	72.62	59.81	65.59	54.47	50.39	32.31	23.05	27.25	37.50
Audio-Flamingo-3	7B	37.51	59.25	40.02	47.78	48.24	36.37	27.83	33.25	29.35	56.91

Table 2: Evaluation Results of the Synthetic and Real-World Datasets on APUN-Bench in various LALMs and cascaded systems. ✯ represents the proprietary LALMs and the symbol ’–’ indicates that the model size is not publicly disclosed. Coa and Fin repesent the accuracy of coarse-grained and fine-grained locations. Hog, hop and heg denote the accuracy of homographic puns, homophonic puns and heterographic puns.

Figure 3: Statistical comparison of predicted in different LALMs and ground-truth pun positions in APUN-Bench, showing the distribution of pun words across different sentence intervals (beginning, middle, end of the sentence).

Figure 4: The distribution of error types in pun inference for Omni-R1 and GPT4o-Audio across heterographic, homographic, and homophonic puns.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Error Type	Definition	Example (Audio Transcription)	Prediction	Ground Truth
Heterographic Puns & Homophonic Puns
Position Error (PE)	The model fails to provide the correct word corresponding to the given pun word and may instead return a word from another position in the sentence.	When asked to picture the perfect modern defensive weapon the Claymore springs to mine .	Claymore	mind
Pun Word Repeat (PWR)	The model outputs the given pun word itself rather than its intended meaning.	Did you hear about the nervous preacher ? He had sweaty psalms .	psalms	palm
Word Inference Error (WIE) (Similar)	The model infer an incorrect word, often manifested as another word with a similar pronunciation.	One ear of corn said to the other’You’re getting husky ’.	hoarse	husk
Word Inference Error (WIE) (No Similar)	The model infer an incorrect word, often manifested as another word with a completely different pronunciation.	George Westinghouse was a refrigerator magnate .	entrepreneur	magnet
Irrelevant Answer (IA)	The model’s answer is unrelated to the pun inference for the pun word.	She was only a Butcher’s daughter, but there wasn’t much more she could loin .	line human given an audio …	learn

Appendix

Table 3: Error types of pun inference in heterographic, homophonic and homographic puns

Laura	Gary	Jon	Lea
Karen	Rick	Brenda	David
Eileen	Jordan	Mike	Yann
Joy	James	Eric	Lauren
Rose	Will	Jason	Aaron
Naomie	Alisa	Patrick	Jerry
Tina	Jenna	Bill	Tom

Appendix

Table 4: List of the 34 speakers used in Parler-TTS.

Model	Heterographic		Homophonic		Homographic
Model	Coa ( $\uparrow$ )	Fin ( $\uparrow$ )	Coa ( $\uparrow$ )	Fin ( $\uparrow$ )	Coa ( $\uparrow$ )	Fin ( $\uparrow$ )
SALMONN	32.37	17.12	35.86	8.55	35.32	28.81
Qwen2-Audio-Instruct	52.14	29.47	44.41	13.81	51.51	37.41
Audio-Reasoner	50.50	29.34	39.47	11.18	52.90	39.11
Qwen-2.5-omni	53.53	36.39	56.91	21.38	66.54	61.42
MiniCPM	56.30	34.13	62.50	20.06	61.58	56.85

Appendix

Table 5: Evaluation results of the Synthetic Dataset on APUN-Bench. We evaluate various LALMs on our proposed APUN-Bench. ✯ represents the proprietary LALMs.

Model	NOUN	ADV	VERB	ADJ	PROPN
Synthetic Dataset	954	312	246	202	146
Real-world Dataset	208	36	131	81	93

Appendix

Table 6: The part-of-speech distributions between Synthetic and Real-world Datasets.

ASR Model	Synthetic			Real-world
ASR Model	Heg ( $\downarrow$ )	Hog ( $\downarrow$ )	Hop ( $\downarrow$ )	Heg ( $\downarrow$ )	Hog ( $\downarrow$ )	Hop ( $\downarrow$ )
SenseVoiceSmall	0.144	0.126	0.151	0.177	0.134	0.269
Whisper-Large	0.111	0.097	0.123	0.154	0.093	0.221

Appendix

Table 7: Word Error Rate (WER) of SenseVoiceSmall and Whisper-Large on synthetic and real-world datasets.

Model	Architecture	ASR (s)	Reasoning (s)	Total (s)
MiniCPM	End-to-End	-	0.61	0.61
Qwen-2.5-omni	End-to-End	-	0.91	0.91
SenseVoiceSmall+MiniCPM	Cascaded	0.42	0.54	0.74
Whisper-Large+MiniCPM	Cascaded	1.31	0.54	1.85
SenseVoiceSmall+Qwen-2.5-Instruct	Cascaded	0.42	0.34	0.76
Whisper-Large+Qwen-2.5-Instruct	Cascaded	1.31	0.34	1.65

Appendix

Table 8: Time consumption analysis in real-world dataset between open-source LALMs and cascaded systems.

Model	Size	Pun Recognition					Pun Location	Pun Inference
Model	Size	Acc. ( $\uparrow$ )	Pre ( $\uparrow$ )	Rec ( $\uparrow$ )	F1 ( $\uparrow$ )	Pun(%)	Pun Location	Heg ( $\uparrow$ )	Hog ( $\uparrow$ )	Hop ( $\uparrow$ )
Audio
MiniCPM	8.7B	74.35	98.08	61.62	75.68	40.80	38.03	43.69	34.86	54.79
Qwen-2.5-omni	7B	71.68	99.09	56.79	72.21	37.12	42.85	50.00	35.78	57.53
Gemini 2.0 Flash ✯	-	87.51	84.82	98.51	91.15	60.98	51.03	62.78	47.36	82.85
GPT4o-Audio ✯	-	85.36	87.02	92.56	89.71	66.89	50.08	60.98	50.70	76.71

Appendix

Table 9: Comparison between text-only and audio models in real-world dataset. ✯ represents the proprietary LALMs and the symbol ’–’ indicates that the model size is not publicly disclosed.

Model	Pun Recognition		Pun Location		Pun Inference
Model	Acc. ( $\uparrow$ )	Pun ( $\%$ )	Coa ( $\uparrow$ )	Fin ( $\uparrow$ )	Heg ( $\uparrow$ )	Hog ( $\uparrow$ )	Hop ( $\uparrow$ )
SALMONN	63.33	50.00	43.33	23.33	18.33	28.33	18.33
Qwen2-Audio-Instruct	51.66	43.33	58.33	33.33	30.00	23.63	28.33
Audio-Reasoner	58.33	50.00	63.33	40.00	31.67	34.61	35.42
Qwen-2.5-omni	78.33	58.33	60.00	36.67	43.33	31.66	65.00
MiniCPM	53.33	33.33	53.33	30.00	46.67	44.06	58.33

Appendix

Table 10: Comparison with Human Evaluation and LALMs. ✯ represents the proprietary LALMs.

Panel of Figure 5 — Figure 5: Prompts used to guide Claude-Opus-4-1 in pun location for auxiliary annotation.

Panel of Figure 6 — Figure 6: Prompts used to guide Claude-Opus-4-1 in pun inference for auxiliary annotation (for heterographic and homophonic puns).

Panel of Figure 7 — Figure 7: Prompts used to guide Claude-Opus-4-1 in pun inference for auxiliary annotation (for homographic puns).

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 205 papers

PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio
Mar 5, 2026Yuanjian Chen, Yang Xiao, Han Yin +3Audio Reasoning Audio-Language Model Evaluation
Harbin University of Science and Technology, China · The University of Melbourne, Australia · KAIST, South Korea +1
All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation
Apr 27, 2026Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2Audio-Language Model Evaluation Modality Reliance in Multimodal Models
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model Evaluation Pairwise Comparison
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1

Mar 5, 2026J/K move · Enter open · S save

PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

Yuanjian Chen, Yang Xiao, Han Yin, Xubo Liu+2

Harbin University of Science and Technology, China · The University of Melbourne, Australia · KAIST, South Korea+1

Audio Reasoning Audio-Language Model Evaluation Compositional Reasoning Audio-Language Models

Open paper AskPDF arXiv

Large Audio Language Models (LALMs) are increasingly capable of reasoning over audio, yet existing benchmarks offer limited coverage of reasoning in polyphonic audio, where multiple sound events co-occur and induce compositional structure. To address this gap, we introduce PolyBench, a benchmark designed to evaluate compositional reasoning in polyphonic audio, comprising five evaluation subsets that cover counting, classification, detection, concurrency, and duration estimation, all of which require reasoning over multiple concurrent events and their relations. Our evaluation of state-of-the-art LALMs reveals consistent performance degradation in polyphonic settings, indicating a fundamental bottleneck in current LALMs.

Abstract

Audio-Language Model Evaluation

Explore similar work

PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge