TVPara: Paralinguistic Captioning of Expressive TV-Series Speech in Complex Acoustic Environments

Ziyang Zhang; Wenyi Yu; Wen Wu; Xuenan Xu; Chao Zhang

cs.SDOct 14, 2026

TVPara: Paralinguistic Captioning of Expressive TV-Series Speech in Complex Acoustic Environments

Authors: Ziyang Zhang, Wenyi Yu, Wen Wu, Xuenan Xu, Chao Zhang

Organizations: Tsinghua University, China · Shanghai Artificial Intelligence Laboratory, China

Abstract

Large audio-language models (LALMs) struggle to parse paralinguistics (e.g., emotion, pace, pitch) in speech that combines intense emotion with complex acoustic environments. Strengthening this ability calls for training data that reflects such conditions. However, existing corpora rely heavily on studio-recorded speech, offering limited emotional and acoustic diversity. To address this gap, we introduce TVPara, a paralinguistic captioning dataset built from TV series. Speech in TV series often carries diverse emotional expression and co-occurs with scene-specific background sound, providing expressive emotion and complex acoustic environments at scale. Constructed via LALM pre-annotation and human correction, TVPara includes two tasks: Single-Utterance (~492h) and Multi-Utterance (~213h) Paralinguistic Captioning. Fine-tuning open-source LALMs on TVPara yields an average relative improvement of 31.26% in paralinguistic understanding under complex acoustic environments. The gains further generalize to an unseen TV series and to cleaner, cross-domain speech.

Figures & tables

Figure 1: Overview of the two TVPara tasks. (a) Single-Utterance Paralinguistic Captioning describes the paralinguistic attributes of a single utterance, including emotion, pace, pitch, and volume. (b) Multi-Utterance Paralinguistic Captioning describes a sequence of adjacent utterances in order, with emphasis on how these attributes change over time ( e.g. , from irony to excitement).

Figure 2: Emotion word clouds of (a) TVPara and (b) VoxCeleb. VoxCeleb is dominated by neutral and calm states, whereas TVPara covers many expressive emotions ( e.g. , frustration, sarcasm, anger).

Statistic	Single	Multi
Total audio clips	256,212	45,402
Total duration (h)	492	213
Avg. clip duration (s)	6.91	16.89
Avg. caption length (words)	32.8	58.1

Table 1: Statistics of the Single- and Multi-Utterance Paralinguistic Captioning tasks of TVPara.

Statistic	TVPara	VoxCeleb
Clips with non-neutral emotion	67.2%	2.9%
Clips with background sound	36.2%	8.0%
Adjacent utterances with emotion change	55.5%	–

Table 2: Emotion and background sound statistics of TVPara and VoxCeleb. VoxCeleb has no multi-utterance data.

Figure 3: Annotation pipeline of TVPara. (a) Single-Utterance Annotation : each clip is pre-annotated by an LALM from the audio and a prompt, the caption is polished by an LLM, and it is then corrected by human annotators, who remove hallucinated content ( e.g. , emotions that the audio does not support) and add omitted content. The corrected data are used to fine-tune the pre-annotation model from $M_{p}$ to $M_{f}$ , which then pre-annotates the remaining audio. (b) Multi-Utterance Construction : a clip covering $N$ adjacent utterances is cut from the source recording, and the corrected captions of the single-utterance clips within it are merged by an LLM into one caption in time order.

Figure 4: Illustration of the inference and LLM-as-a-Judge evaluation pipeline. During inference, the audio and user prompt are processed by the LALM to generate a paralinguistic caption. For evaluation, an LLM extracts atomic attributes (emotion, pace, pitch, volume) from both the generated caption and the ground-truth label. An LLM evaluator then compares these aligned attributes, assigning similarity scores on a discrete scale from 0 to 4.

Model	Training	Single-Utterance Task				Multi-Utterance Task
Model	Training	Emotion	Pace	Pitch	Volume	Emotion	Pace	Pitch	Volume
Qwen2.5-Omni	Zero-shot	0.74	2.02	1.41	2.43	0.99	1.29	1.26	1.12
Qwen2.5-Omni	SFT	0.99	2.40	2.45	2.33	1.46	1.55	1.48	1.35
SALMONN	Zero-shot	1.45	2.74	2.32	2.31	1.09	1.21	1.10	1.04
SALMONN	SFT	1.52	2.73	2.44	3.58	1.60	1.60	1.70	1.81

Table 3: Evaluation results of paralinguistic captioning on the four attributes (emotion, pace, pitch, and volume; scored from 0 to 4) on the TVPara hold-out test set. The better result for each base model is in bold .

Model	Training	Unseen Series (Single)				Unseen Series (Multi)				VoxCeleb (Single)
Model	Training	Emo.	Pace	Pitch	Vol.	Emo.	Pace	Pitch	Vol.	Emo.	Pace	Pitch	Vol.
Qwen2.5-Omni	Zero-shot	0.74	2.21	1.45	1.47	1.07	1.36	1.26	1.11	0.84	2.24	1.45	1.80
Qwen2.5-Omni	SFT	0.95	1.78	2.30	2.44	1.19	1.28	1.42	1.08	0.96	1.86	2.43	2.22
SALMONN	Zero-shot	1.35	2.64	2.08	1.65	1.17	1.48	1.30	1.13	1.25	2.70	2.02	1.79
SALMONN	SFT	1.52	2.89	2.48	3.51	1.64	1.55	1.76	1.83	1.39	2.85	2.34	3.69

Table 4: Out-of-distribution evaluation on the four attributes (scored from 0 to 4). Unseen Series is the unseen-series test set (Section 2.3 ); VoxCeleb is the cleaner, cross-domain test set (Section 3.2 , single-utterance only). The better result for each base model is in bold .

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding
Sep 19, 2026Nishit Anand, Jiaqi Su, Ke Chen +5Audio Understanding Audio-Language Models
Adobe Research, USA · University of Maryland, College Park, USA · OpenAI, USA
Resurfacing Paralinguistic Awareness in Large Audio Language Models
Mar 12, 2026Hao Yang, Minghan Wang, Tongtong Wu +3Audio-Language Models LLM Fine-Tuning
Department of Data Science & AI, Monash University, Australia · Department of Computer Science, University College London, UK
Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
May 26, 2026Jiacheng Pang, Ashutosh Chaubey, Mohammad SoleymaniAudio-Language Model Evaluation Speech Processing
Institute for Creative Technologies, University of Southern California, Los Angeles, USA

Sep 19, 2026J/K move · Enter open · S save

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding

Nishit Anand, Jiaqi Su, Ke Chen, Yunyun Wang+4

Adobe Research, USA · University of Maryland, College Park, USA · OpenAI, USA

Audio Understanding Audio-Language Models Audio Reasoning Audio-Language Model Evaluation

Open paper AskPDF arXiv

Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.

Abstract

Audio-Language Models

Explore similar work

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding

Resurfacing Paralinguistic Awareness in Large Audio Language Models

Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox