Edge-Cloud Collaborative Speech Emotion Captioning via Token-Level Speculative Decoding in Audio-Language Models

Xiangyuan Xue; Jiajun Lu; Yan Gao; Gongping Huang; Ting Dang; Hong Jia

cs.SDMar 12, 2026

Edge-Cloud Collaborative Speech Emotion Captioning via Token-Level Speculative Decoding in Audio-Language Models

Authors: Xiangyuan Xue, Jiajun Lu, Yan Gao, Gongping Huang, Ting Dang, Hong Jia

Organizations: The University of Auckland, New Zealand · The University of Melbourne, Australia · The University of Cambridge, United Kingdom · Wuhan University, China

Abstract

Speech Emotion Captioning (SEC) leverages large audio-language models to generate rich, context-aware affective descriptions from speech. However, real-world deployment remains challenging due to the substantial computational demands on resource-constrained edge devices and the privacy risks of transmitting biometric audio. While smaller audio-language models enable efficient on-device SEC, their limited capacity often weakens subtle paralinguistic modeling and fine-grained affective grounding. We propose an edge-cloud collaborative framework based on Uncertainty-Guided Speculative Decoding (UGSD). A lightweight edge model drafts captions locally, and only high-uncertainty token blocks are selectively escalated to a stronger cloud verifier for validation. Experiments on the MER2024 benchmark demonstrate substantial BLEU improvements up to 62.7%. UGSD further achieves 1.4x lower latency and 8.5x higher token throughput compared to an edge-only model. These results empirically characterize the quality-efficiency-privacy trade-off in deployable SEC systems.

Figures & tables

Figure 1: Overview of the uncertainty-aware collaborative decoding framework for Speech Emotion Captioning.

Type

Prompt

English Prompt

As an expert in the field of emotions, please focus on the acoustic information in the audio to discern clues related to the emotions of the individual. Please provide a detailed description and ultimately predict the emotional state of the individual. In the audio, respond in English only, use third person, avoid dialogue style. Example: “In the audio, the speaker’s tone is raised with rapid pacing, expressing dissatisfaction. Therefore, the speaker exhibits frustration and impatience.”

Chinese Prompt

任务：请基于给定音频，输出一句“情感说明短句”。必须遵守：只输出一句中文短句（12–30个汉字），以“。”结尾；句子中同时包含一个主要情绪和一个简短的声学或韵律线索（如语气、语速、强弱、音高变化等类别层面的描述即可），但不要解释或列举；不要出现客套话、邀请继续对话、表情符号、英文、Markdown、标号或代码；不要提及“音频”“模型”“分析”或“我”；若存在多种可能性，只选择最可能的一种，不要并列罗列。只给出最终这一句短句，不要输出其他内容。

Table 1: Language-specific prompts for speech emotion captioning.

Config	TTFT $\downarrow$	ITPS $\uparrow$	OET $\downarrow$	OTPS $\uparrow$	Total Time $\downarrow$	CPU (%) $\downarrow$	RAM (GB) $\downarrow$	GPU (%) $\downarrow$	GPU Mem (GB) $\downarrow$
Edge (3B)	8.41	2340.11	40.12	1.53	40.21	201.6	20.55	——	——
UGSD ( $L=3$ )	6.23	2504.27	2.89	12.92	28.47	68.8	2.87	44.2	37.75
UGSD ( $L=5$ )	6.18	2561.42	2.73	12.85	28.73	68.8	2.83	44.5	37.75
UGSD ( $L=7$ )	6.30	2428.09	2.79	9.76	29.56	66.6	2.77	43.5	37.76
UGSD ( $L=10$ )	6.40	2439.40	2.77	8.60	28.92	66.8	2.78	43.0	37.76
UGSD ( $L=20$ )	6.75	2395.87	2.78	3.96	29.31	68.0	2.79	42.5	37.75

Table 2: Comparison of latency and resource usage. Lower is better for TTFT, OET, Total Time, CPU, RAM, GPU, and GPU Mem (

\downarrow

). Higher is better for ITPS and OTPS (

\uparrow

). Where bold indicates the best, and underlining indicates the second best.

Model	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	METEOR $\uparrow$	ROUGE-L $\uparrow$
Edge	36.71	0.71	13.92	14.38
Cloud	48.19	1.20	23.68	24.01
UGSD ( $L=3$ )	39.36	1.05	24.15	19.89
UGSD ( $L=5$ )	43.75	1.02	24.32	20.27
UGSD ( $L=7$ )	44.05	1.02	23.65	18.46
UGSD ( $L=10$ )	42.66	1.01	23.27	17.16

Table 3: Comparison of caption quality on the English subset. Spec means speculative decoding.

Method	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	METEOR $\uparrow$	ROUGE-L $\uparrow$
Edge	39.14	2.01	12.35	16.77
Cloud	51.04	3.77	25.01	26.61
UGSD ( $L=3$ )	41.89	3.23	25.62	22.40
UGSD ( $L=5$ )	46.44	3.12	25.84	22.79
UGSD ( $L=7$ )	46.75	3.12	24.97	20.94
UGSD ( $L=10$ )	45.31	3.09	24.48	19.61

Table 4: Comparison of caption quality on the Chinese subset.

Model	BLEU-1 $\uparrow$	BLEU-4 $\uparrow$	METEOR $\uparrow$	ROUGE-L $\uparrow$	Privacy
Cloud (7B, A100)	39.56	0.89	10.60	14.59	$\times$
Edge (3B, CPU)	36.71	0.71	13.92	14.38	✓
UGSD (coll.)	37.63	0.78	13.93	14.41	✓

Table 5: Caption quality and privacy-preserving comparison between edge, cloud, and proposed method in UGSD (coll. denotes collaboration).

Edge-Cloud Collaborative Inference

See all 28 papers

Bandwidth-Efficient and Privacy-Preserving Edge-Cloud Many-to-Many Speech Translation
May 27, 2026Yexing Du, Kaiyuan Liu, Youcheng Pan +4Speech Translation Speech Processing
Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Harbin Institute of Technology, Harbin, China
EmotionAI: A Privacy-Preserving Computational Intelligence Pipeline for Speech-Emotion-Grounded Conversational Analysis
Jun 22, 2026Wai Laam Mak, Isibor Kennedy Ihianle, Pedro MachadoSpeech Processing Emotion Recognition in Conversations
School of Science and Technology, Nottingham Trent University, Nottingham, UK
EMODE: Dynamic Para-Semantic Experts for Emotion-Aware Speech Language Modeling
Oct 3, 2026Jianan Pan, Yiwen Gu, Xinze Li +2Speech Language Models Emotion Representation in Language Models
Zhejiang University Hangzhou, China

May 27, 2026J/K move · Enter open · S save

Bandwidth-Efficient and Privacy-Preserving Edge-Cloud Many-to-Many Speech Translation

Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang+3

Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Harbin Institute of Technology, Harbin, China

Speech Translation Speech Processing Privacy-Preserving Speech Processing Edge-Cloud Computing

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have demonstrated significant potential for speech-to-text translation (S2TT). However, existing deployment paradigms face critical challenges: pure on-device models suffer from resource constraints, while centralized cloud systems incur severe privacy risks and bandwidth bottlenecks by transmitting raw voice data. Furthermore, most models exhibit English-centric biases, restricting many-to-many translation scaling. In this paper, we propose Edge-cloud Speech Recognition and Translation (ESRT), a privacy-preserving and bandwidth-efficient collaborative edge-cloud MLLM framework. Specifically, we design an edge-cloud split inference architecture that retains a lightweight speech encoder and adapter on the device, transmitting only highly compressed intermediate features to the cloud. This fundamentally prevents voiceprint leakage and reduces bandwidth requirements by up to 10

\times

. To overcome English-centric bottlenecks, we introduce a multi-task weighted curriculum learning strategy with data balancing to ensure robust cross-lingual consistency. Extensive experiments on the FLEURS dataset demonstrate that our models, ESRT-4B and ESRT-12B, achieve state-of-the-art many-to-many S2TT performance across 45 languages (

45 \times 44

directions). Code and models are released to facilitate reproducible, privacy-aware MLLM S2TT research. The code and models are released at https://github.com/yxduir/esrt.

Abstract

Edge-Cloud Collaborative Inference

Explore similar work

Bandwidth-Efficient and Privacy-Preserving Edge-Cloud Many-to-Many Speech Translation

EmotionAI: A Privacy-Preserving Computational Intelligence Pipeline for Speech-Emotion-Grounded Conversational Analysis

EMODE: Dynamic Para-Semantic Experts for Emotion-Aware Speech Language Modeling