S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modeling

Xueqi Wang; Zhigang Wang; Runqing Zhang; Zhenqi Jia; Junfeng Zhao

cs.CLAug 14, 2026

S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modeling

Authors: Xueqi Wang, Zhigang Wang, Runqing Zhang, Zhenqi Jia, Junfeng Zhao

Organizations: College of Computer Science, Inner Mongolia University, Hohhot, China · University of Electronic Science and Technology of China, Shenzhen Campus, Shenzhen, China

Abstract

Multimodal dialogue retrieval aims to retrieve dialogues from multimodal dialogue banks that are similar to a target dialogue in terms of both textual semantics and acoustic conversational styles. Such dialogue-level retrieval is crucial for many dialogue-related tasks, including Emotion Recognition in Conversation, Spoken Dialogue Systems, and Conversational Speech Synthesis, where external dialogue examples can provide valuable semantic and stylistic references. However, existing retrieval methods are still largely limited to utterance-level or unimodal matching, and often fail to capture the global semantic coherence and stylistic consistency of an entire dialogue. To address this gap, we propose S2Dialog, a unified framework for dialogue-level semantic-style retrieval from multimodal dialogue banks. Specifically, S2Dialog consists of a Dialogue-level Textual Retriever and a Dialogue-level Acoustic Retriever, which encode the textual and acoustic modalities of a dialogue into dialogue-level representations, respectively. To further enhance multimodal retrieval, we introduce Dialogue-level Textual-Acoustic Contrastive Learning, which aligns semantically and stylistically similar dialogues while distinguishing unrelated ones. Extensive experiments on the multimodal dialogue dataset DailyTalk demonstrate that S2Dialog achieves outstanding retrieval performance.

Figures & tables

Figure 1: From traditional retriever to our proposed multimodal dialogue retriever.

Figure 2: The overview of S2Dialog consists of Dialogue-level Textual Retriever, Dialogue-level Acoustic Retriever, and Dialogue-level Textual-Acoustic Contrastive Learning.

Systems	Recall@10	Recall@20	Recall@30	Recall@40	Recall@50
Text Pooling	11.60	22.00	31.20	40.00	53.20
Multimodal Pooling	21.20	35.20	50.00	60.80	68.40
Text Summarization	16.40	28.00	38.80	45.60	54.00
Multimodal Summarization	25.60	40.40	50.80	61.20	68.40
S2Dialog	50.68	63.01	72.60	77.17	83.56

Table 1: Recall Performance of Different Dialogue Retrieval Methods.

Systems	Recall@10	Recall@20	Recall@30	Recall@40	Recall@50
Abl.1 w/o Textual Retriever	39.60	54.80	69.60	75.13	79.60
Abl.2 w/o Acoustic Retriever	47.60	62.80	71.80	75.20	80.80
Abl.3 w/o $\mathcal{L}^{cl}_{t}$	45.32	59.75	70.24	73.31	79.57
Abl.4 w/o $\mathcal{L}^{cl}_{a}$	42.19	57.73	70.47	73.28	77.58
Abl.5 w/o Bottom-K	21.12	32.74	46.85	57.76	68.22
S2Dialog	50.68	63.01	72.60	77.17	83.56

Table 2: Recall Performance of S2Dialog and Ablation Variants.

Figure 3: Similarity Score Trends under Different Ablation Settings Across Retrieval Ranks.

Systems	Recall@10	Recall@20	Recall@30	Recall@40	Recall@50
Qwen3-Omni	16.45	27.27	41.23	55.02	63.38
Qwen2.5-Omni	14.29	25.54	37.66	47.62	54.11
Kimi-Audio	9.09	15.58	21.65	29.00	37.23
Step-Audio	16.02	30.30	43.72	51.95	60.17
S2Dialog	50.68	63.01	72.60	77.17	83.56

Table 3: Recall Performance of Multimodal Large Models and S2Dialog.

Systems	Recall@10	Recall@20	Recall@30	Recall@40	Recall@50
Reversed Dialogue Modeling	40.69	58.01	68.40	76.19	83.55
Randomly Shuffled Dialogue Modeling	37.66	57.58	66.67	73.16	80.09
Bidirectional Dialogue Modeling	42.86	60.17	72.73	82.25	86.15
S2Dialog (Forward Modeling)	50.68	63.01	72.60	77.17	83.56

Table 4: Recall Performance under Different Dialogue Temporal Modeling Strategies.

Figure 4: Retrieval performance across various $K$ values. The performance follows an inverted U-shaped trend, peaking at $K=5$ for all Recall metrics. This highlights the crucial role of an appropriate sample size in balancing training sufficiency and data redundancy during the optimization process.

Multimodal IR

Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.

See all 128 papers

Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues
Jun 3, 2026Hanbo Bi, Zhiqiang Yuan, Chongyang Li +7Multimodal IR Conversational Search
Pattern Recognition Center, WeChat AI, Tencent Inc, China · Aerospace Information Research Institute, Chinese Academy of Sciences
Efficient and High-Fidelity Omni Modality Retrieval
Mar 2, 2026Chuong Huynh, Manh Luong, Abhinav ShrivastavaMultimodal IR Cross-Modal Retrieval
University of Maryland, College Park, USA · Monash University, Australia
AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition
Mar 7, 2026Yunsheng Wang, Yuntao Shou, Yilong Tan +3Emotion Recognition in Conversations Multimodal Emotion Recognition
College of Computer and Mathematics, Central South University of Forestry and Technology · Department of Computer Science, State University of New York

Jun 3, 2026J/K move · Enter open · S save

Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues

Hanbo Bi, Zhiqiang Yuan, Chongyang Li, Qiwei Yan+6

Pattern Recognition Center, WeChat AI, Tencent Inc, China · Aerospace Information Research Institute, Chinese Academy of Sciences

Multimodal IR Conversational Search

Open paper AskPDF arXiv

With the widespread adoption of multi-modal communication platforms, long-form dialogues interleaving text and images have become increasingly common. Users often need to retrieve coherent dialogue fragments related to specific topics, rather than isolated utterances. We propose Fine-grained Fragment Retrieval (FFR), which locates semantically relevant multi-utterance, multi-image fragments in multi-modal long-form dialogues. We explore two settings: (1) FFR within Single-Dialogue, retrieving fragments from a given dialogue; and (2) FFR within Dialogue Corpus, retrieving from a large-scale corpus for open-domain scenarios. For (1), we introduce F2RVLM, a generation-based retrieval model trained with reinforcement learning, using multi-objective rewards and difficulty-aware curriculum sampling to enhance fragment coherence. For (2), we develop FFRS, a two-stage system combining offline fragment-level indexing with online retrieval. Specifically, each dialogue is decomposed into minimal semantic fragments encoded by a Fragment Embedding Model (FEM) into a vector database; at inference, FEM rapidly recalls Top-K candidates, and F2RVLM performs fine-grained reasoning to identify the most relevant sub-content. To support FFR, we construct MLDR, the longest multi-modal dialogue retrieval dataset to date, and a WeChat-based real-world test set. Experiments on both benchmarks demonstrate that F2RVLM and FFRS consistently achieve superior performance across single-dialogue and corpus-level FFR.

Abstract

Multimodal IR

Explore similar work

Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues

Efficient and High-Fidelity Omni Modality Retrieval

AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition