DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolution

Xin Shen; Zhishu Jiang; Jiaye Yang; Haibo Liu; Yichen Wan; Jiarui Zhang; Tingzhi Dai; Luodong Xu

cs.CLFeb 25, 2026

DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolution

Authors: Xin Shen, Zhishu Jiang, Jiaye Yang, Haibo Liu, Yichen Wan, Jiarui Zhang, Tingzhi Dai, Luodong Xu,

Organizations: Baidu Inc, Beijing, China

Abstract

Immersive conversational systems in production face a persistent trade-off between responsiveness and long-horizon task capability. Real-time interaction is achievable for lightweight turns, but requests involving planning and tool invocation (e.g., search and media generation) produce heavy-tail execution latency that degrades turn-taking, persona consistency, and user trust. To address this challenge, we propose DuCCAE (Conversation while Collaboration with Augmentation and Evolution), a hybrid engine for immersive conversation deployed within Baidu Search, serving millions of users. DuCCAE decouples real-time response generation from asynchronous agentic execution and synchronizes them via a shared state that maintains session context and execution traces, enabling asynchronous results to be integrated back into the ongoing dialogue. The system orchestrates five subsystems-Info, Conversation, Collaboration, Augmentation, and Evolution-to support multi-agent collaboration and continuous improvement. We evaluate DuCCAE through a comprehensive framework that combines offline benchmarking on the Du-Interact dataset and large-scale production evaluation within Baidu Search. Experimental results demonstrate that DuCCAE outperforms strong baselines in agentic execution reliability and dialogue quality while reducing latency to fit strict real-time budgets. Crucially, deployment metrics since June 2025 confirm substantial real-world effectiveness, evidenced by a tripling of Day-7 user retention to 34.2% and a surge in the complex task completion rate to 65.2%. Our hybrid architecture successfully preserves conversational continuity while enabling reliable agentic execution, offering practical guidelines for deploying scalable agentic systems in industrial settings.

Figures & tables

Figure 1. DuCCAE interface and interaction flow in production. Starting from a Baidu Search entry, the system maintains persona-consistent chat responses and supports escalation to real-time calling; complex requests trigger asynchronous collaboration and tool augmentation while preserving conversational continuity.

Figure 2. DuCCAE : an evolving agentic service engine for immersive conversational interaction. Info System converts multimodal signals into policy-aware context and manages memory. Conversation System acts as a low-latency gatekeeper for intent routing and renders persona-consistent responses. Collaboration System supports multi-agent execution for long-horizon tasks via planning and tool use. Augmentation System empowers agents with external tools, retrieval resources, and execution protocols. Evolution System drives continuous improvement through episode-based judging and post-training.

Figure 3. Runtime Execution Dataflow of a Mixed-Intent Interaction Episode. Processing the input “Exhausted… Plan a trip”, the Fast Track (top) utilizes User Memory (“Hobby: Basketball”) for an immediate empathetic inquiry. Simultaneously, the Slow Track (bottom) decomposes the travel request, demonstrating: (1) Parallel Execution of independent flight and activity searches to optimize latency; and (2) Constraint Fusion , which filters dining options via the memory constraint (“Dislikes Raw Fish”) to select the compatible “Wagyu Beef”.

System Ver.	Evolution Stage	Training Source	Dataset Size	Method
DuCCAE-V1	Cold Start	N/A	0	Zero-shot Prompting
DuCCAE-V2	Evolution I	Du-Interact-Evo-V1	15,000	SFT (on V1 logs)
DuCCAE-V3	Evolution II	Du-Interact-Evo-V2	50,000	SFT + RL (on V2 logs)
Benchmark	Evaluation	Du-Interact	5,000	Fixed Golden Set

Table 1. Statistics of Du-Interact-Evo (Training) and Du-Interact (Testing) across Evolutionary Stages.

Backbone Model	Setup	Task Execution		Dialogue Quality			Efficiency
Backbone Model	Setup	Dispatch ( $P_{disp}$ )	Success Rate (SR)	Fidelity (%)	Persona (1-5)	Empathy (1-5)	Avg. Latency
Edge Class (3B)
Qwen2.5-3B-Instruct	Zero-shot	52.4%	28.5%	45.2%	2.5	2.4	1,250 ms
Llama-3.2-3B-Instruct	Zero-shot	51.8%	26.2%	44.1%	2.4	2.3	1,280 ms
Server Class (7B-11B)
Qwen2.5-7B-Instruct	Zero-shot	61.5%	37.2%	54.8%	2.8	2.8	1,480 ms

Table 2. Main Results on the Du-Interact Golden Benchmark.

Figure 4. Evolution of Key Business Metrics across System Iterations. Statistics are derived from large-scale online controlled experiments in Baidu Search. From V1 to V3, DuCCAE demonstrates consistent growth in: (a) User Stickiness; (b) Conversation Quality; (c) Interaction Depth; and (d) Agentic Capability.

Method	Paradigm	Response Fidelity	Avg. Latency
		(%) $\uparrow$	(ms) $\downarrow$
Baseline-E2E	Video-In $\to$ Text-Out	64.0%	2,100
DuCCAE (Ours)	Caption $\to$ LLM	71.1%	480

Table 3. Ablation on Visual Perception Strategies. Reported latency measures the visual perception overhead (pre-LLM).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5. Core system prompt in DuCCAE-V3 for intent routing and planning. The structured schema ensures strict compliance with the dual-track architecture and deterministic execution.

Panel of Figure 6 — Figure 6. Evaluation protocol for the LLM-as-a-Judge framework. The design enforces structured scoring aligned with expert annotations.

Figure 7. Qualitative examples of DuCCAE’s Dual-Track execution. Note: Specific entities, numerical values, and PII have been replaced with placeholders ( e.g. , “XXXX”) to strictly preserve user privacy and commercial confidentiality.

Conversational Agents

Conversational agents are AI systems that interact with people through multi-turn dialogue to answer questions, provide support, or complete tasks.

See all 287 papers

Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling
May 13, 2026Coleman Hooper, Minwoo Kang, Suhong Moon +7Tool-Using Agents LLM Agents
University of California, Berkeley · ICSI · LBNL
DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration
Oct 8, 2026Yingda Shen, Yuxiang Wang, Kunyu Feng +7LLM Agent Harnesses Agent Harness Optimization
The Chinese University of Hong Kong, Shenzhen · Amphion Technology Co., Ltd
ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation
Oct 6, 2026Arkajyoti Chakraborty, Aryan Tayal, Ishika Agarwal +5Tool-Using Agents AI Agent Benchmarks
Uniphore · University of Illinois Urbana-Champaign

May 13, 2026J/K move · Enter open · S save

Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling

Coleman Hooper, Minwoo Kang, Suhong Moon, Nicholas Lee+6

University of California, Berkeley · ICSI · LBNL

Tool-Using Agents LLM Agents LLM Agent Training

Open paper AskPDF arXiv

There is a growing demand for agentic AI technologies for a range of downstream applications like customer service and personal assistants. For applications where the agent needs to interact with a person, real-time low-latency responsiveness is required; for example, with voice-controlled applications, under 1 second of latency is typically required for the interaction to feel seamless. However, if we want the LLM to reason and execute an agentic workflow with tool calling, this can add several seconds or more of latency, which is prohibitive for real-time latency-sensitive applications. In our work, we propose Speculative Interaction Agents to enable real-time interaction even for agents with complex multi-turn tool calling. We propose Asynchronous I/O, which decouples the core agent reason-and-act thread from waiting for additional information from either the user or environment, thereby allowing for overlapping agentic processing while waiting on external delays. We also propose Speculative Tool Calling as a method to manage task execution when the agent is still unsure if it has received the full information or if additional user information may later be provided. For strong cloud models, our method can be applied out-of-the-box to existing real-time cloud APIs, providing 1.3-1.7

\times

speedups with minor accuracy loss. To enable real-time interaction with small edge-scale models, we also present a clock-based training methodology that adapts the model to handle streaming inputs and asynchronous responses, and demonstrate a synthetic data generation strategy for SFT. Altogether, this approach provides 1.6-2.2

\times

speedups with the Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct models across multiple tool calling benchmarks.

Abstract

Conversational Agents

Explore similar work

Speculative Interaction Agents: Building Real-Time Agents with Asynchronous I/O and Speculative Tool Calling

DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration

ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation