JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents

Junseok Ahn; Jaehyun Park; Jeongsoo Choi; Joon Son Chung; Ji-Hoon Kim

cs.CVOct 15, 2026

JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents

Authors: Junseok Ahn, Jaehyun Park, Jeongsoo Choi, Joon Son Chung, Ji-Hoon Kim

Organizations: Korea Advanced Institute of Science and Technology, South Korea · Chung-Ang University, South Korea

Abstract

Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals. However, existing conversational systems have mainly focused on text or speech, leaving out the visual feedback that is essential for natural interaction. In this paper, we propose JAVIS, an end-to-end streaming framework for real-time audio-visual conversational agents. Our key idea is to let the voice and the face speak from a single representation. On top of a shared LLM, a Joint Refined Head produces a unified audio-visual representation, which is fed into dedicated streaming branches for speech and face generation. This end-to-end design avoids intermediate bottlenecks, enabling real-time generation while keeping speech and face tightly synchronized. In addition, we present JAVIS-Syn, a 334-hour synthetic corpus designed to address the lack of expressive audio-visual dialogue data, covering emotionally rich multi-turn conversations and assistant-style exchanges. Experiments show that JAVIS achieves approximately 200 $\times$ lower initial latency than a strong baseline, while outperforming baselines in lip synchronization and facial expression appropriateness.

Figures & tables

Figure 1: Overview of JAVIS. Given user speech, JAVIS jointly generates an audio-visual response in real time.

Figure 2: Network architecture. The Joint Refined Head produces a unified audio-visual representation $H_{t}$ , which is fed into parallel speech and motion branches for streaming generation. The speech branch samples five speech tokens $S_{t}$ from $H_{t}$ , while the Motion Decoder denoises $X_{\tau}$ at timestep $\tau$ to generate the five-frame motion chunk $M_{t}$ , conditioned on $H_{t}$ , $H_{t+1}$ , the previous motion chunk $M_{t-1}$ , and the reference facial state $g$ .

Method	AV sync		Video quality			Speech quality
Method	LSE-C $\uparrow$	LSE-D $\downarrow$	IQ $\uparrow$	SC $\uparrow$	MS $\uparrow$	WER (%) $\downarrow$	UTMOS $\uparrow$	SpkSIM $\uparrow$
FAC (s2s) + FLOAT	7.94	8.26	0.743	0.989	0.9928	0.41	4.20	0.689
FAC (s2s) + InfiniteTalk	9.59	6.86	0.754	0.991	0.9936	0.41	4.20	0.689
FAC (s2t) + Talker-T2AV	8.72	7.04	0.738	0.994	0.9954	0.54	3.79	0.575
Ex-Omni-2D	5.24	11.06	0.734	0.991	0.9947	0.31	4.32	0.525
JAVIS	9.59	6.74	0.744	0.993	0.9935	0.44	4.29	0.701

Table 1: Objective quality of audio-visual responses, evaluated in terms of audio-visual synchronization, video quality, and speech quality. Arrows indicate the preferred direction. Best and second-best results are shown in bold and underlined , respectively.

System	Latency (s) $\downarrow$	RTF $\downarrow$	Len. (s)
FAC + FLOAT	6.73	0.569	11.8
FAC + FLOAT (2 s chunks)	2.09	0.560	11.8
FAC + InfiniteTalk	252.0	20.9	11.8
FAC + Talker-T2AV	113.3	10.8	10.5
Ex-Omni-2D	1210.7	116.6	10.5
JAVIS	1.27	0.541	12.1

Table 3: Streaming latency on two RTX 4090 GPUs, averaged over five queries. Latency: time until both the first audio and video packets are available. RTF: wall-clock time divided by the length of the generated response. Len.: the length of the generated response.

Method	AV	Motion	Facial Expression
Method	Sync $\uparrow$	Naturalness $\uparrow$	Appropriateness $\uparrow$
FAC + FLOAT	3.70 $\pm$ 0.15	3.50 $\pm$ 0.15	3.25 $\pm$ 0.15
FAC + InfiniteTalk	4.25 $\pm$ 0.11	4.02 $\pm$ 0.11	3.44 $\pm$ 0.14
FAC + Talker-T2AV	3.87 $\pm$ 0.13	2.13 $\pm$ 0.12	1.78 $\pm$ 0.11
Ex-Omni-2D	1.93 $\pm$ 0.13	2.19 $\pm$ 0.13	2.15 $\pm$ 0.14
JAVIS	4.37 $\pm$ 0.10	3.92 $\pm$ 0.13	3.76 $\pm$ 0.16

Table 4: Subjective evaluation results. The results are presented with 95% confidence intervals.

Model	IQ $\uparrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	WER (%) $\downarrow$	UTMOS $\uparrow$
JAVIS	0.744	9.59	6.74	0.44	4.29
w/o lookahead	0.744	8.80	7.55	0.37	4.31
w/o JAVIS-Syn	0.737	6.71	8.72	1.21	4.03
Stage 1 only	0.734	6.17	8.79	13.91	3.74

Table 5: Ablation study of JAVIS.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 154 papers

JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
Feb 22, 2026Kai Liu, Yanhao Zheng, Kai Wang +7Audio-Video Generation Audio-Visual Alignment
Zhejiang University · University of Toronto · National University of Singapore +3
EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents
Apr 19, 2026Yu Zhang, Kaiyuan Shen, Yang LiStreaming Video Generation Talking Head Generation
School of Computer Science and Technology, East China Normal University, Shanghai, China · Garabido Shanghai Technology Co., Ltd., Shanghai, China
OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
Jul 25, 2026Quanyue Song, Yishan He, Yanbo Ding +4Audio-Video Generation Real-Time Interactive Video Generation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · Shenzhen Key Laboratory of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

Feb 22, 2026J/K move · Enter open · S save

JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation

Kai Liu, Yanhao Zheng, Kai Wang, Shengqiong Wu+6

Zhejiang University · University of Toronto · National University of Singapore+3

Audio-Video Generation Audio-Visual Alignment Temporal Consistency in Video Generation Direct Preference Optimization

Open paper AskPDF arXiv

AIGC has rapidly expanded from text-to-image generation toward high-quality multimodal synthesis across video and audio. Within this context, joint audio-video generation (JAVG) has emerged as a fundamental task that produces synchronized and semantically aligned sound and vision from textual descriptions. However, compared with advanced commercial models such as Veo3, existing open-source methods still suffer from limitations in generation quality, temporal synchrony, and alignment with human preferences. To bridge the gap, this paper presents JavisDiT++, a concise yet powerful framework for unified modeling and optimization of JAVG. First, we introduce a modality-specific mixture-of-experts (MS-MoE) design that enables cross-modal interaction efficacy while enhancing single-modal generation quality. Then, we propose a temporal-aligned RoPE (TA-RoPE) strategy to achieve explicit, frame-level synchronization between audio and video tokens. Besides, we develop an audio-video direct preference optimization (AV-DPO) method to align model outputs with human preference across quality, consistency, and synchrony dimensions. Built upon Wan2.1-1.3B-T2V, our model achieves state-of-the-art performance merely with around 1M public training entries, significantly outperforming prior approaches in both qualitative and quantitative evaluations. Comprehensive ablation studies have been conducted to validate the effectiveness of our proposed modules. All the code, model, and dataset are released at https://JavisVerse.github.io/JavisDiT2-page.

Abstract

Audio-Video Generation

Explore similar work

JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation

EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents

OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars