Modeling the reactive tempo of human conversation remains difficult because most audio-visual datasets portray isolated speakers delivering short monologues. We introduce Face-to-Face with Jimmy Fallon (F2F-JF), a 70-hour, 14k-clip dataset of two-person talk-show exchanges that preserves the sequential dependency between a guest turn and the host's response. A semi-automatic pipeline combines multi-person tracking, speech diarization, and lightweight human verification to extract temporally aligned host/guest tracks with tight crops and metadata that are ready for downstream modeling. We showcase the dataset with a reactive, speech-driven digital avatar task in which the host video during [t1,t2] is generated from their audio plus the guest's preceding video during [t0,t1]. Conditioning a MultiTalk-style diffusion model on this cross-person visual context yields small but consistent Emotion-FID and FVD gains while preserving lip-sync quality relative to an audio-only baseline. The dataset, preprocessing recipe, and baseline together provide an end-to-end blueprint for studying dyadic, sequential behavior, which we expand upon throughout the paper. Dataset and code are available at https://face2face2026.github.io.
Figures & tables
Figure 1: Face-to-Face with Jimmy Fallon (F2F-JF) in one look. (a) Raw talk-show frames show the variety of topics, lighting, and guest identities captured with aligned audio. (b) Cropped faces emphasize demographic diversity and the fact that every guest is paired with the same recurring host. (c) Each host–guest turn is trimmed into a guest-context/host-response pair, which becomes the supervised signal for reactive avatar generation. All panels are sampled from different videos with synchronized speech.
Figure 2: Face-to-Face data pipeline. (a) Multi-person tracking slices long episodes into segments that contain exactly two visible people. (b) Speaker diarization finds the recurring host in the audio channel. (c) Vetted host face crops form an embedding reference for visual verification. (d) Frame-level tracking labels the host and guest, producing clean dyadic clips with aligned identities.
Figure 3: Histogram of two-person clip lengths after filtering. Most clips fall between 8 and 20 seconds, which ensures enough temporal context for both the guest turn and the host reaction.
# Clips
Total Duration(hrs)
Average Duration(s)
Resolution
Interactive
14,123
70
14.36 ± 4.64
720p to 1080p
Yes
Table 1: Aggregate statistics for the Face-to-Face with Jimmy Fallon (F2F-JF) dataset. The dataset contains 14,123 high-quality dyadic clips totaling 70 hours of synchronized audio-video.
Figure 4: Sample frames from the Face-to-Face with Jimmy Fallon (F2F-JF) dataset. Each row shows the same host interacting with different guests, highlighting the range of poses, clothing, and conversational topics covered by the dataset.
Figure 5: Paired crops for the reactive avatar task. Each triplet shows (left) the guest providing visual context, (middle) the temporal boundary between turns, and (right) the host response generated from the same clip. The crops stay aligned in time and framing, which allows the model to condition on the guest video before synthesizing the host.
Video CFG
Sync-C ↑
Sync-D ↓
Emotion-FID ↓
FVD ↓
0.00 (equiv. no visual cond.)
11.2144
1.9966
1.8930
115.0814
0.10
11.2153
1.9954
1.8923
115.0790
0.20
11.2152
1.9957
1.8924
115.0822
0.30
11.2151
1.9957
1.8920
115.0740
0.40
11.2150
1.9960
1.8926
115.0844
0.50
11.2151
1.9957
1.8922
115.0793
Table 2: Quantitative effect of varying the video guidance scale (CFG) on the held-out host responses. Higher Sync-C and lower Sync-D indicate better lip-speech alignment; lower Emotion-FID/FVD indicate better video distribution modeling.
Figure 6: Smoothed sweep of the video classifier-free guidance (CFG) scale on the quantitative metrics reported in Table 2 . Sync-C/D benefits slightly from weaker visual conditioning, while Emotion-FID and FVD gain small improvements with stronger video guidance.
Figure 7: Qualitative effect of video guidance. Each row shows the same guest context (left) followed by host responses generated with different video CFG scales. CFG = 0 removes visual conditioning, while CFG = 5 enforces strong use of the guest clip. The comparison highlights how stronger guidance encourages head motion and timing that mirror the guest. Larger versions of this figure with video samples are available in the supplementary material.
Large-scale dyadic interactive audio-visual dialogue (DIAD) datasets provide fundamental data resources for developing humanoid interactive virtual agents and digital humans. However, collecting such data is time-consuming, expensive, and ethically sensitive. To address this, we propose CHAT, a new dyadic interactive audio-visual dialogue generation (DIADG) framework that generates diverse, paired, and mutually responsive speech-face dialogue clips from a single textual prompt. CHAT unifies large language models and talking face models with interactive audio and facial behaviour refinement modules, enabling the generation of aligned dyadic dialogue clips with diverse contents and facial identities. Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations. Moreover, our synthesised CHAT-AVD-50k dataset serves as effective pre-training data for downstream interactive head generation, consistently improving PerFRDiff and ReactDiff on REACT 2024. CHAT offers a scalable alternative to the costly and ethically sensitive collection of real dyadic interaction data.
Junhao Song, Lluis Guasch, Xilin He +8
Department of Computing, Imperial College London, UK · Department of Earth Science & Engineering, Imperial College London, UK · Mohamed bin Zayed University of Artificial Intelligence, UAE +8
Face-to-face audiovisual interaction is central to human communication, conveying rich emotional and social cues. However, existing multimodal dialogue datasets remain limited by inadequate emotion annotations, poor emotional diversity, and small scale. We introduce EmotionDialogCN, a large-scale audiovisual-emotional dataset designed to capture authentic face-to-face communication. It contains 21,880 dialogue sessions performed by 119 professional actors across 20 everyday scenarios, covering 18 emotion categories with over 400 hours of recordings, the largest and most comprehensive dataset of its kind. A novel data collection framework minimizes equipment interference, enabling natural and nuanced emotional expressions. EmotionDialogCN achieves an emotion distribution deviation of 0.64 from real human emotion statistics (versus 5.65 for prior datasets) and consistent subject framing (52-59% frame occupancy). Together, these properties translate into stable unimodal and multimodal performance across acoustic, lexical, and visual modalities, with fusion results further underscoring strong multimodal alignment and cross-modal complementarity.
Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial videos into compact discrete tokens, referred to as face tokens, and conversely reconstructs 3D meshes from these tokens. We then extend Moshi with a Face Transformer module that generates face tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face tokens in real time. Experiments show that Moshi-Face achieves audiovisual alignment at low latency while preserving the dialogue quality of the original audio-only model.