Talking-head videos constitute a predominant content type in real-time communication, yet publicly available datasets for video processing research in this domain remain scarce and limited in signal fidelity. In this paper, we open-source a camera-native dataset of 838 talking-head recordings (approximately 210 minutes), each 15s in duration, captured from 799 participants across 443 camera-code categories in their natural environments. All recordings are stored using the FFV1 lossless codec, preserving the camera-native signal---uncompressed (24.7%) or MJPEG-encoded (75.3%)---without additional lossy processing. Each recording is annotated with a Mean Opinion Score (MOS) and ten perceptual quality tokens that jointly explain 64.4% of the MOS variance. From this corpus, we curate a stratified benchmarking subset of 120 clips in three content conditions: original, background blur, and background replacement. Codec efficiency evaluation across four datasets and four codecs, namely H.264, H.265, H.266, and AV1, yields VMAF BD-rate savings up to −71.3 (H.266) relative to H.264, with significant encoder×dataset (ηp2=.112) and encoder×content condition (ηp2=.149) interactions, demonstrating that both content type and background processing affect compression efficiency. A preliminary super-resolution evaluation with four SR models confirms that the dataset significantly affects absolute performance while preserving model rankings, demonstrating applicability beyond codec benchmarking. The dataset offers 5× the scale of the largest prior talking-head webcam dataset (838 vs. 160 clips) and preserves the camera-native signal without additional lossy compression, establishing a resource for benchmarking video compression, super-resolution, quality assessment, and enhancement models in real-time communication.
Figures & tables
Resolution
Uncompressed
MJPEG
Total
720p
10.9%
49.9%
60.7%
1080p
11.2%
21.8%
33.1%
1440p or larger
2.6%
3.6%
6.2%
Total
24.7%
75.3%
100%
TABLE I : Distribution of recordings by resolution and input pixel format (% of 838 published clips). Uncompressed combines YUYV422 and NV12 formats. Percentages are independently rounded and may not sum exactly.
Token
Description
Sel. (%)
Noisy
Random pixel-level luminance/chrominance fluctuations (sensor noise)
Over-/under-exposure, direct glare, white balance deviation, color cast
36.6
No issue
No perceptible quality degradation
29.2
Blurry
Loss of edge sharpness; defocus or motion blur
13.6
Choppy motion
Irregular temporal motion; stuttering or judder
6.0
TABLE II : Perceptual quality tokens and their selection rates across the 838 published clips. A token is considered selected when ≥ 2 assessors independently chose it.
Fig. 1 : Cumulative distribution function of MOS values across the 838 published clips.
Fig. 2 : Thumbnail atlases of the three benchmarking groups. (a) Talking Head, (b) Talking Head - Background Blur, (c) Talking Head - Background Replacement
Fig. 3 : Distribution of clips in the SI–TI space for each benchmarking group, color-coded by MOS category (green: High, blue: Medium, red: Low). Dashed lines indicate population medians.
Token
TH
TH-BB
TH-BR
Noise
62
70
55
Low resolution
45
50
50
Lighting/color
45
42
42
No issue
22
28
40
Blur
15
10
10
Choppy motion
5
2
2
TABLE III : Problem token selection rates (% of clips) per benchmarking group. A token is selected when ≥ 2 assessors chose it.
N
VMAF BD-Rate (%)
PSNR BD-Rate (%)
Dataset
Clips
H.265
AV1
H.266
H.265
AV1
H.266
HEVC [ 4 ]
25
−34.9 (3.0)
−32.6 (4.1)
−65.0 (3.9)
−34.0 (3.0)
−36.1 (5.1)
−64.5 (4.2)
UVG [ 6 ]
16
−34.5 (5.2)
−40.0 (9.9)
−70.4 (7.5)
−42.5 (6.1)
−48.2 (9.4)
−72.6 (7.5)
VCD [ 2 ]
120
−27.9 (2.3)
−36.3 (1.7)
−63.7 (1.8)
−32.1 (1.4)
−40.0 (1.8)
−62.5 (1.6)
VCD2 (ours)
120
−25.9 (2.7)
−42.2 (2.7)
−71.3 (1.7)
−36.0 (1.9)
−52.8 (2.4)
−70.2 (1.5)
VCD2 + VP8
120
−25.6 (2.1)
−35.1 (1.4)
−62.9 (1.9)
−35.0 (1.3)
−38.0 (1.3)
−61.4 (1.5)
TABLE IV : Mean BD-rate (%) relative to H.264 baseline. 95% confidence intervals are shown in parentheses. More negative values indicate greater bitrate savings. VCD2 + VP8 denotes VCD2 content pre-encoded with VP8 to simulate WebRTC capture conditions.
Fig. 4 : Rate–distortion curves for the VCD2 benchmarking subset ( 4(a) – 4(b) ) and VMAF rate–distortion curves for VCD ( 4(c) ) and HEVC ( 4(d) ) datasets. Shaded bands indicate 95% confidence intervals.
Dataset
RealSR
SwinIR
RealVif.
BVSR++
REDS
61.4 (3.6)
72.6 (2.7)
37.5 (6.2)
71.5 (2.8)
VCD
72.6 (3.5)
78.6 (3.1)
51.9 (4.6)
77.9 (3.0)
VCD2 (ours)
79.7 (1.5)
90.9 (1.2)
58.8 (2.5)
90.1 (1.4)
VCD2 + VP8
77.1 (1.6)
84.0 (1.5)
55.7 (2.7)
83.7 (1.6)
TABLE V : Mean VMAF after 4× SR (180p → 720p) with 95% CI. Best model per dataset in bold.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Value
Published clips
838
Unique participants
799
Camera-code categories †
443
Duration per clip
15 s
Codec
FFV1 (lossless)
Container
Matroska (.mkv)
Appendix
TABLE VI : Full statistics of the 838-clip published dataset.
Metric
Group
H.265
AV1
H.266
VMAF
TH
−24.3±5.0
−30.0±3.2
−71.6±2.4
TH-BB
−24.0±4.5
−37.9±1.6
−65.2±3.3
TH-BR
−29.5±4.6
−58.9±3.1
−77.1±2.0
PSNR
TH
−39.4±3.4
−45.4±3.2
−69.5±2.9
TH-BB
−39.1±2.3
−46.3±2.2
−66.5±2.4
TH-BR
−29.4±3.2
−66.5±2.7
−74.6±1.6
Appendix
TABLE VII : Mean BD-rate (%) relative to H.264 baseline per benchmarking group, with 95% t -based confidence intervals computed over clips within each group. VMAF and PSNR metrics are reported separately.
Fig. 5 : Rate–distortion curves per benchmarking group. Top row: PSNR vs. bpp; bottom row: VMAF vs. bpp. Each curve shows the mean metric value across clips in the group.
Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.
Videocalling has become a popular form of communication in the world today, with many companies providing free services for it. However, there are still millions of people around the world that experience poor quality videocalls due to limitations in bandwidth. This despite, most people having the required hardware. In this paper we present a novel framework for enhancing highly compressed videocalls. We show, that with as little as 10 frames of the face, we can rapidly (in under 100 seconds) train a model to enhance that instance of the videocall. The model can be trained either prior to or during the call, enhancing the rest of the call by producing better quality video. The video conferencing application need not be modified - it can be off the shelf with our system as a layer on top that trains quickly then simply lets the video conferencing application (e.g. Zoom) run as usual, where our system intercepts and improves images before they are displayed. The model is designed to run in realtime on low-compute devices such as a typical laptop CPU. Experimentally, we show that the model significantly improves quality of compressed face video both quantitatively as well as perceptually. Code can be found at https://github.com/varun-jois/FSFVE.
Varun Ramesh Jois, Antonella DiLillo, James Storer
Long-form and real-time talking-head generation remains challenging due to a latency-quality trade-off: inefficient multi-step diffusion prohibits streaming generation, whereas real-time autoregressive approaches suffer from error accumulation and identity drift. To address this drawback, we propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step, scaling to arbitrarily long videos. At the heart of our approach lies a single-step bridge distillation scheme. On the one hand, departing from the conventional noise-to-data paradigm, we introduce a data-to-data transport formulation based on a Brownian bridge. Anchored by a persistent reference, this strategy effectively mitigates identity drift and enhances long-term temporal stability. On the other hand, to enable smooth knowledge transfer from a pre-trained diffusion teacher to the student bridge model, we explore a heterogeneous distillation framework with an SNR-aligned time transformation Φ(τ), which bridges the functional discrepancy between the two models. Moreover, we propose an audio-driven classifier-free guidance mechanism to maintain fine-grained lip synchronization under extreme step reduction. Extensive experiments demonstrate that our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS, significantly outperforming existing approaches in both efficiency and stability. Project Page: https://zhangrongxiang.github.io/leaptalk-page/
Rongxiang Zhang, Songhua Liu
School of Artificial Intelligence, Shanghai Jiao Tong University · 2Harbin Institute of Technology
Video traffic constitutes a significant share of global web traffic. To reduce its volume, video codecs have been developed and continuously improved. While the industry has achieved substantial progress in traditional video coding, neural video codecs (NVCs) have recently emerged as a new approach that applies deep learning to video compression. This creates new challenges for compression quality assessment, which is essential for the further development and improvement of such codecs. In particular, it is important to evaluate the novel temporal compression paradigms introduced by NVCs. In this work, we present a large-scale subjective dataset of videos compressed with both neural and traditional video codecs. The subjective scores were collected through crowd-sourced pairwise comparisons. The proposed dataset provides a valuable resource for the development and benchmarking of video quality metrics tailored to neural video codecs. The dataset is available at the following link: https://videoprocessing.github.io/nvc-dataset-benchmark
Nikolay Safonov, Nikita Gornostaev, Alexandra Dubonos +1
AI Center, Lomonosov Moscow State University Moscow, Russia · MSU Institute for Artificial Intelligence Moscow, Russia · Lomonosov Moscow State University Moscow, Russia