A Camera-Native Talking-Head Video Dataset for Various Computer Vision Tasks

Babak Naderi; Ross Cutler; Nabakumar Singh Khongbantabam

cs.CVMar 23, 2026

A Camera-Native Talking-Head Video Dataset for Various Computer Vision Tasks

Authors: Babak Naderi, Ross Cutler, Nabakumar Singh Khongbantabam

Organizations: Microsoft Corporation, Redmond, USA

Abstract

Talking-head videos constitute a predominant content type in real-time communication, yet publicly available datasets for video processing research in this domain remain scarce and limited in signal fidelity. In this paper, we open-source a camera-native dataset of 838 talking-head recordings (approximately 210 minutes), each 15s in duration, captured from 799 participants across 443 camera-code categories in their natural environments. All recordings are stored using the FFV1 lossless codec, preserving the camera-native signal---uncompressed (24.7%) or MJPEG-encoded (75.3%)---without additional lossy processing. Each recording is annotated with a Mean Opinion Score (MOS) and ten perceptual quality tokens that jointly explain 64.4% of the MOS variance. From this corpus, we curate a stratified benchmarking subset of 120 clips in three content conditions: original, background blur, and background replacement. Codec efficiency evaluation across four datasets and four codecs, namely H.264, H.265, H.266, and AV1, yields VMAF BD-rate savings up to $-71.3%$ (H.266) relative to H.264, with significant encoder $\times$ dataset ( $η_p^2 = .112$ ) and encoder $\times$ content condition ( $η_p^2 = .149$ ) interactions, demonstrating that both content type and background processing affect compression efficiency. A preliminary super-resolution evaluation with four SR models confirms that the dataset significantly affects absolute performance while preserving model rankings, demonstrating applicability beyond codec benchmarking. The dataset offers 5 $\times$ the scale of the largest prior talking-head webcam dataset (838 vs. 160 clips) and preserves the camera-native signal without additional lossy compression, establishing a resource for benchmarking video compression, super-resolution, quality assessment, and enhancement models in real-time communication.

Figures & tables

Resolution	Uncompressed	MJPEG	Total
720p	10.9%	49.9%	60.7%
1080p	11.2%	21.8%	33.1%
1440p or larger	2.6%	3.6%	6.2%
Total	24.7%	75.3%	100%

TABLE I : Distribution of recordings by resolution and input pixel format (% of 838 published clips). Uncompressed combines YUYV422 and NV12 formats. Percentages are independently rounded and may not sum exactly.

Token	Description	Sel. (%)
Noisy	Random pixel-level luminance/chrominance fluctuations (sensor noise)	69.8
Low resolution	Insufficient spatial detail; overall poor definition	61.3
Lighting/color	Over-/under-exposure, direct glare, white balance deviation, color cast	36.6
No issue	No perceptible quality degradation	29.2
Blurry	Loss of edge sharpness; defocus or motion blur	13.6
Choppy motion	Irregular temporal motion; stuttering or judder	6.0

TABLE II : Perceptual quality tokens and their selection rates across the 838 published clips. A token is considered selected when

\geq

2 assessors independently chose it.

Fig. 1 : Cumulative distribution function of MOS values across the 838 published clips.

Panel of Figure 4 — Fig. 2 : Thumbnail atlases of the three benchmarking groups. (a) Talking Head, (b) Talking Head - Background Blur, (c) Talking Head - Background Replacement

Panel of Figure 5 — Fig. 3 : Distribution of clips in the SI–TI space for each benchmarking group, color-coded by MOS category (green: High, blue: Medium, red: Low). Dashed lines indicate population medians.

Token	TH	TH-BB	TH-BR
Noise	62	70	55
Low resolution	45	50	50
Lighting/color	45	42	42
No issue	22	28	40
Blur	15	10	10
Choppy motion	5	2	2

TABLE III : Problem token selection rates (% of clips) per benchmarking group. A token is selected when

\geq

2 assessors chose it.

	$N$	VMAF BD-Rate (%)			PSNR BD-Rate (%)
Dataset	Clips	H.265	AV1	H.266	H.265	AV1	H.266
HEVC [ 4 ]	25	$-34.9$ (3.0)	$-32.6$ (4.1)	$-65.0$ (3.9)	$-34.0$ (3.0)	$-36.1$ (5.1)	$-64.5$ (4.2)
UVG [ 6 ]	16	$-34.5$ (5.2)	$-40.0$ (9.9)	$-70.4$ (7.5)	$-42.5$ (6.1)	$-48.2$ (9.4)	$-72.6$ (7.5)
VCD [ 2 ]	120	$-27.9$ (2.3)	$-36.3$ (1.7)	$-63.7$ (1.8)	$-32.1$ (1.4)	$-40.0$ (1.8)	$-62.5$ (1.6)
VCD2 (ours)	120	$-25.9$ (2.7)	$-42.2$ (2.7)	$-71.3$ (1.7)	$-36.0$ (1.9)	$-52.8$ (2.4)	$-70.2$ (1.5)
VCD2 + VP8	120	$-25.6$ (2.1)	$-35.1$ (1.4)	$-62.9$ (1.9)	$-35.0$ (1.3)	$-38.0$ (1.3)	$-61.4$ (1.5)

TABLE IV : Mean BD-rate (%) relative to H.264 baseline. 95% confidence intervals are shown in parentheses. More negative values indicate greater bitrate savings. VCD2 + VP8 denotes VCD2 content pre-encoded with VP8 to simulate WebRTC capture conditions.

Panel of Figure 8 — Fig. 4 : Rate–distortion curves for the VCD2 benchmarking subset ( 4(a) – 4(b) ) and VMAF rate–distortion curves for VCD ( 4(c) ) and HEVC ( 4(d) ) datasets. Shaded bands indicate 95% confidence intervals.

Dataset	RealSR	SwinIR	RealVif.	BVSR++
REDS	$61.4$ (3.6)	$\mathbf{72.6}$ (2.7)	$37.5$ (6.2)	$71.5$ (2.8)
VCD	$72.6$ (3.5)	$\mathbf{78.6}$ (3.1)	$51.9$ (4.6)	$77.9$ (3.0)
VCD2 (ours)	$79.7$ (1.5)	$\mathbf{90.9}$ (1.2)	$58.8$ (2.5)	$90.1$ (1.4)
VCD2 + VP8	$77.1$ (1.6)	$\mathbf{84.0}$ (1.5)	$55.7$ (2.7)	$83.7$ (1.6)

TABLE V : Mean VMAF after

4\times

SR (180p

\to

720p) with 95% CI. Best model per dataset in bold.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Value
Published clips	838
Unique participants	799
Camera-code categories $\dagger$	443
Duration per clip	15 s
Codec	FFV1 (lossless)
Container	Matroska (.mkv)

Appendix

TABLE VI : Full statistics of the 838-clip published dataset.

Metric	Group	H.265	AV1	H.266
VMAF	TH	$-24.3\pm 5.0$	$-30.0\pm 3.2$	$-71.6\pm 2.4$
	TH-BB	$-24.0\pm 4.5$	$-37.9\pm 1.6$	$-65.2\pm 3.3$
	TH-BR	$-29.5\pm 4.6$	$-58.9\pm 3.1$	$-77.1\pm 2.0$
PSNR	TH	$-39.4\pm 3.4$	$-45.4\pm 3.2$	$-69.5\pm 2.9$
	TH-BB	$-39.1\pm 2.3$	$-46.3\pm 2.2$	$-66.5\pm 2.4$
	TH-BR	$-29.4\pm 3.2$	$-66.5\pm 2.7$	$-74.6\pm 1.6$

Appendix

TABLE VII : Mean BD-rate (%) relative to H.264 baseline per benchmarking group, with 95%

t

-based confidence intervals computed over clips within each group. VMAF and PSNR metrics are reported separately.

Panel of Figure 12 — Fig. 5 : Rate–distortion curves per benchmarking group. Top row: PSNR vs. bpp; bottom row: VMAF vs. bpp. Each curve shows the mean metric value across clips in the group.

Encoder	Command
H.264	ffmpeg -init_hw_device qsv=hw -filter_hw_device hw
(Intel QSV)	-i {input} -c:v h264_qsv -load_plugin h264_hw
	-scenario 2 -p_strategy 0 -b_strategy 0 -threads 1
	-sc_threshold 0 -preset fast -bf 0 -look_ahead 0
	-g 6000 -i_qfactor 1.0 -i_qoffset 0 -b_qfactor 1.0
	-b_qoffset 0 -refs 1 -low_power 1 -q {qp} {output}

Appendix

TABLE VIII : Encoder configurations used in the codec benchmarking experiments.

Video Compression

Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.

See all 73 papers

FSFVE: Few Shot Compressed Face Video Enhancement
Jul 13, 2026Varun Ramesh Jois, Antonella DiLillo, James StorerFew-Shot Learning
Brandeis University · Waltham, MA, USA
LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation
Jul 29, 2026Rongxiang Zhang, Songhua LiuAudio-Video Generation Diffusion Model Distillation
School of Artificial Intelligence, Shanghai Jiao Tong University · 2Harbin Institute of Technology
Neural video codecs quality assessment dataset and benchmark
Aug 29, 2026Nikolay Safonov, Nikita Gornostaev, Alexandra Dubonos +1Video Quality Assessment Neural Video Compression
AI Center, Lomonosov Moscow State University Moscow, Russia · MSU Institute for Artificial Intelligence Moscow, Russia · Lomonosov Moscow State University Moscow, Russia

Jul 13, 2026J/K move · Enter open · S save

FSFVE: Few Shot Compressed Face Video Enhancement

Varun Ramesh Jois, Antonella DiLillo, James Storer

Brandeis University · Waltham, MA, USA

Few-Shot Learning

Open paper AskPDF arXiv

Videocalling has become a popular form of communication in the world today, with many companies providing free services for it. However, there are still millions of people around the world that experience poor quality videocalls due to limitations in bandwidth. This despite, most people having the required hardware. In this paper we present a novel framework for enhancing highly compressed videocalls. We show, that with as little as 10 frames of the face, we can rapidly (in under 100 seconds) train a model to enhance that instance of the videocall. The model can be trained either prior to or during the call, enhancing the rest of the call by producing better quality video. The video conferencing application need not be modified - it can be off the shelf with our system as a layer on top that trains quickly then simply lets the video conferencing application (e.g. Zoom) run as usual, where our system intercepts and improves images before they are displayed. The model is designed to run in realtime on low-compute devices such as a typical laptop CPU. Experimentally, we show that the model significantly improves quality of compressed face video both quantitatively as well as perceptually. Code can be found at https://github.com/varun-jois/FSFVE.

Abstract

Video Compression

Explore similar work

FSFVE: Few Shot Compressed Face Video Enhancement

LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

Neural video codecs quality assessment dataset and benchmark