VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes

Yan Ma; Jiadi Su; Zhulin Hu; Ethan Chern; Linhao Zhang; Tiantian Mi; Pengfei Liu

cs.CVSep 6, 2026

VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes

Authors: Yan Ma, Jiadi Su, Zhulin Hu, Ethan Chern, Linhao Zhang, Tiantian Mi, Pengfei Liu

Organizations: Fudan University · Generative Artificial Intelligence Research Lab (GAIR) · Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai University

Abstract

Video foundation models increasingly rely on large-scale pretraining data, yet the end-to-end data pipelines behind them remain largely closed and difficult to inspect or reuse. Researchers seeking to understand how video data recipes affect model pretraining often need to build substantial infrastructure before testing even a focused hypothesis. We present VIDAFORGE, an open research infrastructure that represents a video data recipe as an executable five-stage workflow from raw videos to training datasets. A decision in this workflow can be varied to construct alternative datasets while preserving how every sample was produced. To demon strate this research workflow, we compare data recipes with different coverage and quality in early from-scratch pretraining of Wan 2.1 and V-JEPA 2.1. Across both learning objectives, the broader-coverage recipe achieves the highest downstream benchmark scores, while loss-based evaluation favors different recipes. This study demonstrates how VidaForge connects data-recipe choices to downstream model performance. We further release VIDAFORGE-3M, containing 3.14 million scene level clips totaling 6,475 hours, with fine-grained annotations and curation signals for video data-recipe research.

Figures & tables

Figure 1: Publicly reported detail on data preparation and processing for recent video and visual foundation models, measured by approximate word count.

Figure 2: VidaForge overview. The top row follows six data states from raw videos $\mathcal{D}_{0}$ to target-specific training datasets $\mathcal{D}_{5}$ . Five stages implement the transitions between these states, while the bottom row shows the execution patterns used by their processing steps.

Figure 3: VidaForge data flow for the coverage–quality study. Blue marks clips admitted by Selection; gray branches show clips rejected by individual filters or deduplication thresholds. Each rejected clip is counted under the first rule it fails. The two outcomes form the Selected and Rejected pools.

Panel of Figure 4 — Figure 4: Quality and coverage of the training recipes. Left: average optical, motion, aesthetic, and visible-text scores, together with the within-dataset redundancy measured using PDQ ( Meta, 2019 ) and Cosmos-Embed ( NVIDIA et al., 2025 ) groups, for all four recipes. Random values are averaged over 3 sampled subsets. Right: a t-SNE map of semantic video embeddings generated by Cosmos-Embed, sampled from the candidate pool. Blue regions contain more Selected clips and red regions contain more Rejected clips; $\text{Mixed}=\text{Selected}\cup\text{Rejected}$ covers both.

	Wan 2.1-1.3B: VBench			V-JEPA 2-1B
Recipe	Quality	Semantic	Total	SSv2 (%)	Kinetics-400 (%)
Mixed	$\mathbf{71.65}\pm 0.74$	$\mathbf{16.60}\pm 1.25$	$\mathbf{60.64}\pm 0.83$	$\mathbf{16.256}\pm 0.153$	$\mathbf{27.218}$
Selected	$70.61\pm 0.51$	$16.28\pm 0.81$	$59.74\pm 0.55$	$15.712\pm 0.382$	$26.880$
Random	$70.25\pm 0.80$	$16.43\pm 0.76$	$59.48\pm 0.67$	$15.311\pm 0.194$	$25.519$
Rejected	$70.25\pm 1.19$	$16.08\pm 0.89$	$59.42\pm 1.06$	$15.099\pm 0.338$	$25.776$

Table 1: Final downstream scores under the fixed training budget in each model family. VBench and SSv2 report mean

\pm

std over 3 runs; Kinetics-400 reports one run per recipe. Higher is better.

Figure 5: Wan 2.1-1.3B loss curves for all four recipes across 3 runs (mean $\pm$ std). Left: training loss. Right: loss on the Selected, Mixed, and Rejected validation sets. Rejected has the lowest final mean training loss; Selected has the lowest final mean validation loss.

Figure 6: V-JEPA 2 loss curves across 3 runs (mean

\pm

std). Left: training loss. Right: loss on the Selected, Mixed, and Rejected validation sets. Rejected has the lowest final mean training and validation losses. An enlarged training curve appears in Fig. 17 .

Figure 7: Processing time as CPU or GPU resources increase. Transcode and scene detection process 1,000 videos; aesthetic scoring and captioning process 10,000 clips. Lower is better. Crossed legend entries indicate framework–operator combinations not included in this comparison. Each point averages 3 runs under the configurations described in the text.

	Data-Juicer	NeMo Curator	Cosmos Curator	VidaForge
Video data processing
Multi-node processing	✓	✓	✓	✓
Scene clipping	✓	✓	✓	✓
Quality filtering	✓	✓	✓	✓
Perceptual near-duplicates	—	—	—	✓
Semantic duplicates	—	✓	✓	✓

Table 2: Support for video processing and data-recipe research in released workflows. ✓ indicates that the full row definition is met; — indicates that it is not, including partial support. Official extensions are included; Data-Juicer includes Sandbox ( Chen et al., 2025b ) .

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Fields	Allowed labels
Motion type	motion_type	no-motion , minor-motion , simple-motion , complex-motion
Steadiness	steadiness	static , no-shaking , minimal-shaking , unsteady , very-unsteady
Rotation	pan , tilt , roll	pan-left/right/no-pan ; tilt-up/down/no-tilt ; roll-CW/CCW/no-roll
Translation	dolly , pedestal , truck	dolly-in/out/no-dolly ; pedestal-up/down/no-pedestal ; truck-left/right/no-truck
Intrinsic	zoom	zoom-in , zoom-out , no-zoom
Object-centric	arc	arc-CW , arc-CCW , no-arc

Appendix

Table 3: Camera schema. Nested groups contain the listed leaf fields. Every leaf field also permits unknown .

Field	Target length	Required content
level_0	$<20$	Main subject, primary action, and scene; camera motion and audio are omitted.
level_1	20–50 words	Concise event flow, important subjects and changes, noticeable camera motion, and a summary of important audio when available.
level_2	50–100 words	Initial state, temporal progression, subject–object relations, position changes, background, main camera motion, and summarized speech or other relevant audio.
level_3	100–200 words	Dense reconstruction of subjects, attributes, actions, environment, lighting, style, composition, temporal changes, camera behavior, visible text, and relevant audio. In video_audio mode, clearly audible speech is included.

Appendix

Table 4: Caption schema. Target lengths summarize the intended granularity of the four caption levels, informed by observed caption lengths.

Field	Allowed labels
domain	real_world , animation , game , screen_recording , synthetic_render , mixed , unknown
scene	general_indoor , general_outdoor , urban , nature , driving , sports , food , product , portrait , screen , other , unknown
subjects	person , vehicle , animal , object , food , landscape , building , text , screen , robot , other , unknown
actions	talking , locomotion , driving , sports , cooking , object_manipulation , natural_motion , camera_motion_only , timelapse , none , other , unknown
style	photorealistic , cinematic , documentary , anime , cartoon , cg_render , gameplay , graphic , unknown
text	none , incidental , subtitle , screen_ui , document , signage , overlay_text , unknown

Appendix

Table 5: Tag schema. subjects and actions are multi-label fields; all other fields are single-label.

Property	Value
Source videos processed	800,000
Released scene-level clips	3,141,246
Total duration	6,475.1 hours
Mean / median clip duration	7.42 / 9.70 seconds
Clip duration range	1.0–10.0 seconds
Video format	H.265/MP4

Appendix

Table 6: VidaForge-3M at a glance. Statistics are computed over all 3,141,246 released clips.

Stage	Step	Input	Success / fail	Time	Throughput	Hardware
I	Transcode	800k videos	800,000 / 0	5h00m	44.36 videos/s	800 CPU cores
II	Detect	800k videos	800,000 / 0	5h27m	40.74 videos/s	800 CPU cores
II	Clip	800k videos	3,144,490 / 0	12h42m	68.71 clips/s	800 CPU cores
III	Context	3.144M clips	3,144,490 / 0	54m38s	959.34 clips/s	800 CPU cores
III	Optical + Motion	3.144M clips	3,144,490 / 0	1h35m	550.13 clips/s	800 CPU cores
III	Aesthetic	3.144M clips	3,144,490 / 0	2h26m	357.15 clips/s	32 H200

Appendix

Table 7: Execution of the VidaForge-3M pipeline. Output entries report successful and failed items. CPU steps use 800 Intel Xeon Gold 6530 cores; GPU steps use NVIDIA H200 GPUs.

Figure 8: Media composition of VidaForge-3M . Left: clip-duration distribution over all released clips. Right: shares of the three most common spatial resolutions, with the remaining resolutions grouped as Other.

Figure 9: Quality-score distributions across all released VidaForge-3M clips. Vertical lines mark the 10th, 50th, and 90th percentiles for each measurement.

Method	Groups	Clips (share)	Size P50/P95
PDQ	74,508	185,387 (5.90%)	2 / 3
Cosmos-Embed	262,498	714,089 (22.73%)	2 / 3

Appendix

Table 8: Non-singleton duplicate-group statistics within VidaForge-3M . Clip shares are computed over the full release; group-size percentiles are computed over non-singleton groups.

Annotation	Released structure	Coverage
Camera	21 structured motion fields	3,141,246 (100%)
Caption	Four levels of increasing detail	3,141,246 (100%)
Tag	Seven semantic taxonomy fields	3,141,246 (100%)

Appendix

Table 9: Annotation coverage in VidaForge-3M . Coverage is computed over all 3,141,246 released clips.

Figure 10: Distributions of two complementary high-level annotations in VidaForge-3M . Steadiness describes camera stability; scene dynamics describes temporal activity in the visible content.

Figure 11: Character-length distributions of the four caption levels in VidaForge-3M . Vertical lines mark the 10th, 50th, and 90th percentiles.

Figure 12: Selected Tag distributions in VidaForge-3M . Scene, Text, and Watermark are single-label fields; Subjects and Actions are multi-label fields, so percentages within either multi-label panel need not sum to 100%.

Figure 13: Distributions of the four selection scores in the final Mixed, Selected, and Rejected training datasets. Each histogram is normalized by the number of valid clips in its dataset. Optical and text scores equal to 1.0 are omitted from the histogram bins and reported separately inside their panels; they remain in the normalization denominator. The visible-text panel focuses on scores

\geq 0.9

for readability.

Figure 14: PDQ and semantic redundancy by within-dataset group size. A group of size

n

contributes

n-1

redundant clips; each bar is reported as a percentage of the corresponding final training dataset.

Model	Experiment	Count
Wan + V-JEPA 2	From-scratch pretraining	24 runs
Wan	Validation loss	252 evaluations
V-JEPA 2	Validation loss	720 evaluations
Wan	VBench	36 evaluations
V-JEPA 2	SSv2	60 evaluations
V-JEPA 2	Kinetics-400	4 evaluations

Appendix

Table 10: Training and evaluation counts for the reported study. Each loss evaluation uses one checkpoint and one validation set.

Figure 15: Wan 2.1 flow-matching training loss for all four recipes across 3 runs (mean $\pm$ std). The inset enlarges the later training phase.

Figure 16: VBench dimension scores for Wan 2.1-1.3B, averaged over 3 runs. Left: Quality dimensions. Right: Semantic dimensions. Axes are rescaled to highlight differences; Tab. 13 gives per-dimension mean

\pm

std.

Recipe	Quality	Semantic	Total	Total †
Mixed	$\mathbf{71.65}\pm 0.74$	$\mathbf{16.60}\pm 1.25$	$\mathbf{60.64}\pm 0.83$	$\mathbf{64.95}\pm 1.52$
Selected	$70.61\pm 0.51$	$16.28\pm 0.81$	$59.74\pm 0.55$	$64.04\pm 0.74$
Random	$70.25\pm 0.80$	$16.43\pm 0.76$	$59.48\pm 0.67$	$63.90\pm 0.64$
Rejected	$70.25\pm 1.19$	$16.08\pm 0.89$	$59.42\pm 1.06$	$63.73\pm 1.47$

Appendix

Table 11: Final VBench aggregate scores for all four recipes across 3 Wan runs (mean

\pm

std). Total † excludes Dynamic Degree. Higher is better.

Training seed	Mixed	Selected	Rejected
Seed 1	60.95	59.81	59.50
Seed 2	59.69	59.16	58.32
Seed 3	61.27	60.26	60.43

Appendix

Table 12: Final VBench Total for Mixed, Selected, and Rejected under each Wan training seed. Mixed ranks first among these three under all 3 seeds; the ordering of Selected and Rejected varies.

Quality dimensions
Dimension	Mixed	Selected	Random	Rejected
Subject Consistency	$\mathbf{95.58}\pm 2.40$	$94.57\pm 1.73$	$93.66\pm 2.08$	$94.02\pm 2.73$
Background Consistency	$\mathbf{93.73}\pm 0.75$	$93.15\pm 1.58$	$93.25\pm 1.32$	$93.16\pm 1.10$
Temporal Flickering	$95.58\pm 2.38$	$95.04\pm 0.66$	$95.04\pm 0.96$	$\mathbf{96.22}\pm 1.33$
Motion Smoothness	$\mathbf{97.07}\pm 1.96$	$96.61\pm 0.55$	$96.34\pm 1.32$	$96.92\pm 1.36$
Dynamic Degree †	$\mathbf{6.94}\pm 9.63$	$6.17\pm 2.38$	$4.01\pm 1.75$	$5.56\pm 5.26$

Appendix

Table 13: Normalized VBench dimension scores for all four recipes at the final Wan checkpoint across 3 runs (mean

\pm

std). Higher is better; bold marks the highest displayed mean in each row.

Figure 17: V-JEPA 2 masked-prediction training loss for all four recipes across 3 runs. Each run’s curve is first smoothed; solid lines and shaded bands then show mean

\pm

std. Rejected has the lowest final mean loss.

Recipe / comparison	Run 1	Run 2	Run 3	Mean $\pm$ std
Mixed	16.105	16.412	16.250	$16.256\pm 0.153$
Selected	15.387	16.133	15.617	$15.712\pm 0.382$
Random	15.535	15.213	15.185	$15.311\pm 0.194$
Rejected	15.153	15.407	14.737	$15.099\pm 0.338$
Mixed–Selected	$+0.719$	$+0.278$	$+0.633$	$+0.543\pm 0.233$
Mixed–Rejected	$+0.953$	$+1.005$	$+1.513$	$+1.157\pm 0.309$

Appendix

Table 14: Final SSv2 top-1 accuracy (%) and paired gains (percentage points) across 3 runs per recipe (mean

\pm

std). Mixed, Selected, and Rejected use matched training seeds; Random varies both subset sampling and training seed. Paired gains use matched seeds before aggregation; displayed values are rounded.

Figure 18: VBench Total for all four recipes during Wan pretraining (mean

\pm

std across 3 runs). Random varies both subset sampling and training seed; the other recipes vary the training seed.

Figure 19: SSv2 frozen-probe accuracy for all four recipes during V-JEPA 2 pretraining (mean

\pm

std across 3 runs). Random varies both subset sampling and training seed; the other recipes vary the training seed.

Figure 20: Optical filtering. Two clips from the same cooking video illustrate optical filtering. Overexposed regions cause the first clip to fail the 0.9 threshold; the second clip passes.

Figure 21: Motion filtering. Two clips from the same source video illustrate motion filtering. The nearly static clip fails the 0.1 threshold; the clip showing movement of a hand-held tool passes.

Figure 22: Aesthetic filtering. The clip scoring 0.05 is rejected, while the clip scoring 0.30 passes the 0.1 aesthetic threshold.

Figure 23: Visible-text filtering. Plant-related clips illustrate the 0.5 threshold. A higher score indicates a smaller detected text area; the images show each clip’s recorded score and selection outcome.

Figure 24: Perceptual deduplication. Four clips from different source videos show the same rainy street scene. All four pass the quality filters. PDQ identifies them as near-duplicates, and the recipe retains only 1 clip in the group.

Figure 25: Semantic deduplication. Six market clips from different source videos pass the quality filters and PDQ selection. Cosmos-Embed groups them by semantic similarity. Keeping 20% of the group, rounded up, retains 2 clips and removes 4. Each similarity score compares the displayed clip with the clip named beside it.

Panel of Figure 26 — Figure 26: Annotations for a selected clip. A valley landscape is shown with its quality scores, selection outcome, Camera and Tag labels, and four caption levels, from a short summary to a detailed description.

Panel of Figure 27 — Figure 27: Annotations for a rejected clip. An engine-repair clip fails the optical, motion, and aesthetic thresholds, yet retains Camera and Tag labels and all four caption levels for use in alternative recipes.

Explore similar work

Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models
May 18, 2026Svetlana Orlova, Niccolò Cavagnero, Gijs DubbelmanVideo Foundation Models Frozen Vision-Language Models
Eindhoven University of Technology
HumanForge: A Human-Centric Deepfake Video Benchmark with Multi-Agent Forgery Rationales
Jul 9, 2026Wenbo Xu, Zhimin Chen, Xiaojie Liang +2Deepfake Detection Ai-Generated Video Detection
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China
Paris 2.0: A Decentralized Diffusion Model for Video Generation
May 25, 2026Ali Rouzbayani, Bidhan Roy, Marcos Villagra +1Generative Video Models Video Generation
Bagel Labs (bagel.com)

May 18, 2026, cs.CVJ/K move · Enter open · S save

Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

Svetlana Orlova, Niccolò Cavagnero, Gijs Dubbelman

Eindhoven University of Technology

Video Foundation Models Frozen Vision-Language Models Video Compression Spatial Encoding

Open paper AskPDF arXiv

Video foundation models achieve strong performance across many video understanding tasks, but typically require large-scale pre-training on massive video datasets, resulting in substantial data and compute costs. In contrast, modern image foundation models already provide powerful spatial representations. This raises an important question: can competitive video models be built by reusing these spatial representations and pre-training only for temporal reasoning? We take initial steps toward exploring a lightweight training paradigm that freezes a pre-trained image foundation model and trains only a recurrent temporal module to process streaming video. By reusing an image foundation model as a spatial encoder, this approach could significantly reduce the amount of video data and compute required compared to end-to-end video pre-training. In this work, we explore the feasibility of this approach before investing in computing for video pre-training. Our empirical findings across multiple video understanding tasks suggest that strong temporal performance can emerge without large-scale video pre-training, motivating future work on recurrent video foundation models obtained by pre-training a temporal module on top of a frozen image foundation model. Code: https://github.com/tue-mps/towards-video-image-frozen .

Abstract

Explore similar work

Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

HumanForge: A Human-Centric Deepfake Video Benchmark with Multi-Agent Forgery Rationales

Paris 2.0: A Decentralized Diffusion Model for Video Generation