cs.CVFeb 26, 2026
SaveSummer-22B: A Systematic Approach to Dataset Engineering and Training at Scale for Video Foundation Model
Organizations: fal
Abstract
We describe our experience training Summer-22B, a video foundation model developed from scratch. This report documents the engineering challenges, design decisions, and lessons learned while scaling from raw footage collection to a functional model trained on approximately 50 million clips. We outline our approach combining metadata-driven dataset curation, multi-stage filtering, P parameterization, and hypersphere-constrained optimization. We developed the Lavender Data system for dataset management and adopted inference-aware architectural choices. We share observations on what worked in our setting: dataset engineering consumed the majority of effort, architectural variants showed smaller differences than we expected, and P hyperparameter transfer appeared effective even under geometric constraints. We hope this account proves useful to others undertaking similar projects.
Figures & tables
Figure 1: Benchmark results of video decoding libraries across resolutions and codecs (720p H.264, 1080p H.264, 1080p VP9, 2160p VP9). PyNvVideoCodec consistently achieves the highest throughput and was selected as the default decoder in our pipeline.
| Library | Task | CUDA type |
|---|---|---|
| PyNvVideoCodec | decoding | torch.Tensor |
| OpenCV cv2.cudacodec | encoding | cv2.GpuMat |
| RapidOCR | OCR | torch.Tensor |
| OpenCV Farnebäck | Flow estimation | cv2.GpuMat |
| DOVER (ONNX) | Aesthetic scoring | torch.Tensor |
Table 1: GPU matrix types used across libraries in our video preprocessing pipeline. Different libraries operate on different CUDA tensor representations; zero-copy conversion between these types is critical for avoiding costly memory transfers during decoding, encoding, and inference stages.
| Task | Network | CPU | GPU |
|---|---|---|---|
| Footage download | Heavy | Light | – |
| Decoding | – | Medium | Heavy |
| Shot detection (PySceneDetect) | – | Heavy | – |
| Aesthetic scoring | – | Light | Heavy |
| Optical flow | – | Heavy | – |
| Encoding | – | Medium | Heavy |
Table 2: Qualitative resource intensity by task in the preprocessing pipeline. Each stage has a dominant resource bottleneck (network, CPU, or GPU). We exploit this heterogeneity by pipelining stages so that CPU-bound and GPU-bound work overlap, maximizing overall throughput.
Figure 2: Within-node Ray actor design: network I/O overlapped with processing; CPU and GPU jobs overlapped via threads inside the actor.
Figure 3: Cluster scaling via a task distributor front-end dispatching work to many Ray actors across nodes.
Figure 4: Complete video preprocessing pipeline showing the flow from raw footage through segmentation, filtering, and encoding stages. Multiple parallel filters operate on different quality dimensions to ensure only high-quality clips proceed to training.
| Consistency | Uniformity | Preferred | Characteristics |
|---|---|---|---|
| High | High | Bad | Sliding images, static 2D pans |
| High | Low | Good | Complex camera motion, parallax, tracking shots |
| Low | High | Bad | Shaky camera, sudden jumps |
| Low | Low | Neutral | Complex patterns (rain, snow, anime motion) |
Table 3: Classification of video dynamics based on optical flow angle consistency (temporal stability of motion direction) and spatial uniformity (similarity of motion across pixels). The combination of these two metrics discriminates between desirable training content (e.g., parallax and tracking shots) and undesirable artifacts (e.g., sliding images, shaky camera).
Figure 5: Performance profile of the fine-tuned Qwen 2.5 VL captioning model showing inference throughput across different batch sizes and model configurations.
Figure 6: Distribution of ultra-short three-word captions across the dataset, showing the long-tail distribution with approximately 80% of clips having unique captions.
Figure 7: Lavender Data web interface showing dataset visualization.
Figure 8: Lavender Data web interface showing stream-merging architecture. New features can be added to the dataset without requiring re-writing existing dataset.
Figure 9: Lavender Data web interface showing data loading simulation. The system ensures what is displayed matches exactly what the model receives during training.
Figure 11: Empirical validation that P hyperparameter transfer works under hypersphere-constrained Riemannian optimization. Optimal learning rates transfer across model scales (30M to 1B parameters), demonstrating compatibility between P and our geometric constraints.
Figure 12: Comparison of training dynamics. (a) shows weights remaining within the P band with smooth evolution. (b) shows weights escaping the predicted range, indicating training instability.
Figure 13: HEBO optimization convergence for layer-specific learning rates. Multiple independent runs (shown in different colors) converge to similar performance levels within 50 iterations, indicating redundancy in the sensitivity to these hyperparameters.
Figure 14: Weight distribution for post-hoc exponential moving average using power schedule. Later checkpoints receive substantially higher weight, reflecting their superior quality. The optimal value is selected based on validation loss.
Figure 15: Comparison of validation CLIP similarity between Cross-attention DiT and Simple DiT architectures over training. Both architectures achieve similar final performance, demonstrating that architectural choices have minimal impact compared to optimization and data quality.
| Model | Creativity | Commonsense | Controllability | Human Fidelity | Physics |
|---|---|---|---|---|---|
| Summer-22B (Ours) | 0.387 | 0.622 | 0.311 | 0.745 | 0.629 |
| Wan 2.2-5B | 0.503 | 0.581 | 0.324 | 0.803 | 0.662 |
| Wan 2.2-A14B | 0.516 | 0.595 | 0.454 | 0.793 | 0.690 |
| Veo3 Fast | 0.556 | 0.619 | 0.415 | 0.822 | 0.679 |
Table 4: VBench 2.0 evaluation results across five quality dimensions. Summer-22B is compared against Wan 2.2 variants (5B and A14B, open-source models of similar or larger scale) and Veo3 Fast (a proprietary system included as an upper reference). Higher is better for all metrics. Our model is competitive on commonsense and physics but shows gaps in creativity and controllability, likely reflecting limited prompt diversity during training.
| Model | Total Score |
|---|---|
| Summer-22B (Ours) | 0.539 |
| Wan 2.2-5B | 0.575 |
| Wan 2.2-A14B | 0.610 |
| Veo3 Fast | 0.618 |
Table 5: VBench 2.0 total scores (weighted average across all dimensions). Summer-22B achieves 0.539, within 0.036 of the open-source Wan 2.2-5B baseline trained on substantially more data, demonstrating the effectiveness of our data-efficient training methodology.
Figure 16: Radar plots of per-dimension VBench scores. Left: VBench 1.0 evaluates 16 quality dimensions including temporal consistency, motion smoothness, and semantic alignment. Right: VBench 2.0 evaluates 20 advanced dimensions covering creativity, physics, and human fidelity. Summer-22B (blue) is compared against Wan 2.2-5B, Wan 2.2-A14B, and Veo3 Fast. Our model is competitive on low-level quality metrics (flickering, smoothness) but lags on high-level semantic dimensions.