Rethinking Video Storage for AI Training: Breaking Cross-Frame Dependencies for Efficient Random Access

Zetian Song; Qize Yu; Yue Chen; Bohan Zhou; Bowen Ping; Yuqi Ye; Haocheng Tang; Haotian Zhang

cs.CVOct 21, 2026

Rethinking Video Storage for AI Training: Breaking Cross-Frame Dependencies for Efficient Random Access

Authors: Zetian Song, Qize Yu, Yue Chen, Bohan Zhou, Bowen Ping, Yuqi Ye, Haocheng Tang, Haotian Zhang,

Organizations: Peking University · Tsinghua University

Abstract

Large-scale AI training increasingly relies on massive video datasets with random temporal sampling patterns. However, existing video storage paradigms are optimized for sequential playback and employ sophisticated inter-frame reference structures to maximize compression efficiency. When applied to the random-sampling workloads of AI training, these paradigms face substantial decoding-efficiency bottlenecks. Workarounds such as independent JPEG frames and short groups of pictures (GOPs) reduce decoding work at the cost of larger storage and transfer volumes. In this paper, we identify two sources of decode amplification: sequential reconstruction of unrelated frames and reconstruction of the reference chains required by sampled targets. We address both sources by co-designing the decoding schedule and video reference structure. Dependency-aware sparse decoding reconstructs only requested frames and their required references, skipping unrelated reconstruction without transcoding the source videos. Optional offline Dependency-Decoupled Random Access (DDRA) encoding further reduces reference reconstruction by restricting each interior frame to two bounding intra-coded anchors. Together, sparse decoding and DDRA retain inter-frame prediction while bounding reconstruction to at most three frames per requested target, independent of the intra period. Experiments on a large-scale robotics corpus show that, at comparable reconstruction quality, our method achieves $1.45\times$ the loading throughput of LeRobot v3 with 58.7% less media payload.

Figures & tables

Figure 2: Codec–loader co-design for video-based training. Optional offline DDRA encoding and online dependency-aware sparse decoding deliver requested RGB frames. (a) Hierarchical coding requires intermediate B-frame references; (b) DDRA coding restricts each interior frame to two intra-coded anchors, reducing the illustrated dependency closure.

1	$Q\leftarrow\operatorname{set}(\bm{q})$
2	$(\mathcal{S},\operatorname{Ref})\leftarrow\operatorname{SelectAndParse}(V,Q)$
3	$D(Q)\leftarrow\operatorname{BFSClosure}(\operatorname{Ref},Q)$
4	$\sigma\leftarrow\operatorname{InitState}(V,\mathcal{S});\hskip 9.24994ptX\leftarrow\varnothing$
5	for $f\in\mathcal{S}$ in order $\pi$ do
6	if $f\in D(Q)$ then
7	$(x,\sigma)\leftarrow\operatorname{DecodeFrame}(V[f],\sigma)$

Algorithm 1 Dependency-aware sparse decoding with a validated reference graph

Figure 3: Exact expected decode amplification for Sequential, Sparse, and DDRA + Sparse under the periodic counting model. Sparse denotes dependency-aware sparse decoding (Section 3.1 ) and DDRA denotes the encoding configuration in Section 3.2 .

Loading time (ms/batch) $\downarrow$
		No offline transcoding				Offline transcoding required
Pattern	Stat.	PyAV	Decord	Torch Codec	Ours native	JPEG	LeRobot v3	Ours-DDRA H.264	Ours-DDRA H.265
Contiguous	Average	336.5	305.2	278.2	226.7	251.0	170.2	123.6	118.2
Contiguous	p95	2286.0	1916.3	1963.7	1348.3	1274.6	982.2	573.2	521.5
Random	Average	594.2	407.1	383.3	297.8	248.8	185.1	134.1	140.6
Random	p95	2895.7	2088.3	1952.8	1921.3	1238.9	1218.2	633.4	714.9

Table 1: Batch-loading efficiency on three datasets. Average and p95 loading times are macro-averaged across datasets. For methods requiring offline transcoding, reconstruction distortion is aligned to LeRobot v3 using RGB PSNR. bold marks the best value within each method group.

Figure 4: Training performance under compression. DDRA (H.265) approximately halves storage and bandwidth requirements relative to LeRobot v3 at comparable VLA/WAM performance, while high-quality transcoding need not reduce VLM accuracy. Compression factors use the original visual representations as anchors, explaining their different ranges.

Figure 5: Loading ten contiguous frames per camera under node-wide bandwidth caps of 50–800 MiB/s. (a) Average loading time and (b) p95 loading time over 100 measured batches, both macro-averaged across datasets. (c) Aggregate object-storage response throughput over the full proxy window. The dashed line in (c) marks the cap. Arrows label values beyond the axis limits.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Representation	Quality setting	RGB PSNR (dB) $\uparrow$
LeRobot v3 (AV1)	CRF 30	41.473
JPEG	Quality 79, 4:4:4	41.393
Ours-DDRA (H.264)	CRF 16	41.294
Ours-DDRA (H.265)	CRF 16	41.496

Appendix

Table 3: Quality settings and dataset-macro-averaged RGB PSNR for the loading experiments.

		LIBERO			MVBench
Method	Setting	$C$ ( $\times$ )	VLA (%)	WAM (%)	$C$ ( $\times$ )	VLM (%)
Original	–	1.000	98.00	99.20	1.000	65.87
DDRA (H.264)	CRF18	28.885	97.75	98.90	0.703	66.13
DDRA (H.264)	CRF28	92.307	97.00	98.55	2.067	66.42
DDRA (H.264)	CRF38	249.160	85.60	86.40	5.511	64.89
DDRA (H.264)	CRF48	574.095	39.45	45.25	15.510	63.82

Appendix

Table 4: Numerical results corresponding to Figure 4 . Compression factor

C

uses PNG as the LIBERO anchor and H.264 as the VLM anchor; relative storage is

S=1/C

. VLA and WAM report success rate, and VLM reports accuracy. Higher values are better for all columns. Dashes indicate configurations not shown for that model family. Scores below the figure’s vertical range are listed in full.

Figure 7: Sensitivity to access patterns and loader resources. The top row varies requested frame count, fixed stride/temporal span, and random candidate span; the bottom row varies batch size, worker count, and CPU allocation. Each point reports average loading time over 100 measured batches, macro-averaged across datasets. Panels share a linear loading time scale; lower is better.

Video Compression

Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.

See all 73 papers

LightVAE: Towards Compact and Efficient Video Autoencoders
Oct 10, 2026Shiqiao Gu, Yang Yong, Fengjun Zhong +5Variational Autoencoders Video Compression
SenseTime · Beihang University · Nanyang Technological University
Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolation
Apr 3, 2026Hanshuai Cui, Zhiqing Tang, Zhi Yao +3Autoregressive Video Generation Diffusion Model Inference Acceleration
School of Artificial Intelligence, Beijing Normal University, Beijing, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai, China · Shenzhen University of Advanced Technology, Shenzhen, China
ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding
Mar 20, 2026Quan Kong, Yuhao Shen, Yicheng Ji +2Efficient VLM Inference Video-Language Models
Zhejiang University, Hangzhou, China

Oct 10, 2026J/K move · Enter open · S save

LightVAE: Towards Compact and Efficient Video Autoencoders

Shiqiao Gu, Yang Yong, Fengjun Zhong, Ruihao Gong+4

SenseTime · Beihang University · Nanyang Technological University

Variational Autoencoders Video Compression Low-Rank Compression

Open paper AskPDF arXiv

Latent diffusion models enable high-quality video generation, but video VAE encoding and decoding remain computationally expensive. Existing methods use stage-specific lightweight operators, yet either constrain spatial and temporal computation jointly or remove temporal computation altogether, without distinguishing their compression sensitivities in pretrained Conv3D. To address these limitations, we present LightVAE, a compression framework that exploits redundancy in temporal modeling and network depth. LightVAE exactly reparameterizes eligible causal Conv3D layers into an aggregate spatial response and a difference-driven temporal correction. Based on their distinct compression sensitivities, temporal low-rank compression (TLR) keeps the aggregate spatial kernel dense and applies layer-specific low-rank approximation only to the temporal correction. An algebraically equivalent folded implementation converts the arithmetic savings into practical acceleration. LightVAE further uses width-preserving block pruning to reduce depth and probe-guided feature alignment, guided by feedback from a frozen teacher suffix, to recover quality after compression. The framework also extends to video encoders. Across three video VAE backbones and two GPU platforms, LightVAE consistently accelerates decoding. For the Wan VAE family, it achieves up to

2.24\times

and

1.11\times

the decoding throughput of state-of-the-art efficient VAE decoders on NVIDIA H100 and RTX~5090, respectively, with higher DAVIS PSNR and lower LPIPS. On MiniMax-H3, it achieves up to

2.97\times

the original decoding throughput while retaining 99.51% of the original model's overall VBench-2.0 score. Our code is available at https://github.com/ModelTC/LightVAE.

Abstract

Video Compression

Explore similar work

LightVAE: Towards Compact and Efficient Video Autoencoders

Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolation

ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding