VersaDB: A High-Performance AI Storage Database for Unifying Mutimodal Datasets
Authors: Cong Wang, Zelin Liu, Yang Luo Ran Zhang, Zhijian Guo, Hui Zhang, Fan Yu, Yanfei Cao, Naijie Gu, Jun Yu
Organizations: Huawei, China · Shanghai Jiao Tong University, Shanghai, China · South China University of Technology, Guangzhou, China · University of Science and Technology of China, Anhui, China
The AI field has been rapidly developing, leading to the emergence of a large number of AI training datasets of various types. These datasets contain different modalities, including text, images, audio, etc., and may come in various data storage formats. With the advancement of AI hardware, AI computation units like GPUs, TPUs, and NPUs can greatly accelerate the training speed of AI models, which in turn increases the demand for faster data processing. When using existing AI processing frameworks to handle datasets with different modalities and storage formats, processing speeds may be suboptimal due to issues such as data layout and the way users handle the data. Therefore, using a unified database to store multiple data formats can better manage and optimize data access. In this paper, we introduce VersaDB, a database designed specifically for AI datasets with various modalities. We implemented a page-based storage system, separating structured and unstructured data. Additionally, we generated B+ tree-based index files to accelerate data access. VersaDB supports automatic sharding and maintains a hierarchical metadata management system, with corresponding metadata maintained at the page, shard, and global levels, forming the foundation for the efficient operation of the database. We also focused on ease of use by providing APIs for directly converting datasets into VersaDB, as well as APIs for converting popular AI data storage formats (e.g., CSV, TFRecord, .bin) into VersaDB.Our experiments show that using VersaDB can achieve up to 5.35x acceleration and maintain consistent performance across different parallelism levels.
Figures & tables
Fig. 1: VersaDB file generation workflow. VersaDB is designed to provide users with an efficient and convenient AI data storage management method. To store datasets of different modalities using VersaDB, users need to define the schema information based on the content to be stored. Then, different data types in the dataset will be stored separately in Blob data pages and Raw data pages. For example, data labels will be stored in the raw data pages. Subsequently, an index file will be generated based on the stored data, and corresponding metadata will be included in the header.
Fig. 2: This paper introduces the architecture of VersaDB. VersaDB is divided into two main components: the Data file and the Index file. The Data file is used to store the actual data and metadata, while the Index file is used to assist in locating the physical storage location of the data. In this paper, we will primarily focus on the data storage methods, the APIs required to interact with VersaDB, the statistics and structure of metadata, data sharding management, dynamic page management, as well as random sampling and parallel reading optimizations.
Fig. 3: Three simple definition for Schema format.
Fig. 4: Statistical counting based on distributed aggregation tree
Fig. 5: When the local work queue of write thread 2 is empty, it will steal tasks from other queues to ensure load balancing.
Domain
Dataset
Number of Samples
CV
VOC
17,152
Kitti
7,481
COCO
118,287
ImageNet
20,000
NLP
WikiText-103
100,000
SQuAD2.0
130,319
TABLE I: Number of Samples for Different Datasets in Various Domains
Experiment
Categories
Hardware Configuration
Machine 1 CPU INFO: Architecture: x86-64, 64bits+64cores, Clock Speed: 3.9GHz DRAM INFO: Total memory size: 251Gi,Configured Memory Speed: 2934 MT/s Framework Versions: PyTorch Version 2.4.0,TensorFlow version 2.13.1,MindSpore Version 2.2.14.
Machine 2 CPU INFO: Model: Kunpeng 920, Architecture: aarch64, 64bits+192cores, Clock Speed: 2.6GHz DRAM INFO: Total Memory Size: 755Gi,Configured Memory Speed: 2933 MT/s Framework Versions: PyTorch Version 2.4.1,TensorFlow version 2.12.0,MindSpore Version 2.4.0.
TABLE II: Experiment Setting
Fig. 6: Performance comparation of three different work
Fig. 7: Figures (a) and (b) represent the memory consumption rates of LMDB, VersaDB, and TFRecord across the eleven datasets in machine environments 1 and 2, respectively. The time consumed by the three methods is scaled to a unified scale. The x-axis represents the time interval, and the y-axis represents the memory consumption ratio, with consistent units of measurement.
Fig. 8: Figures (a) and (b) show the test data for LMDB, VersaDB, and TFRecord on eleven datasets in machine environments 1 and 2, respectively. We set five parallelism levels: 1, 2, 4, 8, and 16. The x-axis represents the number of parallel workers, and the y-axis represents the consumption time.
Distributed reading
Shard num
VersaDB
TFRecord
Lmdb
Kitti
2
4.62
8.03
9.9
4
1.15
6.4
6.13
8
1.04
6.08
4.34
Ljspeech
2
0.1
2.09
1.24
4
0.051
1.25
1.16
TABLE III: Performance comparison under different number of distributed nodes
Modern model hubs store hundreds of petabytes of large language models (LLMs), with fine-tuned variants dominating the storage footprint. These variants contain substantial cross-model redundancy that delta compression can exploit by storing only the difference between a target and a reference model. However, compression effectiveness depends critically on choosing a similar reference. At model-hub scale, this is challenging because model lineage metadata is often missing or unreliable, and different tensors within the same model may be most similar to tensors from different models. Consequently, model-level pairing leaves substantial redundancy unexploited. We present TensorDex, a lossless, tensor-centric storage system that performs delta compression at tensor granularity. TensorDex decomposes models into tensors, uses compact bit-level fingerprints to predict tensor-pair compressibility, and incrementally organizes tensors into multi-center clusters to select effective bases as new models arrive. Evaluated on 2,890 randomly sampled Hugging Face models, TensorDex reduces storage footprint by 70.5%, 37% lower than the state-of-the-art design. It achieves 22.9 GB/s compression and 28.4 GB/s decompression throughput, 3.86x and 1.49x faster than the next-best system, respectively.
AI-driven analytics are increasingly crucial to data-centric decision-making. Executing relational and AI operators in separate runtimes prevents the database optimizer and runtime from coordinating operator ordering, model placement, batching, and state reuse. Integrating AI operators into database engines enables such coordination but raises challenges in jointly optimizing query processing and model execution, scheduling under resource contention, and reusing relational intermediates and AI artifacts. This paper formalizes AIxDB workloads as iterative, concurrent, and shareable executions that interleave relational and AI operators. We then advocate database-native orchestration as a paradigm for redesigning database engines for these workloads and distill two design principles: holistic AIxDB co-optimization and unified AIxDB cache management. We present NeurEngine as a proof-of-concept prototype and report preliminary results illustrating the performance benefits of database-native orchestration
Naili Xing, Haotian Gao, Zhanhao Zhao +6
National University of Singapore · Beijing Institute of Technology · Renmin University of China +1
We introduce Maia 200, an advanced AI accelerator delivering high performance-10 145 Tflop/s FP4 and 5072 Tflop/s FP8 within a 750W TDP and 7 TB/s HBM bandwidth. Maia exemplifies a new class of Software Defined Locally Accessed Dataflow Architectures (SDLA), which explicitly program dataflow engines to orchestrate highly specialized memories and data movement engines. This approach shifts the focus from today's thread-centric to data-movement-centric architecture, improving efficiency and scalability. Our taxonomy of data management, inspired by Flynn's classification, highlights how SDLA addresses challenges in modern AI computing. Maia 200 achieves significant cost and energy savings while supporting massive parallelism for AI inference workloads, making it a compelling solution for next-generation high-performance computing systems.