VersaDB: A High-Performance AI Storage Database for Unifying Mutimodal Datasets
Authors: Cong Wang, Zelin Liu, Yang Luo Ran Zhang, Zhijian Guo, Hui Zhang, Fan Yu, Yanfei Cao, Naijie Gu, Jun Yu
Organizations: Huawei, China · Shanghai Jiao Tong University, Shanghai, China · South China University of Technology, Guangzhou, China · University of Science and Technology of China, Anhui, China
The AI field has been rapidly developing, leading to the emergence of a large number of AI training datasets of various types. These datasets contain different modalities, including text, images, audio, etc., and may come in various data storage formats. With the advancement of AI hardware, AI computation units like GPUs, TPUs, and NPUs can greatly accelerate the training speed of AI models, which in turn increases the demand for faster data processing. When using existing AI processing frameworks to handle datasets with different modalities and storage formats, processing speeds may be suboptimal due to issues such as data layout and the way users handle the data. Therefore, using a unified database to store multiple data formats can better manage and optimize data access. In this paper, we introduce VersaDB, a database designed specifically for AI datasets with various modalities. We implemented a page-based storage system, separating structured and unstructured data. Additionally, we generated B+ tree-based index files to accelerate data access. VersaDB supports automatic sharding and maintains a hierarchical metadata management system, with corresponding metadata maintained at the page, shard, and global levels, forming the foundation for the efficient operation of the database. We also focused on ease of use by providing APIs for directly converting datasets into VersaDB, as well as APIs for converting popular AI data storage formats (e.g., CSV, TFRecord, .bin) into VersaDB.Our experiments show that using VersaDB can achieve up to 5.35x acceleration and maintain consistent performance across different parallelism levels.
Figures & tables
Fig. 1: VersaDB file generation workflow. VersaDB is designed to provide users with an efficient and convenient AI data storage management method. To store datasets of different modalities using VersaDB, users need to define the schema information based on the content to be stored. Then, different data types in the dataset will be stored separately in Blob data pages and Raw data pages. For example, data labels will be stored in the raw data pages. Subsequently, an index file will be generated based on the stored data, and corresponding metadata will be included in the header.
Fig. 2: This paper introduces the architecture of VersaDB. VersaDB is divided into two main components: the Data file and the Index file. The Data file is used to store the actual data and metadata, while the Index file is used to assist in locating the physical storage location of the data. In this paper, we will primarily focus on the data storage methods, the APIs required to interact with VersaDB, the statistics and structure of metadata, data sharding management, dynamic page management, as well as random sampling and parallel reading optimizations.
Fig. 3: Three simple definition for Schema format.
Fig. 4: Statistical counting based on distributed aggregation tree
Fig. 5: When the local work queue of write thread 2 is empty, it will steal tasks from other queues to ensure load balancing.
Domain
Dataset
Number of Samples
CV
VOC
17,152
Kitti
7,481
COCO
118,287
ImageNet
20,000
NLP
WikiText-103
100,000
SQuAD2.0
130,319
TABLE I: Number of Samples for Different Datasets in Various Domains
Experiment
Categories
Hardware Configuration
Machine 1 CPU INFO: Architecture: x86-64, 64bits+64cores, Clock Speed: 3.9GHz DRAM INFO: Total memory size: 251Gi,Configured Memory Speed: 2934 MT/s Framework Versions: PyTorch Version 2.4.0,TensorFlow version 2.13.1,MindSpore Version 2.2.14.
Machine 2 CPU INFO: Model: Kunpeng 920, Architecture: aarch64, 64bits+192cores, Clock Speed: 2.6GHz DRAM INFO: Total Memory Size: 755Gi,Configured Memory Speed: 2933 MT/s Framework Versions: PyTorch Version 2.4.1,TensorFlow version 2.12.0,MindSpore Version 2.4.0.
TABLE II: Experiment Setting
Fig. 6: Performance comparation of three different work
Fig. 7: Figures (a) and (b) represent the memory consumption rates of LMDB, VersaDB, and TFRecord across the eleven datasets in machine environments 1 and 2, respectively. The time consumed by the three methods is scaled to a unified scale. The x-axis represents the time interval, and the y-axis represents the memory consumption ratio, with consistent units of measurement.
Fig. 8: Figures (a) and (b) show the test data for LMDB, VersaDB, and TFRecord on eleven datasets in machine environments 1 and 2, respectively. We set five parallelism levels: 1, 2, 4, 8, and 16. The x-axis represents the number of parallel workers, and the y-axis represents the consumption time.
Distributed reading
Shard num
VersaDB
TFRecord
Lmdb
Kitti
2
4.62
8.03
9.9
4
1.15
6.4
6.13
8
1.04
6.08
4.34
Ljspeech
2
0.1
2.09
1.24
4
0.051
1.25
1.16
TABLE III: Performance comparison under different number of distributed nodes