Vector-Quantized Variational Autoencoder

A VQ-VAE encodes data into vectors, replaces them with entries from a learned codebook, and trains a decoder to reconstruct the input. The resulting discrete tokens support compression and generative modeling across images, audio, video, motion, and other structured signals, and can also aid interpretation or control. Common approaches include hierarchical or modality-specific tokenizers, improved codebook updates and priors, and continuous pretraining to reduce codebook or dimensional collapse. The main tension is balancing compact, well-used token vocabularies with faithful reconstruction and reliable downstream performance, especially at low bitrates or on unfamiliar data.