Balancing hierarchical representation and quantization: A performance trade-off in text embeddings
Organizations: Ludwig-Maximilian-University Munich, Geschwister-Scholl-Platz 1, Munich, Germany · Otto von Guericke University Magdeburg, Universitätsplatz 2, Magdeburg, Germany · University of Bari Aldo Moro, Via E. Orabona 4, Bari, Apulia, Italy · European Commission, Joint Research Centre (JRC), Via E. Fermi 2749, Ispra, Italy ·
Abstract
Hierarchical text embedding models offer variable-length representations, while quantization reduces numerical precision; both aim to improve computational efficiency. This work is an empirical benchmark study of how these two orthogonal, already-established techniques interact. We evaluate 38 tasks from the Massive Text Embedding Benchmark (MTEB) using four transformer-based embedding models, covering 1D and 2D hierarchical representations combined with 32-bit, 8-bit, and binary quantization, for 168 configurations per task. We report each efficiency dimension separately: theoretical per-vector storage and end-to-end evaluation time. Under a Friedman test with Nemenyi post-hoc analysis, int8 quantization is statistically indistinguishable from float32 for pair classification and re-ranking and costs at most 1.5 percentage points overall, whereas binary quantization is significantly worse across every task category. Halving the embedding dimension once is not statistically detectable above ; two successive halvings always are. Combining an 8x dimensionality reduction with int8 yields a 32x reduction in per-vector storage at 94.4% of baseline effectiveness. Because evaluation times were measured once, we treat them as indicative: we find no evidence that quantization speeds up evaluation, whereas sampling from intermediate layers shortens it substantially. Retrieval degrades far faster than every other category, so we report findings per task category.
Figures & tables
| Configuration | Vector excerpt (q) | sim / sim | Bits per vector | Index, 1M vectors |
|---|---|---|---|---|
| float32, (baseline) | / | 32,768 | 4.10 GB | |
| float32, (1D MR) | / | 4096 | 0.51 GB | |
| int8, (MR + int8) | / | 1024 | 0.13 GB | |
| binary, (MR + 1-bit) | Hamming 0 / 3 | 128 | 0.02 GB |
| Method | Compression | Fitting | Pros | Cons |
|---|---|---|---|---|
| vs float32 | required | |||
| Float16 / BFloat16 [ 21 ] | 2 | No | Plug-and-play; hardware-native on modern GPUs; small reported accuracy loss | Modest compression; bfloat16 trades precision for range |
| Int8 scalar quantization [ 10 ] | 4 | Value ranges | Integer arithmetic is fast and energy-efficient; widely supported | Ranges must be estimated from data; outlier values reduce the resolution |
| Binary quantization [ 11 ] | 32 | No | Extreme compression; fast Hamming-distance search on CPU; data-independent | Large semantic loss; close neighbors collapse to identical codes; usually needs rescoring |
| Product quantization (PQ, OPQ) [ 22 , 23 ] | 8–32 | Codebook | Strong compression-accuracy trade-off; mature nearest-neighbor search support | Codebook must be trained on in-domain data and retrained under drift; more complex distance computation |
| PCA projection [ 14 , 15 ] | 2–10 | Projection | Cheap to fit and apply; orthogonal to precision reduction | Linear, so it misses non-linear structure; projection must be stored and refitted per corpus |
| Symbol | Domain | Meaning |
| , | n/a | Set of evaluated embedding models; an individual model |
| Number of transformer layers of model | ||
| Default (maximum) output dimensionality of a model | ||
| Hierarchical (truncated) embedding size, | ||
| Set of hierarchical sizes, successive halvings of | ||
| Full embedding vector produced by a model |
Embedding Compression
Embedding compression reduces the storage and computation needed for vector representations while preserving task-relevant information.
See all 55 papers