Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization

Jiacheng Liu; Yaxin Luo; Xiaohan Zhao; Peng Sun; Xinyi Shang; Jiacheng Cui; Yonina C. Eldar; Zhiqiang Shen

cs.LGOct 22, 2026

Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization

Authors: Jiacheng Liu, Yaxin Luo, Xiaohan Zhao, Peng Sun, Xinyi Shang, Jiacheng Cui, Yonina C. Eldar, Zhiqiang Shen

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Westlake University · Zhejiang University · University College London ·

Abstract

Scaling embedding capacity alongside Transformer computation is an emerging trend in language models. This trend makes high compression rates increasingly useful. Hence, we study extreme compression of Per-Layer Embeddings (PLE), which follow this trend and account for 35.2% of the parameters of Gemma 4 E4B. At such rates, conventional quantizers give each token-layer vector its own code of only a few bytes. We instead develop OPTIC (Output-Path-fitted, Token-shared Indexed Codes), which stores one quantized code tuple per token and decodes it at every layer with a layer-specific affine decoder. Codes and decoders are fitted to the PLE output path rather than to the stored vectors. At about $100\times$ PLE compression of bfloat16 (BF16) E4B, OPTIC retains an aggregate 97.8% of the uncompressed model's performance across seven benchmarks, whereas product quantization, the strongest of six conventional baselines, retains 66.3%. At $594\times$ and $1,002\times$ PLE compression of BF16 E4B, OPTIC retains 90.5% and 84.1%. On the official E4B release with 4-bit linear weights and 16-bit activations (W4A16), OPTIC retains 96.9% at about $100\times$ PLE compression. During inference at about $100\times$ PLE compression, BF16 E4B uses 29.7% less GPU memory than with uncompressed PLE at the same key-value cache size, while running at most 0.40% slower.

Figures & tables

Figure 1: One shared code per token serves all PLE layers. Token

t

retrieves its stored codebook indices and its input embedding. Each index

q_{tm}

selects the highlighted codeword of codebook

C_{m}

. Concatenating these codewords forms one shared latent

\mathbf{z}_{t}

, from which each affine decoder reconstructs its layer’s PLE row. Matching colors identify corresponding layers, and ellipses mark omitted layers and codebooks. Each cell of

\mathbf{E}

contains a

d

-dimensional vector. The original PLE is used only for fitting, with frozen gates and projections (Equation 4 ). The frozen backbone shows the PLE residual connections and omits attention and feed-forward operations.

Baselines
	Original BF16		Official W4A16
Method	E2B $R$ / Retention	E4B $R$ / Retention	E2B $R$ / Retention	E4B $R$ / Retention
PQ	101.24/22.9	101.43/66.3	101.24/0.0	101.43/82.4
RVQ	100.96/0.0	98.64/37.6	100.96/29.3	101.20/82.0
OPQ-P	100.92/0.0	101.16/52.4	100.92/0.0	101.16/57.9
AQ	100.96/0.0	101.20/45.6	100.96/0.0	101.20/62.8

Table 1: PLE compression on the original BF16 and official W4A16 releases at the nominal

100\times

tier. Cells report compression ratio

R

/ seven-task retention (%) relative to the matched uncompressed PLE. A zero denotes at least one zero task score. Bold marks the highest retention in each column.

Setting	Our method	Retention (95% interval)	Strongest baseline	Retention (95% interval)
E2B, BF16	OPTIC	89.1 [85.8, 91.8]	PQ	22.9 [0.0, 28.8]
E4B, BF16	OPTIC	97.8 [95.5, 99.0]	PQ	66.3 [60.6, 70.5]
E2B, W4A16	OPTIC	91.5 [89.3, 93.3]	AQLM Euclidean beam encoder	32.0 [0.0, 38.9]
E4B, W4A16	OPTIC	96.9 [93.9, 97.8]	AQLM Euclidean beam encoder	82.8 [76.8, 87.1]

Table 2: Paired-bootstrap confidence intervals for the four primary nominal-

100\times

comparisons. Retention is reported in percent.

Figure 2: Rate–retention curves on BF16 Gemma 4 E4B, normalized to the uncompressed PLE. Dashed curves show the PQ and OPQ-P baselines, and solid curves show our methods. OPTIC retains 97.8% near

100\times

, and its advantage over the shown baselines remains large at higher compression ratios. A zero geometric mean means that at least one task scored zero.

Representation	E2B BF16	E4B BF16	E2B W4A16	E4B W4A16
Independent latent codes	28.5	79.3	53.7	86.8
Additive codes	42.1	90.0	83.5	94.4
OPTIC	89.1	97.8	91.5	96.9

Table 3: Seven-task retention (%) near

100\times

PLE compression with fit seed 0, relative to the uncompressed checkpoints of Tables 12 and 13 . Independent latent codes use the main method’s fitting recipe and calibration data, with 0.8–0.9% fewer resident bytes. OPTIC values are the main results of Table 1 . Output-path-fitted additive codes keep their standard initializer.

Figure 3: Cumulative share of each layer’s token-specific variation captured by its leading principal directions in the original BF16 E4B PLE table, with every token in the vocabulary weighted equally, in the stored coordinates (top) and at the PLE projection output (bottom). Each curve is one of the 42 layers. The dashed line marks 20 directions, the latent width of independent latent codes (Section 5.2 ).

Decoder refit with fixed codes	Retention (%)
Rank unlimited	$95.09\pm 0.48$
Fitted rank at most 20 per layer	$70.55\pm 1.18$

Table 4: Decoder refits on the three E4B BF16 fits of Table 17 , which use a common Euclidean initialization, near

100\times

. Values are means and sample standard deviations over fit seeds 0, 1 and 2. Refits keep each fit’s indices, codebooks and biases, constrain rank before float16 storage, and share the uncompressed reference of Table 17 .

Fixed-code objective	Retention (%)
Euclidean	$92.76\pm 1.16$
Diagonal output path	$94.44\pm 0.94$
Context-dependent output path	$94.74\pm 0.59$

Table 5: Fixed-code fitting on E4B BF16 near

100\times

, with the common Euclidean initialization and 5,000 continuous updates. Each loss is divided by its value for an all-zero reconstruction. Values are means and sample standard deviations over three fits.

Optimization recipe	Retention (%)
Initialization only	$63.60\pm 3.68$
Continuous updates only	$94.67\pm 0.53$
Continuous updates + reassignment	$95.49\pm 0.40$

Table 6: Optimization on E4B BF16 near

100\times

. Values are means and sample standard deviations over three fits. The last two rows use 5,000 continuous updates, and the final row adds 18 reassignment sweeps.

Backbone	Dense (GB)	Compact (GB)	Saved (%)
BF16	18.57	13.05	29.7
W4A16	12.95	7.37	43.1

Table 7: Post-initialization process GPU memory for E4B on an RTX 4090 with a fixed 1 GiB key-value cache, averaged over two engine initializations. Resident PLE storage falls from 5.64 GB to 56.4 MB.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Scale	Role	Hugging Face model	Version
E2B	Evaluated BF16	google/gemma-4-E2B-it	b446025c61ec
E4B	Evaluated BF16	google/gemma-4-E4B-it	ee0ef6023621
E2B	PLE fit for W4A16	google/gemma-4-E2B-it-qat-q4_0-unquantized	6befbaca7398
E4B	PLE fit for W4A16	google/gemma-4-E4B-it-qat-q4_0-unquantized	476025a01dbf
E2B	Evaluated W4A16	google/gemma-4-E2B-it-qat-w4a16-ct	971342c08f60
E4B	Evaluated W4A16	google/gemma-4-E4B-it-qat-w4a16-ct	6cd26aaa2357

Appendix

Table 8: Model releases used for fitting and evaluation. The final column gives a 12-character revision prefix.

Weights	Phase	Batch	Input tokens	Output tokens	Wall time $\Delta$ (%)	Throughput $\Delta$ (%)
BF16	Prefill	16	128	1	-0.23	+0.19
BF16	Prefill	4	512	1	-0.55	+0.55
BF16	Prefill	1	2,048	1	+0.16	-0.16
BF16	Prefill	1	8,192	1	+0.31	-0.31
BF16	Decode	32	128	32	+0.17	-0.32
BF16	Decode	16	128	128	+0.20	-0.30

Appendix

Table 9: Direct compact-serving comparison with uncompressed PLE for matched BF16 and W4A16 E4B checkpoints. Input and output lengths are per sequence; values are means over six timed measurements. Positive wall-time change means slower, and positive throughput change means faster.

		Same budget			Smaller
Input	Requests	Dense	Compact	Speedup	Compact
BF16 E4B: 21.6 GiB; smaller budget 16 GiB
32K	1	17.3	17.3	1.00 $\times$	17.2
32K	2	22.2	22.2	1.00 $\times$	22.0
32K	4	30.1	29.9	1.01 $\times$	29.7
32K	8	59.9	46.1	1.30 $\times$	59.9

Appendix

Table 10: Seconds until all concurrent requests finish on one RTX 4090, with 1,024 output tokens per request. Speedup compares the two conditions at the same budget and uses unrounded times; dense PLE does not start at the smaller budget.

Tokens	GPU	Host	Host	Speedup
	Dense		Compact
1	18	27	46	0.6 $\times$
8	18	63	66	0.9 $\times$
32	18	126	69	1.8 $\times$
128	18	380	90	4.2 $\times$
512	26	1,092	154	7.1 $\times$

Appendix

Table 11: Median time in microseconds to return the BF16 PLE rows of all 42 layers for uniformly random token IDs on an RTX 4090. Speedup compares compact and dense lookup from host memory and uses unrounded medians.

E2B
Method	$R$	PIQA	GSM8K	GPQA Diamond	AIME 2026	IFBench	Infographic VQA	MMLU- Redux	Retention
Uncompressed PLE	1.00	72.4	90.5	43.2	42.5	36.0	63.1	71.5	100.0
Baselines
PQ	101.24	68.3	53.1	29.8	0.8	0.7	19.2	55.7	22.9
RVQ	100.96	70.7	39.7	23.0	0.0	1.0	23.6	58.9	0.0
OPQ-P	100.92	69.2	51.5	1.5	0.0	1.3	42.1	56.4	0.0

Appendix

Table 12: Per-task results (%) for the original BF16 releases at the nominal

100\times

tier.

R

is the actual resident-byte PLE compression ratio. Bold marks the highest compressed-model retention for each model size.

E2B
Method	$R$	PIQA	GSM8K	GPQA Diamond	AIME 2026	IFBench	Infographic VQA	MMLU- Redux	Retention
Uncompressed PLE	1.00	74.7	88.1	41.8	25.0	33.7	56.0	68.1	100.0
Baselines
PQ	101.24	71.3	42.5	26.0	0.0	23.0	20.8	53.9	0.0
RVQ	100.96	72.5	51.9	22.9	0.8	6.3	6.4	56.2	29.3
OPQ-P	100.92	71.6	52.7	23.6	0.0	5.3	11.5	55.3	0.0

Appendix

Table 13: Per-task results (%) for the official W4A16 releases at the nominal

100\times

tier. Notation follows Table 12 . Bold marks the highest compressed-model retention for each model size.

Baselines
	$20\times$		$40\times$		$100\times$
Method	Actual $R$	Retention	Actual $R$	Retention	Actual $R$	Retention
PQ	21.17	98.6	42.00	77.0	101.43	66.3
RVQ	20.26	94.2	40.52	86.0	98.64	37.6
OPQ-P	21.15	98.1	41.95	90.4	101.16	52.4
AQLM Euclidean beam encoder	20.24	98.2	42.16	82.5	101.20	46.7

Appendix

Table 14: Original BF16 E4B at the nominal

20\times

40\times

, and

100\times

PLE compression tiers.

R

is the actual resident-byte compression ratio; retention (%) is the seven-task geometric mean. Bold marks the highest retention at each tier.

Baselines
	E2B W4A16			E4B W4A16
Method	$20\times$	$40\times$	$100\times$	$20\times$	$40\times$	$100\times$
PQ	21.13/ 95.0	41.87/85.6	101.24/0.0	21.17/96.0	42.00/88.6	101.43/82.4
RVQ	20.17/85.4	40.35/41.4	100.96/29.3	20.26/96.5	40.52/89.3	101.20/82.0
OPQ-P	21.12/94.9	41.81/83.7	100.92/0.0	21.15/93.6	41.95/91.6	101.16/57.9
Our methods

Appendix

Table 15: Official W4A16 results at the nominal

20\times

40\times

, and

100\times

PLE compression tiers. Each cell reports actual

R

/seven-task retention (%) relative to the matched uncompressed PLE. Bold marks the highest retention for each model and tier.

Nominal tier	Actual $R$	Resident PLE state	BF16 retention	W4A16 retention
$20\times$	19.994	281.9 MB	97.4	96.8
$40\times$	40.035	140.8 MB	96.4	95.9
$100\times$	99.941	56.4 MB	97.8	96.9
$500\times$	594.431	9.5 MB	90.5	87.7
$1000\times$	1002.189	5.6 MB	84.1	81.8

Appendix

Table 16: E4B OPTIC across weight formats. Retention (%) is the seven-task geometric mean; nominal labels denote target compression tiers, while

R

reports the measured resident-byte ratio.

Representation	E4B BF16	E4B W4A16
Independent latent codes	$75.90\pm 3.38$	$81.51\pm 2.06$
Additive codes	$82.66\pm 5.06$	$90.95\pm 1.23$
OPTIC	$95.49\pm 0.40$	$95.16\pm 0.42$

Appendix

Table 17: Seven-task retention (%) near

100\times

PLE compression. Each cell reports the mean and sample standard deviation over fit seeds 0, 1, and 2. All rows use output-path fitting and a common Euclidean initialization policy. Table 3 reports the seed-0 comparison under the main fitting recipe.

Embedding Compression

Embedding compression reduces the storage and computation needed for vector representations while preserving task-relevant information.

See all 49 papers

FastE: Readout-Triggered Token Compression for LLM Embedding Inference
Sep 8, 2026Jinsong Shu, Jinyong Wen, Baokun Wang +4Token Pruning LLM Inference Acceleration
Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Ant Group +1
SPQ: An Ensemble Technique for Large Language Model Compression
Feb 20, 2026Jiamin Yao, Eren GultepeLLM Compression LLM Quantization
Dept. of Computer Science, Southern Illinois University Edwardsville, Edwardsville, USA
Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
Jul 3, 2026Hamish OgilvyLLM Inference Acceleration Mixed-Precision Quantization
Algolia Research

Sep 8, 2026J/K move · Enter open · S save

FastE: Readout-Triggered Token Compression for LLM Embedding Inference

Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie+3

Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Ant Group+1

Token Pruning LLM Inference Acceleration Embedding Compression Text Embedding Models

Open paper AskPDF arXiv

In this study, we identify depth-dependent prefix redundancy in final-readout LLM embedding models, notably across representative backbones including Qwen3-Embedding and Qwen3-VL-Embedding. We find that removing prefix states is substantially more damaging in shallow layers than at greater depth, showing that prefix states become increasingly compressible as the prefix and readout states propagate through the network. To this end, we introduce FastE, a training-free, plug-and-play method. FastE uses a shared fixed threshold on batch-mean readout-prefix alignment as a lightweight online heuristic for selecting when compression occurs, and ranks prefix states by the attention scores they receive from the readout position to determine which states are retained in subsequent layers. Our evaluations demonstrate FastE's ability to substantially reduce computational costs: on NarrativeQA with Qwen3-Embedding-0.6B, it reduces decoder-backbone FLOPs by 40.11% while retaining 99.53% of Full Forward nDCG@10. Across five text embedding benchmarks, two backbone scales, and three cross-modal retrieval tasks, the quality-efficiency trade-off is directly customizable through the maximum removal ratio without retraining. We believe FastE offers practical value for scalable embedding generation in retrieval, indexing, clustering, and multimodal representation systems.

Abstract

Embedding Compression

Explore similar work

FastE: Readout-Triggered Token Compression for LLM Embedding Inference

SPQ: An Ensemble Technique for Large Language Model Compression

Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models