cs.LGNov 11, 2026

KVPath: Connecting Key-Value Representations Across Transformer Layers

Authors: Xuan Luo, Zhiyuan Wang, Hanzhong Cao, Xifeng Yan

Organizations: University of California, Santa Barbara

Abstract

Transformers typically cache an independent key-value (KV) representation for each token at every layer. Recent work reveals that a token's KV representations across neighboring layers are correlated and contain shared information. We hypothesize that this correlation implies incremental inter-layer changes, which can be effectively captured by low-dimensional deltas. Motivated by this, we propose KVPath, a new KV representation paradigm that connects KV representations across the network depth into a path. The path establishes a full-dimensional base KV at the first layer, while subsequent layers generate low-dimensional delta KVs to update the preceding KV, thereby constructing their own KV states. By caching the base KV once alongside these compact delta KVs, KVPath preserves distinct layer-specific representations without storing full KV states at every layer. We develop two variants of KVPath: KVPath-Residual, which updates vector values via residual addition, and KVPath-Rolling, which progressively updates feature dimensions in a rolling manner. Under matched parameter counts, KVPath reduces KV-cache storage by up to 68% for Grouped-Query Attention (GQA) and 55% for Multi-Head Latent Attention (MLA), while maintaining or even improving model performance.

Figures & tables

Explore similar work

CardsList
  1. Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

    Feb 16, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +1Memory-Efficient AttentionKV-Cache Compression

  2. Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

    Jun 20, 2026Xin GaoTransformer InferenceKV Caching