YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference

You Wu; Ziheng Chen; Yizhen Zhang; Haoyi Wu; Chengting Yu; Yuchi Xu; Wenbo Su; Bo Zheng

cs.CLApr 15, 2026

YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference

Authors: You Wu, Ziheng Chen, Yizhen Zhang, Haoyi Wu, Chengting Yu, Yuchi Xu, Wenbo Su, Bo Zheng,

Organizations: ShanghaiTech University · Alibaba Group · Zhejiang University

Abstract

Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the memory consumption of the KV cache, such methods usually introduce non-negligible performance degradation. In this work, we aim to enhance the performance of YOCO, a cross-layer KV compression method that shares the KVs of the middle layer with the top-half layers. We propose YOCO++, an enhanced YOCO that incorporates a weighted residual connection between the KVs of each bottom-half layer and the bottom layer. Compared to YOCO, YOCO++ increases model capacity while maintaining the same training and inference efficiency. Our experiments show that YOCO++ achieves state-of-the-art performance among the cross-layer KV compression methods at a 50% KV cache compression rate, outperforming the standard Transformer.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of YOCO and YOCO++. KVs that need to be cached are indicated by red boxes. (a) YOCO caches the KVs of the bottom-half layers and shares the KVs of the middle layer with the top-half layers. (b) Based on YOCO, YOCO++ introduces a weighted residual connection between the KVs of each bottom-half layer and the bottom layer and caches the combined KVs.

Panel of Figure 2 — Figure 2: Learned residual weights of a 22-layer YOCO++ model without (left) and with (right) the scaling factor.

Figure 3: Prefilling latency (left) and decoding throughput (right) on an NVIDIA H20 (96GB) GPU at different sequence lengths.

Model	HellaSwag	OBQA	WinoGrande	ARC-c	ARC-e	PIQA	Avg
Transformer	51.25	33.0	56.04	27.56	52.1	70.29	48.37
YOCO	51.21	32.0	53.59	29.01	51.43	70.62	47.98
FusedKV-Lite	52.28	32.8	54.93	28.24	52.19	70.13	48.43
FusedKV	52.04	32.0	55.48	28.67	51.98	70.35	48.42
YOCO++	52.30	34.0	55.48	27.90	53.11	71.16	48.99

Table 1: Zero-shot accuracy on commonsense reasoning tasks.

Figure 4: Training loss curves on a 100B subset of the SlimPajama dataset.

Model	Valid Loss
Transformer	2.4266
YOCO	2.4372
FusedKV-Lite	2.4258
FusedKV	2.4204
YOCO++	2.4194
w/o scaling	2.4333

Table 2: Ablations of YOCO++ trained on a 10B subset of the SlimPajama dataset. We report the validation loss on a 10M subset of the SlimPajama validation set.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
Mar 24, 2026Shiyu Ji, Yixuan Wang, Yijun Liu +2KV-Cache Compression LLM Inference Acceleration
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Jul 7, 2026Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
Exploring a Layer-Wise Design Space for KV Cache Eviction
Jun 13, 2026Chao Fei, Kaihua Liang, Hanzhi Hu +4KV Caching KV-Cache Eviction
King Abdullah University of Science and Technology · Fudan University

Mar 24, 2026J/K move · Enter open · S save

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

Shiyu Ji, Yixuan Wang, Yijun Liu, Qingfu Zhu+1

Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology, China

KV-Cache Compression LLM Inference Acceleration

Open paper AskPDF arXiv

The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.

Abstract

LLM Inference Acceleration

Explore similar work

EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

Exploring a Layer-Wise Design Space for KV Cache Eviction