ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation

Zhongpan Tang

cs.LGAug 24, 2026

ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation

Authors: Zhongpan Tang

Organizations: Independent Researcher

Abstract

The quadratic growth of attention computation and key-value (KV) cache with respect to sequence length is a central bottleneck for ultra-long-context language models and high-resolution generative models. We propose ProxyFormer, a general dual-stream architecture built upon proxy tokens. In each layer, fine-grained local features are compressed bottom-up into a small set of proxy states; expensive global interactions are performed only in the compressed proxy space; the globally contextualized proxies are then decompressed and injected top-down back into the local stream. Because the local stream persists across layers, fine-grained information that is not captured by one compression step remains accessible for later refinement, alleviating the irreversible information loss of conventional one-shot compression. We further introduce factorized multi-level compression/decompression, layer-wise dynamic compression ratios, asymmetric dual embeddings, and a proxy-only KV-cache inference scheme. On a 16GB GPU with batch size 1, a standard decoder-only model can train sequences of only about 20K tokens, whereas ProxyFormer with a compression ratio of 64 extends the trainable sequence length to about 0.7M. A model trained with a 64K window retains 92%-95% retrieval accuracy on a multi-needle retrieval task with 1,048,576 tokens, and a model trained with an 8K window exceeds 94% accuracy when extrapolated to 256K tokens. Preliminary image-generation experiments demonstrate the feasibility of ProxyFormer for both pixel-space and latent-space flow matching.

Figures & tables

Figure 1: Local stream and proxy stream dual-stream architecture. Fine-grained local features update proxy states through ”micro guiding macro”; after proxy states complete global interaction, they are injected into the local stream through ”macro guiding micro”.

Figure 2: Multi-level cascaded factorized compression and symmetric decompression.

P=\prod_{j=1}^{M}k_{j}

, and intermediate features exist only on shorter sequences with narrower channels.

Figure 3: Language model based on proxy features. Long history is compressed into an extremely small proxy KV cache; during decoding, the generation region only performs gated cross-attention with the proxy cache.

Figure 4: ProxyFormer in text-to-image and diffusion conditional generation. Global conditions undergo cross-modal cross-attention only in the compressed image proxy space.

Model	History	PPL $\downarrow$
Base (decoder-only)	None	21.36
ProxyFormer	None	22.10
ProxyFormer	Yes	21.01

Table 1: Perplexity on WikiText-103-v1 (NoPE).

Panel of Figure 5 — Figure 5: Multi-needle in a haystack heatmap. The horizontal axis is context length (tokens), the vertical axis is the depth interval of the needle, and the values are retrieval accuracy.

Model	History len.	Gen. len.	VRAM (GB)	Thr. (it/s)
Standard decoder-only	0	20,496	15.6	2.6
Base (full-history attention)	20,992	128	15.6	1.3
ProxyFormer ( $P=64$ )	20,992	128	2.9	16.10
ProxyFormer ( $P=64$ )	716,800	128	15.1	2.70

Table 2: VRAM and training speed comparison under a 16 GB GPU with batch size 1.

Dataset	Method	Space	FID $\downarrow$	IS $\uparrow$
CIFAR-10	JIT	Pixel	16.21	8.42
MNIST	JIT	Pixel	2.06	1.95
MNIST	JLT	Latent	4.36	1.93

Table 3: Image Generation Evaluation (Lower FID is better, higher IS is better).

Long-Context Language Modeling

Long-context language modeling studies how language models can use, retain, and efficiently process information across very long sequences.

See all 189 papers

ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference
May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM Pruning KV Caching
Shanghai Jiao Tong University
End-to-End Context Compression at Scale
Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12LLM Compression Long-Context Language Model Inference
New York University · Modal Labs · Harvard University +5
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA

May 9, 2026J/K move · Enter open · S save

ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference

Junjie Li, Jiong Lou, Jie Li

Shanghai Jiao Tong University

LLM Pruning KV Caching Long-Context Language Model Inference KV-Cache Eviction LLM Inference Acceleration KV-Cache Pruning

Open paper AskPDF arXiv

Efficient long-context inference in Large Language Models (LLMs) is severely constrained by the Key-Value (KV) cache memory wall, yet existing pruning methods force a choice between low-latency heuristics that sacrifice precision and high-precision reconstruction methods that incur prohibitive prefilling overhead. To bridge this scoring-cost--accuracy gap, we propose ProxyKV, a cross-model proxy pruning framework that offloads importance scoring to a lightweight intra-family Small-Model Proxy executed asynchronously to the Large-Model Target. To bridge the architectural gap between heterogeneous models, we design the HybridAxialMapper, which disentangles temporal feature extraction from cross-head alignment, together with a Multi-Granularity Hybrid Loss that shifts the learning objective from rigid regression to relative ranking consistency. Across the Llama-3.1, Qwen-2.5, and Qwen-3 families spanning targets from 7B up to 32B parameters on LongBench, SCBench, and RULER, ProxyKV matches KVZip on aggregate (recovering

\sim$$98.7\%

of its mean accuracy) while delivering up to a

3.21\times

prefilling speedup on Llama-3.1-8B (dual-GPU;

\sim$$1.5\times

shared single-GPU) and sustaining the speedup at contexts up to 170k tokens on Qwen-2.5-7B.

Abstract

Long-Context Language Modeling

Explore similar work

ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference

End-to-End Context Compression at Scale

Learning When to Attend: Conditional Memory Access for Long-Context LLMs