As large language models (LLMs) advance toward increasingly long inputs, soft context compression offers a promising approach to efficient inference by converting long inputs into compact memory slots. Under limited memory budgets, effective context compression remains challenging because task-relevant evidence varies substantially across queries, ranging from localized evidence to multiple distant spans or information dispersed throughout the full document. Instead of compressing everything indiscriminately, we propose CoMem, a query-guided selective context compression framework that learns memory representations conditioned on both the document and the query. CoMem develops Contrastive Compressor Learning to achieve query-sensitive yet context-preserving memory representations. To inject a strong prior over query-dependent evidence distributions, we devise a same-context hard negative mining strategy by pairing a single document with multiple queries, requiring the compressor to selectively retain information according to each query's distinct evidence needs. To address the restricted information flow imposed by causal masking, we further enable bidirectional attention among memory slots. Experiments demonstrate that CoMem consistently outperforms existing context compression methods across compression ratios on question answering and RULER-NIAH benchmarks. Our code is available at https://github.com/scar-on/CoMem.
Figures & tables
Figure 1. Comparison between the query-agnostic compression and CoMem. (a) The query-agnostic method learns memory slots supervised by the QA loss. (b) Our method further introduces contrastive compressor learning and trains on different queries from the same context. Comparison between the query-agnostic compression and \method{}. (a) The query-agnostic method learns memory slots supervised by the QA loss. (b) Our method further introduces contrastive compressor learning and trains on different queries from the same context.
Figure 2. Similarity patterns of compressed memory slots. (a) and (b) show memory slots from different contexts, (c) and (d) show memory slots from the same context under different queries. In both settings, our method achieves lower inter-slot similarity, demonstrating more discriminative query-guided compression. Similarity patterns of compressed memory slots. (a) and (b) show memory slots from different contexts, (c) and (d) show memory slots from the same context under different queries. In both settings, our method achieves lower inter-slot similarity, demonstrating more discriminative query-guided compression.
Figure 3. Overview of CoMem. The QA objective LQA provides an end-task generation signal, while the contrastive learning objective Lcl imposes a more explicit representation-level signal, aligning memory slots with their source context while using same-context, different-query examples as hard negatives to encourage query-guided memory representations. Overview of \method{}. The QA objective \textcolor{red}{LQA} provides an end-task generation signal, while the contrastive learning objective \textcolor{red}{Lcl} imposes a more explicit representation-level signal, aligning memory slots with their source context while using same-context, different-query examples as hard negatives to encourage query-guided memory representations.
Type
Model
2Wiki
HotpotQA
MuSiQue
NQ
NIAH
ACC
LLM Judge
ACC
LLM Judge
ACC
LLM Judge
ACC
LLM Judge
ACC
RAG (No Compression)
Mistral-7B
0.7654
0.7088
0.7751
0.8249
0.5333
0.5085
0.7905
0.7671
0.886
Qwen3-4B
0.7927
0.6236
0.8164
0.8220
0.3814
0.2880
0.8007
0.7405
1.0
Mistral-7B: 4 × Compression
Hard compression
LLMLingua2
0.3905
0.3379
0.3612
0.4848
0.1614
0.2284
0.5081
0.6121
0.790
Table 1. Overall performance on four question-answering benchmarks and RULER-NIAH. All rows use Mistral-7B except the final two CoMem rows, which use Qwen3-4B. SeleCom-82 × ∗ denotes the best-performing configuration reported in the original paper, and △ marks query-aware methods. Bold and underlined numbers denote the best and second-best results within each Mistral-7B compression setting.
Model
2Wiki
HotpotQA
MuSiQue
NQ
Mistral-7B-8 ×
0.5863
0.6437
0.2826
0.7524
Mistral-7B-16 ×
0.5988
0.6411
0.2627
0.7524
Mistral-7B-64 ×
0.5724
0.5925
0.2466
0.7223
Mistral-7B-128 ×
0.5756
0.5975
0.2594
0.7294
Qwen3-4B-8 ×
0.5833
0.6578
0.3434
0.7010
Qwen3-4B-16 ×
0.5540
0.6366
0.2974
0.6997
Table 2. Accuracy (ACC) of CoMem across different compression ratios.
Contrastive
Bi-Mask
2Wiki
HotpotQA
MuSiQue
NQ
Stage I
×
✓
0.2785
0.2440
0.0709
0.2560
×
×
0.2852
0.2447
0.0861
0.2665
✓
×
0.2627
0.1492
0.0203
0.1160
✓
✓
0.3761
0.5211
0.2280
0.6197
Stage II
Table 3. Ablation study on Stage I and Stage II training components.
Figure 4. Optimization dynamics for same-context and different-context negatives. Optimization dynamics for same-context and different-context negatives.
Negative Type
2Wiki
HotpotQA
MuSiQue
NQ
Different-Context
0.3681
0.4947
0.2201
0.6047
Same-Context
0.3761
0.5211
0.2280
0.6197
Table 4. Ablation study on same-context hard negatives and different-context negatives.
Method
2Wiki
HotpotQA
MuSiQue
NQ
CoMem w/ true query
0.5591
0.6349
0.2971
0.7013
CoMem w/ general query
0.3786
0.4607
0.2098
0.5597
Table 5. ACC with the true downstream query or a general-purpose query at 32 × compression ratios.
GFlops
Compression ratio
4 ×
16 ×
32 ×
64 ×
Full Context
10031
Ours
14074
10800
10256
9985
Time/sample (s)
Compression ratio
4 ×
16 ×
32 ×
64 ×
Full Context
0.053
Table 6. Efficiency comparison of CoMem across different compression ratios.
Figure 5. Needle-signal retention under different compression ratios. Needle-signal retention under different compression ratios.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Batch size
128
Micro batch size per GPU
8
Gradient accumulation steps
16
Learning rate
1×10−4
LR scheduler
cosine with minimum LR
Optimizer
DeepSpeed FusedAdam
Appendix
Table 7. Training hyperparameters.
Shared context. This document describes the history of Azerbaijani press. Hasan bey Zardabi founded publishing in Azerbaijan and sought permission to publish the native-language newspaper “Akinchi” under the Tsar’s colonial system. The newspaper drew a wide response in the Caucasus, and Zardabi served as its publisher, editor, corrector, and typesetter. From 1875 to 1877, 56 issues were published. Zardabi distributed the newspaper gratuitously to arouse public interest and used it to promote national enlightenment, improve peasants’ agricultural literacy, and support national unity. During the Russo-Turkish War of 1877–1878, Tsarist censorship prohibited political discussion, and in 1877 a decree ordered the newspaper’s liquidation.
ID
Query
Answer
1
Who is considered the founder of publishing in Azerbaijan?
Hasan bey Zardabi is considered the founder of publishing in Azerbaijan.
2
During which period was the newspaper “Akinchi” published?
The newspaper “Akinchi” was published during 1875–1877.
3
How many issues of the “Akinchi” newspaper were published between 1875 and 1877?
56 issues.
4
What role did Hasan bey Zardabi play in the publication of the “Akinchi” newspaper?
Hasan bey Zardabi was the founder, publisher, editor, corrector, and typesetter of the “Akinchi” newspaper.
5
What was the main goal of Hasan bey Zardabi in publishing the “Akinchi” newspaper?
His main goal was the propagation of ideas of the nation’s enlightenment.
Appendix
Table 8. A same-context example from the contrastive data. One context about the history of Azerbaijani press supports eight different queries and answers.
Question
What contest did Drew Barry’s younger brother win in 1996?
Gold Answer
the 1996 NBA Slam Dunk Contest
Qwen3-4B
There is no information in the provided documents indicating that Drew Barry’s younger brother won any contest in 1996. In fact, the documents mention that Brent Barry won the 1996 NBA Slam Dunk Contest , but Brent is not Drew Barry’s younger brother. However, the documents do not state that Brent won a contest in 1996 as a younger brother of Drew. (mentions the answer but rejects it)
CoMem
Drew Barry’s younger brother, Brent, won the NBA Slam Dunk Contest in 1996.
Question
Who is the actor who plays the title character in The Bourne Betrayal?
Gold Answer
Matt Damon
Qwen3-4B
The actor who plays the title character in The Bourne Betrayal is Matt Damon . However, it is important to clarify: The Bourne Betrayal is a novel by Eric Van Lustbader, not a film. Matt Damon does not appear in the novel, as it is a book, not a movie. Therefore, there is no actor who plays the title character in the novel The Bourne Betrayal. (self-contradictory final answer)
Appendix
Table 9. Qualitative examples where lexical ACC marks the Qwen3-4B original output as correct because the gold answer is mentioned, while LLM-as-a-Judge rejects it due to refusal, contradiction, or over-qualification. In contrast, CoMem produces a direct answer that is accepted by both ACC and LLM-as-a-Judge.
Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latency, energy consumption, and GPU memory demand as context length scales. Existing soft-compression methods either lack query-guided memory selection at inference time, train without answer-targeted supervision, or couple compression tightly to a specific decoder architecture. We propose a Context-to-Answer-Aligned Memory Compression (CMC) framework, which compresses long input contexts into compact Context Memory Embeddings (CMEs) aligned to any frozen decoder's embedding space, reducing inference costs without modifying decoder weights. CMC introduces a two-tier KV cache that combines question-guided CME selection with a local context window, and trains the compressor with answer-targeted distillation from a frozen LLM. Experiments across nine encoder-decoder combinations and four QA benchmarks show that CMC consistently outperforms the baseline, achieving up to 7.3 EM and 4.0 F1 point gains on SQuAD, while reducing inference time and energy consumption by up to 20% and peak reserved GPU memory by up to 50% at 3,000 generation tokens. Ablation studies confirm that each architectural component and training objective contributes to the performance.
Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA
Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and the key-value (KV) cache grows with the number of processed tokens. Larger context windows also do not ensure reliable evidence use. Context compression reduces this cost, but many soft-compression methods use LLMs as compressors and rely on compact memory tokens both to preserve information and to condition the decoder. We propose SeDeM, a selective decompression framework that decouples compact memory storage from decoder conditioning. SeDeM stores context as compact hidden-state memory blocks, selects query-relevant blocks, and decompresses only the selected blocks for decoder conditioning. Thus, the decoder avoids both full-context processing and direct generation from highly compressed memory slots. On four long-context QA benchmarks, SeDeM achieves higher QA scores than the compression baselines in our main comparison in both 1B and 3B same-backbone settings, and with the 3B backbone exceeds full-context fine-tuning on three datasets. SeDeM also provides favorable quality--efficiency trade-offs, achieving 1.74--2.46× lower online time-to-first-token and 1.08--1.10× higher autoregressive decoding throughput relative to ICAE while maintaining strong answer quality.
Efficient long-context inference is not only about reducing memory cost, but also about keeping useful contextual evidence accessible as generation proceeds. However, existing compression-oriented approaches, such as key-value (KV) cache compression and context compression, often either make an early decision about which contextual information to keep or rely on an external compressor. Such designs make it difficult to adapt the compressed context to the evidence needed by later reasoning steps. This paper introduces PReM (Preserve and Refresh Memory), a context-compression framework that maintains the long context as the model's internal layer-wise KV memory and learns what to preserve and when to refresh it. Specifically, PReM uses a dedicated memory layer to make memory-selection decisions, and a special memory token <m> to trigger refreshes during generation. To train this behavior, PReM introduces Phase-Separated Refresh Training, aligning memory selection with memory-conditioned generation while preserving continuity across refreshes. Experiments with 32K-token contexts show that PReM outperforms strong baselines under both 16x and 32x compression, while maintaining a favorable balance between answer quality and inference efficiency.