Beyond Compressing Everything: Contrastive Memory Learning for Selective Context Compression
Organizations: Institute of Automation, Chinese Academy of Sciences, China · Alibaba Group China · Hello Group China
Abstract
As large language models (LLMs) advance toward increasingly long inputs, soft context compression offers a promising approach to efficient inference by converting long inputs into compact memory slots. Under limited memory budgets, effective context compression remains challenging because task-relevant evidence varies substantially across queries, ranging from localized evidence to multiple distant spans or information dispersed throughout the full document. Instead of compressing everything indiscriminately, we propose CoMem, a query-guided selective context compression framework that learns memory representations conditioned on both the document and the query. CoMem develops Contrastive Compressor Learning to achieve query-sensitive yet context-preserving memory representations. To inject a strong prior over query-dependent evidence distributions, we devise a same-context hard negative mining strategy by pairing a single document with multiple queries, requiring the compressor to selectively retain information according to each query's distinct evidence needs. To address the restricted information flow imposed by causal masking, we further enable bidirectional attention among memory slots. Experiments demonstrate that CoMem consistently outperforms existing context compression methods across compression ratios on question answering and RULER-NIAH benchmarks. Our code is available at https://github.com/scar-on/CoMem.
Figures & tables
| Type | Model | 2Wiki | HotpotQA | MuSiQue | NQ | NIAH | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| ACC | LLM Judge | ACC | LLM Judge | ACC | LLM Judge | ACC | LLM Judge | ACC | ||
| RAG (No Compression) | ||||||||||
| Mistral-7B | 0.7654 | 0.7088 | 0.7751 | 0.8249 | 0.5333 | 0.5085 | 0.7905 | 0.7671 | 0.886 | |
| Qwen3-4B | 0.7927 | 0.6236 | 0.8164 | 0.8220 | 0.3814 | 0.2880 | 0.8007 | 0.7405 | 1.0 | |
| Mistral-7B: 4 Compression | ||||||||||
| Hard compression | LLMLingua2 | 0.3905 | 0.3379 | 0.3612 | 0.4848 | 0.1614 | 0.2284 | 0.5081 | 0.6121 | 0.790 |
| Model | 2Wiki | HotpotQA | MuSiQue | NQ |
|---|---|---|---|---|
| Mistral-7B-8 | 0.5863 | 0.6437 | 0.2826 | 0.7524 |
| Mistral-7B-16 | 0.5988 | 0.6411 | 0.2627 | 0.7524 |
| Mistral-7B-64 | 0.5724 | 0.5925 | 0.2466 | 0.7223 |
| Mistral-7B-128 | 0.5756 | 0.5975 | 0.2594 | 0.7294 |
| Qwen3-4B-8 | 0.5833 | 0.6578 | 0.3434 | 0.7010 |
| Qwen3-4B-16 | 0.5540 | 0.6366 | 0.2974 | 0.6997 |
| Contrastive | Bi-Mask | 2Wiki | HotpotQA | MuSiQue | NQ |
|---|---|---|---|---|---|
| Stage I | |||||
| 0.2785 | 0.2440 | 0.0709 | 0.2560 | ||
| 0.2852 | 0.2447 | 0.0861 | 0.2665 | ||
| 0.2627 | 0.1492 | 0.0203 | 0.1160 | ||
| 0.3761 | 0.5211 | 0.2280 | 0.6197 | ||
| Stage II | |||||
| Negative Type | 2Wiki | HotpotQA | MuSiQue | NQ |
|---|---|---|---|---|
| Different-Context | 0.3681 | 0.4947 | 0.2201 | 0.6047 |
| Same-Context | 0.3761 | 0.5211 | 0.2280 | 0.6197 |
| Method | 2Wiki | HotpotQA | MuSiQue | NQ |
|---|---|---|---|---|
| CoMem w/ true query | 0.5591 | 0.6349 | 0.2971 | 0.7013 |
| CoMem w/ general query | 0.3786 | 0.4607 | 0.2098 | 0.5597 |
| GFlops | ||||
|---|---|---|---|---|
| Compression ratio | 4 | 16 | 32 | 64 |
| Full Context | 10031 | |||
| Ours | 14074 | 10800 | 10256 | 9985 |
| Time/sample (s) | ||||
| Compression ratio | 4 | 16 | 32 | 64 |
| Full Context | 0.053 | |||
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Batch size | 128 |
| Micro batch size per GPU | 8 |
| Gradient accumulation steps | 16 |
| Learning rate | |
| LR scheduler | cosine with minimum LR |
| Optimizer | DeepSpeed FusedAdam |
| Shared context. This document describes the history of Azerbaijani press. Hasan bey Zardabi founded publishing in Azerbaijan and sought permission to publish the native-language newspaper “Akinchi” under the Tsar’s colonial system. The newspaper drew a wide response in the Caucasus, and Zardabi served as its publisher, editor, corrector, and typesetter. From 1875 to 1877, 56 issues were published. Zardabi distributed the newspaper gratuitously to arouse public interest and used it to promote national enlightenment, improve peasants’ agricultural literacy, and support national unity. During the Russo-Turkish War of 1877–1878, Tsarist censorship prohibited political discussion, and in 1877 a decree ordered the newspaper’s liquidation. | ||
| ID | Query | Answer |
| 1 | Who is considered the founder of publishing in Azerbaijan? | Hasan bey Zardabi is considered the founder of publishing in Azerbaijan. |
| 2 | During which period was the newspaper “Akinchi” published? | The newspaper “Akinchi” was published during 1875–1877. |
| 3 | How many issues of the “Akinchi” newspaper were published between 1875 and 1877? | 56 issues. |
| 4 | What role did Hasan bey Zardabi play in the publication of the “Akinchi” newspaper? | Hasan bey Zardabi was the founder, publisher, editor, corrector, and typesetter of the “Akinchi” newspaper. |
| 5 | What was the main goal of Hasan bey Zardabi in publishing the “Akinchi” newspaper? | His main goal was the propagation of ideas of the nation’s enlightenment. |
| Question | What contest did Drew Barry’s younger brother win in 1996? |
|---|---|
| Gold Answer | the 1996 NBA Slam Dunk Contest |
| Qwen3-4B | There is no information in the provided documents indicating that Drew Barry’s younger brother won any contest in 1996. In fact, the documents mention that Brent Barry won the 1996 NBA Slam Dunk Contest , but Brent is not Drew Barry’s younger brother. However, the documents do not state that Brent won a contest in 1996 as a younger brother of Drew. (mentions the answer but rejects it) |
| CoMem | Drew Barry’s younger brother, Brent, won the NBA Slam Dunk Contest in 1996. |
| Question | Who is the actor who plays the title character in The Bourne Betrayal? |
| Gold Answer | Matt Damon |
| Qwen3-4B | The actor who plays the title character in The Bourne Betrayal is Matt Damon . However, it is important to clarify: The Bourne Betrayal is a novel by Eric Van Lustbader, not a film. Matt Damon does not appear in the novel, as it is a book, not a movie. Therefore, there is no actor who plays the title character in the novel The Bourne Betrayal. (self-contradictory final answer) |