Retrieval-augmented generation (RAG) is bottlenecked by the token budget of large language models: feeding long documents into the context window is expensive, while aggressive textual compression discards fine-grained information that is irreversibly lost in a one-dimensional token sequence. We argue that this bottleneck is not fundamental to information density but to the choice of compression modality. Modern visual encoders can pack the contents of a rendered text page into roughly a hundred visual tokens with little semantic loss, opening a new compression-fidelity tradeoff that purely textual methods cannot reach. We present OpticalRAG, a framework that renders documents as page images, encodes them with a frozen vision encoder, and distills the retrieved visual tokens into as few as 16 injected tokens before passing them to the LLM. Two ingredients make this work without expensive cross-modal training: (i) Encoder-Bridged Retrieval, which projects text-query states into the visual feature space of the frozen encoder, through a projection shared with the compressor, to retrieve visual tokens from a text query, with no contrastive training; and (ii) Query-Driven Distillation, a lightweight cross-attention compressor that condenses the retrieved visual tokens into a small set of query-conditioned prefix tokens. On LongBench-v2, BAMBOO, and LooGLE-v2, OpticalRAG achieves the best compression-aware accuracy (CAP) on all three benchmarks while injecting only 16 tokens, and improves the closed-book Qwen2.5-7B-Instruct backbone by 5.17 points on LongBench-v2. Code is available at https://github.com/brighterr/OpticalRAG.
Figures & tables
Figure 1: Illustration of OpticalRAG. (a) Token-space methods prune or summarize the 1D text sequence, causing permanent information loss at high compression ratios. (b) Dense-vector methods squeeze each document into a single embedding, collapsing fine-grained details. (c) Our OpticalRAG renders documents into page images and encodes them into compact visual tokens via a frozen ViT. At query time, a cross-attention module distills retrieved visual tokens into a small set of injected tokens conditioned on the user query.
Figure 2: OpticalRAG framework overview. ① Optical Encoding : document chunks are rendered as page images and encoded by a frozen VLM encoder into visual tokens; each visual token is indexed individually. ② Encoder-Bridged Retrieval : the query is projected into visual space via Wq and matched to the index by inner product to retrieve top- K visual tokens. ③ Query-Driven Distillation : a cross-attention module with learned compression queries distills retrieved visual tokens into a compact prefix fed into the frozen LLM.
Backbone
Std. RAG
Sel. Context
LLMLingua-v2
Xrag
RECOMP
Ours
LongBench-v2
Score
30.62
31.41
32.40
28.23
26.24
32.01
35.79
Len
0
1622.68
5511.73
2669.19
1
6953.93
16
CAP ↑
30.62
34.71
33.69
28.23
26.24
32.90
49.68
Bamboo
Score
56.75
69.50
70.00
73.50
51.00
75.50
61.00
Len
0
814.22
3141.29
1667.27
1
2415.46
16
CAP ↑
56.75
79.50
74.85
81.12
51.00
81.80
84.67
Table 1: Main results on LongBench-v2, Bamboo, and LooGLE-v2.
Figure 3: Ablation on the number of injected tokens.
Figure 4: Ablation on query-driven versus query-agnostic compression on LongBench-v2.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Format
Samples
Notes
LongBench-v2 train
HuggingFace Arrow
503
6 domains / 16 sub-domains
BAMBOO choice subset
JSONL
400
4 tasks, 100 samples each
LooGLE-v2 choice subset
JSONL
1334
6 tasks with mixed label spaces
Appendix
Table 2: Statistics of the datasets used in our experiments.
Asset
Source / provider
Published license notice
LongBench-v2
THUDM / Z.ai
Apache-2.0 (dataset card).
BAMBOO
RUCAIBox
MIT (repository release); underlying source content retains its own terms.
LooGLE-v2
GraphPKU
CC BY-SA 4.0 (dataset card).
Wikipedia
Atlas / Wikimedia
CC BY-SA 3.0; enwiki-dec2021 release.
C4
Google / Ai2 distribution
ODC-BY for the Ai2 distribution, together with Common Crawl terms.
SQuAD v2
Stanford SQuAD
CC BY-SA 4.0.
Appendix
Table 3: Dataset sources and published license notices. Source-specific qualifications are explained below.
Asset
Source / provider
Published license notice
DeepSeek-OCR
DeepSeek
MIT (model card and repository ).
Qwen2.5-7B-Instruct
Qwen / Alibaba
Apache-2.0.
SAM implementation
Meta
Apache-2.0.
CLIP implementation
OpenAI
MIT.
DejaVu Sans Mono
DejaVu / Bitstream
Bitstream Vera font terms; DejaVu modifications are public domain, with additional notices for imported glyphs.
Appendix
Table 4: Pretrained components and rendering font: upstream license notices.
Module
Sub-components
Params
Status
Role
DeepEncoder
SAM-ViT-B + CLIP-L; loaded from DeepSeek-OCR checkpoint via safetensors
—
FROZEN
Maps 640×640 page images to T=100 visual tokens of dim dvis=1024 .
Qwen2.5-7B-Instruct
Hidden dim 3584; loaded from the pinned Qwen checkpoint
—
FROZEN
Provides query hidden states; scores final answer choices via next-token logits.
Projects query into visual space; downsamples and cross-attends visual tokens into M=16 compressed tokens conditioned on the query.
VisualToLLM Bridge
Linear (1024→3584) , GELU, Dropout (0.1) , Linear (3584→3584) , LN
16,529,408
TRAINABLE
Projects 16 compressed visual tokens into the LLM’s embedding space; prepended as a soft prefix to the query.
Appendix
Table 5: Model architecture and parameter budget. Compressor and bridge counts are computed from the instantiated modules. Their allocated parameters total 30,716,928 ( ≈30.72 M).
Tier
Context length threshold
sf (px)
sm (px)
sl
1
≤300,000 chars
15
20
1.30
2
≤900,000 chars
15
22
1.35
3
>900,000 chars
16
24
1.45
Appendix
Table 6: Length-adaptive rendering density tiers.
Flag value
Formula
Notes
mean
nq1i∑h~q,i
Smooth global average; the natural default.
max
imaxh~q,i (element-wise)
Released inference configuration. Element-wise maximum; pooling does not mask padding positions.
last
h~q,nq
Last-token summary; analogous to causal LM hidden state.
max_mean
21(imaxh~q,i+nq1i∑h~q,i)
Arithmetic mean of max and mean pooling.
attn
i∑wih~q,i , wi=softmaxi(a⊤h~q,i)
Learned scalar scoring with a∈R1024 . Allocated in all modes, but used only in this mode.
none
0∈RB×1×1024
Disables query bias in the compressor only; retrieval still uses max-pooled projected query states.
Appendix
Table 7: Query pooling strategies. Input: H~q∈RB×nq×1024 . Output: qg∈RB×1×1024 . The max strategy (shaded) is used by the released LongBench-v2 entry point.
score_type
Token score formula s(v)
Multi- Q ?
Notes
dot
v⊤q1/d
No
Default. Single global query; plain inner product.
cosine
v^⊤q^1/d
No
L2-normalised dot product; scale-invariant.
maxsim
maxjv⊤qj/d
Yes
ColBERT-style MaxSim over the full projected query sequence.
maxsim_cosine
maxjv^⊤q^j/d
Yes
Normalised MaxSim.
dot_multi
∣Q∣1∑jv⊤qj/d
Yes
Mean dot product across all query vectors.
cosine_multi
∣Q∣1∑jv^⊤q^j/d
Yes
Mean cosine similarity.
Appendix
Table 8: Scoring functions in QueryVisualScorer (FAISS scaling differs). v∈Rd is a visual token; (⋅)^ denotes L2 normalisation; qj is the j -th query vector; d=1024 . The dot entry (shaded) is the default.
context_sampling
Window start positions
Description
head
{0}
Default. Always encodes the first Lwin characters. Appropriate when evidence is front-loaded (e.g. news articles, paper abstracts, legal preambles).
uniform
{round(W−1i(∣D∣−Lwin))}i=0W−1
Evenly-spaced windows spanning the full document length. When W=1 , reduces identically to head .
head_tail
{0,∣D∣−Lwin} + W−2 uniform midpoints
Anchors at the document’s head and tail, with uniformly-spaced intermediate windows. Useful when critical evidence is concentrated near document boundaries.
Appendix
Table 9: Context-window placement strategies. D is the full document of length ∣D∣ ; windows are of width Lwin . The head strategy is the library default for this optional path.
Table 11: Two-stage training settings for reconstruction pretraining and task-aware fine-tuning.
Priority
Regex pattern
Example match
1
The correct answer is\s*([A−D])
“The correct answer is (B)”
2
The correct answer is\s*([A-D])
“The correct answer is B”
3
([A−D])
“…the choice (C) is…”
4
\b([A-D])\b
“…therefore D.”
Appendix
Table 12: Answer-extraction regex cascade ( eval.py ). All patterns are applied case-insensitively via re.IGNORECASE . The first match is returned; if none match, the empty string is returned.
Method
Easy
Hard
Short
Medium
Long
Avg. ↑
Mean-Pooled
36.98
28.30
30.56
31.63
33.33
32.16
Random-Proj.
36.46
28.30
32.78
31.16
29.63
31.67
Query-Driven
37.50
34.73
38.33
35.81
31.48
35.57
Appendix
Table 13: Ablation on query-driven compression on LongBench-v2.
Cat.
1
2
4
16∗
8
32
64
Overall
31.0
31.6
31.4
35.8
32.8
30.0
34.2
Diff.
Easy
33.9
34.4
34.4
37.5
38.0
30.7
38.5
Hard
29.3
29.9
29.6
34.7
29.6
29.6
31.5
Len.
Sht.
34.4
35.0
32.8
38.3
36.1
34.4
34.4
Med.
27.4
30.2
33.0
35.8
30.7
25.6
33.5
Lng.
32.4
28.7
25.9
31.5
31.5
31.5
35.2
Appendix
Table 14: Ablation on the number of injected tokens.
Retrieval-Augmented Generation (RAG) has become essential for knowledge-intensive question answering, yet scaling RAG pipelines remains challenging due to the prohibitive computational cost of processing lengthy retrieved contexts. Existing compression approaches face a fundamental trade-off: hard compression methods operate online in a query-aware fashion but achieve only modest compression rates and typically require fine-tuning the generative model, while soft compression methods attain higher ratios but rely on costly offline encoding that is entirely agnostic to the input query. To bridge this gap, we introduce RAGOCR, a novel framework that compresses retrieved documents into compact visual representations conditioned on the input query. To further balance compression rate and information fidelity, we introduce a query-aware dynamic resolution mechanism that adaptively allocates visual granularity based on each document's estimated relevance and complexity: highly relevant passages are rendered at higher resolution to preserve fine-grained details, while peripheral documents are aggressively compressed at lower resolution. Experiments on five QA benchmarks using the MedOmniKB retrieval corpus demonstrate that RAGOCR surpasses naive RAG by over 15% in accuracy while requiring only one-eighth the number of input tokens, and consistently outperforms both hard and soft compression baselines across varying retrieval depths.
Jiayang Yu, Jialun Zhong, Lei Zou
Wangxuan Institute of Computer Technology, Peking University Beijing, China
The expansion of retrieval-augmented generation (RAG) into multimodal domains has intensified the challenge for processing complex visual documents, such as financial reports. While page-level chunking and retrieval is a natural starting point, it creates a critical bottleneck: delivering entire pages to the generator introduces excessive extraneous context. This not only overloads the generator's attention mechanism but also dilutes the most salient evidence. Moreover, compressing these information-rich pages into a limited visual token budget further increases the risk of hallucinations. To address this, we introduce AgenticOCR, a dynamic parsing paradigm that transforms optical character recognition (OCR) from a static, full-text process into a query-driven, on-demand extraction system. By autonomously analyzing document layout in a "thinking with images" manner, AgenticOCR identifies and selectively recognizes regions of interest. This approach performs on-demand decompression of visual tokens precisely where needed, effectively decoupling retrieval granularity from rigid page-level chunking. AgenticOCR has the potential to serve as the "third building block" of the visual document RAG stack, operating alongside and enhancing standard Embedding and Reranking modules. Experimental results demonstrate that AgenticOCR improves both the efficiency and accuracy of visual RAG systems, achieving expert-level performance in long document understanding. Code and models are available at https://github.com/OpenDataLab/AgenticOCR.
Zhengren Wang, Dongsheng Ma, Huaping Zhong +4
Shanghai Artificial Intelligence Laboratory · Peking University · SenseTime
Retrieval-augmented generation (RAG) improves knowledge-intensive large language model (LLM) applications by conditioning generation on retrieved documents, but longer contexts increase latency, key-value (KV) cache memory, and token cost. Post-retrieval compression can reduce this cost, yet existing compressors often operate independently for each query, rely on auxiliary models or rewriting, and introduce online overhead that can offset the benefit of shorter prompts. We revisit RAG compression from a data-mining perspective by aggregating historical query--document--model interactions into reusable evidence views. We first show that modern compressors have unstable gains over simple truncation and can add substantial inference-time latency. We then propose Reusable Evidence View Aggregation (REVA), a framework that mines the target generator's historical attention traces into a document-keyed, budget-agnostic score store. REVA maps token-level attention to readable word units, aggregates importance across repeated document accesses, and renders budget-specific plain-text views that preserve document order and the standard RAG interface. Across four representative benchmarks and modern LLMs, REVA improves generation quality by 1.0--5.8 points over existing advances, while reducing compression overhead by a factor of 5.3 to 15.6, adding less than 40 ms of latency.
Tuan Nguyen, Qiran Hu, Banruo Liu +3
VinUni-Illinois Smart Health Center, VinUniversity, Vietnam · University of Illinois Urbana-Champaign, USA