Retrieval-augmented generation (RAG) is bottlenecked by the token budget of large language models: feeding long documents into the context window is expensive, while aggressive textual compression discards fine-grained information that is irreversibly lost in a one-dimensional token sequence. We argue that this bottleneck is not fundamental to information density but to the choice of compression modality. Modern visual encoders can pack the contents of a rendered text page into roughly a hundred visual tokens with little semantic loss, opening a new compression-fidelity tradeoff that purely textual methods cannot reach. We present OpticalRAG, a framework that renders documents as page images, encodes them with a frozen vision encoder, and distills the retrieved visual tokens into as few as 16 injected tokens before passing them to the LLM. Two ingredients make this work without expensive cross-modal training: (i) Encoder-Bridged Retrieval, which projects text-query states into the visual feature space of the frozen encoder, through a projection shared with the compressor, to retrieve visual tokens from a text query, with no contrastive training; and (ii) Query-Driven Distillation, a lightweight cross-attention compressor that condenses the retrieved visual tokens into a small set of query-conditioned prefix tokens. On LongBench-v2, BAMBOO, and LooGLE-v2, OpticalRAG achieves the best compression-aware accuracy (CAP) on all three benchmarks while injecting only 16 tokens, and improves the closed-book Qwen2.5-7B-Instruct backbone by 5.17 points on LongBench-v2. Code is available at https://github.com/brighterr/OpticalRAG.
Figures & tables
Figure 1: Illustration of OpticalRAG. (a) Token-space methods prune or summarize the 1D text sequence, causing permanent information loss at high compression ratios. (b) Dense-vector methods squeeze each document into a single embedding, collapsing fine-grained details. (c) Our OpticalRAG renders documents into page images and encodes them into compact visual tokens via a frozen ViT. At query time, a cross-attention module distills retrieved visual tokens into a small set of injected tokens conditioned on the user query.
Figure 2: OpticalRAG framework overview. ① Optical Encoding : document chunks are rendered as page images and encoded by a frozen VLM encoder into visual tokens; each visual token is indexed individually. ② Encoder-Bridged Retrieval : the query is projected into visual space via Wq and matched to the index by inner product to retrieve top- K visual tokens. ③ Query-Driven Distillation : a cross-attention module with learned compression queries distills retrieved visual tokens into a compact prefix fed into the frozen LLM.
Backbone
Std. RAG
Sel. Context
LLMLingua-v2
Xrag
RECOMP
Ours
LongBench-v2
Score
30.62
31.41
32.40
28.23
26.24
32.01
35.79
Len
0
1622.68
5511.73
2669.19
1
6953.93
16
CAP ↑
30.62
34.71
33.69
28.23
26.24
32.90
49.68
Bamboo
Score
56.75
69.50
70.00
73.50
51.00
75.50
61.00
Len
0
814.22
3141.29
1667.27
1
2415.46
16
CAP ↑
56.75
79.50
74.85
81.12
51.00
81.80
84.67
Table 1: Main results on LongBench-v2, Bamboo, and LooGLE-v2.
Figure 3: Ablation on the number of injected tokens.
Figure 4: Ablation on query-driven versus query-agnostic compression on LongBench-v2.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Format
Samples
Notes
LongBench-v2 train
HuggingFace Arrow
503
6 domains / 16 sub-domains
BAMBOO choice subset
JSONL
400
4 tasks, 100 samples each
LooGLE-v2 choice subset
JSONL
1334
6 tasks with mixed label spaces
Appendix
Table 2: Statistics of the datasets used in our experiments.
Asset
Source / provider
Published license notice
LongBench-v2
THUDM / Z.ai
Apache-2.0 (dataset card).
BAMBOO
RUCAIBox
MIT (repository release); underlying source content retains its own terms.
LooGLE-v2
GraphPKU
CC BY-SA 4.0 (dataset card).
Wikipedia
Atlas / Wikimedia
CC BY-SA 3.0; enwiki-dec2021 release.
C4
Google / Ai2 distribution
ODC-BY for the Ai2 distribution, together with Common Crawl terms.
SQuAD v2
Stanford SQuAD
CC BY-SA 4.0.
Appendix
Table 3: Dataset sources and published license notices. Source-specific qualifications are explained below.
Asset
Source / provider
Published license notice
DeepSeek-OCR
DeepSeek
MIT (model card and repository ).
Qwen2.5-7B-Instruct
Qwen / Alibaba
Apache-2.0.
SAM implementation
Meta
Apache-2.0.
CLIP implementation
OpenAI
MIT.
DejaVu Sans Mono
DejaVu / Bitstream
Bitstream Vera font terms; DejaVu modifications are public domain, with additional notices for imported glyphs.
Appendix
Table 4: Pretrained components and rendering font: upstream license notices.
Module
Sub-components
Params
Status
Role
DeepEncoder
SAM-ViT-B + CLIP-L; loaded from DeepSeek-OCR checkpoint via safetensors
—
FROZEN
Maps 640×640 page images to T=100 visual tokens of dim dvis=1024 .
Qwen2.5-7B-Instruct
Hidden dim 3584; loaded from the pinned Qwen checkpoint
—
FROZEN
Provides query hidden states; scores final answer choices via next-token logits.
Projects query into visual space; downsamples and cross-attends visual tokens into M=16 compressed tokens conditioned on the query.
VisualToLLM Bridge
Linear (1024→3584) , GELU, Dropout (0.1) , Linear (3584→3584) , LN
16,529,408
TRAINABLE
Projects 16 compressed visual tokens into the LLM’s embedding space; prepended as a soft prefix to the query.
Appendix
Table 5: Model architecture and parameter budget. Compressor and bridge counts are computed from the instantiated modules. Their allocated parameters total 30,716,928 ( ≈30.72 M).
Tier
Context length threshold
sf (px)
sm (px)
sl
1
≤300,000 chars
15
20
1.30
2
≤900,000 chars
15
22
1.35
3
>900,000 chars
16
24
1.45
Appendix
Table 6: Length-adaptive rendering density tiers.
Flag value
Formula
Notes
mean
nq1i∑h~q,i
Smooth global average; the natural default.
max
imaxh~q,i (element-wise)
Released inference configuration. Element-wise maximum; pooling does not mask padding positions.
last
h~q,nq
Last-token summary; analogous to causal LM hidden state.
max_mean
21(imaxh~q,i+nq1i∑h~q,i)
Arithmetic mean of max and mean pooling.
attn
i∑wih~q,i , wi=softmaxi(a⊤h~q,i)
Learned scalar scoring with a∈R1024 . Allocated in all modes, but used only in this mode.
none
0∈RB×1×1024
Disables query bias in the compressor only; retrieval still uses max-pooled projected query states.
Appendix
Table 7: Query pooling strategies. Input: H~q∈RB×nq×1024 . Output: qg∈RB×1×1024 . The max strategy (shaded) is used by the released LongBench-v2 entry point.
score_type
Token score formula s(v)
Multi- Q ?
Notes
dot
v⊤q1/d
No
Default. Single global query; plain inner product.
cosine
v^⊤q^1/d
No
L2-normalised dot product; scale-invariant.
maxsim
maxjv⊤qj/d
Yes
ColBERT-style MaxSim over the full projected query sequence.
maxsim_cosine
maxjv^⊤q^j/d
Yes
Normalised MaxSim.
dot_multi
∣Q∣1∑jv⊤qj/d
Yes
Mean dot product across all query vectors.
cosine_multi
∣Q∣1∑jv^⊤q^j/d
Yes
Mean cosine similarity.
Appendix
Table 8: Scoring functions in QueryVisualScorer (FAISS scaling differs). v∈Rd is a visual token; (⋅)^ denotes L2 normalisation; qj is the j -th query vector; d=1024 . The dot entry (shaded) is the default.
context_sampling
Window start positions
Description
head
{0}
Default. Always encodes the first Lwin characters. Appropriate when evidence is front-loaded (e.g. news articles, paper abstracts, legal preambles).
uniform
{round(W−1i(∣D∣−Lwin))}i=0W−1
Evenly-spaced windows spanning the full document length. When W=1 , reduces identically to head .
head_tail
{0,∣D∣−Lwin} + W−2 uniform midpoints
Anchors at the document’s head and tail, with uniformly-spaced intermediate windows. Useful when critical evidence is concentrated near document boundaries.
Appendix
Table 9: Context-window placement strategies. D is the full document of length ∣D∣ ; windows are of width Lwin . The head strategy is the library default for this optional path.
Table 11: Two-stage training settings for reconstruction pretraining and task-aware fine-tuning.
Priority
Regex pattern
Example match
1
The correct answer is\s*([A−D])
“The correct answer is (B)”
2
The correct answer is\s*([A-D])
“The correct answer is B”
3
([A−D])
“…the choice (C) is…”
4
\b([A-D])\b
“…therefore D.”
Appendix
Table 12: Answer-extraction regex cascade ( eval.py ). All patterns are applied case-insensitively via re.IGNORECASE . The first match is returned; if none match, the empty string is returned.
Method
Easy
Hard
Short
Medium
Long
Avg. ↑
Mean-Pooled
36.98
28.30
30.56
31.63
33.33
32.16
Random-Proj.
36.46
28.30
32.78
31.16
29.63
31.67
Query-Driven
37.50
34.73
38.33
35.81
31.48
35.57
Appendix
Table 13: Ablation on query-driven compression on LongBench-v2.
Cat.
1
2
4
16∗
8
32
64
Overall
31.0
31.6
31.4
35.8
32.8
30.0
34.2
Diff.
Easy
33.9
34.4
34.4
37.5
38.0
30.7
38.5
Hard
29.3
29.9
29.6
34.7
29.6
29.6
31.5
Len.
Sht.
34.4
35.0
32.8
38.3
36.1
34.4
34.4
Med.
27.4
30.2
33.0
35.8
30.7
25.6
33.5
Lng.
32.4
28.7
25.9
31.5
31.5
31.5
35.2
Appendix
Table 14: Ablation on the number of injected tokens.