cs.CLAug 16, 2026
SaveSEER: Long-Context Reasoning via Selective Visual-Text Compression
Organizations: School of Information, The University of Texas at Austin · Department of Computer Science, The University of North Carolina at Chapel Hill · Department of Engineering, University of Cambridge · Faculty of Engineering and Information Technology, University of Technology Sydney ·
Abstract
Long-context reasoning remains computationally expensive for large language models due to the quadratic complexity of attention over text tokens. Visual-text compression offers a promising alternative by rendering text into images and processing them with vision-language models, often reducing token usage. However, existing approaches apply uniform compression regardless of query relevance, potentially sacrificing precision where detailed extraction is required. We present SEER, a framework that learns to select query-relevant images through visual scanning and retrieve textual content only where needed, combining the efficiency of visual compression with the precision of text-based reasoning. Through supervised fine-tuning on tool-interaction trajectories, SEER learns adaptive tool invocation for selection and retrieval. Experiments on long-context benchmarks show that SEER improves extraction precision through selective text retrieval while retaining average prompt-token savings relative to full-text baselines. On LongBench, SEER achieves 51.11% average accuracy, outperforming the visual-text baseline Glyph-9B by 2.33 points and Qwen3-8B by 3.49 points. Code can be accessed at https://github.com/jiaweixu98/SEER
Figures & tables
Figure 1: Long-context QA trade-off among LLMs, visual-text compression, and Seer .
| Model | Single-Doc QA | Multi-Doc QA | Summarization | Few-shot | Synthetic | Avg | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| QP | NQA | HQA | 2QA | QSUM | GovRep | TREC | TriQA | PR Zh | PR En | Pa C | ||
| Text-based Inference | ||||||||||||
| GPT-4.1 | 51.60 | 35.73 | 69.10 | 74.15 | 23.50 | 33.36 | 77.00 | 93.36 | 100.00 | 100.00 | 26.50 | 56.56 |
| LLaMA-3.1-8B-Instruct | 44.56 | 26.34 | 56.88 | 46.67 | 23.28 | 32.36 | 19.25 | 89.12 | 62.20 | 99.50 | 7.13 | 40.52 |
| Qwen2.5-7B-Instruct-1M | 45.29 | 25.61 | 60.70 | 40.51 | 22.95 | 29.97 | 59.37 | 86.93 | 98.50 | 100.00 | 3.50 | 42.41 |
| Qwen3-8B | 43.86 | 27.08 | 65.60 | 73.62 | 19.68 | 26.97 | 72.00 | 89.11 | 100.00 | 96.88 | 11.57 | 47.62 |
Table 1: Main LongBench results (%). Avg is the official category-level macro-average computed over all 21 tasks (the remaining task columns, including the Code category, are reported in Table 10 ). Glyph-9B uses pure visual reasoning. Seer augments visual reasoning with selective text retrieval and shows improved accuracy on several extraction-heavy tasks. The row in gray (GPT-4.1) is a large proprietary reference point included only to situate the open-weight results on a common scale; it is excluded from the ranking. Among the remaining models, bold marks the best and underline the second-best value per column.
| Model | Prompt Tokens | Output Efficiency | ||||
|---|---|---|---|---|---|---|
| Textual Prompt | Visual Prompt | Retrieved Text | Total Prompt | Completion Tokens | Prompt Compression | |
| (all) | (textual) | ( , ) | ||||
| GLM-4-9B-Chat-1M | 10175.8 | 0.0 | 0.0 | 10175.8 | 80.4 | 1.00 |
| Glyph-9B | 174.8 | 3347.8 | 0.0 | 3522.5 | 427.4 | 3.14 |
| Seer (Ours) † | 4051.1 | 3572.7 | 3741.8 | 7623.8 | 577.1 | 1.28 |
Table 2: Token budget comparison on LongBench under a unified setup, reported as per-task means averaged over all 21 tasks. Textual Prompt (all) denotes all textual prompt tokens seen by the model; Visual Prompt and Retrieved Text denote stage-1 visual tokens and stage-2 retrieved textual tokens, respectively. Total Prompt is Textual Prompt (all) Visual Prompt ; Retrieved Text is the retrieval-induced subset of Textual Prompt (all) and is broken out separately rather than added again. Completion Tokens count generated textual outputs only. Prompt Compression ( , ) is the mean over the 21 tasks of the per-task ratio GLM total prompt / model total prompt , with GLM fixed to 1.00; because it averages per-task ratios rather than dividing the averaged totals, it does not equal the quotient of the Total Prompt column. Full per-task breakdown is reported in Appendix A.6 . denotes our final model.
| Variant | LongBench Avg (%) | vs. Full |
|---|---|---|
| Seer (Full) | 51.11 | 0.00 |
| w/o Retrieve (Selection only) | 43.00 | |
| w/o Selection + Retrieval (Glyph-9B) | 48.78 | |
| w/o mechanism, same data (Glyph-9B-SFT-direct) | 45.20 |
Table 3: Ablation on LongBench using the overall category-level macro-average (across six major task categories). The Full variant is Seer with the complete Select-Retrieve-Reason pipeline. w/o Retrieve keeps Stage-1 selection but disables Stage-2 text retrieval. w/o Selection + Retrieval corresponds to Glyph-9B-style pure visual reasoning. w/o mechanism, same data is an iso-training control trained on the same trajectories, initialization, optimizer, seed and step budget as Seer , with only the select/retrieve/observation turns removed (Appendix A.8 ). is computed as Avg(variant) Avg(full) ; negative values indicate performance degradation after removing components.
Figure 2: Selective retrieval behavior on LongBench. Left: Distribution of selected images per query; counts here are the model’s raw predicted selections prior to the canonicalization of Section 4.1 . Right: For each total-page bucket (1–3, 4–6, 7–10, 11+), we show the fraction of samples selecting each image-count bucket (0, 1, 2, 3, 4+). The mode shifts from one to three images as contexts grow, but selections almost never exceed three: the budget saturates rather than scaling with context length.
Figure 3: Representative-task analysis of retrieval effectiveness by selected-image bucket. Blue bars show Seer(full) bucket-level mean accuracy for selected-count buckets (0,1,2,3,4+), with per-bucket sample counts annotated as . Horizontal reference lines indicate task-level mean scores of Glyph , Seer(k=0) , Seer(k=1) , and Seer(all) . The figure highlights task heterogeneity: retrieval is beneficial on some tasks but not uniformly so across all tasks.
| Task (Chinese) | Glyph-9B | Seer | |
|---|---|---|---|
| MultiFieldQA-Zh | 37.82 | 56.99 | |
| PassageRetrieval-Zh | 91.00 | 97.00 | |
| VCSUM | 12.28 | 14.20 | |
| Mean (3 tasks) | 47.03 | 56.06 |
Table 4: Error-analysis summary on the three Chinese-language LongBench tasks. Values are task scores (%) under official LongBench metrics. is computed as Seer Glyph-9B .
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Specification |
|---|---|
| Language Model | |
| Hidden size | 4096 |
| Intermediate size | 13696 |
| Number of layers | 40 |
| Attention heads | 32 (2 KV heads) |
| Max position embeddings | 131072 |
Table 5: Model architecture specifications.
| Token | Purpose | ID |
| Conversation Structure | ||
| <|endoftext|> | End of text / Padding | 151329 |
| <|user|> | User turn marker | 151336 |
| <|assistant|> | Assistant turn marker | 151337 |
| <|observation|> | Tool response marker | 151338 |
| Reasoning and Tool Use | ||
Table 6: Special tokens used in our framework and their token IDs.
| Stage | # Datasets | # Samples |
|---|---|---|
| Download (raw) | 10 | 90,777 |
| Render | 10 | 38,577 |
| Phase1 kept ( relevant_indices ) | 10 | 30,697 |
| Phase2 final filtered (used) | 7 | 10,041 |
Table 7: Dataset and sample retention across the data construction pipeline.
| Dataset | Download | Render | Phase1 kept | Phase2 final |
|---|---|---|---|---|
| 2wikimqa | 12,000 | 980 | 722 | 325 |
| gov_report | 12,000 | 8,000 | 8,000 | 966 |
| hotpotqa | 12,000 | 2,650 | 2,235 | 1,270 |
| lcc_python | 9,995 | 8,000 | 4,604 | 1,035 |
| musique | 12,000 | 8,000 | 5,704 | 2,080 |
| qasper | 2,742 | 2,707 | 2,383 | 364 |
Table 8: Per-dataset sample retention for the included training datasets from download to final filtering.
| Hyperparameter | Value |
|---|---|
| Base model | Glyph-9B (full-parameter) |
| Learning rate | |
| Optimizer | AdamW, , weight decay |
| Warmup ratio | 0.05 |
| Gradient clipping | 1.0 |
| Batch size (total) | 2 |
Table 9: Training hyperparameters. The three loss weights refer to the per-token weighting in Section 4.2 : tokens inside the tool call and the structural format tags are upweighted relative to ordinary generated tokens, while non-generated observation tokens are masked out entirely.
| Model | Single-Doc QA | Multi-Doc QA | Summarization | Few-shot | Code | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| QA Zh | QA En | Mus | Dur | News | VCSUM | Sam | LSHT | RB | LCC | |
| Text-based Inference | ||||||||||
| GPT-4.1 | 63.90 | 51.27 | 55.63 | 24.58 | 23.70 | 14.66 | 41.25 | 50.00 | 67.94 | 68.43 |
| LLaMA-3.1-8B-Instruct | 62.20 | 54.98 | 31.61 | 33.75 | 24.21 | 16.23 | 7.61 | 0.00 | 42.81 | 46.35 |
| Qwen2.5-7B-Instruct-1M | 62.98 | 53.62 | 34.72 | 21.85 | 21.02 | 12.20 | 39.17 | 28.68 | 29.80 | 21.72 |
| Qwen3-8B | 60.64 | 47.61 | 47.32 | 16.89 | 18.45 | 12.19 | 37.06 | 41.83 | 40.71 | 41.84 |
Table 10: Remaining task-level LongBench results (%); the main table is Table 1 . Column abbreviations: QA Zh/En MultiFieldQA-Zh/En, Mus MuSiQue, Dur DuReader, News MultiNews, Sam SAMSum, RB RepoBench-P. As in Table 1 , the gray GPT-4.1 row is a proprietary reference point excluded from the ranking; among the remaining models bold marks the best and underline the second-best value per column.
| Task | Model | Prompt Tokens | Completion | Compression ( ) | |||
|---|---|---|---|---|---|---|---|
| Textual | Visual | Retrieved | Total | ||||
| 2wikimqa | GLM-4-9B-Chat-1M | 7197.465 | 0.000 | 0.000 | 7197.465 | 6.230 | 1.000 |
| 2wikimqa | Glyph-9B | 44.770 | 3106.955 | 0.000 | 3151.725 | 402.135 | 2.284 |
| 2wikimqa | Seer (Ours) † | 2111.755 | 3331.335 | 1936.705 | 5443.090 | 478.390 | 1.322 |
| dureader | GLM-4-9B-Chat-1M | 9803.400 | 0.000 | 0.000 | 9803.400 | 230.980 | 1.000 |
| dureader | Glyph-9B | 22.695 | 1798.830 | 0.000 | 1821.525 | 201.465 | 5.382 |
Table 11: Per-task token breakdown on LongBench (Part I: 11 tasks). Values are per-sample means.
| Task | Model | Prompt Tokens | Completion | Compression ( ) | |||
|---|---|---|---|---|---|---|---|
| Textual | Visual | Retrieved | Total | ||||
| passage_count | GLM-4-9B-Chat-1M | 14908.325 | 0.000 | 0.000 | 14908.325 | 10.360 | 1.000 |
| passage_count | Glyph-9B | 93.000 | 4761.505 | 0.000 | 4854.505 | 1505.105 | 3.071 |
| passage_count | Seer (Ours) † | 6680.260 | 4988.545 | 6441.020 | 11668.805 | 1186.850 | 1.278 |
| passage_retrieval_en | GLM-4-9B-Chat-1M | 12546.610 | 0.000 | 0.000 | 12546.610 | 5.000 | 1.000 |
| passage_retrieval_en | Glyph-9B | 239.850 | 3607.695 | 0.000 | 3847.545 | 153.040 | 3.261 |
Table 12: Per-task token breakdown on LongBench (Part II: 10 tasks). Values are per-sample means.
| Model | Prompt form | Prefill (s) | E2E | Decode | Completion | Prompt | s | |
|---|---|---|---|---|---|---|---|---|
| server | client server | median (s) | (tok/s) | tok (med.) | tok (mean) | |||
| GLM-4-9B-Chat-1M | full text | 0.23 | 0.23 | 1.07 | 76 ‡ | 8 | 5,811 | 1 |
| Glyph-9B | images | 0.21 | 0.29 | 3.61 | 75.5 | 235 | 1,983 | 10 |
| Seer | images retrieved text | 0.22 | 0.29 | 5.47 | 76.3 | 381 | 7,678 | 14 |
Table 13: Wall-clock measurements on LongBench under a controlled protocol: a single NVIDIA H100 80GB, batch size 1, 10 samples per task across all 21 tasks ( completed samples), and the same checkpoints and Stage-2 pipeline as our main results. Prefill / time-to-first-token is measured with a cold max_tokens 1 probe; server prefill is the server-side time, while client server prefill additionally includes client-side image construction and transfer. Medians are reported because a small number of samples generate to the token cap; the count of samples exceeding 30 s is given in the last column. : GLM’s raw decode rate (36 tok/s) is an artifact of its 8-token outputs; bucketed at a matched output length ( 260 tokens) it also reaches 76 tok/s.
| Full text (GLM-4-9B-Chat-1M) | Rendered images (Glyph-9B) | ||
| Text tokens | Prefill (s) | Visual tokens | Prefill (s) |
| 5.4k | 0.26 | 5.6k | 0.78 |
| 20k | 0.70 | 14.6k | 2.23 |
| 40k | 1.89 | 29.4k | 4.75 |
| 79k | 5.53 | 59k | 11.5 |
| – | – | 105k | 26.0 |
Table 14: Prefill scaling with context length (batch size 1, cold measurements with distinct documents per repetition, so no prefix-cache reuse). Both text and visual prefill grow super-linearly. Because a document with text tokens renders to roughly visual tokens, the two columns should be compared at matched document size: below roughly 90k text tokens (which covers all of LongBench), text prefill is comparable to or faster than the rendered form, so visual compression yields no single-request prefill speedup in this regime. Beyond that point the ordering reverses, and beyond roughly 128k text tokens the text form no longer fits at all while the rendered form still does. The visual column is measured through an external HTTP probe and therefore includes client-side image transfer (approximately 9 MB at 180 pages), so it is an upper bound on true server-side visual prefill and the real crossover is somewhat earlier.
| Model / condition | Select-Retrieve | LongBench Avg (%) | QA-7 mean (%) |
|---|---|---|---|
| Glyph-9B (off-the-shelf) | ✗ | 48.78 | 45.98 |
| Glyph-9B-SFT-direct (iso-training control) | ✗ | 45.20 | 44.67 |
| Seer without retrieval ( ) | selection only | 43.00 | – |
| Seer | ✓ | 51.11 | 52.54 |
Table 15: Iso-training control isolating the mechanism from the effect of fine-tuning on our data. Glyph-9B-SFT-direct is trained on exactly the same trajectories as Seer , from the same Glyph-9B initialization, with the same optimizer, learning rate, seed, batch size, maximum sequence length, loss, and step budget (6,000 steps); the only difference is that the select, retrieve, and observation turns are removed, so the model maps rendered images directly to an answer. LongBench Avg is the official category-level macro-average over all 21 tasks; QA-7 mean is the unweighted mean over the seven LongBench QA tasks (Qasper, NarrativeQA, MultiFieldQA-En/Zh, HotpotQA, 2WikiMQA, MuSiQue). Fine-tuning on our data alone does not reproduce Seer ’s accuracy: both no-mechanism conditions land at 43–45, whereas the Select-Retrieve-Reason mechanism recovers over the iso-training control and over the condition.
| Model | Single-doc QA | Multi-doc QA | Summ. | Few-shot | Synthetic | Code |
|---|---|---|---|---|---|---|
| Glyph-9B | 35.45 | 51.48 | 19.50 | 61.47 | 70.50 | 54.29 |
| Glyph-9B-SFT-direct | 34.44 | 49.15 | 21.09 | 49.23 | 66.83 | 50.48 |
| Seer | 46.18 | 52.20 | 21.67 | 60.44 | 71.75 | 54.42 |
Table 16: Category-level breakdown for the iso-training control. Direct fine-tuning without the select-retrieve turns over-specializes to the direct answer format and loses most of its accuracy on the few-shot category (LSHT , SAMSum ), which is what drives its lower overall average. Seer , trained on the same data but with the mechanism retained, does not show this degradation, so the comparison is conservative with respect to our claim rather than favorable to it.
| Strategy | Page P | Page R | Page F1 | All-sample F1 | Selected pages |
|---|---|---|---|---|---|
| Random match- | 0.429 | 0.447 | 0.413 | 0.262 | 1.77 |
| First-page match- | 0.507 | 0.518 | 0.486 | 0.308 | 1.77 |
| All pages | 0.433 | 1.000 | 0.547 | 0.346 | 6.43 |
| BM25 match- | 0.630 | 0.643 | 0.606 | 0.384 | 1.77 |
| Seer | 0.592 | 0.620 | 0.578 | 0.425 | 1.62 |
| Oracle gold pages | 1.000 | 1.000 | 1.000 | 1.000 | 1.85 |
Table 17: Intrinsic page-selection analysis on the LongBench evaluation set. Gold page labels are newly annotated with the same answer-conditioned Qwen3.5-9B protocol used for training-data annotation, and are used only for analysis. The top block reports page precision, recall, and F1 on the 3,007 examples with at least one gold page, plus all-sample F1 over all 4,750 examples; the oracle row is the upper bound rather than a competing strategy. Each match- baseline is given the same per-example page budget as Seer (one-page minimum), which is why they average 1.77 pages against Seer ’s 1.62. Bold marks the best non-oracle value per column. The bottom block compares Seer and BM25 on diagnostic subsets. BM25 is a strong lexical baseline and is slightly ahead on gold-nonempty page F1, while Seer selects fewer pages, is best on the all-sample metric, and is ahead of BM25 on low lexical-overlap examples.
| Category | Task | BM25-forced selection | Seer selection |
|---|---|---|---|
| Single-doc QA | Qasper | 46.23 | 47.74 |
| NarrativeQA | 25.63 | 26.05 | |
| MultiFieldQA-En | 53.30 | 53.94 | |
| MultiFieldQA-Zh | 50.86 | 56.99 | |
| Multi-doc QA | HotpotQA | 62.78 | 60.22 |
| 2WikiMQA | 57.24 | 68.65 |
Table 18: End-to-end comparison of page selectors under an identical answering pipeline. Both columns use the same Seer Stage-2 answerer, the same retrieval mechanism, and the same token budget; the only difference is which pages are selected. In the left column the selection is forced to the pages returned by a BM25 retriever over rendered page text (with the per-example page budget matched to Seer ); in the right column the pages are chosen by Seer ’s learned visual selector. We report the 15 tasks for which the BM25-forced run completed under the same retry and merge policy as Seer ; the remaining six tasks are excluded rather than reported from partial runs. BM25 is a strong lexical selector and wins on three lexically-driven tasks, but it degrades sharply where page relevance is not lexical (VCSUM) or where pages are code (LCC).
| Task | RLM-Qwen3-8B | Qwen3-8B (direct) | Glyph-9B | Seer |
|---|---|---|---|---|
| Qasper | 35.61 | 43.86 | 38.99 | 47.74 |
| NarrativeQA | 16.65 | 27.08 | 21.83 | 26.05 |
| MultiFieldQA-En | 43.58 | 47.61 | 43.18 | 53.94 |
| MultiFieldQA-Zh | 50.72 | 60.64 | 37.82 | 56.99 |
| HotpotQA | 51.84 | 65.60 | 56.42 | 60.22 |
| 2WikiMQA | 61.74 | 73.62 | 71.62 | 68.65 |
Table 19: Comparison with Recursive Language Models on the seven LongBench QA tasks, where a text-side recursive reader is the applicable comparison. All systems are scored with the same official LongBench evaluator used elsewhere in this paper. RLM is run with the authors’ released recursive framework and their post-trained checkpoint ( mit-oasys/rlm-qwen3-8b-v0.1 ); the only code we add is a LongBench adapter that loads examples and writes predictions in LongBench format. Documents are not truncated: the full document is placed in RLM’s local REPL as the context variable, and the root prompt keeps the LongBench QA template with the context field replaced by an instruction that the full context is available in the REPL. Decoding is greedy (temperature , top- ) with max_tokens 2048, max_depth 1, and max_iterations 8, served through a vLLM OpenAI-compatible endpoint; the checkpoint’s native 40,960-token window applies per model call. The only RLM-specific post-processing is deterministic removal of wrapper labels such as “FINAL ANSWER:” before scoring, which raises rather than lowers RLM’s score.
Figure 4: Task-wise retrieval heterogeneity view on LongBench, over the 14 tasks with sufficient per-bucket coverage. Color/value in each cell is Seer(full) bucket mean accuracy minus Glyph task-level accuracy ; blank cells indicate empty buckets. Cells backed by few samples (e.g. the narrativeqa 4+ bucket) are correspondingly noisy and should not be read as task-level effects.
| Task | Query (abridged) | Seer | Glyph-9B |
| Successes: localized evidence, an exact list, magnitude, or polarity | |||
| Qasper | Which neural network modules are included in NeuronBlocks? | Embedding Layer, Neural Network Layers, Loss Function, Metrics (matches gold) | Word/character embedding, RNN, CNN, Transformer, Focal Loss, Accuracy, Precision, F1, CRF, distillation (invented list) |
| MultiFieldQA-Zh | Amount of local fiscal spending on social security over the past five years | 650 million yuan (matches gold) | over 5 billion yuan (misread magnitude) |
| DuReader | Does a decayed wisdom tooth always have to be extracted? | Yes: wisdom teeth are hard to clean and hard to restore, so early extraction is advised (matches gold) | No: it depends on severity and position (contradicts gold) |
| Failures: multi-hop questions whose evidence is distributed | |||
| MuSiQue | Multi-hop bridge question; gold is Manhattan Project | 7th Solvay Conference (correct intermediate hop, chain not completed) | – |
Table 20: Representative successes and failures, taken verbatim from the Seer and Glyph-9B prediction files. The successes are the intended regime: the answer hinges on an exact list, number, or polarity that is present on one or two pages, and retrieving the source text for those pages resolves what purely visual decoding gets wrong. The failures are the regime our analysis identifies as hardest: multi-hop questions whose evidence is distributed, where a single round of selection can surface a correct intermediate hop without completing the chain. A dash in the last column marks a case we report only for Seer . This is the qualitative counterpart to the distributed-evidence subset in Table 17 , and it motivates iterative selection as future work.
More in this topic
Long-Context Language Model Inference
Long-context language model inference makes processing and generating over lengthy prompts faster and more memory-efficient.
See all 227 papers