Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model

Athos Georgiou

cs.CVMar 30, 2026

Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model

Authors: Athos Georgiou

Organizations: Independent Researcher

Abstract

Visual document understanding typically requires separate retrieval and generation models, doubling memory and system complexity. We present Hydra, a dual-head approach that provides both ColBERT-style late-interaction retrieval and autoregressive generation from a single vision-language model. A single LoRA adapter, trained only for retrieval, is toggled at inference: enabling it produces multi-vector embeddings; disabling it recovers the base model's generation quality, with 426 of 426 language-model weight tensors byte-for-byte identical to a freshly-loaded Qwen3.5-4B. We identify two failure modes that can silently break generation in retrieval-fine-tuned VLMs (attention-mode restoration and lm_head preservation) plus an efficiency requirement (KV-cache-aware decoding); Hydra sidesteps the first two structurally and addresses the third in the decode loop. We release two scales, Hydra-4B and Hydra-0.8B, sharing LoRA hyperparameters (r=32, alpha=32) and optimisation recipe; data mix and projection dim differ across scales. The single-model design cuts peak GPU memory from 28.85 GB to 10.77 GB at 4B (62.7% reduction) and from 5.79 GB to 2.37 GB at 0.8B (59.1%) relative to a co-resident two-model deployment. A controlled ablation finds GritLM-style joint training matches Hydra's retrieval-only training on the evaluated modes while its LoRA-on generation mode collapses. A proof-of-concept on Qwen2.5-Omni-3B preserves generation equivalence on a non-Qwen3.5 backbone and transfers image retrieval within 2-8 pp of Hydra-4B, with zero-shot audio retrieval emerging through the frozen Whisper encoder.

Figures & tables

Panel of Figure 1 — Figure 1: Hydra architecture. A single VLM serves two modes by toggling a LoRA adapter at inference time. Left : Retrieval mode (LoRA-on, bidirectional attention) produces multi-vector embeddings via custom_text_proj (320-dim shown here for Hydra-4B; Hydra-0.8B uses 128-dim). Right : Generation mode (LoRA-off, causal attention) produces autoregressive text via the base lm_head with KV-cache. The vision encoder is frozen and shared. No weight copying or model reloading occurs between modes. Arrow style indicates the data path active in each mode: solid blue = LoRA-on retrieval flow; dashed red = LoRA-bypassed generation flow (the dashed line marks that LoRA is disabled along this path, not that the layer is absent).

Panel of Figure 2 — Figure 2: RAG pipeline comparison. Top: A conventional two-model pipeline (ColQwen3.5 for retrieval + Qwen3.5 for generation) holds both models co-resident in GPU memory ( $\sim$ 8.1B parameters, 28,850 MB peak VRAM). Bottom: Hydra uses a single 4B-parameter model for both indexing (retrieval head for embeddings) and querying (retrieval head finds top- $k$ pages, generation head answers from them). Both heads share one model in GPU memory, reducing peak VRAM to 10,770 MB (savings quantified in Figure 3 ). Solid blue borders = retrieval; red borders = generation.

Table 1: ViDoRe retrieval performance at both scales. (a) Hydra-4B paired against a same-recipe single-head baseline (

r{=}32

, seed 123, identical data and step count). (b) Hydra-0.8B. Per-task results in Appendices A and B (Appendix).

Inference Mode	Hydra	GritLM-style
LoRA-on, bidirectional (retrieval)	0.8842 nDCG@5	0.8893 nDCG@5
LoRA-off, causal (shared base generation) ‡	0.543 / 0.561 ANLS, 76.5% match
LoRA-on, causal (target of joint training)	N/A	image-blind †

Table 2: Three-mode comparison of Hydra (retrieval-only training) vs. GritLM-style joint training, both at

r{=}16

. Retrieval: 9 ViDoRe V1 tasks. Generation: DocVQA validation (

n{=}200

). Both models use the same base and LoRA config; batch size differs (32 for the joint model vs. 112 for the Hydra partner) due to the additional memory cost of interleaving generation batches. The LoRA-off causal row reports a single shared measurement: with the LoRA adapter disabled the two checkpoints share the same base lm_head and embed_tokens , so the LoRA-off forward pass is identical for both. Per-checkpoint LoRA-off ANLS for the Hydra partner is therefore not separately measured; the column-spanning entry reflects this. Despite different adapter weights when LoRA is on (max element-wise diff: 0.50), the two functional modes are equivalent on retrieval and on LoRA-off generation; the mode that joint training was designed to unlock (LoRA-on causal) fails.

Panel of Figure 3 — Figure 3: Peak GPU memory for a retrieve-then-generate workflow at both scales. A conventional two-model deployment holds a ColQwen3.5 retriever and a Qwen3.5 generator simultaneously (red); Hydra holds a single set of weights and toggles the LoRA adapter in place (blue). The single-model design cuts peak VRAM by 59.1% at the 0.8B scale and 62.7% at the 4B scale; the ratio is stable across backbone size. Numbers from bench_mode_switch_vram_4b.py (4B) and bench_mode_switch_vram.py (0.8B).

Panel of Figure 4 — Figure 4: Mode-switching latency is negligible. A full round-trip (retrieval $\rightarrow$ generation $\rightarrow$ retrieval) takes 6.4 ms mean (median 6.3 ms; min–max 5.9–7.6 ms) over 50 iterations— 1.9% of a single generation call (331 ms). The switch cost is dominated by enabling/disabling the LoRA adapter plus swapping the attention mode on 8 full-attention layers; both are pointer-level operations rather than matmul. Source: results/efficiency/mode_switch/mode_switch_latency_report.json in the public Hydra-4B repo; measured on a single NVIDIA B200.

Metric	Hydra-4B	Hydra-0.8B
ViDoRe V1 mean nDCG@5	0.9082 (10 tasks)	0.8560 (10 tasks)
ViDoRe V2 mean nDCG@5	0.5697	0.5317
ViDoRe V3 mean nDCG@5	0.5639	0.4434
Gen equivalence (LM weights bitwise match)	426/426	320/320
Trainable parameters	65.75M (1.43% of 4.60B)	scaled by backbone
Peak VRAM (Hydra)	10.77 GB	2.37 GB

Table 3: Summary of Hydra. Efficiency measured on a single GPU.

Property	Hydra	SV-RAG	URaG	GritLM
Adapters needed	1	2 (swapped)	0 (custom)	0 (full FT)
Generation training	None	Yes	Yes	Yes
Retriever independence	Yes	Yes	No	N/A
Multi-vector retrieval	Yes	Yes	Yes	No
Resident backbones	1 (10.77 GB ∗ )	1 (2 adapters)	1 (custom module)	1

Table 4: Structural comparison of unified retrieval-generation architectures. Hydra is the only approach requiring no generation training and a single adapter. VRAM figures for Hydra are measured at the 4B scale; rows for other systems are structural rather than measured (we did not run identical co-resident benchmarks against SV-RAG, URaG, or GritLM).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

ViDoRe V1 (10 tasks)			ViDoRe V2 & V3
Task	Hydra-4B	Baseline	Task	Hydra-4B	Baseline
ArxivQA	0.9193	0.9223	V2
DocVQA	0.6593	0.6570	BioMedical Lectures	0.5939	0.6096
InfoVQA	0.9328	0.9405	ESG Reports (HL)	0.7328	0.6831
ShiftProject	0.9133	0.9233	ESG Reports	0.5207	0.5264
SynthDocQA-AI	1.0000	1.0000	Economics Reports	0.4314	0.4256

Appendix

Table 5: Per-task retrieval performance of Hydra-4B and the matched single-head baseline (same recipe,

r{=}32

, seed 123) on ViDoRe V1, V2, and V3 (MTEB v2.10.13, MaxSim scoring).

ViDoRe V1 (10 tasks)		ViDoRe V2 & V3
Task	nDCG@5	Task	nDCG@5
ArxivQA	0.8511	V2
DocVQA	0.6134	BioMedical Lectures	0.5423
InfoVQA	0.9061	ESG Reports (HL)	0.5678
ShiftProject	0.7682	ESG Reports	0.4954
SynthDocQA-AI	0.9819	Economics Reports	0.5212

Appendix

Table 6: Per-task retrieval performance of Hydra-0.8B. Source: results/vidore/ in the public model repo.

Benchmark	$n$	Base ANLS	Hydra-4B ANLS	$\Delta$	Exact Match%
DocVQA	5,000	0.5449	0.5490	$+$ 0.0041	71.08%
ChartQA †	2,500	0.0015	0.0015	$+$ 0.0000	22.92%
InfoVQA	2,801	0.1804	0.1761	$-$ 0.0043	31.06%
TextVQA †	5,000	0.0566	0.0574	$+$ 0.0008	16.20%
Total	15,301

Appendix

Table 7: Generation equivalence across four VQA benchmarks. Base: Qwen3.5-4B via model.generate() ; Hydra: same weights with LoRA disabled, using the custom KV-cache path. Greedy decoding, 128 new tokens. Max

|\Delta|{=}0.0043

across all four benchmarks (DocVQA, the largest sample with the highest base ANLS, dominates the comparison).

Section	Script	Claim	Report
Section 3.4 (Req. 3)	scripts/bench_kv_cache.py	KV-cache speedup ( $7.65\times$ mean)	results/efficiency/kv_cache_bench/report.json
Section 5.3	scripts/test_gen_equivalence_4b.py (4B), scripts/test_gen_equivalence.py (0.8B)	426/426 bitwise match	results/generation_equivalence/
Section 5 (Eff.)	scripts/bench_mode_switch_vram_4b.py (4B), scripts/bench_mode_switch_vram.py (0.8B)	Peak VRAM at both scales	results/mode_switch_vram/
Section 5.5	results/kv_cache_isolation/run.py	50/50 byte-identical	results/kv_cache_isolation/kv_isolation_report.json
Section 6	(in Omni repo)	InfoVQA generation equivalence	results/infovqa_report.json

Appendix

Table 8: Released artifacts: script

\rightarrow

claim

\rightarrow

report JSON. Repos: athrael-soju/HydraQwen3.5-4B , -0.8B , HydraQwen2.5-Omni-3B .

ViDoRe V1		ViDoRe V2		ViDoRe V3
Task	nDCG@5	Task	nDCG@5	Task	nDCG@5
SynthDocQA-AI	0.9852	ESG Reports (HL)	0.6050	Computer Science	0.6727
SynthDocQA-Healthcare	0.9663	BioMedical Lectures	0.5827	Energy	0.5574
SynthDocQA-Energy	0.9566	ESG Reports	0.4860	Pharmaceuticals	0.5446
SynthDocQA-Gov	0.9529	Economics Reports	0.4676	HR	0.4953
InfoVQA	0.9341			Finance (EN)	0.4636

Appendix

Table 9: Hydra-Omni image retrieval on ViDoRe V1, V2, and V3. Model: vidore/colqwen-omni-v0.1 (Qwen2.5-Omni-3B backbone, 4.4B total parameters). No Hydra-specific training. V1 is the full 10-task set, directly comparable to the Hydra-4B V1 column in Table 5 .

Benchmark	$n$	Base ANLS	Hydra-Omni ANLS	$\Delta$ (95% CI)	Exact Match%
InfoVQA	2,801	0.7257	0.7257	$+$ 0.0000 [ $+$ 0.0000, $+$ 0.0000]	100.00%

Appendix

Table 10: Hydra-Omni generation equivalence on InfoVQA, using the same protocol as Table 7 . Base: Qwen2.5-Omni-3B thinker via Qwen2_5OmniThinkerForConditionalGeneration ; Hydra-Omni: vidore/colqwen-omni-v0.1 with LoRA disabled and lm_head extracted from the thinker. Greedy decoding, 128 new tokens, with a short-answer prompt suffix applied identically to both paths (needed because Qwen2.5-Omni’s default outputs are sentence-form). 95% CI via 10,000-sample bootstrap.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 722 papers

NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval
Mar 13, 2026Zhuchenyang Liu, Yao Zhang, Yu XiaoVisual Document Retrieval Vision-Language Retrieval
Aalto University, Espoo, Finland
HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models
Oct 16, 2026Oscar Kemme, Janek Haberer, Ali Hojjat +1Elastic Neural Networks Efficient VLM Inference
Hamburg University of Technology (TUHH), Germany · Kiel University, Germany · UNU-INWEH, Germany
ReToken: Improving Long-Context VLMs with Visual Retrieval Token
Jul 30, 2026Yao Xiao, Reuben Tan, Zhen Zhu +3Vision-Language Models Vision-Language Retrieval
University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind

Mar 13, 2026J/K move · Enter open · S save

NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval

Zhuchenyang Liu, Yao Zhang, Yu Xiao

Aalto University, Espoo, Finland

Visual Document Retrieval Vision-Language Retrieval VLM Distillation Efficient VLM Inference

Open paper AskPDF arXiv

Vision-Language Model (VLM) based retrievers have advanced visual document retrieval (VDR) to impressive quality. They require the same multi-billion parameter encoder for both document indexing and query encoding, incurring high latency and GPU dependence even for plain-text queries. We observe that this design is unnecessarily symmetric: documents are visually complex and demand strong visual understanding, whereas queries are just short text strings. NanoVDR exploits this query--document asymmetry by decoupling the two encoding paths: a frozen 2B VLM teacher indexes documents offline, while a distilled text-only student as small as 69M parameters encodes queries at inference. The key design choice is the distillation objective. Through systematic comparison of six objectives across three backbones and 22 ViDoRe benchmark datasets, we find that pointwise cosine alignment on query text consistently outperforms ranking-based and contrastive alternatives, while requiring only pre-cached teacher query embeddings and no document processing during training. Furthermore, we identify cross-lingual transfer as the primary performance bottleneck, and resolve it cheaply by augmenting training data with machine-translated queries. The resulting NanoVDR-S-Multi (DistilBERT, 69M) retains 95.1% of teacher quality and outperforms DSE-Qwen2 (2B) on v2 and v3 with 32

\times

fewer parameters and 50

\times

lower CPU query latency, at a total training cost under 13 GPU-hours.

Abstract

Vision-Language Models

Explore similar work

NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval

HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models

ReToken: Improving Long-Context VLMs with Visual Retrieval Token