Visual document understanding typically requires separate retrieval and generation models, doubling memory and system complexity. We present Hydra, a dual-head approach that provides both ColBERT-style late-interaction retrieval and autoregressive generation from a single vision-language model. A single LoRA adapter, trained only for retrieval, is toggled at inference: enabling it produces multi-vector embeddings; disabling it recovers the base model's generation quality, with 426 of 426 language-model weight tensors byte-for-byte identical to a freshly-loaded Qwen3.5-4B. We identify two failure modes that can silently break generation in retrieval-fine-tuned VLMs (attention-mode restoration and lm_head preservation) plus an efficiency requirement (KV-cache-aware decoding); Hydra sidesteps the first two structurally and addresses the third in the decode loop. We release two scales, Hydra-4B and Hydra-0.8B, sharing LoRA hyperparameters (r=32, alpha=32) and optimisation recipe; data mix and projection dim differ across scales. The single-model design cuts peak GPU memory from 28.85 GB to 10.77 GB at 4B (62.7% reduction) and from 5.79 GB to 2.37 GB at 0.8B (59.1%) relative to a co-resident two-model deployment. A controlled ablation finds GritLM-style joint training matches Hydra's retrieval-only training on the evaluated modes while its LoRA-on generation mode collapses. A proof-of-concept on Qwen2.5-Omni-3B preserves generation equivalence on a non-Qwen3.5 backbone and transfers image retrieval within 2-8 pp of Hydra-4B, with zero-shot audio retrieval emerging through the frozen Whisper encoder.
Figures & tables
Figure 1: Hydra architecture. A single VLM serves two modes by toggling a LoRA adapter at inference time. Left : Retrieval mode (LoRA-on, bidirectional attention) produces multi-vector embeddings via custom_text_proj (320-dim shown here for Hydra-4B; Hydra-0.8B uses 128-dim). Right : Generation mode (LoRA-off, causal attention) produces autoregressive text via the base lm_head with KV-cache. The vision encoder is frozen and shared. No weight copying or model reloading occurs between modes. Arrow style indicates the data path active in each mode: solid blue = LoRA-on retrieval flow; dashed red = LoRA-bypassed generation flow (the dashed line marks that LoRA is disabled along this path, not that the layer is absent).
Figure 2: RAG pipeline comparison. Top: A conventional two-model pipeline (ColQwen3.5 for retrieval + Qwen3.5 for generation) holds both models co-resident in GPU memory ( ∼ 8.1B parameters, 28,850 MB peak VRAM). Bottom: Hydra uses a single 4B-parameter model for both indexing (retrieval head for embeddings) and querying (retrieval head finds top- k pages, generation head answers from them). Both heads share one model in GPU memory, reducing peak VRAM to 10,770 MB (savings quantified in Figure 3 ). Solid blue borders = retrieval; red borders = generation.
Table 1: ViDoRe retrieval performance at both scales. (a) Hydra-4B paired against a same-recipe single-head baseline ( r=32 , seed 123, identical data and step count). (b) Hydra-0.8B. Per-task results in Appendices A and B (Appendix).
Inference Mode
Hydra
GritLM-style
LoRA-on, bidirectional (retrieval)
0.8842 nDCG@5
0.8893 nDCG@5
LoRA-off, causal (shared base generation) ‡
0.543 / 0.561 ANLS, 76.5% match
LoRA-on, causal (target of joint training)
N/A
image-blind †
Table 2: Three-mode comparison of Hydra (retrieval-only training) vs. GritLM-style joint training, both at r=16 . Retrieval: 9 ViDoRe V1 tasks. Generation: DocVQA validation ( n=200 ). Both models use the same base and LoRA config; batch size differs (32 for the joint model vs. 112 for the Hydra partner) due to the additional memory cost of interleaving generation batches. The LoRA-off causal row reports a single shared measurement: with the LoRA adapter disabled the two checkpoints share the same base lm_head and embed_tokens , so the LoRA-off forward pass is identical for both. Per-checkpoint LoRA-off ANLS for the Hydra partner is therefore not separately measured; the column-spanning entry reflects this. Despite different adapter weights when LoRA is on (max element-wise diff: 0.50), the two functional modes are equivalent on retrieval and on LoRA-off generation; the mode that joint training was designed to unlock (LoRA-on causal) fails.
Figure 3: Peak GPU memory for a retrieve-then-generate workflow at both scales. A conventional two-model deployment holds a ColQwen3.5 retriever and a Qwen3.5 generator simultaneously (red); Hydra holds a single set of weights and toggles the LoRA adapter in place (blue). The single-model design cuts peak VRAM by 59.1% at the 0.8B scale and 62.7% at the 4B scale; the ratio is stable across backbone size. Numbers from bench_mode_switch_vram_4b.py (4B) and bench_mode_switch_vram.py (0.8B).
Figure 4: Mode-switching latency is negligible. A full round-trip (retrieval → generation → retrieval) takes 6.4 ms mean (median 6.3 ms; min–max 5.9–7.6 ms) over 50 iterations— 1.9% of a single generation call (331 ms). The switch cost is dominated by enabling/disabling the LoRA adapter plus swapping the attention mode on 8 full-attention layers; both are pointer-level operations rather than matmul. Source: results/efficiency/mode_switch/mode_switch_latency_report.json in the public Hydra-4B repo; measured on a single NVIDIA B200.
Metric
Hydra-4B
Hydra-0.8B
ViDoRe V1 mean nDCG@5
0.9082 (10 tasks)
0.8560 (10 tasks)
ViDoRe V2 mean nDCG@5
0.5697
0.5317
ViDoRe V3 mean nDCG@5
0.5639
0.4434
Gen equivalence (LM weights bitwise match)
426/426
320/320
Trainable parameters
65.75M (1.43% of 4.60B)
scaled by backbone
Peak VRAM (Hydra)
10.77 GB
2.37 GB
Table 3: Summary of Hydra. Efficiency measured on a single GPU.
Property
Hydra
SV-RAG
URaG
GritLM
Adapters needed
1
2 (swapped)
0 (custom)
0 (full FT)
Generation training
None
Yes
Yes
Yes
Retriever independence
Yes
Yes
No
N/A
Multi-vector retrieval
Yes
Yes
Yes
No
Resident backbones
1 (10.77 GB ∗ )
1 (2 adapters)
1 (custom module)
1
Table 4: Structural comparison of unified retrieval-generation architectures. Hydra is the only approach requiring no generation training and a single adapter. VRAM figures for Hydra are measured at the 4B scale; rows for other systems are structural rather than measured (we did not run identical co-resident benchmarks against SV-RAG, URaG, or GritLM).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
ViDoRe V1 (10 tasks)
ViDoRe V2 & V3
Task
Hydra-4B
Baseline
Task
Hydra-4B
Baseline
ArxivQA
0.9193
0.9223
V2
DocVQA
0.6593
0.6570
BioMedical Lectures
0.5939
0.6096
InfoVQA
0.9328
0.9405
ESG Reports (HL)
0.7328
0.6831
ShiftProject
0.9133
0.9233
ESG Reports
0.5207
0.5264
SynthDocQA-AI
1.0000
1.0000
Economics Reports
0.4314
0.4256
Appendix
Table 5: Per-task retrieval performance of Hydra-4B and the matched single-head baseline (same recipe, r=32 , seed 123) on ViDoRe V1, V2, and V3 (MTEB v2.10.13, MaxSim scoring).
ViDoRe V1 (10 tasks)
ViDoRe V2 & V3
Task
nDCG@5
Task
nDCG@5
ArxivQA
0.8511
V2
DocVQA
0.6134
BioMedical Lectures
0.5423
InfoVQA
0.9061
ESG Reports (HL)
0.5678
ShiftProject
0.7682
ESG Reports
0.4954
SynthDocQA-AI
0.9819
Economics Reports
0.5212
Appendix
Table 6: Per-task retrieval performance of Hydra-0.8B. Source: results/vidore/ in the public model repo.
Benchmark
n
Base ANLS
Hydra-4B ANLS
Δ
Exact Match%
DocVQA
5,000
0.5449
0.5490
+ 0.0041
71.08%
ChartQA †
2,500
0.0015
0.0015
+ 0.0000
22.92%
InfoVQA
2,801
0.1804
0.1761
− 0.0043
31.06%
TextVQA †
5,000
0.0566
0.0574
+ 0.0008
16.20%
Total
15,301
Appendix
Table 7: Generation equivalence across four VQA benchmarks. Base: Qwen3.5-4B via model.generate() ; Hydra: same weights with LoRA disabled, using the custom KV-cache path. Greedy decoding, 128 new tokens. Max ∣Δ∣=0.0043 across all four benchmarks (DocVQA, the largest sample with the highest base ANLS, dominates the comparison).
Table 9: Hydra-Omni image retrieval on ViDoRe V1, V2, and V3. Model: vidore/colqwen-omni-v0.1 (Qwen2.5-Omni-3B backbone, 4.4B total parameters). No Hydra-specific training. V1 is the full 10-task set, directly comparable to the Hydra-4B V1 column in Table 5 .
Benchmark
n
Base ANLS
Hydra-Omni ANLS
Δ (95% CI)
Exact Match%
InfoVQA
2,801
0.7257
0.7257
+ 0.0000 [ + 0.0000, + 0.0000]
100.00%
Appendix
Table 10: Hydra-Omni generation equivalence on InfoVQA, using the same protocol as Table 7 . Base: Qwen2.5-Omni-3B thinker via Qwen2_5OmniThinkerForConditionalGeneration ; Hydra-Omni: vidore/colqwen-omni-v0.1 with LoRA disabled and lm_head extracted from the thinker. Greedy decoding, 128 new tokens, with a short-answer prompt suffix applied identically to both paths (needed because Qwen2.5-Omni’s default outputs are sentence-form). 95% CI via 10,000-sample bootstrap.
Vision-Language Model (VLM) based retrievers have advanced visual document retrieval (VDR) to impressive quality. They require the same multi-billion parameter encoder for both document indexing and query encoding, incurring high latency and GPU dependence even for plain-text queries. We observe that this design is unnecessarily symmetric: documents are visually complex and demand strong visual understanding, whereas queries are just short text strings. NanoVDR exploits this query--document asymmetry by decoupling the two encoding paths: a frozen 2B VLM teacher indexes documents offline, while a distilled text-only student as small as 69M parameters encodes queries at inference. The key design choice is the distillation objective. Through systematic comparison of six objectives across three backbones and 22 ViDoRe benchmark datasets, we find that pointwise cosine alignment on query text consistently outperforms ranking-based and contrastive alternatives, while requiring only pre-cached teacher query embeddings and no document processing during training. Furthermore, we identify cross-lingual transfer as the primary performance bottleneck, and resolve it cheaply by augmenting training data with machine-translated queries. The resulting NanoVDR-S-Multi (DistilBERT, 69M) retains 95.1% of teacher quality and outperforms DSE-Qwen2 (2B) on v2 and v3 with 32× fewer parameters and 50× lower CPU query latency, at a total training cost under 13 GPU-hours.
LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fixed-size architectures limit the ability to reuse a single model across deployment settings with different compute, latency, and model-size targets. We introduce HydraLLM, an elastic MQA-based architecture. Its number of active attention heads determines the effective widths of the embeddings, attention projections, feed-forward layers, and the output head, forming a nested family of subnetworks within a single parameter set. We introduce a static-to-elastic conversion pipeline that, starting from a pre-trained model checkpoint, includes importance-driven weight reordering and stochastic subnetwork training with knowledge distillation. We extend this principle to multimodal inference through an elastic projection module whose output width follows the active text width. HydraLLM covers a wider range of active parameter configurations than FFN-only elastic architectures. With 75% of heads active, HydraLLM achieves 2.2 percentage points higher TriviaQA accuracy than a state-of-the-art nested baseline configuration which has 17.25% more parameters. With 50% of heads active, our multimodal model achieves a CLIPScore comparable to that of the 7× larger Gemma 3 4B model. With only 25% of heads active, HydraLLM increases token-generation throughput by 75.3% over the full-width configuration.
Oscar Kemme, Janek Haberer, Ali Hojjat +1
Hamburg University of Technology (TUHH), Germany · Kiel University, Germany · UNU-INWEH, Germany
Long visual contexts challenge vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once can exceed GPU memory limits. We present RETOKEN, a single learnable embedding that extracts retrieval signals from the VLM's internal representations to select query-relevant visual tokens from the pre-filled KV cache. This enables retrieval within the answering VLM, without a separate retriever or re-encoding. Despite being trained on only a small image-QA dataset, RETOKEN generalizes across image and video benchmarks. On Visual Haystacks, it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative). On LVBench, it transfers zero-shot to long video and improves Qwen3VL-8B by 8.0 points. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken
Yao Xiao, Reuben Tan, Zhen Zhu +3
University of Illinois at Urbana-Champaign · Microsoft Research · Google DeepMind