HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models

Oscar Kemme; Janek Haberer; Ali Hojjat; Olaf Landsiedel

cs.CLOct 16, 2026

HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models

Authors: Oscar Kemme, Janek Haberer, Ali Hojjat, Olaf Landsiedel

Organizations: Hamburg University of Technology (TUHH), Germany · Kiel University, Germany · UNU-INWEH, Germany

Abstract

LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fixed-size architectures limit the ability to reuse a single model across deployment settings with different compute, latency, and model-size targets. We introduce HydraLLM, an elastic MQA-based architecture. Its number of active attention heads determines the effective widths of the embeddings, attention projections, feed-forward layers, and the output head, forming a nested family of subnetworks within a single parameter set. We introduce a static-to-elastic conversion pipeline that, starting from a pre-trained model checkpoint, includes importance-driven weight reordering and stochastic subnetwork training with knowledge distillation. We extend this principle to multimodal inference through an elastic projection module whose output width follows the active text width. HydraLLM covers a wider range of active parameter configurations than FFN-only elastic architectures. With 75% of heads active, HydraLLM achieves 2.2 percentage points higher TriviaQA accuracy than a state-of-the-art nested baseline configuration which has 17.25% more parameters. With 50% of heads active, our multimodal model achieves a CLIPScore comparable to that of the 7 $\times$ larger Gemma 3 4B model. With only 25% of heads active, HydraLLM increases token-generation throughput by 75.3% over the full-width configuration.

Figures & tables

Figure 1: HydraLLM width slicing. The number of active heads determines the embedding dimension, the attention projection, and the feed-forward network width, producing nested subnetworks from a single parameter set.

Figure 2: Pipeline overview. Starting from a static text checkpoint, such as Gemma 3 270M, the sliced axes can first be reordered to improve prefix quality. The resulting checkpoint is then trained as an elastic text model whose runtime width is controlled by the active number of query heads

k

. Finally, we combine the elastic text decoder with, for example, Gemma 3 4B vision components to form a multimodal model whose projection output follows the same active width.

Figure 3: Elastic LVLM architecture. The projection output width follows the active text width.

Recipe	Perplexity $\downarrow$
Recipe	H1	H2	H3	H4
no reordering	168.6M	1.1M	41.3k	49.0
mag-cett-cett	37.2k	2.43k	1.22k	49.0
mag-mag-mag	8.45M	378.9k	1.54k	49.0
vxc-vxc-vxc	38.0M	4.87M	60.4M	49.0

Table 1: Perplexity of reordered checkpoints measured on samples of plays by William Shakespeare. Magnitude for embeddings combined with CETT for heads and FFN yields the strongest subnetwork perplexities, while VxC consistently hurts.

Recipe	Perplexity $\downarrow$
Recipe	H1	H2	H3	H4
no reordering	123.9	77.4	56.7	45.2
mag-cett-cett	106.6	71.6	54.4	44.3
mag-mag-mag	110.1	71.9	54.4	44.8
vxc-vxc-vxc	118.1	71.7	55.0	44.1

Table 2: Perplexity of reordered and trained (20k steps) checkpoints measured on samples of plays by William Shakespeare. Magnitude for embeddings combined with CETT for heads and FFN yields the strongest subnetwork perplexities.

Figure 4: ARC-Easy accuracy as a function of active parameter size for HydraLLM and MatFormer configurations. HydraLLM covers a wider active parameter range than MatFormer while offering competitive accuracies.

Figure 5: TriviaQA accuracy as a function of active parameter size for HydraLLM and MatFormer configurations. HydraLLM is stronger at H3 clearly outperforming the similarly sized MatFormer W0.25, while MatFormer surpasses HydraLLM at the full configuration.

Config	HydraLLM		MatFormer
Config	Prefill	Decode	Prefill	Decode
H1 / W0.25	55,891.3	1,057.0	51,549.1	669.6
H2 / W0.50	52,851.7	785.9	51,353.8	648.5
H3 / W0.75	51,165.4	594.2	49,945.2	623.4
H4 / W1.00	50,487.9	603.0	50,509.8	602.3

Table 3: Throughput in tokens per second for HydraLLM and MatFormer variants. Prompt-processing of 2048 tokens. Decode 512 tokens. Prompt-processing throughput gains are modest for both methods, while token-generation scaling is much larger for HydraLLM than for MatFormer.

Figure 6: Qualitative captioning example. The elastic LVLM is prompted to describe an image of a car across the four head configurations ( $k=1$ to $k=4$ ). Hallucinated or incorrect statements are highlighted in orange, illustrating the qualitative gap that persists alongside the encouraging CLIPScore trends.

No training
Training Recipe	H1	H2	H3	H4
elastic	32.4	31.5	32.0	32.7
Gemma 3 4B	–	–	–	65.3
Projection only (500 steps)
non-elastic	–	–	–	28.9
elastic	38.2	31.4	31.1	33.5

Table 4: CLIPScore (

\times 100

) across head configurations for each multimodal training recipe. Bold marks the best score per head configuration. non-elastic : starting from pre-trained Gemma 3 270m and only head configuration

k{=}4

is trained; elastic : starting from elastic text model and all head configurations are trained stochastically; fine-tune : an additional text-only fine-tuning step is applied before multimodal training; pre-aligned : the projection is pre-aligned on captioning data before joint training; Gemma 3 4B : the original frozen Gemma 3 4B multimodal model provided as a baseline.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Package	Version
torch	2.8.0
torchvision	0.23.0
transformers	4.56.1
accelerate	1.10.1
datasets	4.2.0
lighteval	0.11.0

Appendix

Table 5: Key Python package versions used in the implementation.

	H1	H2	H3	H4
$H_{\mathrm{kv}}$	1	1	1	1
query heads $k$	1	2	3	4
$d_{E,\mathrm{active}}$	160	320	480	640
$d_{\mathrm{ff},\mathrm{active}}$	512	1024	1536	2048
$d_{Q,\mathrm{active}}$	256	512	768	1024

Appendix

Table 6: HydraLLM’s Gemma 3 270M runtime configurations and active component dimensions.

d_{E,\mathrm{active}}

is the active embedding width,

d_{\mathrm{ff},\mathrm{active}}

the active FFN neuron count, and

d_{Q,\mathrm{active}}

the total query projection dimension. MatFormer’s W0.25-W1.0 have the same

d_{\mathrm{ff},\mathrm{active}}

as H1-H4, but otherwise keep the dimensions of H4.

Captioning & Pretraining
Dataset	Num Samples
PixMo-Cap	50,000
COCO Recaptioned	12,000
LLaVA Pretrain	50,000
Question Answering
PixMo Ask Model Anything	30,000

Appendix

Table 7: Composition of the multimodal dataset of the main multimodal experiments.

Captioning & Pretraining
Dataset	Num Samples
PixMo-Cap	500,000
COCO Recaptioned	100,000
LLaVA Pretrain	500,000
Question Answering
PixMo Ask Model Anything	125,000

Appendix

Table 8: Composition of the multimodal dataset of the scaled 16,000-step multimodal training runs.

Hyperparameter	Text	Multimodal
Sequence length	3072	3072
Learning rate (text backbone)	$3\times 10^{-4}$	$2\times 10^{-5}$
Learning rate (projection)	–	$2\times 10^{-4}$
LR schedule	cosine	cosine
Weight decay	0.1	0.1
Optimizer	AdamW	AdamW

Appendix

Table 9: Training hyperparameters for the text-only and default multimodal runs. The full multimodal run where all components are trained differs only in vision encoder learning rate (

6\times 10^{-6}

) and warmup steps (text: 75, vision: 75). Where not listed, the multimodal configuration inherits the text default.

ARC-Easy (Raw)
Variant	H1/W0.25	H2/W0.5	H3/W0.75	H4/W1.0
MatFormer	$53.9\pm 1.02$	$55.6\pm 1.02$	$54.6\pm 1.02$	$56.0\pm 1.02$
default	$39.0\pm 1.00$	$46.3\pm 1.02$	$51.9\pm 1.03$	$56.4\pm 1.02$
1M steps	$43.6\pm 1.02$	$51.1\pm 1.03$	$56.9\pm 1.02$	$60.3\pm 1.00$
no teacher	$37.2\pm 1.00$	$46.2\pm 1.03$	$51.9\pm 1.02$	$56.1\pm 1.02$
uniform sampling	$38.8\pm 0.99$	$48.2\pm 1.02$	$52.2\pm 1.02$	$55.4\pm 1.02$

Appendix

Table 10: Training recipe ablation across text benchmarks after conversion of the static Gemma 3 270M model, with MatFormer as a width-pruning comparison. Values are accuracies in percent, with standard error, for the four runtime head configurations H1–H4 and MatFormer widths W0.25–W1.0. ARC-Easy Norm is answer-length-normalized accuracy. TriviaQA is five-shot exact-match accuracy.

ARC-Easy (Raw)
Variant	H1	H2	H3	H4
no reordering	$39.0\pm 1.00$	$46.3\pm 1.02$	$51.9\pm 1.03$	$56.4\pm 1.02$
Mag-CETT-CETT	$38.4\pm 1.00$	$46.3\pm 1.02$	$52.2\pm 1.02$	$56.6\pm 1.02$
Mag-Mag-Mag	$37.6\pm 0.99$	$46.8\pm 1.02$	$52.9\pm 1.02$	$57.1\pm 1.02$
VxC-VxC-VxC	$36.9\pm 0.99$	$46.8\pm 1.02$	$51.9\pm 1.03$	$56.7\pm 1.02$
Mag-no-Mag	$38.3\pm 1.00$	$46.3\pm 1.02$	$53.2\pm 1.02$	$56.5\pm 1.02$

Appendix

Table 11: Effect of importance reordering on the four nested runtime head configurations after 20,000 training steps. Values are accuracies in percent with standard error. ARC-Easy Norm is answer-length-normalized accuracy. TriviaQA is five-shot exact-match accuracy. The no-reordering variant is the default baseline from Table 10 .

Variant	H1	H2	H3	H4
text, 1M steps	$0.13$	$0.66$	$2.35$	$3.55$
multimodal, default	$0.80$	$2.16$	$4.08$	$6.48$
text fine-tune, 270M teacher	$0.12$	$1.10$	$3.64$	$5.07$
text fine-tune, 4B teacher	$0.08$	$2.13$	$4.56$	$7.32$

Appendix

Table 12: TriviaQA accuracy (exact match, %) across head configurations for text-only continued pre-training and subsequent multimodal or text-only fine-tuning variants.

Figure 7: Training loss per-head configuration during projection alignment. The one-head model converges to a higher loss, while configurations with two or more heads reduce their loss rapidly and stabilize within the first 100 steps.

Elastic Neural Networks

See all 9 papers

Stepped MoE: Segment-Level Routing with Configurable Inference Complexity
Oct 5, 2026Arnav Kundu, Zhaoyang Xu, Bairu Hou +3On-Device Language Model Inference Dynamic Neural Networks
Apple
Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control
May 8, 2026Ali Taghibakhshi, Ruisi Cai, Saurav Muralidharan +17LLM Compression LLM Post-Training
NVIDIA
Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model
Mar 30, 2026Athos GeorgiouVision-Language Models Visual Document Retrieval
Independent Researcher

Oct 5, 2026J/K move · Enter open · S save

Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

Arnav Kundu, Zhaoyang Xu, Bairu Hou, Chang Gao+2

Apple

On-Device Language Model Inference Dynamic Neural Networks Mixture of Experts Elastic Neural Networks LLM Routing Efficient Language Model Inference

Open paper AskPDF arXiv

Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adaptive computation, existing approaches treat these dimensions independently. Moreover, models catered towards on-device edge inference need to conform to the memory and compute limitations of the serving devices. In this paper, we introduce a unified framework that combines elastic structures with sparsely gated architectures to create models that adapt simultaneously to both deployment constraints and task requirements. Our approach employs a model backbone that conditions on both the context and target efficiency specifications, enabling fine-grained control over the accuracy-efficiency trade-off at inference time. The model learns to activate task-relevant parameters within elastically-nested sub-networks, allowing a single model to span multiple capacity points while maintaining input-adaptive routing. Through experiments we demonstrate that we can create a model that allows the flexibility to use 1,2,3,4 billion parameters while being more accurate than their dense counter-parts (2-5% on knowledge-intensive benchmarks) and at par with their static versions while delivering similar latency metrics as dense models. Overall, we save on device disk space by sharing the model parameters, allow flexibility of serving based on DRAM and compute available while delivering more accurate results.

Abstract

Elastic Neural Networks

Explore similar work

Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control

Hydra: Unifying Document Retrieval and Generation in a Single Vision-Language Model