Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications

Hector Borobia; Elies Seguí-Mas; Guillermina Tormo-Carbó

cs.CLMar 23, 2026

Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications

Authors: Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-Carbó

Organizations: VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain

Abstract

Hybrid language models combine softmax attention with linear-time sequence mechanisms such as state-space or linear-attention layers, but the functional contribution of each component type remains insufficiently characterized. We study component-level ablation in two sub-1B hybrid language models, Qwen3.5-0.8B and Falcon-H1-0.5B, using likelihood-based evaluation, downstream benchmarks, layer-wise interventions, random controls, and representation-level diagnostics. Across the tested models, removing either attention or the alternative sequence-processing pathway substantially degrades performance, indicating that both component types contribute to model behavior. Likelihood metrics are especially sensitive to the linear-attention or state-space pathway, while downstream benchmark degradation depends on task and architecture. Layer-wise ablations show that component importance is position-dependent, with the strongest effects concentrated in early or mid-network components rather than uniformly across depth. Random-removal controls further show that hybrid architectures and same-family Transformer baselines degrade differently under structural perturbation. These results suggest that component ablation is a useful diagnostic for understanding hybrid language model architectures. The findings provide evidence relevant to efficient model design, compression, robustness analysis, and deployment decisions in architectures that combine attention with alternative sequence-processing mechanisms.

Figures & tables

Model	Condition	MMLU	GSM8K	ARC-C	HSwag	TFQA
Qwen3.5 0.8B	Baseline	.489	.350	.742	.516	.285
	Linear off ∗	.250 (-.239)	.000 (-.350)	.261 (-.482)	.268 (-.248)	.230 (-.054)
	Attention off ∗	.214 (-.274)	.000 (-.350)	.227 (-.515)	.341 (-.175)	.267 (-.018)
Falcon-H1 0.5B	Baseline	.568	.400	.736	.512	.223
	SSM off ∗	.224 (-.344)	.030 (-.370)	.211 (-.525)	.265 (-.247)	.263 (+.041)
	Attention off ∗	.228 (-.340)	.000 (-.400)	.214 (-.522)	.369 (-.143)	.227 (+.004)

Table 1: Group ablation results across five benchmarks. Values show absolute score; parentheses show change from baseline (

\Delta

). Conditions marked with

\ast

indicate that the mean score drop across all benchmarks has a bootstrap 95% CI excluding zero. Note that individual benchmarks may vary: TruthfulQA shows non-significant drops under some conditions.

Figure 1: Benchmark accuracy under baseline and group-level component ablations for Qwen3.5-0.8B and Falcon-H1-0.5B. Values correspond to the scores in Table 1 . Lower values indicate greater performance degradation after removing the specified component type. The plot summarizes task-level effects and should be interpreted together with the per-benchmark deltas in Table 1 .

Model	Condition	Loss	Perplexity	$\times$ Baseline
Qwen3.5 0.8B	Baseline	2.03	7.6	1.0 $\times$
	Linear off	12.50	268,337	35,200 $\times$
	Attention off	6.44	625	82 $\times$
Falcon-H1 0.5B	Baseline	1.73	5.6	1.0 $\times$
	SSM off	5.69	295	53 $\times$
	Attention off	2.88	17.7	3.2 $\times$

Table 2: WikiText-2 perplexity under group ablation. Removing the linear-attention pathway in Qwen3.5-0.8B and the SSM pathway in Falcon-H1-0.5B produces larger likelihood degradation than removing attention in the corresponding model. Multipliers are computed from full-precision values; displayed values are rounded.

Figure 2: WikiText-2 perplexity under group-level component ablation, shown on a logarithmic scale. Removing the linear-attention pathway in Qwen3.5-0.8B and the SSM pathway in Falcon-H1-0.5B produces larger likelihood degradation than removing attention in the corresponding model. Multipliers are computed from full-precision values; displayed table values are rounded.

Model	Condition	Targeted Mean	Random Ctrl Mean
Qwen3.5 0.8B	Linear off (18 layers)	.202	.199 $\pm$ .011
Qwen3.5 0.8B	Attention off (6 layers)	.210	.206 $\pm$ .005
Falcon-H1 0.5B	SSM off (all blocks)	.198	.203 $\pm$ .012

Table 3: Comparison of targeted ablation versus matched random controls (mean

\pm

standard deviation over 3 trials, averaged across all 5 benchmarks). Random controls remove the same number of computational units as the corresponding targeted condition. The comparison separates degradation due to the quantity and position of disrupted units from degradation associated with a specific component type.

Condition	Targeted PPL	Random PPL	Redundancy
Linear off (18 layers)	268,337	1,748,920 $\pm$ 766,270	6.5 $\times$
Attention off (6 layers)	625	41,154 $\pm$ 81,609	66 $\times$

Table 4: WikiText-2 perplexity under targeted vs. matched random ablation for Qwen3.5-0.8B (mean

\pm

standard deviation over 5 trials). Targeted removal causes less mean PPL damage than random removal of the same number of layers. Ratios are computed from mean PPL values; the 6-layer random control has high variance because some samples include critical early layers.

Layers removed	Hybrid ( $\times$ base)	Transformer ( $\times$ base)	Ratio
18 of 24	229,408 $\times$	4,701,068 $\times$	20 $\times$
6 of 24	5,398 $\times$	640,850 $\times$	119 $\times$

Table 5: Perplexity under random layer removal: hybrid (Qwen3.5-0.8B) vs. pure Transformer (Qwen2.5-0.5B). Values are normalized as multiples of each model’s baseline perplexity to enable cross-architecture comparison. Under this protocol, the Transformer control shows 20–119

\times

larger normalized PPL increases than the hybrid model.

Figure 3: Mean benchmark accuracy under targeted component ablation and matched random ablation. Random controls remove the same number of computational units as the corresponding targeted condition and are averaged over trials; error bars show standard deviation across random trials. The comparison separates degradation due to the quantity and position of disrupted units from degradation associated with a specific component type.

Figure 4: Layer-wise ablation impact, measured as the mean accuracy drop from baseline across MMLU and ARC-Challenge after ablating one component at one layer. Points correspond only to layers where the component is present. Larger positive values indicate greater degradation. The strongest effects are concentrated in early or mid-network components, while late layers generally show smaller drops.

Figure 5: Per-benchmark layer-wise ablation impact for each model. Cell values show score drop from the corresponding baseline after ablating a single component at the indicated layer. Warmer colors indicate larger degradation; blank cells indicate layers where that component type is not present. The heatmaps provide a task-specific view of the positional sensitivity summarized in Figure 4 .

Component	Position	MMLU	GSM8K	ARC-C	HSwag	TFQA
Baseline		.568	.400	.736	.512	.223
SSM off	Early	.255	.020	.294	.274	.248
	Middle	.454	.280	.592	.457	.224
	Late	.538	.160	.676	.485	.234
Attn off	Early	.278	.040	.244	.442	.230
	Middle	.441	.000	.555	.412	.225

Table 6: Falcon-H1 positional ablation: removing all components of one type within a positional third. Early-layer ablation causes substantially more damage than late-layer ablation for both SSM and attention.

Figure 6: Relationship between layer-wise hidden-state diagnostics and behavioral ablation impact. Each point corresponds to one ablated component/layer condition. The $x$ -axis reports either normalized hidden-state change or logit-lens KL divergence; the $y$ -axis reports mean accuracy drop across MMLU and ARC-Challenge. Pearson correlations and $p$ -values are computed over the plotted points. Norm change correlates positively with ablation impact in Qwen and more weakly in Falcon, whereas logit-lens KL does not significantly predict ablation impact in either model.

Figure 7: Score drop by benchmark and group-ablation condition. Columns correspond to benchmarks and rows to model–condition pairs. Warmer colors indicate larger degradation relative to the corresponding baseline; negative values indicate improvement under ablation. In the tested models, GSM8K shows the largest drops, HellaSwag shows clear component asymmetry, and TruthfulQA is comparatively insensitive to several ablations.

Hybrid Language Models

See all 14 papers

Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
Apr 24, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóLow-Rank Adaptation Recurrent Transformers
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Component-Aware Self-Speculative Decoding in Hybrid Language Models
May 1, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóSpeculative Decoding Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model
Aug 28, 2026Christos KoutsiarisHybrid Language Models Decoder-Only Language Models
Development Expert, SAP P&E, Cloud ERP, UX Foundation

Apr 24, 2026J/K move · Enter open · S save

Where Should LoRA Go? Component-Type Placement in Hybrid Language Models

Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-Carbó

VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain

Low-Rank Adaptation Recurrent Transformers Language Modeling Parameter-Efficient Fine-Tuning Hybrid Language Models

Open paper AskPDF arXiv

Hybrid language models that interleave attention with recurrent components are increasingly competitive with pure Transformers, yet standard LoRA practice applies adapters uniformly without considering the distinct functional roles of each component type. We systematically study component-type LoRA placement across two hybrid architectures -- Qwen3.5-0.8B (sequential, GatedDeltaNet + softmax attention) and Falcon-H1-0.5B (parallel, Mamba-2 SSM + attention) -- fine-tuned on three domains and evaluated on five benchmarks. We find that the attention pathway -- despite being the minority component -- consistently outperforms full-model adaptation with 5-10x fewer trainable parameters. Crucially, adapting the recurrent backbone is destructive in sequential hybrids (-14.8 pp on GSM8K) but constructive in parallel ones (+8.6 pp). We further document a transfer asymmetry: parallel hybrids exhibit positive cross-task transfer while sequential hybrids suffer catastrophic forgetting. These results establish that hybrid topology fundamentally determines adaptation response, and that component-aware LoRA placement is a necessary design dimension for hybrid architectures.

Abstract

Hybrid Language Models

Explore similar work

Where Should LoRA Go? Component-Type Placement in Hybrid Language Models

Component-Aware Self-Speculative Decoding in Hybrid Language Models

An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model