An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model

Christos Koutsiaris

cs.CLAug 28, 2026

An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model

Authors: Christos Koutsiaris

Organizations: Development Expert, SAP P&E, Cloud ERP, UX Foundation

Abstract

We report an exploratory, single-seed ablation of TALH (Adaptive Latent Hybrid), a decoder-only language model with parallel Multi-head Latent Attention (MLA) and a custom recurrent state-space (SSM) branch. Five variants, spanning 117--217M estimated active parameters per token, are trained from scratch on a FineWeb sample for the same number of optimisation steps and tokens. In this specific setup, removing the SSM branch gives the largest degradation in validation perplexity (MLA-only PPL 315), whereas removing MLA has a much smaller effect (SSM-only PPL 239). A dense-FFN hybrid obtains PPL 231, compared with 240 for the tested top-2 ternary-MoE hybrid, while using 3.87 GB less peak training memory. We also preserve a preliminary Apple M3 timing observation: among the five unoptimised implementations, MLA-only has the flattest measured time-to-first-token curve from 512 to 2,048 prompt tokens, although the dense Transformer is much faster in absolute terms. Because the runs are single-seed, parameter counts are unmatched, the evaluation stream may overlap the training source, and raw repeated timing records are unavailable, these results support implementation-specific hypotheses rather than general conclusions about MLA, SSMs, or mixture-of-experts models.

Figures & tables

Variant	Components	Total	Active
full	MLA+SSM+ternary MoE	493M	$\approx$ 217M
ssm_only	SSM+MoE (no MLA)	480M	$\approx$ 203M
mla_only	MLA+MoE (no SSM)	453M	$\approx$ 177M
dense_ffn	MLA+SSM+dense FFN	171M	171M
transformer_dense	MHA+dense FFN	117M	117M

TABLE I: Architectural variants (

d_{\text{model}}=800

d_{\text{ff}}=1{,}600

, 12 layers, 8 experts, top-

k=2

). Total = all stored weights; Active = weights touched per forward token pass.

Fig. 1: 15M pilot: MLA reduces KV-cache size by $8\times$ relative to standard multi-head attention.

Fig. 2: 15M pilot: total memory on Apple Silicon stays within the 18 GB budget up to 4,096-token context.

Fig. 3: Step- and token-matched quality at step 4,000. Parameter counts, operations, and wall-clock training time are not matched.

Fig. 4: Validation loss curves through Phase-1 for all five variants. The hybrid models ( dense_ffn , full ) lead throughout; mla_only consistently trails.

Variant	Val. loss	PPL	GPU memory
dense_ffn	5.443	231	24.01 GB
ssm_only	5.476	239	26.50 GB
full	5.480	240	27.88 GB
transformer_dense	5.572	263	5.36 GB
mla_only	5.753	315	16.20 GB

TABLE II: Phase-1 results at step 4,000 (same steps and nominal tokens). PPL =

e^{\text{val\_loss}}

at 512-token context.

Fig. 5: Extended training: dense_ffn and full variants through the 1,024-token curriculum stage. dense_ffn reaches the same loss as full approximately 1,000 steps sooner.

Step	Context	full	dense_ffn
2,000	256	6.117	6.074
4,000	512	5.480	5.443
6,000	1024	5.106	5.093
7,000	1024	5.093	5.074

TABLE III: Extended training: validation loss at key steps. These numbers cannot be compared against the three variants not extended to 7K.

Fig. 6: Step-4,000 quality vs. peak training memory. All variants use the same steps and nominal tokens but differ in size and operations.

Variant	512 tok	1024 tok	2048 tok
transformer_dense	114 ms	256 ms	631 ms
mla_only	1,599 ms	1,739 ms	2,110 ms
dense_ffn	2,560 ms	4,508 ms	8,390 ms
ssm_only	3,312 ms	5,100 ms	8,643 ms
full	3,364 ms	5,221 ms	9,050 ms

TABLE IV: Time to first token (TTFT, ms) on MacBook M3, 18 GB unified memory. Unoptimised prototype; no quantisation, no kernel fusion.

Fig. 7: Preliminary single-run TTFT observations on an M3 MacBook. mla_only has the flattest observed curve (1.32 $\times$ from 512 to 2,048 tokens), but no repetitions or uncertainty estimates are available.

Hybrid Language Models

See all 14 papers

Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications
Mar 23, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóHybrid Language Models Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Component-Aware Self-Speculative Decoding in Hybrid Language Models
May 1, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóSpeculative Decoding Language Modeling
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Hybrid Latent Attention for Looped Language Models
Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model Inference LLM Inference Acceleration
Virginia Tech · Independent Researcher

Mar 23, 2026J/K move · Enter open · S save

Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications

Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-Carbó

VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain

Hybrid Language Models Language Modeling Hybrid Attention Language Model Robustness

Open paper AskPDF arXiv

Hybrid language models combine softmax attention with linear-time sequence mechanisms such as state-space or linear-attention layers, but the functional contribution of each component type remains insufficiently characterized. We study component-level ablation in two sub-1B hybrid language models, Qwen3.5-0.8B and Falcon-H1-0.5B, using likelihood-based evaluation, downstream benchmarks, layer-wise interventions, random controls, and representation-level diagnostics. Across the tested models, removing either attention or the alternative sequence-processing pathway substantially degrades performance, indicating that both component types contribute to model behavior. Likelihood metrics are especially sensitive to the linear-attention or state-space pathway, while downstream benchmark degradation depends on task and architecture. Layer-wise ablations show that component importance is position-dependent, with the strongest effects concentrated in early or mid-network components rather than uniformly across depth. Random-removal controls further show that hybrid architectures and same-family Transformer baselines degrade differently under structural perturbation. These results suggest that component ablation is a useful diagnostic for understanding hybrid language model architectures. The findings provide evidence relevant to efficient model design, compression, robustness analysis, and deployment decisions in architectures that combine attention with alternative sequence-processing mechanisms.

Abstract

Hybrid Language Models

Explore similar work

Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications

Component-Aware Self-Speculative Decoding in Hybrid Language Models

Hybrid Latent Attention for Looped Language Models