TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

Kabeh Mohsenzadegan; Vahid Tavakkoli; Kyandoghere Kyamakya

cs.AISep 17, 2026

TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

Authors: Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya

Organizations: Institute for Smart System Technologies University of Klagenfurt Klagenfurt, Austria · University of Klagenfurt / Inst. f. Smart Systems Technologies, Austria & Faculte Polytechnique Universite de Kinshasa, DR-Congo

Abstract

Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers. TinyCeNN-LM introduces a quality-gated post-training conversion framework using CeNN-inspired cellular-recurrent layers with bounded local processing, compact recurrent memory, routing, fusion, and accept-or-rollback validation. Three implementations are studied: Integrated Memory, MemoryFusion, and PDelta3-GDN2-CLVR+Local32. Strict PDelta3 conversion accepts a layer only when representation and NLL criteria pass fixed thresholds. On SmolLM2-135M, layers 0-2 are accepted with cumulative $Δ\mathrm{NLL}=+0.01209$ , while layer 3 is rejected despite acceptable NLL because representation fidelity fails. On Qwen3.5-0.8B, full-attention layers 3, 7, and 11 are accepted with final $Δ\mathrm{NLL}=+0.02073$ . Integrated Memory keeps perplexity within $-0.07\%$ to $+0.93\%$ while reducing total cache by up to $6.01\%$ . A sampled 200-item downstream sanity check gives $28.5\%$ -- $32.0\%$ overall accuracy for converted Qwen releases. The results support conservative, quality-gated structural conversion rather than universal attention replacement or speedup.

Figures & tables

Fig. 1: CeNN-inspired conversion layer used throughout TinyCeNN-LM. The local operator

\Phi_{\ell}

captures bounded neighborhood evidence, the recurrent operator

\Psi_{\ell}

transports compact long-range state,

\mathcal{R}_{\ell}

optionally injects cross-layer context,

\mathcal{G}_{\ell}

fuses the paths, and

\mathcal{Q}_{\ell}

validates the proposed structural change. The dashed feedback path represents rollback: a candidate that violates the declared quality contract is not committed.

Panel of Figure 2 — Fig. 2: Three concrete CeNN-inspired layer instantiations under the abstraction in Fig. 1 . Integrated Memory is the conservative quality/cache design; MemoryFusion is an exploratory local/global fusion design; PDelta3-GDN2-CLVR+Local32 is the strongest strict replacement track in the present experiments.

Model	Extra params	CeNN state	Native KV@256	Native KV@2048
SmolLM2	0.153M	65.6 KiB	192 KiB	1.50 MiB
Qwen3.5	0.753M	178 KiB	512 KiB	4.00 MiB

TABLE I: Per-Layer Analytical State for PDelta3+Local32 (FP16, Batch 1)

Model	Layer	NMSE $\downarrow$	Cosine $\uparrow$	$\Delta$ NLL inc	$\Delta$ NLL cum	Local gate	Decision
SmolLM2-135M	0	0.0069	0.9974	+0.00239	+0.00239	0.749	Accept
	1	0.0108	0.9960	+0.00972	+0.01211	0.766	Accept
	2	0.0856	0.9587	-0.00002	+0.01209	0.766	Accept
	3	0.2265	0.8949	+0.00278	+0.01487	0.441	Reject
Qwen3.5-0.8B	3	0.0621	0.9745	-0.00325	-0.00325	0.721	Accept
	7	0.0389	0.9705	+0.01327	+0.01003	0.721	Accept

TABLE II: Strict PDelta3-GDN2-CLVR+Local32 Acceptance Across Two Pretrained Models

Panel of Figure 5 — Fig. 3: Observed quality-gated conversion frontiers. The SmolLM2 experiment exposes a representation-fidelity boundary at layer 3. The Qwen experiment validates the targeted prefix of three full-attention anchors; later anchors are not claimed as converted in this result.

Model / release	Custom	MMLU	PIQA	MMMLU	GPQA-D	Overall
	layers	Pro		DE
Qwen3.5-0.8B (native)	–	12.0	48.0	34.0	26.0	30.0
PDelta3-CLVR+Local32	3,7,11	14.0	48.0	30.0	22.0	28.5
MemoryFusion	3,7	10.0	54.0	28.0	30.0	30.5
Integrated Memory	3,23	12.0	42.0	44.0	30.0	32.0

TABLE III: Sampled Qwen3.5-0.8B FastEval Sanity Check (50 Items per Benchmark; 200 Total). These Are Not Official Full-Benchmark Scores.

Context	PPL/orig.	Cache/orig.	Save	Prefill / Decode
256	0.9993	0.9857	1.43%	0.830 / 0.893 $\times$
512	1.0021	0.9595	4.05%	0.849 / 0.910 $\times$
1024	1.0034	0.9464	5.36%	0.829 / 0.892 $\times$
2048	1.0093	0.9399	6.01%	0.830 / 0.891 $\times$

TABLE IV: SmolLM2 Integrated Memory V3 Held-Out Results

Panel of Figure 8 — Fig. 4: Integrated Memory V3 preserves test perplexity close to the baseline while the relative total-cache footprint decreases as context grows.

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 193 papers

Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion
Apr 7, 2026Zhen Cheng, Hao-Bo Yang, Wan-Yi Huang +1LLM Inference Acceleration
China Merchants Bank Artificial Intelligence Laboratory
Capability-Stratified Degradation in Ternary Language Models
Aug 28, 2026Anirudh Malik, M Sparsh Mehra, Poojith DevanLLM Quantization Ternary Quantization
OneBit AI
Data-Free Pruning of Self-Attention Layers in LLMs
Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Inference Acceleration LLM Compression
School of Computer Science University of St Andrews United Kingdom

Apr 7, 2026J/K move · Enter open · S save

Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion

Zhen Cheng, Hao-Bo Yang, Wan-Yi Huang, Jin-Long Li

China Merchants Bank Artificial Intelligence Laboratory

LLM Inference Acceleration

Open paper AskPDF arXiv

Key-Value (KV) cache memory and bandwidth increasingly dominate large language model inference cost in long-context and long-generation regimes. Architectures such as multi-head latent attention (MLA) and hybrid sliding-window attention (SWA) can alleviate this bound, but integrating them into existing models remains difficult. Prior methods impose fine-grained structural requirements on both source and target attention modules, which cannot meet the feasible requirement in practical deployment. We present Attention Editing, a practical framework for converting already-trained large language models (LLMs) with new attention architectures without re-pretraining from scratch. Attention editing replaces the original attention with a learnable target module and trains it using progressive distillation, consisting of (1) layer-wise teacher-forced optimization with intermediate activation supervision to prevent cold-start error accumulation, and (2) model-level distillation on next-token distributions, optionally regularized by weak feature matching. We instantiate the framework on two different target--MLA and GateSWA, a gated hybrid SWA design, and apply it to Qwen3-8B and Qwen3-30B-A3B. The resulting models maintain competitive performance while delivering substantial efficiency improvements, demonstrating that large-scale attention conversion is both feasible and robust. Notably, experiments are conducted on an Ascend 910B clusters, offering a practical training case study on domestic hardware.

Abstract

Language Model Post-Training

Explore similar work

Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion

Capability-Stratified Degradation in Ternary Language Models

Data-Free Pruning of Self-Attention Layers in LLMs