Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attention

George Fountzoulas

cs.CLApr 9, 2026

Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attention

Authors: George Fountzoulas

Organizations: Department of Computer Engineering & Informatics, Frederick University, Nicosia, Cyprus

Abstract

We present Kathleen, a text classification architecture that operates directly on raw UTF-8 bytes using frequency-domain processing -- requiring no tokenizer, no attention mechanism, and under 470K parameters. Kathleen introduces several novel components: (1) RecurrentOscillatorBanks -- damped sinusoid convolutions with temporal memory for O(L) sequence processing; (2) an FFT-Rotate Wavetable Encoder that maps all 256 byte values using a single learnable vector (256 floats); (3) PhaseHarmonics -- a sinusoidal non-linearity with just 6 learnable phase parameters (+2.6% accuracy, <0.001% of model parameters); (4) Content-Dependent Reverb with Positional Decay Modulation -- a temporal memory mechanism whose decay rate is jointly conditioned on input content and a learned position-indexed bias vector; (5) Token-Level Module Sequencer with consonance and dissonance interference channels. Through iterative architecture evolution from an initial 733K-parameter baseline (Kathleen-Clean) to the current Kathleen-V9 (469K parameters), we demonstrate that pretraining can be entirely eliminated while improving accuracy. Kathleen-V9 achieves 88.5% +/- 0.2% on IMDB, 92.4% +/- 0.2% on AG News, and 85.8% +/- 0.5% on SST-2 (3-seed averages) -- matching or exceeding the pretrained baseline on all benchmarks with 36% fewer parameters. On SST-2, the improvement is +2.5% absolute over the pretrained predecessor. Kathleen processes sequences in O(L) time and memory.

Figures & tables

Encoder	Accuracy	Params (encoder)
nn.Embedding(256, 256)	83.1%	65,536
Fourier features ( $K\!=\!32$ )	82.3%	8,256
FFT-Rotate wavetable	83.7%	256

Table 1: Byte encoder comparison (IMDB, same architecture otherwise).

Dataset	Task	Train	Test	Classes
IMDB ( Maas et al., 2011 )	Sentiment	25,000	25,000	2
AG News ( Zhang et al., 2015 )	Topic	120,000	7,600	4
SST-2 ( Socher et al., 2013 )	Sentiment	67,349	872	2

Table 2: Benchmark datasets.

Pretrained Transformers (reference)
Model	IMDB	AG News	SST-2	Params	Pretrain
BERT-base ( Devlin et al., 2019 )	93.0	94.0	93.0	110M	Ext. corpus
DistilBERT ( Sanh et al., 2019 )	92.0	93.0	91.0	66M	Ext. corpus
Byte/char-level Transformers
CANINE-S ( Clark et al., 2022 )	—	—	85.8	132M	Ext. corpus
ByT5-Small ( Xue et al., 2022 )	—	—	${\sim}$ 92	300M	Ext. corpus

Table 3: Main classification results. All Kathleen variants operate on raw UTF-8 bytes with no tokenizer and no attention. V0 uses MLM pretraining; V9 uses no pretraining. All byte-level results report mean

\pm

std over 3 seeds.

Dataset	seed=42	seed=123	seed=456
SST-2	86.4%	85.3%	85.6%
IMDB	88.8%	88.5%	88.2%
AG News	92.3%	92.2%	92.6%

Table 4: Individual seed results for Kathleen-V9 (no pretraining).

Model	$L=1024$	$L=2048$	$L=4096$
Transformer (byte)	82.1%	OOM	OOM
Kathleen (byte)	83.7%	84.4%	85.1%

Table 5: Accuracy vs. sequence length (IMDB). Transformer runs out of memory (OOM) beyond

L=1024

bytes on a single T4 GPU.

Variant	IMDB	$\Delta$
Full model	87.0%	—
$-$ Adaptive Gate	84.8%	$-$ 2.2%
$-$ ConvLiteC	85.8%	$-$ 1.2%
$-$ PowerLawGate	87.0%	0.0%
$-$ ResonanceCodebook	87.1%	+0.1%

Table 6: Tokenized Kathleen ablation (IMDB).

Variant	SST-2 (%)	Params	$\Delta$
Full (all components)	84.4	1,823K	—
$-$ PhaseHarmonics	81.8	1,364K	$-$ 2.6
$-$ HierarchicalKeys	82.7	1,293K	$-$ 1.7
$-$ PhaseShift	83.1	1,790K	$-$ 1.3
$-$ SDM Memory	83.4	1,823K	$-$ 1.0
$-$ Phantasy (560K params)	84.2	1,264K	$-$ 0.2

Table 7: Component ablation of the predecessor model (SST-2, seed=42). Each row removes one component from the full model.

	SST-2		IMDB		AG News
Model	Mean	$\Delta$	Mean	$\Delta$	Mean	$\Delta$
V0 (733K, pretrain)	83.3%	—	88.6%	—	92.3%	—
V9 (469K, no pretrain)	85.8%	+2.5	88.5%	$-$ 0.1	92.4%	+0.1

Table 8: V0 vs. V9 comparison (3-seed averages). V0 uses MLM pretraining; V9 uses no pretraining.

Context	PLG effect
Tokenized (word embeddings)	0.0%
Byte + nn.Embedding	0.0%
Byte + FFT-Rotate	+0.9%

Table 9: PowerLawGate effect depends on input representation.

Model	IMDB (%)	Params	Acc/M-params
BERT-base	93.0	110M	0.85
DistilBERT	92.0	66M	1.39
CANINE-S	—	132M	—
Kathleen-Clean (V0)	88.6	733K	120.9
Kathleen-V9	88.5	469K	188.7

Table 10: Parameter efficiency comparison (IMDB accuracy).

Text Classification

Text classification assigns text to categories such as topic, sentiment, intent, stance, or risk, supporting search, moderation, and decisions.

See all 226 papers

Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention
Aug 5, 2026George FountzoulasLanguage Model Generation Evaluation Autoregressive Language Modeling
Department of Computer Engineering & Informatics · Frederick University, Nicosia, Cyprus
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model Scaling Byte-Level Language Model
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Mar 3, 2026Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3Language Modeling Byte-Level Language Model
Rice University · Amazon Science

Aug 5, 2026J/K move · Enter open · S save

Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention

George Fountzoulas

Department of Computer Engineering & Informatics · Frederick University, Nicosia, Cyprus

Language Model Generation Evaluation Autoregressive Language Modeling Language Model Scaling Language Modeling Generative AI Evaluation

Open paper AskPDF arXiv

Papers 1-2 of the Kathleen series showed that a byte-level, attention-free architecture built from a wavetable encoder and multi-scale reverberant state can match strong baselines on classification at ~450-700K parameters, without pretraining. We ask whether the same ingredients can generate. (1) Scaling: on byte-level language modeling (WikiText-103, raw UTF-8, no tokenizer), the reverberant model beats a parameter-matched transformer at every dataset scale measured (2-512 MB), e.g. 1.84 vs 2.04 bits/byte at 512 MB with ~0.5M parameters; the transformer needs more than 512 MB to match what the attention-free model learns from 32 MB. (2) Measurement: we introduce FORM DISTANCE, a non-parametric, gaming-resistant instrument for "reads like text": nine statistical axes of human text define a reference cloud, and five constructed fakes are all rejected. (3) Generation: decoding policy dominates architecture -- widening the sampler halves the same model's distance (3.17 to 1.52), and a retrieval-augmented decoding scheme takes the frozen model further (1.52 to 1.14) with no training step involved; the ablation attributes the gain to the sparse phrase dose itself, not the selection gate. The gain has a sharp boundary condition: the phrases must come from the model's own training corpus -- a 40x larger foreign library helps not at all, an effect the attention twin shares, consistent with in-context integration being a capability of scale. We also report four architectural additions that did not help, and a computed lexicon reaching 94% of a learned table's top-1 accuracy at one fifth of the parameters. Everything runs offline; all experiments are reproducible on a free Kaggle T4.

Abstract

Text Classification

Explore similar work

Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer