CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling

Dejan Čugalj; Aleksandar Jevremovic

cs.CLApr 5, 2026

CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling

Authors: Dejan Čugalj, Aleksandar Jevremovic

Organizations: Singidunum University

Abstract

Modern Large Language Models (LLMs) rely on Transformer self-attention, which scales quadratically with sequence length. Recent linear-time alternatives, like State Space Models (SSMs), often suffer from signal degradation over extended contexts. We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture. Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors, achieving sequence mixing through a causal, $O(L)$ Phase Accumulation mechanism. To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism incorporating Frequency-Dependent Retention, Hard-Threshold Gating via Straight-Through Estimation, and a Temporal Syntax Cache to capture short-term local dependencies. We also replace standard dense linear projections with Depth-wise Harmonic Convolutions for optimal spatial frequency mixing, augmented by Block Attention Residuals for depth-wise state routing. Scaled to a 150M-parameter model, CAWN utilizes custom Triton kernels for hardware-efficient, true-complex phase accumulation in float32. Trained via a continuous streaming loop on a 100-Billion-token corpus, the prototype is evaluated at a 5-Billion-token milestone. Empirical evaluations via a Targeted Semantic Retrieval protocol demonstrate robust vocabulary acquisition and extended explicitly learned contextual denoising. By leveraging $O(1)$ state-passing via chunked prefill, the model retrieves targeted information across 2,000,000 tokens while strictly plateauing at 8.72 GB of Peak VRAM, empirically overcoming the $O(L^2)$ context memory wall.

Figures & tables

Panel of Figure 1 — Figure 1: CAWN-150M architecture. Quadratic attention is supplanted by complex-domain phasor synthesis with Temporal Cache and Depth-wise Harmonic Convolution, alongside GELU [ 10 ] feed-forward layers. Residual connections are restricted to block-local scope, with historical states retrieved via depth-wise attention residuals [ 2 ] .

*Evaluated utilizing chunked prefill continuous state-passing.
Length	Trans.	CAWN	Length	Trans.	CAWN
512	2.45 GB	2.01 GB	37,831	—	7.34 GB
1,024	2.82 GB	2.26 GB	100k	—	8.72 GB *
2,048	3.89 GB	2.64 GB	1M	—	8.72 GB *
4,096	OOM	3.40 GB	2M	—	8.72 GB *
8,192	—	4.91 GB

Table 1: Peak VRAM utilization. The

\mathcal{O}(L^{2})

Transformer baseline exhibits quadratic memory expansion, crashing (OOM) at 4k tokens on an 8GB GPU. CAWN compresses context into a fixed-size phase state, bypassing the inflating Key-Value (KV) cache bottleneck. CAWN’s VRAM strictly plateaus at 8.72 GB up to 2 Million tokens.

Panel of Figure 2 — Figure 2: Empirical VRAM scaling trajectory for the CAWN-150M architecture evaluated on an NVIDIA H100. Extended bounds testing demonstrates predictable linear growth until the 32k chunk limit, after which the $\mathcal{O}(1)$ phase caching mechanism strictly caps peak VRAM at 8.72 GB. This allows the model to continuously process up to 2 Million tokens without memory degradation or expansion.

Context ( $L$ )	Llama Baseline ( $\mathcal{O}(L^{2})$ )	Pythia Baseline ( $\mathcal{O}(L^{2})$ )	CAWN-150M ( $\mathcal{O}(1)$ )
1,024 Tokens	75.86 tok/s	96.81 tok/s	51.75 tok/s
2,048 Tokens	74.18 tok/s	96.09 tok/s	52.06 tok/s
4,096 Tokens	74.51 tok/s	95.19 tok/s	52.29 tok/s
8,192 Tokens	75.42 tok/s	95.19 tok/s	52.39 tok/s
16,384 Tokens	75.79 tok/s	95.04 tok/s	52.38 tok/s

Table 2: Autoregressive generation throughput evaluated at Batch Size = 1. At the 150M-parameter scale, the KV cache for

L=16{,}384

remains small enough (

\approx 600

MB) to avoid the hardware memory wall, allowing the highly-fused FlashAttention baselines to maintain high speeds. CAWN successfully demonstrates a perfectly flat,

\mathcal{O}(1)

generation speed, validating its constant-memory phase cache regardless of sequence distance.

Panel of Figure 3 — Figure 3: Validation perplexity descent of the CAWN-150M prototype compared to $\mathcal{O}(L^{2})$ baselines. CAWN demonstrates strict, monotonic convergence, successfully crossing below the parameter-matched standard Transformer baseline (Pythia-160M) as training progresses (where 800k micro-batch steps equates to $\sim$ 5.7 Billion tokens).

Architecture	Parameters	Training Tokens	PIQA (Acc)	ARC-Easy (Acc)
Pythia Baseline ( $\mathcal{O}(L^{2})$ )	160M	$\sim$ 2.1 Billion	55.50%	30.64%
CAWN (Ours, $\mathcal{O}(1)$ )	150M	$\sim$ 5 Billion	60.23%	45.45%
SmolLM Ceiling ( $\mathcal{O}(L^{2})$ )	135M	$\sim$ 600 Billion	68.55%	61.74%

Table 3: Zero-shot semantic reasoning evaluation via the EleutherAI Language Model Evaluation Harness [ 23 ] . Operating with a smaller parameter footprint, CAWN at the

\sim

5B token milestone demonstrates strong emergent reasoning compared to the early-stage standard Transformer reference (Pythia at 2.1B tokens). This confirms that continuous complex-domain wave interference successfully captures discrete logical reasoning.

Testing Distance	Approx. Tokens	Red	Blue	Green	Peak VRAM
Standard Context	650	✓	✓	✓	2.42 GB
Extended (Short)	19,000	✓	✓	✓	3.78 GB
Extended (Medium)	37,800	✓	✓	✓	7.34 GB
Extreme (Long)	100,000	✓	✓	✓	8.72 GB
Extreme (Limit)	1,000,000	✓	✓	✓	8.72 GB
Phase Boundary	2,000,000	✓	✓	FAIL	8.72 GB

Table 4: Targeted Semantic Retrieval results for CAWN-150M. The model maintains 100% retrieval accuracy up to 1,000,000 tokens. At the 2,000,000 token mark (Phase Boundary), a selective failure occurs due to harmonic phase cancellation.

Long-Context Language Modeling

Long-context language modeling studies how language models can use, retain, and efficiently process information across very long sequences.

See all 189 papers

Caracal: Causal Architecture via Spectral Mixing
Apr 30, 2026Bingzheng Gan, Tianyi Zhang, Yusu Li +4Long-Context Language Modeling Autoregressive Language Modeling
Huawei Technologies Co., Ltd., China.
ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling
Jul 6, 2026Archie ChaudhuryLong-Context Language Model Inference Long-Context Language Modeling
Axionic Labs
Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling
May 30, 2026Petr NyomaLong-Context Language Modeling State Space Models
Independent Researcher

Apr 30, 2026J/K move · Enter open · S save

Caracal: Causal Architecture via Spectral Mixing

Bingzheng Gan, Tianyi Zhang, Yusu Li, Jing Huang+3

Huawei Technologies Co., Ltd., China.

Long-Context Language Modeling Autoregressive Language Modeling LLM Inference Acceleration

Open paper AskPDF arXiv

The scalability of Large Language Models to long sequences is hindered by the quadratic cost of attention and the limitations of positional encodings. To address these, we introduce Caracal, a novel architecture that replaces attention with a parameter-efficient, O(L log(L)) Multi-Head Fourier (MHF) module. Our contributions are threefold: (1) We leverage the Fast Fourier Transform (FFT) for sequence mixing, inherently addressing both bottlenecks mentioned above. (2) We apply a frequency-domain causal masking technique that enforces autoregressive capabilities via asymmetric padding and truncation, overcoming a critical barrier for Fourier-based generative models. (3) Unlike efficient models relying on hardware-specific implementations (e.g., Mamba), we uses standard library operators. This ensures robust portability, eliminating common deployment barriers. Evaluations demonstrate that Caracal performs competitively with Transformer and SSM baselines, offering a scalable and simple pathway for efficient long-sequence modeling. Code is available in Appendix.

Abstract

Long-Context Language Modeling

Explore similar work

Caracal: Causal Architecture via Spectral Mixing

ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modeling

Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling