Modern Large Language Models (LLMs) rely on Transformer self-attention, which scales quadratically with sequence length. Recent linear-time alternatives, like State Space Models (SSMs), often suffer from signal degradation over extended contexts. We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture. Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors, achieving sequence mixing through a causal, O(L) Phase Accumulation mechanism. To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism incorporating Frequency-Dependent Retention, Hard-Threshold Gating via Straight-Through Estimation, and a Temporal Syntax Cache to capture short-term local dependencies. We also replace standard dense linear projections with Depth-wise Harmonic Convolutions for optimal spatial frequency mixing, augmented by Block Attention Residuals for depth-wise state routing. Scaled to a 150M-parameter model, CAWN utilizes custom Triton kernels for hardware-efficient, true-complex phase accumulation in float32. Trained via a continuous streaming loop on a 100-Billion-token corpus, the prototype is evaluated at a 5-Billion-token milestone. Empirical evaluations via a Targeted Semantic Retrieval protocol demonstrate robust vocabulary acquisition and extended explicitly learned contextual denoising. By leveraging O(1) state-passing via chunked prefill, the model retrieves targeted information across 2,000,000 tokens while strictly plateauing at 8.72 GB of Peak VRAM, empirically overcoming the O(L2) context memory wall.
Figures & tables
Figure 1: CAWN-150M architecture. Quadratic attention is supplanted by complex-domain phasor synthesis with Temporal Cache and Depth-wise Harmonic Convolution, alongside GELU [ 10 ] feed-forward layers. Residual connections are restricted to block-local scope, with historical states retrieved via depth-wise attention residuals [ 2 ] .
Table 1: Peak VRAM utilization. The O(L2) Transformer baseline exhibits quadratic memory expansion, crashing (OOM) at 4k tokens on an 8GB GPU. CAWN compresses context into a fixed-size phase state, bypassing the inflating Key-Value (KV) cache bottleneck. CAWN’s VRAM strictly plateaus at 8.72 GB up to 2 Million tokens.
Figure 2: Empirical VRAM scaling trajectory for the CAWN-150M architecture evaluated on an NVIDIA H100. Extended bounds testing demonstrates predictable linear growth until the 32k chunk limit, after which the O(1) phase caching mechanism strictly caps peak VRAM at 8.72 GB. This allows the model to continuously process up to 2 Million tokens without memory degradation or expansion.
Context ( L )
Llama Baseline ( O(L2) )
Pythia Baseline ( O(L2) )
CAWN-150M ( O(1) )
1,024 Tokens
75.86 tok/s
96.81 tok/s
51.75 tok/s
2,048 Tokens
74.18 tok/s
96.09 tok/s
52.06 tok/s
4,096 Tokens
74.51 tok/s
95.19 tok/s
52.29 tok/s
8,192 Tokens
75.42 tok/s
95.19 tok/s
52.39 tok/s
16,384 Tokens
75.79 tok/s
95.04 tok/s
52.38 tok/s
Table 2: Autoregressive generation throughput evaluated at Batch Size = 1. At the 150M-parameter scale, the KV cache for L=16,384 remains small enough ( ≈600 MB) to avoid the hardware memory wall, allowing the highly-fused FlashAttention baselines to maintain high speeds. CAWN successfully demonstrates a perfectly flat, O(1) generation speed, validating its constant-memory phase cache regardless of sequence distance.
Figure 3: Validation perplexity descent of the CAWN-150M prototype compared to O(L2) baselines. CAWN demonstrates strict, monotonic convergence, successfully crossing below the parameter-matched standard Transformer baseline (Pythia-160M) as training progresses (where 800k micro-batch steps equates to ∼ 5.7 Billion tokens).
Architecture
Parameters
Training Tokens
PIQA (Acc)
ARC-Easy (Acc)
Pythia Baseline ( O(L2) )
160M
∼ 2.1 Billion
55.50%
30.64%
CAWN (Ours, O(1) )
150M
∼ 5 Billion
60.23%
45.45%
SmolLM Ceiling ( O(L2) )
135M
∼ 600 Billion
68.55%
61.74%
Table 3: Zero-shot semantic reasoning evaluation via the EleutherAI Language Model Evaluation Harness [ 23 ] . Operating with a smaller parameter footprint, CAWN at the ∼ 5B token milestone demonstrates strong emergent reasoning compared to the early-stage standard Transformer reference (Pythia at 2.1B tokens). This confirms that continuous complex-domain wave interference successfully captures discrete logical reasoning.
Testing Distance
Approx. Tokens
Red
Blue
Green
Peak VRAM
Standard Context
650
✓
✓
✓
2.42 GB
Extended (Short)
19,000
✓
✓
✓
3.78 GB
Extended (Medium)
37,800
✓
✓
✓
7.34 GB
Extreme (Long)
100,000
✓
✓
✓
8.72 GB
Extreme (Limit)
1,000,000
✓
✓
✓
8.72 GB
Phase Boundary
2,000,000
✓
✓
FAIL
8.72 GB
Table 4: Targeted Semantic Retrieval results for CAWN-150M. The model maintains 100% retrieval accuracy up to 1,000,000 tokens. At the 2,000,000 token mark (Phase Boundary), a selective failure occurs due to harmonic phase cancellation.
The scalability of Large Language Models to long sequences is hindered by the quadratic cost of attention and the limitations of positional encodings. To address these, we introduce Caracal, a novel architecture that replaces attention with a parameter-efficient, O(L log(L)) Multi-Head Fourier (MHF) module. Our contributions are threefold: (1) We leverage the Fast Fourier Transform (FFT) for sequence mixing, inherently addressing both bottlenecks mentioned above. (2) We apply a frequency-domain causal masking technique that enforces autoregressive capabilities via asymmetric padding and truncation, overcoming a critical barrier for Fourier-based generative models. (3) Unlike efficient models relying on hardware-specific implementations (e.g., Mamba), we uses standard library operators. This ensures robust portability, eliminating common deployment barriers. Evaluations demonstrate that Caracal performs competitively with Transformer and SSM baselines, offering a scalable and simple pathway for efficient long-sequence modeling. Code is available in Appendix.
Contemporary language models are dominated by the transformer architecture, which leverages self-attention mechanisms to enable more efficient, parallelized training across a wide set of documents and corpora. This has allowed transformers to effectively model data across a wide range of modalities and contexts. However, transformers, along with their conventional counterparts such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), often struggle to maintain efficiency when processing long contexts. We introduce ResonatorLM, a new mechanism that replaces attention with a physics-derived alternative. ResonatorLM treats token sequences as a single, driven one-dimensional latent field and replaces attention dot products with causal functions of damped resonators. We implement ResonatorLM on a traditional network architecture and test it on standard long-context modeling tasks. We find that in a small, 6M matched setting, training and prefill speedups increase with sequence length, decode speed reaches 6.47x compared to that of a standard, optimized transformer at 32K tokens, and accuracy reaches 61.31 percent (compared to 55.32 percent) on WikiText.
We present Harmonic, a hierarchical state space model (SSM) for language modeling. The architecture stacks three recurrent levels at progressively slower timescales; each level receives the prediction error of the level below as input, rather than its raw hidden state. On enwiki8 with equal token budgets, Harmonic outperforms a comparable Transformer (28M params) by +1.4% at 1K tokens, +6.7% at 8K tokens, and +11.4% at 32K tokens (bpt, lower is better). It also outperforms Mamba at every tested length by 0.7--1.8%. At 64K tokens, both Mamba and Transformer run out of memory on an 80GB H100; Harmonic trains successfully, reaching 6.169 bpt. Results replicate on WikiText-103 (H-TF gap +1.7% to +7.2% across 1K--32K). At 1B parameter scale, replacing all attention layers in TinyLlama 1.1B with HarmonicBlock eliminates the RoPE positional encoding limit: the resulting Hallamonic model maintains stable loss across sequence lengths 1K--8K on two independent clean benchmarks (Lambada and fineweb-edu held-out), while TinyLlama degrades catastrophically past its 2K-token RoPE limit (gap: +9.4 bpt at seq=8K on Lambada). Compute is O(L) per forward pass vs. O(L^2) for attention. Logs: https://github.com/Omibranch/harmonic-logs.