cs.CLMar 12, 2026

LPC-SM: Local Predictive Coding and Sparse Memory for Long-Context Language Modeling

Authors: Keqin Xie

Organizations: Independent Researcher, Suzhou, China

Abstract

Most current long-context language models still rely on attention to handle both local interaction and long-range state, which leaves relatively little room to test alternative decompositions of sequence modeling. We propose LPC-SM, a hybrid autoregressive architecture that separates local attention, persistent memory, predictive correction, and run-time control within the same block, and we use Orthogonal Novelty Transport (ONT) to govern slow-memory writes. We evaluate a 158M-parameter model in three stages spanning base language modeling, mathematical continuation, and 4096-token continuation. Removing mHC raises the Stage-A final LM loss from 12.630 to 15.127, while adaptive sparse control improves the Stage-B final LM loss from 12.137 to 10.787 relative to a matched fixed-ratio continuation. The full route remains stable at sequence length 4096, where Stage C ends with final LM loss 11.582 and improves the delayed-identifier diagnostic from 14.396 to 12.031 in key cross-entropy. Taken together, these results show that long-context autoregressive modeling can be organized around a broader division of labor than attention alone.

Figures & tables

Explore similar work

CardsList
  1. MMLA: Memory-Mediated Learning Architecture for Predictive Dual-State Adaptation

    Jun 27, 2026Junyi Zou, Avrova DonzContinual Learning for LLMsTest-Time Adaptation

  2. Fractional State Space Transition for Long Sequence Modeling

    Sep 28, 2026Ivan Kobyzev, Abbas Ghaddar, Ali Nasiri-Sarvi +2State Space ModelsLong-Context Language Modeling

  3. Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

    Jul 3, 2026Xiang Hu, Xinyu Wei, Hao Gu +10Long-Context Language Model InferenceLong-Context Language Modeling