Beyond the Parameter Monolith: Reconstructive Memories, Executable Skills, and Residual Assembly for Language Models
Abstract
Language-model systems can separate contextual computation, persistent storage, and exact execution instead of updating all capabilities through one shared parameter system. We investigate FEM-ASM, a finite-element-method-inspired organization in which independently constructed document states and deterministic executable skills contribute typed proposals to a shared language-model state. An explicit residual operator reconciles proposals attached to common interface nodes. We evaluate this organization through controlled experiments and negative results rather than claiming a physical finite-element formulation of language. An attention-free Multi-Mesh prototype learns causal language modeling but does not establish competitive general capability. A versioned store contains 52,809 reconstructive memory elements near a 1.7-billion-floating-value budget; reconstruction is incomplete, with approximately 75% token accuracy. Support-aware lexical indices make these elements addressable under provenance-controlled query construction. For executable arithmetic, positional result observations substantially improve neural rendering relative to a repeated global result vector, and output substitutions change the model's preferred answer. A bounded attachment demonstration further measures the effect of making selected evidence available, without establishing the utility of loading an entire multi-billion-value store. The results support a separation of storage, execution, and neural coordination, while identifying unresolved limitations in question-only retrieval, unrestricted answer generation, and end-to-end efficiency.
Figures & tables
| Metric | Multi-Mesh LM |
|---|---|
| HellaSwag accuracy | 29.00 |
| HellaSwag normalized accuracy | 30.77 |
| ARC-Easy accuracy | 53.37 |
| ARC-Challenge accuracy | 21.59 |
| PIQA accuracy | 62.95 |
| WinoGrande accuracy | 49.33 |
| Quantity | Value |
|---|---|
| Evidence blocks | 2,910 |
| Elements represented | 98 |
| QA task references | 10,000 |
| Valid source tokens | 186,031 |
| Exactly reconstructed tokens | 139,468 |
| Token reconstruction accuracy | 74.97% |
| Return interface | In-range exact (%) | Out-of-range exact (%) |
|---|---|---|
| Repeated global vector | 8.59 | 6.25 |
| Character-oriented positional | 88.28 | 35.94 |
| Tokenizer-aware positional | 100.00 | 85.94 |
| Token plus character positional | 100.00 | 88.28 |
| Source condition | Coverage | NLL | Token accuracy (%) |
|---|---|---|---|
| G1p7 | 0.00 | 6.5716 | 12.10 |
| G6p5 | 1.00 | 2.8073 | 58.72 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Component | Reported configuration |
|---|---|
| Multi-Mesh large-run specification | Width 2,048; five parameterized stages; two iterations per stage; three coarse levels; ratio four; tile span 64; fine/coarse/scratch kernels 7/5/3; FFN multiplier 4.25; tile width 64; tile-state dimension 2,048. |
| Modular interface pilot | Width 512; context 512; four shared iterations; local/preconditioner kernels 7/3; FFN multiplier two; memory input dimension 768; VM input dimension 128; consistency step scale 0.25. |
| Reconstruction codec-v2 production specification | Hidden and memory dimensions 768; six decoder operators; kernel seven; FFN multiplier 2.5 rounded to a multiple of 256; tile span 64; tile width 48; optional coarse-parent ratio four. |
Memory-Augmented Language Models
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.
See all 292 papers