We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning. Using hidden states as cache and carrier, reasoners repeatedly query memory for required knowledge-vectors, while the knowledge is transmitted back to reasoning operators. Through this knowledge-reasoning-separation architecture, Mobius achieves better knowledge compression and reasoning efficiency. Built upon Mobius-v0 architecture: 1) Our 7B model trained-from-scratch achieves similar downstream score as a 7B Transformer baseline with 62.6% of baseline's training data. 2) Our Intern-S2-Mobius, continually-pretrained from Qwen3.5-35B, achieves similar downstream score while delivering nearly 4x end-to-end inference speedup.
Figures & tables
Figure 1: The Comparison between Transformer and Mobius.
Figure 2: The MMLU score of Mobius and Transformer pre-trained from scratch.
Figure 3: The inference efficiency of Mobius continual pre-trained from Qwen3.5.
Figure 4: The average output length of Mobius continual pre-trained from Qwen3.5.
Figure 5: The comparison between RNN, Transformer, and Mobius.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Expert-Activation pattern across layers under two training recipes. The left panel shows Mobius-7B trained from scratch; the right panel shows Intern-S2-Mobius-35B obtained by architecture conversion and continual pre-training. Rows denote expert IDs, columns denote layer indices, and color represents the base-10 logarithm of selection frequency. Expert IDs in the right panel are reordered to reveal the inherited routing structure.
Figure 7: The Compositional Generalization task selected from Physics of LLM. A future-version of Mobius(unreleased) enjoys faster convergence, higher final scores than Transformer. The train set contains 500-entity single-hop + 400-entity two-hop, while the test set contains 100-entity two-hop holdout. This task evaluates whether models learn cross-knowledge connections, not mere memorization.
Figure 8: Layerwise prediction lens for Mobius and the Qwen3.5 baseline under an identical teacher-forced context. The t+1 column represents standard next-token prediction, while t+2 through t+5 are subsequent MTP predictions. Each cell shows the token predicted from a given hidden-state view; color intensity denotes its probability, and a black outline marks agreement with the target continuation. Mobius exhibits more target-aligned intermediate predictions and produces a five-token draft accepted in full, while the baseline produces only a two-token accepted prefix.
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.