cs.LGOct 18, 2026
SaveDynamic System Inspired Continuous Large Language Model for AI4Science Tasks
Organizations: Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University, Beijing, 100191, Beijing, China. · School of Artificial Intelligence, Beihang University, Beijing, 100191, Beijing, China. · GigaAI, Beijing, 100191, Beijing, China.
Abstract
Scientific foundation models increasingly operate on continuous phenomena, yet still use discrete residual computation. We ask whether pretrained residual transformations can support additional latent computation without expanding the backbone. To test this possibility, we introduce Continuum-LM, which reuses these transformations as dynamical operators in a continuous-depth language model. Its hidden states evolve through an ordinary differential equation evaluated by higher-order numerical integration, with solver order and step size controlling computation. Across eight scientific tasks, Continuum-LM trains stably and improves mean exact-match accuracy by 7.77 and 10.96 percentage points over discrete counterparts at the 0.6B and 1.7B scales, respectively. At 0.6B, it also outperforms an Ouro-style recurrent control under matched parameter and residual-function evaluation budgets. Together, these results support extensible computational depth through parameter reuse, identifying continuous hidden-state evolution as a practical design dimension for scientific language models beyond parameter scaling.
Figures & tables
Figure 1: Continuous-depth conversion of pretrained language models. a, A standard residual network propagates hidden representations through discrete layer-wise updates. The smooth curve is an illustrative continuous-depth path rather than an assumed ground-truth trajectory; coarse discretization can produce local endpoint discrepancies that accumulate with depth. b, Numerical solvers evaluate the learned update function at intermediate hidden states within the same normalized depth interval. Higher-order quadrature and Runge–Kutta (RK) methods are shown schematically as representative integration strategies. c, Starting from a pretrained discrete language model, we replace each residual update with a local continuous-depth initial-value problem initialized from the corresponding pretrained parameters, followed by continued pretraining to adapt the model to the converted computation. d, Within each Transformer block, the self-attention and feed-forward network (FFN) residual branches are converted separately and integrated sequentially using the default two-step midpoint RK2 solver. Solver stages share the parameters of the corresponding pretrained branch but recompute its normalization and transformation at each stage-specific hidden state. e, The converted model is subsequently adapted to eight AI4Science tasks using low-rank adaptation (LoRA). f, LoRA adapters are inserted into the query, key, value and output projections of self-attention and the up, gate and down projections of the FFN, while the continuous-depth backbone is frozen.
Figure 2: Performance across eight AI4Science tasks. a, Exact-match accuracy of Continuum-LM and baseline models across eight tasks spanning four scientific task families. b, Parameter–performance trade-off, measured by mean exact-match accuracy across the eight tasks. Circles indicate supervised fine-tuned (SFT) Qwen3 baselines, the square indicates Ouro-1.4B, and stars indicate Continuum-LM models; the dashed line represents the empirical scaling trend fitted to the Qwen3 SFT results. Continuum-LM -0.6B outperforms Qwen3-1.7B@SFT, whereas Continuum-LM -1.7B outperforms Qwen3-4B@SFT and achieves performance comparable to that of Qwen3-8B@SFT. c, Task-specific evaluation across five complementary metrics per task. Each axis is direction-aligned and min–max scaled across the models displayed in the panel. d, Mean exact-match accuracy before and after LoRA fine-tuning for Continuum-LM and parameter-matched Qwen3 models. Post-LoRA scores are averaged over three independent fine-tuning runs from each fixed base checkpoint.
Figure 3: Inference-compute trade-offs and recurrent-control comparisons. a, Numbers of function evaluations for the attention and feed-forward residual branches within a Transformer block. Each branch is evaluated once in the discrete model, twice with one-step RK2, and four times with either one-step RK4 or two-step midpoint RK2. Stacked bars show the combined attention and feed-forward evaluation counts. b, Mean exact-match accuracy across eight AI4Science tasks versus estimated inference FLOPs. Circles denote Qwen3 baselines, stars denote Continuum-LM models, and the dashed line shows the empirical Qwen3 compute–performance trend. c, Task-level exact-match accuracy per TFLOP. Each cell reports exact-match accuracy expressed as a percentage divided by estimated inference cost in TFLOPs per example. The upper heatmap compares Continuum-LM -0.6B with Qwen3-1.7B; the lower heatmap compares Continuum-LM -1.7B with Qwen3-8B. d, Marginal gain in mean exact-match accuracy per additional TFLOP. Continuous-depth conversion is compared with discrete scaling from 0.6B to 1.7B and from 1.7B to 8B, respectively. e, Task-level comparison of Continuum-LM -0.6B and Qwen3-0.6B-Ouro under a common pretrained initialization, base-training corpus and downstream adaptation configuration, with four function evaluations per residual branch. Bars show mean exact-match accuracy over three LoRA fine-tuning runs from each fixed base checkpoint.
Figure 4: Ablations of continuous-depth conversion, numerical integration and downstream adaptation. a, Layer-wise parameter-scale sweep for RK2-based conversion of Qwen3-0.6B and Qwen3-1.7B. Attention and feed-forward residual branches are evaluated separately. Colours encode the base-10 logarithm of teacher-forced next-token cross-entropy on 1,000 randomly sampled training examples, and red crosses mark the loss-minimizing coefficient for each layer–branch configuration. Most selected coefficients lie near , corresponding to the original pretrained parameter scale. b,c, Training-loss trajectories over 20,000 optimization steps for Continuum-LM -0.6B (b) and Continuum-LM -1.7B (c), comparing one-step RK2, one-step RK4 and two-step midpoint RK2 under a common pretrained initialization and training protocol. These solver-ablation runs use a shorter budget than the 100,000-step continued pretraining of the main models. d, Power-law fits of WikiText-103 test loss against estimated cumulative training FLOPs for the 1.7B one-step RK2 and two-step midpoint RK2 ablation runs. The vertical axis shows excess test loss, , where is the asymptotic loss fitted separately for each configuration. Shading visually distinguishes the fitted trends and does not indicate confidence intervals. e, Task-level exact-match accuracy at LoRA ranks 8, 16 and 32 for Continuum-LM -0.6B (left) and Continuum-LM -1.7B (right). Colours distinguish retrosynthesis, catalytic activity prediction, chemical–disease relation extraction and text-to-structure generation. Increasing rank from 8 to 16 improves three of the four tasks at both model scales, whereas increasing rank further produces task-dependent gains and losses. The rank-16 scores are the corresponding task results used in the main benchmark comparison.
Figure 5: Token-level logit-margin analysis on AI4Science tasks. Distributions of reference-token logit margins on the AI4Science test sets. Under teacher-forced evaluation, the margin at each answer position is defined as the logit assigned to the reference token minus the largest logit assigned to any alternative vocabulary token. Positive values indicate that the reference token is ranked first, whereas larger values indicate stronger separation from its closest competitor. Dashed vertical lines mark the mean margin of each distribution. Relative to the corresponding Qwen3 baselines, Continuum-LM exhibits higher mean margins across the evaluated tasks, with clearer distributional separation across a broader range of tasks at the 1.7B scale.
More in this topic
Neural ODEs
Neural ordinary differential equations (Neural ODEs) learn continuous-time dynamics by representing a network’s evolution as a differential equation.
See all 91 papers