Scientific foundation models increasingly operate on continuous phenomena, yet still use discrete residual computation. We ask whether pretrained residual transformations can support additional latent computation without expanding the backbone. To test this possibility, we introduce Continuum-LM, which reuses these transformations as dynamical operators in a continuous-depth language model. Its hidden states evolve through an ordinary differential equation evaluated by higher-order numerical integration, with solver order and step size controlling computation. Across eight scientific tasks, Continuum-LM trains stably and improves mean exact-match accuracy by 7.77 and 10.96 percentage points over discrete counterparts at the 0.6B and 1.7B scales, respectively. At 0.6B, it also outperforms an Ouro-style recurrent control under matched parameter and residual-function evaluation budgets. Together, these results support extensible computational depth through parameter reuse, identifying continuous hidden-state evolution as a practical design dimension for scientific language models beyond parameter scaling.
Figures & tables
Figure 1: Continuous-depth conversion of pretrained language models. a, A standard residual network propagates hidden representations through discrete layer-wise updates. The smooth curve is an illustrative continuous-depth path rather than an assumed ground-truth trajectory; coarse discretization can produce local endpoint discrepancies that accumulate with depth. b, Numerical solvers evaluate the learned update function at intermediate hidden states within the same normalized depth interval. Higher-order quadrature and Runge–Kutta (RK) methods are shown schematically as representative integration strategies. c, Starting from a pretrained discrete language model, we replace each residual update with a local continuous-depth initial-value problem initialized from the corresponding pretrained parameters, followed by continued pretraining to adapt the model to the converted computation. d, Within each Transformer block, the self-attention and feed-forward network (FFN) residual branches are converted separately and integrated sequentially using the default two-step midpoint RK2 solver. Solver stages share the parameters of the corresponding pretrained branch but recompute its normalization and transformation at each stage-specific hidden state. e, The converted model is subsequently adapted to eight AI4Science tasks using low-rank adaptation (LoRA). f, LoRA adapters are inserted into the query, key, value and output projections of self-attention and the up, gate and down projections of the FFN, while the continuous-depth backbone is frozen.
Figure 2: Performance across eight AI4Science tasks. a, Exact-match accuracy of Continuum-LM and baseline models across eight tasks spanning four scientific task families. b, Parameter–performance trade-off, measured by mean exact-match accuracy across the eight tasks. Circles indicate supervised fine-tuned (SFT) Qwen3 baselines, the square indicates Ouro-1.4B, and stars indicate Continuum-LM models; the dashed line represents the empirical scaling trend fitted to the Qwen3 SFT results. Continuum-LM -0.6B outperforms Qwen3-1.7B@SFT, whereas Continuum-LM -1.7B outperforms Qwen3-4B@SFT and achieves performance comparable to that of Qwen3-8B@SFT. c, Task-specific evaluation across five complementary metrics per task. Each axis is direction-aligned and min–max scaled across the models displayed in the panel. d, Mean exact-match accuracy before and after LoRA fine-tuning for Continuum-LM and parameter-matched Qwen3 models. Post-LoRA scores are averaged over three independent fine-tuning runs from each fixed base checkpoint.
Figure 3: Inference-compute trade-offs and recurrent-control comparisons. a, Numbers of function evaluations for the attention and feed-forward residual branches within a Transformer block. Each branch is evaluated once in the discrete model, twice with one-step RK2, and four times with either one-step RK4 or two-step midpoint RK2. Stacked bars show the combined attention and feed-forward evaluation counts. b, Mean exact-match accuracy across eight AI4Science tasks versus estimated inference FLOPs. Circles denote Qwen3 baselines, stars denote Continuum-LM models, and the dashed line shows the empirical Qwen3 compute–performance trend. c, Task-level exact-match accuracy per TFLOP. Each cell reports exact-match accuracy expressed as a percentage divided by estimated inference cost in TFLOPs per example. The upper heatmap compares Continuum-LM -0.6B with Qwen3-1.7B; the lower heatmap compares Continuum-LM -1.7B with Qwen3-8B. d, Marginal gain in mean exact-match accuracy per additional TFLOP. Continuous-depth conversion is compared with discrete scaling from 0.6B to 1.7B and from 1.7B to 8B, respectively. e, Task-level comparison of Continuum-LM -0.6B and Qwen3-0.6B-Ouro under a common pretrained initialization, base-training corpus and downstream adaptation configuration, with four function evaluations per residual branch. Bars show mean exact-match accuracy over three LoRA fine-tuning runs from each fixed base checkpoint.
Figure 4: Ablations of continuous-depth conversion, numerical integration and downstream adaptation. a, Layer-wise parameter-scale sweep for RK2-based conversion of Qwen3-0.6B and Qwen3-1.7B. Attention and feed-forward residual branches are evaluated separately. Colours encode the base-10 logarithm of teacher-forced next-token cross-entropy on 1,000 randomly sampled training examples, and red crosses mark the loss-minimizing coefficient for each layer–branch configuration. Most selected coefficients lie near c=1 , corresponding to the original pretrained parameter scale. b,c, Training-loss trajectories over 20,000 optimization steps for Continuum-LM -0.6B (b) and Continuum-LM -1.7B (c), comparing one-step RK2, one-step RK4 and two-step midpoint RK2 under a common pretrained initialization and training protocol. These solver-ablation runs use a shorter budget than the 100,000-step continued pretraining of the main models. d, Power-law fits of WikiText-103 test loss against estimated cumulative training FLOPs for the 1.7B one-step RK2 and two-step midpoint RK2 ablation runs. The vertical axis shows excess test loss, L−E , where E is the asymptotic loss fitted separately for each configuration. Shading visually distinguishes the fitted trends and does not indicate confidence intervals. e, Task-level exact-match accuracy at LoRA ranks 8, 16 and 32 for Continuum-LM -0.6B (left) and Continuum-LM -1.7B (right). Colours distinguish retrosynthesis, catalytic activity prediction, chemical–disease relation extraction and text-to-structure generation. Increasing rank from 8 to 16 improves three of the four tasks at both model scales, whereas increasing rank further produces task-dependent gains and losses. The rank-16 scores are the corresponding task results used in the main benchmark comparison.
Figure 5: Token-level logit-margin analysis on AI4Science tasks. Distributions of reference-token logit margins on the AI4Science test sets. Under teacher-forced evaluation, the margin at each answer position is defined as the logit assigned to the reference token minus the largest logit assigned to any alternative vocabulary token. Positive values indicate that the reference token is ranked first, whereas larger values indicate stronger separation from its closest competitor. Dashed vertical lines mark the mean margin of each distribution. Relative to the corresponding Qwen3 baselines, Continuum-LM exhibits higher mean margins across the evaluated tasks, with clearer distributional separation across a broader range of tasks at the 1.7B scale.
Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we present Sigma, the first large-scale (3B/8B) continuous dLM built on steerable, low-dimensional ODE/SDE latent trajectories. Trained blockwise via likelihood optimization, Sigma jointly denoises Gaussian-corrupted token embeddings while learning an optimal embedding geometry. To accelerate training, Sigma leverages pre-trained weights from autoregressive (AR) models for warm-starting. During inference, we identify classifier-free guidance and score temperature as essential for high-fidelity reasoning and coding. Across comprehensive math reasoning and coding evaluations against state-of-the-art discrete counterparts (masked dLMs and AR baselines), Sigma achieves competitive performance with discrete models on standard benchmarks (e.g., GSM8K, Minerva, HumanEval, MBPP) after pre-training and on challenging reasoning tasks (e.g., MATH-500, AIME) after supervised fine-tuning. Beyond performance parity, we uncover key structural properties unique to continuous dLMs: (i) embedding-space steering effectively governs the quality-diversity trade-off, yielding strong pass@k performance and (ii) continuous trajectories enable graceful degradation for low NFEs and efficient distillation. These establish continuous dLMs as a promising paradigm for efficient language generation.
Zhihan Yang, Wei Guo, Jean-Marie Lemercier +14
NVIDIA · Cornell University · Georgia Institute of Technology
While frontier large language models demonstrate strong reasoning and mathematical capabilities, the practical process of training domain-specialized scientific language models from raw sources remains under-documented. In this work, we present a detailed case study of training a 1.36B-parameter scientific language model directly from raw arXiv LaTeX sources spanning mathematics, computer science, and theoretical physics. We describe an end-to-end pipeline covering metadata filtering, archive validation, LaTeX extraction, text normalization, domain-aware tokenization, and dense transformer training under constrained compute (2xA100 GPUs). Through 24 experimental runs, we analyze training stability, scaling behavior, data yield losses, and infrastructure bottlenecks. Our findings highlight how preprocessing decisions significantly affect usable token volume, how tokenization impacts symbolic stability, and how storage and I/O constraints can rival compute as limiting factors. We further analyze convergence dynamics and show stable training behavior in a data-rich regime (52B pretraining tokens). Rather than proposing a novel architecture, this work provides an engineering-grounded, transparent account of training a small scientific language model from scratch. We hope these insights support researchers operating under moderate compute budgets who seek to build domain-specialized models.
While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only 20× compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of 22.1 among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.