cs.LGNov 16, 2026

In-Context Estimation and Prediction for Parametric ODEs

Authors: Abdullah Ahmed, Yun Yang

Organizations: Department of Mathematics, University of Maryland, College Park

Abstract

Transformers have demonstrated the ability to perform in-context learning, that is, to adapt to new tasks from a few examples without updating their parameters. Existing theory has largely focused on emulating gradient-based algorithms for convex statistical problems or establishing general function approximation guarantees without exploiting the underlying statistical structure. We study in-context parameter estimation and trajectory prediction for parametric ordinary differential equations (ODEs), where estimation is generally nonconvex. Under suitable identifiability and regularity conditions, we construct a transformer that estimates unknown system parameters from noisy trajectory observations through a two-stage approximate maximum likelihood procedure: a coarse search provides an initialization within a locally strongly convex region, followed by gradient descent with local linear contraction. For prediction, we compose this estimator with a neural approximation of the ODE solution map. We further show that, through parameter sharing, a single transformer can perform both parameter estimation and trajectory prediction simultaneously. For these constructions, we establish nonasymptotic guarantees and finite-pretraining oracle inequalities that separate errors due to finite pretraining data, finite context length, and finite-sized transformer approximation. Numerical experiments on linear and nonlinear ODEs further show that pretrained transformers can benefit from longer inference contexts without updating their weights.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction

    Oct 14, 2026Jeffrey B. Lai, John Quinn, Anthony Bao +1Dynamical SystemsIn-Context Learning

  2. Transformers Learn Latent Mixture Models In-Context via Mirror Descent

    Apr 12, 2026Francesco D'Angelo, Nicolas FlammarionIn-Context LearningTransformer Attention

  3. In-context learning of closed form solution to simple linear regression task using transformer with linear self-attention

    Jul 17, 2026Katsuyuki HagiwaraIn-Context LearningLinear Regression