In-Context Estimation and Prediction for Parametric ODEs
Organizations: Department of Mathematics, University of Maryland, College Park
Abstract
Transformers have demonstrated the ability to perform in-context learning, that is, to adapt to new tasks from a few examples without updating their parameters. Existing theory has largely focused on emulating gradient-based algorithms for convex statistical problems or establishing general function approximation guarantees without exploiting the underlying statistical structure. We study in-context parameter estimation and trajectory prediction for parametric ordinary differential equations (ODEs), where estimation is generally nonconvex. Under suitable identifiability and regularity conditions, we construct a transformer that estimates unknown system parameters from noisy trajectory observations through a two-stage approximate maximum likelihood procedure: a coarse search provides an initialization within a locally strongly convex region, followed by gradient descent with local linear contraction. For prediction, we compose this estimator with a neural approximation of the ODE solution map. We further show that, through parameter sharing, a single transformer can perform both parameter estimation and trajectory prediction simultaneously. For these constructions, we establish nonasymptotic guarantees and finite-pretraining oracle inequalities that separate errors due to finite pretraining data, finite context length, and finite-sized transformer approximation. Numerical experiments on linear and nonlinear ODEs further show that pretrained transformers can benefit from longer inference contexts without updating their weights.
Figures & tables
| ODE | Equation | Parameter range | Main test characteristic | |
|---|---|---|---|---|
| Forced linear (FL) | Convex | |||
| Logistic growth (LG) | , | Nonconvex | ||
| Birth-death (BD) | , | Non-identifiable | ||
| Stuart-Landau (SL) | , | Higher dimension |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter Estimation
Parameter estimation studies how to infer unknown quantities in statistical, physical, and machine-learning models from observed data.
See all 156 papers