In-context operator learning enables neural networks to infer solution operators from contextual examples without weight updates. While prior work has demonstrated the effectiveness of this paradigm in leveraging vast datasets, a systematic comparison against single-operator learning using identical training data has been absent. We address this gap through controlled experiments comparing in-context operator learning against classical operator learning (single-operator models trained without contextual examples), under the same training steps and dataset. To enable this investigation on real-world spatiotemporal systems, we propose GICON (Graph In-Context Operator Network), combining graph message passing for geometric generalization with example-aware positional encoding for cardinality generalization. Experiments on air quality prediction across two Chinese regions show that in-context operator learning outperforms classical operator learning on complex tasks, generalizing across spatial domains and scaling robustly from few training examples to 100 at inference.
Figures & tables
Figure 1: Illustration of In-Context Operator Networks (ICON). Given k contextual examples, each consisting of a key-value pair related by the governing operator F , ICON infers the underlying operator from these in-context examples and applies it to map a new query input to its predicted output—all in a single forward pass without any weight updates.
Figure 2: Overview of the Graph In-Context Operator Network (GICON) architecture. Left : The overall pipeline selects contextual examples from historical frames, producing an interleaved sequence of keys and values along with the query. Each frame is processed by separate key and value encoders, passed through N GICON layers, and decoded at key and query positions to produce predictions (during inference, only the query prediction is used). Right : Detailed structure of a single GICON layer, consisting of (1) spatial update via message passing that aggregates information from neighboring nodes within each graph, and (2) contextual per-node update using transformers that perform in-context learning across the example sequence with positional encodings.
Figure 3: Example cardinality generalization on BTHSA for simple to moderate operators. Top: PM 2.5 . Bottom: O 3 . Left to right: Δt=1,4,12 h. Classical single-operator learning achieves lower RMSE for simple operators ( Δt=1,4 h), while ICON with operator diversity outperforms the baseline at Δt=12 h given sufficient examples. All models are evaluated with up to 100 examples despite training with at most 5.
Figure 4: Example cardinality generalization on BTHSA at Δt=24 h. Left: PM 2.5 . Right: O 3 . For this complex operator, ICON with operator diversity surpasses the single-operator baseline with sufficient examples, with error decreasing for PM 2.5 and a sharp initial drop followed by stable performance for O 3 .
Figure 5: Operator extrapolation to Δt=48 (out-of-distribution) on BTHSA. Left: PM 2.5 . Right: O 3 . Single-operator shows flat curves, while example-trained ICON models improve with examples. Models with k=5 achieve best extrapolation, with sustained improvement for PM 2.5 and a sharp initial drop for O 3 .
Figure 6: Geometric generalization: models trained on BTHSA or YRD, both evaluated on YRD at Δt=24 h without fine-tuning. Left: PM 2.5 . Right: O 3 . BTHSA-trained models transfer to YRD with moderate performance gap for PM 2.5 and minimal gap for O 3 , suggesting cross-region generalization.
Figure 7: Single-operator setting ( Δt=24 , on BTHSA). Left: PM 2.5 . Right: O 3 . Models trained without contextual examples show no difference between noise and quality examples. The model trained with k=5 approaches and slightly surpasses the baseline for PM 2.5 at 100 quality examples. However, when given noise, they show clear degradation, confirming that these models attend to example content.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Table 8
Figure 8: Training dynamics: multi-operator vs. single-operator comparison at Δt=24 h. Left: PM 2.5 . Right: O 3 . Top: BTHSA. Bottom: YRD. ICON with operator diversity exhibits negligible overfitting while single-operator models overfit.
Figure 9: Training dynamics: single-operator ablation at Δt=24 h with varying k . Left: PM 2.5 . Right: O 3 . Top: BTHSA. Bottom: YRD. All settings exhibit overfitting, with k=0 achieving the best validation performance across all single-operator settings.
Figure 10: Example cardinality generalization on YRD for simple operators. Left: PM 2.5 . Right: O 3 . Top: Δt=1 h. Bottom: Δt=4 h. Classical single-operator learning achieves lower RMSE for these simple operators.
Figure 11: Example cardinality generalization on YRD for complex operators. Left: PM 2.5 . Right: O 3 . Top: Δt=12 h. Bottom: Δt=24 h. ICON with operator diversity surpasses the baseline with sufficient examples.
Neural operators approximate mappings between function spaces, but often generalize poorly to other operators and usually require fine-tuning or retraining. In-Context Operator Networks (ICON) addresses this issue by prompting the model with numerical context so that the model learns specific operators from prompts and adapt to different operators without fine-tuning. However, ICON may still fail to generalize to out-of-distribution (OOD) operator tasks. Inpired by the success of harness engineering of Large Language models (LLMs), we introduce Chain of Operators (CHOP), a framework that harness a frozen ICON to OOD operator tasks without updating its parameters. Specifically, CHOP constructs a chain of operators consisting of explicit elementary transformations and the frozen ICON. Experiments on a scalar conservation law and a mean-field control problem show that CHOP reduces relative inference error over direct ICON evaluation, while each operator in the chain remains interpretable and in closed form. A chain constructed on one PDE family further generalizes to a different family, indicating shared mechanisms across harness systems.
Minghui Yang, Ling Guo, Liu Yang
Department of Mathematics, Shanghai Normal University, Shanghai, China · Department of Mathematics, National University of Singapore, Singapore
We investigate the generalization capabilities of In-Context Operator Networks (ICONs), a new class of operator networks that build on the principles of in-context learning, for higher-order partial differential equations. We extend previous work by expanding the type and scope of differential equations handled by the foundation model. We demonstrate that while processing complex inputs requires some new computational methods, the underlying machine learning techniques are largely consistent with simpler cases. Our implementation shows that although point-wise accuracy degrades for higher-order problems like the heat equation, the model retains qualitative accuracy in capturing solution dynamics and overall behavior. This demonstrates the model's ability to extrapolate fundamental solution characteristics to problems outside its training regime.
Jamie Mahowald, Tan Bui-Thanh
Department of Mathematics, University of Texas at Austin, Austin, Texas · Department of Aerospace Engineering and Engineering Mechanics, Oden Institute for Computational Engineering and Sciences, University of Texas at Austin, Texas
Neural operators learn mappings between function spaces, but are typically developed with dense input-output training fields and fully observed inputs at inference. Many scientific problems require instead predicting solution fields from sparse, irregular, or partial observations under uncertainty. We introduce Neural Operator Processes (NOPs), a framework that unifies neural-process conditioning with neural-operator decoding to predict full output fields from limited context. NOPs condition on sparse joint input-output observations and support deterministic and probabilistic prediction within a shared encoder-decoder architecture. We study two conditioning strategies, convolutional pooled summaries and query-aligned attention, and analyze how their interaction with latent stochastic variables depends on PDE geometry. Across function regression and three PDE benchmarks, we find that sparse conditional operator learning is viable and can match dense-grid behavior in several regimes, that preserving local context-query geometry is essential in non-periodic settings but less so in spectrally smooth periodic regimes, and that uncertainty-aware operator learning succeeds when latent conditioning complements rather than overwrites the local geometric pathway. These results provide a basis for probabilistic operator learning under partial observations and help bridge operator learning and probabilistic meta-learning in function space.