Checkable NTK Positivity and Finite-Width Gradient Descent for Scalar- and Vector-Valued PINNs with Strong-Form, Weak-Form, and Nonlocal Linear Constraints

Zifan Lyu

cs.LGOct 3, 2026

Checkable NTK Positivity and Finite-Width Gradient Descent for Scalar- and Vector-Valued PINNs with Strong-Form, Weak-Form, and Nonlocal Linear Constraints

Authors: Zifan Lyu

Organizations: ETH Zürich

Abstract

We give checkable positive-definiteness criteria for the limiting neural tangent kernel (NTK) and high-probability finite-width gradient-descent guarantees for scalar- or vector-valued physics-informed neural networks (PINNs) with linear constraints. The constraints may be strong-form differential rows of any fixed finite order, including coupled systems, with any linear initial or boundary conditions; weak-form residual and boundary functionals; or finite-measure nonlocal observations such as integral, nonlocal-diffusion, and Dirac-data rows, all in any dimension. For each class, positive definiteness of the limiting NTK is equivalent to a rank condition on a finite coefficient, functional, or moment matrix of the fixed design: a certificate computed before training that detects structural zero modes. The model hypotheses are those of an ordinary two-layer PINN, a smooth nonpolynomial activation with bounded symmetric initialization, and are met by standard choices such as $\tanh$ with uniform initialization. Given a certificate, explicit width and step-size conditions ensure, with high probability, that the empirical NTK retains at least half the limiting gap and that full-batch gradient descent decreases the training loss geometrically at every iteration. Controlled experiments check the certificates and the finite-width mechanism.

Figures & tables

Class	Rows and representative PDEs	$\ker\mathbf{K}_{(0)}^{\infty}$ under “pd”	Result; certificate
Strong-form pointwise	Fixed finite-order linear differential rows, scalar or coupled: Poisson and variable-coefficient elliptic, transport, heat, wave, biharmonic, Stokes, linear elasticity, and linear initial or boundary rows	Sitewise relations among the loss-scaled coefficient vectors	Proposition 3.1 ; Section E.3.1
Weak-form functional	Fixed bounded residual and boundary functionals on one trial space: fixed-test variational elliptic or parabolic systems, tested traces, and the discontinuous-coefficient weak Poisson problem of Corollary D.12	Joint relations among the functionals in $\mathcal{X}^{*}$	Proposition 3.2 ; Section E.3.2
Finite-measure nonlocal	Finite signed-measure tuples on compact $D$ : averages and moments, bounded-kernel integral and nonlocal-diffusion rows, Fredholm or Volterra rows, Dirac data, and atomic quadrature	Joint relations among the loss-scaled measure tuples	Proposition 3.3 ; Section E.3.3

Table 1: The three observation classes, each a fixed finite family of linear rows under Assumption 2.5 . In every class the “reg” clause gives existence of

\mathbf{K}_{(0)}^{\infty}

and

\mathbf{K}_{(0)}\to\mathbf{K}_{(0)}^{\infty}

; the “pd” clause identifies

\ker\mathbf{K}_{(0)}^{\infty}

with the listed relations, so row independence certifies

\mathbf{K}_{(0)}^{\infty}\succ 0

through the output block; and that certificate, conditional centering, and Equation 9 give geometric decay of the empirical training loss, not of the PDE solution error ( Corollary 4.3 ).

Figure 1: (a) Pooled medians over the 110 runs (eleven PDE designs, ten seeds) at each width of the initial relative kernel error against the randomized quasi-Monte Carlo (QMC) population reference

\widehat{\mathbf{K}}^{\infty}

, the maximum relative kernel drift, and the endpoint discrepancy from the frozen-kernel trajectory ( Table 9 ); the gray curve is the fraction of runs with

\lambda_{\min}(\mathbf{K}_{(k)})\geq\lambda_{\min}(\widehat{\mathbf{K}}^{\infty})/2

at every saved checkpoint. (b) Five-seed median final-to-initial loss ratio at

d_{1}=16{,}384

for the contrast target of each two-row family at row mixing

\varepsilon\in\{1,0.1,0.01\}

(markers), against the measured initial condition number

\kappa_{0}=\lambda_{\max}(\mathbf{K}_{(0)})/\lambda_{\min}(\mathbf{K}_{(0)})

; five medians at

\varepsilon=1

lie below the

10^{-16}

display floor.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Work and result	Setting and assumptions	Main conclusion	Additional scope treated here
Panel A: kernel limits, positivity, rank, and spectrum
Wang et al. (2022) , limiting PINN NTK	The block kernel identity holds for general fully connected PINNs, but the infinite-width theorems treat one hidden layer with standard-Gaussian initialization, the one-dimensional Poisson model problem with boundary-value and $u_{xx}$ residual rows, and smooth activations with bounded derivatives; freezing further assumes uniformly bounded parameters and time-integrated residual bounds along GF.	For that model problem, proves the deterministic initialization limit of the joint block DNTK and its freezing along GF as the width grows; residual decay is then read off along eigendirections of the PSD limiting kernel, and the kernel-regression limit assumes an invertible limiting matrix. Exact rank criteria and finite-width discrete-GD guarantees are outside its scope.	Two equal rows give a singular joint Gram even when the ordinary input NTK is PD. Proposition 3.1 identifies the null direction, Corollary H.4 gives GD convergence for equal targets and convergence to the loss floor for unequal ones, and Lemma H.3 identifies that floor; see Example B.4 .
Carvalho et al. (2025) , ordinary-NTK PD	Sequential infinite-width fully connected network with nondegenerate Gaussian initialization and distinct raw inputs. With bias, the activation is continuous, almost everywhere differentiable, and nonpolynomial, with the integrability that keeps the Gaussian kernel expectations finite; without bias, sites must also be nonzero and pairwise nonproportional.	Proves strict PD for ordinary value-observation NTKs under its distinct-input conditions. Differential, weak, and nonlocal observation rows are outside its scope.	Our result treats operator-transformed and co-located rows through their coefficient rank and uses compact initialization to ensure the required moments are finite on compact input and parameter sets.
Gao et al. (2023) , population PD and finite-width GD	Biased scalar two-layer cubic-ReLU network; Gaussian hidden and Rademacher output initialization; both layers trained. The joint heat-residual, Dirichlet, and initial sites are pairwise distinct, and the boundary weight is positive.	Establishes positivity of both population Gram blocks for the specified heat-equation design and gives GF and GD decay after a finite-width transfer.	A fourth-order strong row is identically zero for cubic ReLU away from its kink, whereas Proposition 3.1 under $\mathsf{P}_{\mathrm{pd}}$ covers every fixed finite order; see Example B.1 .
Gan et al. (2025) , operator-informed NTK	Bias-free scalar deep network, sequential widths, one fixed linear differential operator on a compact convex set, continuous coefficients, and a $C^{k+2}$ activation for an operator of order $k$ , with polynomially bounded derivatives. The training theorem assumes a positive limiting Gram and uniform gradient continuity.	Proves uniform convergence in probability of the transformed kernel and, under those two training assumptions, uniform closeness of the GF and limiting-kernel dynamics; it does not characterize rank for arbitrary joint constraint rows.	Duplicate compatible rows have zero full gap but a positive one-dimensional compression. Joint residual–boundary families, weak rows, and nonlocal rows also lie outside the single-operator theorem.
Jin and Wu (2026a) , qualitative DNTK PD	Shallow or deep fully connected networks with standard-Gaussian parameters and positive bias scale; pairwise-distinct physical sites; one nonzero evaluated scalar differential row per site; smooth nonpolynomial or RePU activation with order-matched regularity.	Theorem 2 establishes infinite-width covariance and DNTK positivity under its distinct-site assumptions. Exact nullspaces for arbitrary repeated-site and coupled-output rows are outside the stated setting.	Co-located value/derivative rows can be independent, and coupled-output rows can be dependent. The sitewise matrices $\mathbf{A}_{\mathbf{x}}$ used with Proposition 3.1 give both the exact PD criterion and the nullspace; see Examples B.2 and B.3 .

Appendix

Table 2: Comparison of direct theorem-level results. PD and PSD mean positive definite and positive semidefinite; GF, GD, IGD, and NGD denote gradient flow, gradient descent, implicit GD, and natural GD; SGD denotes stochastic GD. The final column identifies additional constraint structures treated in this paper.

Study	Instances	Runs	Grid
Rank/conditioning controls	600 kernels	1,080	six families, five row designs, widths $\{16,256,4{,}096,16{,}384\}$ , five paired seeds
PDE training diagnostics	11 reference problems	770	11 PDEs, widths $\{4,16,64,256,1{,}024,4{,}096,16{,}384\}$ , ten seeds, 3,000 GD steps
Initialization sweep	7,700 kernels	7,700	11 PDEs, the same seven widths, 100 independent seed banks
Joint-nullspace audit	72 kernels	36	three designs, four perturbation sizes, widths $\{32,128\}$ , three seeds, 5,000 GD steps
Per-iterate audit	30 kernels	30	one two-row design, widths $\{256,1{,}024\}$ , three row mixings, five seeds, 3,000 GD steps

Appendix

Table 3: Experimental matrix. Independent row designs use two compatible targets with a shared initial kernel, so optimization-run counts exceed distinct-kernel counts.

Problem	$N$	$d_{2}$	PDE and sampled data
Poisson 1D	7	1	$-u_{xx}=f$ ; five interior and two endpoint Dirichlet rows.
Variable elliptic 2D	13	1	$-\nabla\!\cdot(a\nabla u)+0.35u=f$ with $a(x,y)=2+0.3x+0.2y$ ; five interior rows and eight Robin rows $\partial_{n}u+\beta u=g$ , where $\beta=0.75+0.08x-0.05y$ .
Heat 1D	17	1	$u_{t}-0.35u_{xx}=f$ ; six interior, five initial, and six endpoint Dirichlet rows.
Transport 1D	15	1	$u_{t}+c(t,x)u_{x}+0.25u=f$ with $c(t,x)=0.65+0.12x+0.08t$ ; six interior, five initial, and four inflow rows at $x=-1$ .
Wave 1D	22	1	$u_{tt}-0.8^{2}u_{xx}=f$ ; six interior rows, five co-located initial displacement/velocity pairs, and six endpoint Dirichlet rows.
Biharmonic 2D	21	1	$\Delta^{2}u=f$ ; five interior rows and eight co-located clamped pairs $u=g_{0}$ , $\partial_{n}u=g_{1}$ .

Appendix

Table 4: Eleven PDE benchmarks in the 770-run training study. Here

N

counts scalar constraints after splitting vector equations into rows, and

d_{2}

is the network output dimension. Applying the displayed value or derivative operator to each manufactured solution yields the corresponding data.

Observation class	Included problems and rows	Runs	Loss decreased
Strong-form local	Seven scalar or vector strong-form systems: point evaluations and derivatives through order four, including the biharmonic and Stokes systems	490	490/490
Weak variational	Weak Poisson with four fixed test functions and endpoint traces	70	70/70
Integral/nonlocal	Nonlocal diffusion, an integro-differential equation, and transformed Caputo-time rows	210	210/210
All classes	Eleven fixed finite constraint systems	770	770/770

Appendix

Table 5: Finite-horizon GD coverage by observation class. “Loss decreased” means only

\mathcal{L}_{3{,}000}<\mathcal{L}_{0}

; it neither asserts monotonicity at unsaved iterates nor verifies the theorem’s sufficient width and step-size conditions. Each problem contributes seven widths and ten initialization seeds. Weak and integral/nonlocal entries refer to the stored quadrature functionals.

Family	Independent base functionals $(\ell_{1},\ell_{2})$
Pointwise Poisson	$(-u^{\prime\prime}(0.07),u(0.07))$ .
Vector Stokes	$(-0.2\Delta v_{1}+p_{x},-0.2\Delta v_{2}+p_{y})$ at $(x,y)=(-0.66,-0.41)$ .
Weak Poisson	The $n=1$ and $n=2$ rows $\int_{-1}^{1}u^{\prime}v_{n}^{\prime}\,dx$ under the 16-node Gauss–Legendre rule.
Nonlocal diffusion	The displayed nonlocal residuals at anchors $x=-0.5$ and $x=0.5$ under the 18-node rule.
Weak boundary–residual	The Dirichlet value $u(-1)$ and the $n=1$ weak-Poisson row.
Nonlocal boundary–residual	The Dirichlet value $u(-1)$ and the nonlocal residual at $x=-0.5$ .

Appendix

Table 6: Independent base functionals for the six two-row controls.

Rows	Kernels	GD runs	Gram status (pred./obs.)	GD outcome
Independent	480	960	nonsingular/nonsingular in 480 of 480	Output-block certificate passed in 480 of 480; loss below its initial value in 959 of 960
Exact duplicate	120	120	singular/singular in 120 of 120	loss reduced $>100\times$ in 120 of 120

Appendix

Table 7: Structural and observed finite-width Gram status. The output-block entry reports the single design-side rank certificate; it is not a second hidden-block test. The 960 qualifying runs use 480 independent kernels and two compatible targets. The Gram-status and certificate tallies are values of the registered campaign; the tracked aggregate stores the GD-outcome counts.

Figure 2: Exact structural nullspace control in three PDE settings. For each setting the three loss-scaled prediction rows are

A

B

, and

A+B

, so every prediction

\mathbf{v}_{P}(\theta)\in\mathbb{R}^{3}

satisfies

\mathbf{z}^{\top}\mathbf{v}_{P}(\theta)=0

for

\mathbf{z}=(1,1,-1)^{\top}/\sqrt{3}

. Define

\mathcal{N}=\linspan\{\mathbf{z}\}

\mathcal{U}=\mathcal{N}^{\perp}

, and the training error

\mathbf{e}_{k}=\mathbf{v}_{P}(\theta_{k})-\mathbf{v}_{T}

; the plotted loss is

\mathcal{L}_{k}=\|\mathbf{e}_{k}\|_{2}^{2}

. A compatible target has

\Pi_{\mathcal{N}}\mathbf{e}_{k}=0

. Its paired incompatible target is displaced by

0.25\mathbf{z}

, hence

\|\Pi_{\mathcal{N}}\mathbf{e}_{k}\|_{2}^{2}=0.25^{2}=0.0625

at every step, while both targets induce the same parameter updates and therefore share the same active loss

\|\Pi_{\mathcal{U}}\mathbf{e}_{k}\|_{2}^{2}

. The 18 paired settings comprise a co-located vector PDE, a weak-residual–boundary PDE, and a nonlocal-residual–boundary PDE, with widths

d_{1}\in\{32,128\}

and three seeds. Full-batch GD uses

\eta=0.2/\lambda_{\max}(\mathbf{K}_{(0)})

for

5{,}000

steps. Curves are medians across the 18 settings; the blue band is the interquartile range. Compatible nullspace loss is exactly zero and therefore is stated rather than drawn on the logarithmic axis. Active losses below

10^{-30}

are placed at the display boundary. At all 19 saved checkpoints, the paired total-loss offset agrees with

0.0625

within

4.44\times 10^{-16}

. The weak and nonlocal rows are the fixed stored quadrature functionals, so this is a discrete-loss diagnostic rather than a continuum-error result.

	$\varepsilon=1$		$\varepsilon=0.1$		$\varepsilon=0.01$
Family	$\rho_{0}$	$\mathcal{L}_{T}/\mathcal{L}_{0}$	$\rho_{0}$	$\mathcal{L}_{T}/\mathcal{L}_{0}$	$\rho_{0}$	$\mathcal{L}_{T}/\mathcal{L}_{0}$
Pointwise Poisson	$8.778\times 10^{-2}$	$8.03\times 10^{-31}$	$2.416\times 10^{-3}$	$4.17\times 10^{-4}$	$2.667\times 10^{-5}$	$1.73\times 10^{-1}$
Vector Stokes	$2.037\times 10^{-1}$	$5.22\times 10^{-32}$	$2.665\times 10^{-3}$	$1.10\times 10^{-3}$	$2.637\times 10^{-5}$	$8.00\times 10^{-1}$
Weak Poisson	$5.007\times 10^{-2}$	$6.55\times 10^{-29}$	$5.949\times 10^{-4}$	$5.93\times 10^{-2}$	$5.956\times 10^{-6}$	$3.41\times 10^{-1}$
Nonlocal diffusion	$3.871\times 10^{-3}$	$6.36\times 10^{-30}$	$2.359\times 10^{-5}$	$5.93\times 10^{-1}$	$1.974\times 10^{-7}$	$9.33\times 10^{-1}$
Weak boundary–residual	$2.703\times 10^{-1}$	$0$	$3.301\times 10^{-3}$	$1.04\times 10^{-4}$	$3.195\times 10^{-5}$	$3.70\times 10^{-1}$

Appendix

Table 8: Conditioning and finite-horizon loss at

d_{1}=16{,}384

. Entries are five-seed medians after

T=3{,}000

GD steps, with

\mathcal{L}_{T}/\mathcal{L}_{0}

for the compatible contrast target; every entry is taken from the tracked aggregate that also generates Figure 1 (b). Here

\rho_{0}

is the normalized initial minimum eigenvalue defined above. Every row design is independent and every contrast target is compatible. The weak boundary–residual entry at

\varepsilon=1

is exactly zero in float64.

$d_{1}$	Initial kernel error	Maximum kernel drift	Final frozen discrepancy	Gap retained
16	$2.23\times 10^{-1}$	$6.11\times 10^{-1}$	$9.08\times 10^{-1}$	17/110
256	$5.46\times 10^{-2}$	$4.84\times 10^{-2}$	$2.74\times 10^{-1}$	110/110
$4{,}096$	$1.56\times 10^{-2}$	$3.33\times 10^{-3}$	$2.25\times 10^{-2}$	110/110
$16{,}384$	$8.09\times 10^{-3}$	$8.62\times 10^{-4}$	$6.91\times 10^{-3}$	110/110

Appendix

Table 9: Width diagnostics pooled over 110 PDE runs per width, transcribed from the registered 770-run campaign; the tracked aggregate stores this transcription with its content hash. All three quantities are relative; initial-kernel error and maximum drift use operator norm. Frozen discrepancy compares the nonlinear residual with the trajectory obtained by holding the empirical Gram matrix at initialization. Gap retained counts runs that preserve at least half the numerical population-reference gap at every saved checkpoint, not at every GD iterate.

$d_{1}$	$\varepsilon$	Violations	Scored iterates	Worst ratio	Predicted
$256$	$1$	$0$	$2{,}845$	—	—
$256$	$0.1$	$0$	$15{,}000$	$0.998359$	$0.998371$
$256$	$0.01$	$0$	$15{,}000$	$0.999983$	$0.999983$
$1{,}024$	$1$	$0$	$3{,}015$	—	—
$1{,}024$	$0.1$	$0$	$15{,}000$	$0.998387$	$0.998389$
$1{,}024$	$0.01$	$0$	$15{,}000$	$0.999983$	$0.999983$

Appendix

Table 10: Per-iterate audit. Violations are steps with

\mathcal{L}(\theta_{(k+1)})>(1+10^{-12})\mathcal{L}(\theta_{(k)})

among scored iterates, pooled over five seeds. The last two columns are the largest observed per-step loss ratio and the frozen-kernel prediction

(1-2\eta\lambda_{\min}(\mathbf{K}_{(0)}))^{2}

, reported for the first seed.

Physics-Informed ML

Physics-informed machine learning combines data-driven models with physical laws to solve, predict, and infer scientific systems.

See all 836 papers

Jul 11, 2026

The Differential Neural Tangent Kernel and Its Positivity

The Neural Tangent Kernel (NTK) is one powerful tool for analyzing the training dynamics of neural networks in the over-parameterized regime. Recently, the theoretical framework has been extended to physics-informed neural networks (PINNs) for solving linear PDEs, one highly popular class of neural PDE solvers. In the analysis, the positivity of the associated NTK plays a fundamental role. However, establishing the positivity of the NTK for PINNs is highly challenging, due to the presence of multiple differential operators. In this work, we propose a new theoretical framework, called Differential Neural Tangent Kernel (DNTK), for analyzing PINNs through the lens of the NTK, and establish the positivity of the infinite width DNTK for both shallow and deep neural networks for a wide class of activation functions, including RePU and smooth but non-polynomial activations, for all linear differential operators. These theoretical results lay the foundation for the analysis of gradient type algorithms for training PINNs.

Bangti Jin, Longjun Wu

Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, P.R. China

Neural PDE Solvers Physics-Informed ML Neural Network Training Dynamics Neural Tangent Kernel

Ask PDF

May 30, 2026

Taming the Loss Landscape of PINNs with Noisy Feynman-Kac Supervision: Operator Preconditioning and Non-Asymptotic Error Bounds

Physics-Informed Neural Networks (PINNs) often train slowly or fail to converge on challenging partial differential equations (PDEs), a behavior recently linked to severely ill-conditioned loss landscapes inherited from the underlying differential operator. We study PINNs augmented with a pointwise data-fidelity term, added at a few points in the domain to the standard residual and boundary losses. We show that this supervision term acts as an operator-level preconditioner: for suitable weights, our comparison bounds guarantee a substantially smaller condition number than under the standard PINN loss, independently of how the pointwise labels are obtained. For a broad class of PDEs admitting a Feynman-Kac (FK) representation, we generate such labels by Monte Carlo averages of the FK functional, resulting in what we call ``FK-PINNs", and using the excess risk decomposition approach, we derive non-asymptotic

L^2(Ω)

-error bounds for FK-PINNs with

\tanh

activation trained by finitely many steps of gradient descent. Along the way, we establish pseudo-dimension bounds for first- and second-order derivatives of

\tanh

neural networks, which are of independent interest and, to the best of our knowledge, new. Numerical experiments on Poisson, Schrödinger, mean exit time, and committor problems corroborate the theory, and show that FK-PINNs can successfully solve PDEs for which standard PINNs exhibit severe failure modes.

Nathanael Tepakbong, Hanyu Hu, Chengyu Liu +1

Department of Data Science, City University of Hong Kong, 83 Tat Chee Ave, Hong Kong · Department of Mathematics, City University of Hong Kong, 83 Tat Chee Ave, Hong Kong

Neural Network Approximation Theory Physics-Informed ML

Ask PDF

Jun 10, 2026

Reliable Error Estimation for PINNs: Lower and Upper A Posteriori Bounds

Physics-informed neural networks (PINNs) combine machine learning with physical laws to solve differential equations. While existing results provide rigorous a posteriori upper bounds for PINN prediction errors, complete certification also requires complementary lower information in order to obtain computable two-sided error enclosures. In this paper, we derive computable a posteriori lower bounds for PINN errors in ordinary differential equations on suitable certified state-space domains under a localized strong monotonicity condition. We combine these estimates with complementary localized upper bounds under a one-sided Lipschitz condition, which is weaker than the global Lipschitz assumption used in previous work and can yield sharper upper error bands. The resulting bounds depend only on the neural-network approximation, the ODE residual, and local monotonicity and growth constants, and therefore do not require access to the exact solution. For linear time-invariant and time-varying systems, we further derive explicit formulas in terms of the minimal and maximal eigenvalues of the symmetric part of the system matrix. We also discuss the distinction between soft and hard enforcement of initial conditions in PINNs and explain why exact enforcement can make the scalar lower certificate uninformative. To recover nontrivial lower information in the linear setting, we use a signed-residual finite-probe certificate based on coordinate unit vectors. We also formulate a certificate-informed training strategy in which the propagated upper certificate is used as an auxiliary regularizer, while lower certificates remain post-training diagnostics. Altogether, the proposed framework provides rigorous and practically computable error certificates for PINN approximations of ODEs, while making explicit the domains and model classes for which the assumptions can be verified.

Ismail Huseynov, Arzu Ahmadova, Agamirza Bashirov

Physikalisch-Technische Bundesanstalt (PTB), Germany · Technical University of Berlin, Germany · Weierstrass Institute for Applied Analysis and Stochastics, Berlin, Germany +1

Dynamical Systems Neural Network Verification Ordinary Differential Equations Physics-Informed ML

Ask PDF

Abstract

Physics-Informed ML

Explore similar work

The Differential Neural Tangent Kernel and Its Positivity

Taming the Loss Landscape of PINNs with Noisy Feynman-Kac Supervision: Operator Preconditioning and Non-Asymptotic Error Bounds

Reliable Error Estimation for PINNs: Lower and Upper A Posteriori Bounds