cs.LGAug 24, 2026
SaveResponse Renormalization for Critical Deep Equilibrium Models
Organizations: Federal University of Bahia, Department of Geophysics Salvador, BA 40170-115, Brazil · Grupo de Estudos e Aplicação de Inteligência Artificial em Geofísica (GAIA) Federal University of Bahia, Salvador, BA 40170-115, Brazil
Abstract
Deep Equilibrium Models (DEQs) compute predictions from a hidden representation unchanged by the model update. Training through this equilibrium uses implicit differentiation and requires solving an adjoint system built from the residual Jacobian. If this Jacobian is nearly singular along loss-sensitive directions, small perturbations can be strongly amplified in the adjoint response, producing large, highly sensitive gradients that can make optimization unreliable. We introduce Response Renormalization, a backward-pass framework that lifts selected near-pole denominators while leaving unlifted response channels unchanged. Collective Mode Response Renormalization (CMR) applies this correction in a low-dimensional critical subspace, while Phi-adaptive CMR computes a bounded response mass from a positive susceptibility rule. We derive dense and matrix-free collective formulations, distinguish exact gradients of a modified frozen-anchor residual from backward-response surrogates, and extend the construction to Structured Implicit Layers and Vector Attractors (SILVA). Across 23 multiphysics families spanning partial differential equations, three-dimensional fields, operator maps, complex geometries, and particle systems, CMR and Phi-CMR yield test errors no more than five percent higher than those from models trained with exact implicit differentiation in more than 98% of static and 95% of transient family-seed comparisons. Solver-index experiments show convergence toward the static adjoint, while physical-time rollouts retain predictive fidelity under the evaluated conditions. These results demonstrate that selective response renormalization can control near-critical adjoint amplification without globally damping well-conditioned sensitivity. Therefore, the method can make parameter updates more reliable while preserving the useful gradient information needed for learning.
Figures & tables
Figure 1: Pole-response maps with (a) dense pole strength , (b) the Phi-CMR mass multiplier, (c) at critical rank one, and (d) the same difference at critical rank two over target gap and state dimension.
Figure 2: Selective renormalization of the near-pole response with (a) exact and filtered response amplification versus singular value, (b) original and renormalized denominators for stable and critical modes, (c) Phi-CMR mass versus pole pressure by state dimension and critical rank, and (d) retained response versus residual against the original adjoint equation.
| Block | Method | |||||
|---|---|---|---|---|---|---|
| Collective grid | Phi-CMR | 36 | 0.303 0.239 | 0.947 0.061 | 0.290 0.234 | 0.726 0.202 |
| Collective grid | Tikhonov | 36 | 0.231 0.250 | 0.815 0.191 | 0.206 0.245 | 0.812 0.208 |
| Source rotation | CMR | 48 | 0.471 0.283 | 0.999 0.001 | 0.471 0.283 | 0.530 0.283 |
| Source rotation | Tikhonov | 48 | 0.242 0.307 | 0.973 0.055 | 0.241 0.308 | 0.760 0.308 |
| Held-out gate | Delta-Phi | 20 | 0.440 0.351 | 0.999 0.002 | 0.440 0.351 | 0.560 0.351 |
| Held-out gate | Phi-CMR | 20 | 0.432 0.350 | 0.997 0.005 | 0.432 0.350 | 0.569 0.350 |
Table 1: Directional dense-adjoint evidence. Entries are mean standard deviation over configurations. is the norm ratio, is directional cosine, is signed projection onto the exact adjoint, and is relative adjoint error.
Figure 3: Darcy field bridge with (a) target field, (b) Phi-CMR absolute error, (c) Tikhonov-minus-Phi-CMR absolute-error difference, (d) source-weighted pole mode, (e) coefficient field with solution contours, (f) Delta-Phi absolute error with pole contours.
| Method | One-step error | Ratio | Time | Horizon-8 error | Ratio |
|---|---|---|---|---|---|
| Implicit | 1.000 | 50/50 | 1.00 | 1.000 | 45/45 |
| Phantom | 0.988 | 48/50 | 0.65 | 0.997 | 43/45 |
| Neumann | 1.015 | 43/50 | 0.58 | 1.007 | 42/45 |
| JFB | 1.011 | 43/50 | 0.52 | 0.988 | 42/45 |
| SHINE | 1.060 | 35/50 | 0.55 | 1.052 | 31/45 |
| Tikhonov | 1.011 | 47/50 | 2.49 | 1.016 | 43/45 |
Table 2: All native SILVA baselines over ten one-step families and nine dynamic families. Errors and time are normalized by exact implicit differentiation.
Appendix figures & tables47 assets
Supplementary material from the paper’s appendix.
Appendix
| Native dataset | GB | Cross-benchmark dataset | GB |
| PDEBench Darcy 2D (five regimes) | 6.554 | PDEArena Navier–Stokes 2D | 0.826 |
| PDEBench Advection 1D | 8.233 | PDEArena Shallow Water 2D | 0.089 |
| PDEBench Burgers 1D | 8.233 | PDEArena Maxwell 3D | 1.887 |
| PDEBench Reaction–Diffusion 1D | 4.137 | PDEArena Kuramoto–Sivashinsky 1D | 0.931 |
| PDEBench Diffusion–Sorption 1D | 4.217 | DynaBench Advection | 2.558 |
| PDEBench Diffusion–Reaction 2D | 13.244 | DynaBench Burgers | 5.090 |
Table 3: Storage footprints of the selected data subsets. Values are decimal gigabytes ( bytes) and exclude unused portions of the complete source collections.
Figure 4: Native SILVA transient analysis with (a) family-weighted free physical-time rollout error relative to models trained with exact implicit differentiation on nine dynamic datasets and five seeds, (b) finite causal-response error relative to the converged static adjoint, and (c) fixed-point state error relative to equilibrium, where panels (b,c) contain 45 dataset–seed models per method. Shading in panel (a) denotes error ratios at most 1.05.
| Diagnostic | Observed | Reference level |
|---|---|---|
| Maximum final forward residual | ||
| Maximum implicit/CMR/Phi adjoint residual | ||
| Structured-method realizations improving from initialization | 57.0% | Descriptive |
| Families with collective activation | 10/10 | 10/10 |
| Phi-CMR error ratio | 48/50 pairs | 40/50 pairs |
| Phi-CMR lower error than Tikhonov | 27/50 pairs | 25/50 pairs |
Table 4: SILVA convergence, fidelity, and activation diagnostics. Statistics use physical-family weighting, with the five Darcy regimes contributing one family-level unit.
Figure 5: Native SILVA multiphysics evaluation with (a) final inverse-standardized sample-relative Euclidean error normalized by exact implicit differentiation and paired-seed 95% intervals, (b) paired Phi-CMR-minus-Tikhonov differences, (c) the fraction of training updates with a lifted collective response, and (d) measured forward-solve plus backward wall time relative to implicit differentiation, with the Darcy regimes aggregated as one physical family. Shading in panel (a) denotes error ratios at most 1.05. The Darcy confidence interval in (a) extends beyond the displayed range.
| Symbol | Meaning | Symbol | Meaning |
|---|---|---|---|
| equilibrium Jacobian | response denominator | ||
| loss source | implicit adjoint | ||
| singular triplet of | critical cutoff | ||
| base and effective masses | critical source fraction | ||
| local and low-rank factors | collective denominator | ||
| original and lifted residuals | projection onto interval |
Table 5: Response-renormalization notation.
| Block | Unit | Fixed design |
|---|---|---|
| Controlled response | 36 collective, 48 source-rotation, 20 held-out Delta-Phi configurations | Dimensions 12–24, ranks 1–2, gaps 0.005–0.02, prescribed source fractions, double precision, triplet tolerance , and GMRES relative tolerance . |
| Native and cross-benchmark SILVA | 70 and 65 family/regime–seed model units, 1,080 method evaluations | Five seeds, eight backward modes, disjoint dataset-provided splits, identical forward models and ordered minibatches within every paired comparison. |
| Capacity extension | 50 exploratory aligned settings and 400 evaluations, plus 40 transient-aligned evaluations | Six high-error families, hidden widths 96 or 128, resolutions up to or , and the same eight backward modes. |
| Transient evaluation | 2,880 physical-time and 945 solver-time repeated evaluations | Free autoregressive rollouts without retraining and solver horizons , compared with converged equilibria and GMRES adjoints. |
Table 6: Evaluation blocks and statistical units.
| Block | Metric | A | B | A–B | |
|---|---|---|---|---|---|
| Phi-CMR vs Tikhonov | response-norm ratio | 36 | 0.303 | 0.231 | 0.073 |
| CMR vs Tikhonov, source sweep | response-norm ratio | 48 | 0.471 | 0.242 | 0.229 |
| High vs low source alignment | CMR gain over Tikhonov | 36 | 0.263 | 0.172 | 0.091 |
| Phi-CMR vs Tikhonov, Darcy | relative field error | 1 | 1.211 | 1.558 | -0.347 |
| Delta-Phi vs Phi-CMR | low-source norm ratio | 20 | 0.668 | 0.623 | 0.045 |
| Delta-Phi vs Phi-CMR | high-source suppression | 20 | 0.816 | 0.758 | 0.057 |
Table 7: Extended quantitative evidence. Negative field-error differences favor the first method.
Figure 6: Rank-two response with (a) pole strength and (b) the rank-two minus rank-one Phi-CMR advantage over Tikhonov.
Figure 7: Source-alignment experiment with (a) retention gain over Tikhonov and (b) response norm versus critical-mode suppression.
Figure 8: Delta-Phi sweep with (a) bounded mass, (b) strength utility, and (c) response norm versus critical-mode suppression.
Figure 9: Delta-Phi ablation with (a) gate activation, (b) mass correction, and (c) suppression gain.
| Source | Dataset | Physical setting | Representation |
|---|---|---|---|
| PDEArena | Navier–Stokes 2D | incompressible flow | multi-field grid |
| PDEArena | Shallow Water 2D | free-surface flow | multi-field grid |
| PDEArena | Maxwell 3D | electromagnetic waves | vector-field volume |
| PDEArena | Kuramoto–Sivashinsky 1D | chaotic dissipative dynamics | scalar grid |
| DynaBench | Advection | linear transport | structured grid |
| DynaBench | Burgers | nonlinear conservation law | structured grid |
Table 8: Cross-benchmark evaluation. Dynamic entries use one-step training and seven-step free physical-time rollouts, while Poisson–Gauss is evaluated as a static source-to-solution operator.
| Method | Static error | Ratio | Time | Horizon-7 error | Ratio |
|---|---|---|---|---|---|
| Implicit | 1.0000 | 65/65 | 1.00 | 1.0000 | 60/60 |
| Phantom | 0.9994 | 64/65 | 0.59 | 0.9969 | 60/60 |
| Neumann | 0.9975 | 64/65 | 0.54 | 0.9922 | 60/60 |
| JFB | 0.9982 | 63/65 | 0.48 | 0.9962 | 60/60 |
| SHINE | 0.9818 | 62/65 | 0.52 | 0.9922 | 55/60 |
| Tikhonov | 0.9989 | 65/65 | 2.48 | 0.9987 | 58/60 |
Table 9: All-method cross-benchmark results. Static values use 65 family–seed pairs, while horizon seven uses 60 pairs from the 12 dynamic datasets. Error and measured solve-plus-backward time are normalized by exact implicit differentiation.
Figure 10: Cross-benchmark evaluation of SILVA response renormalization with (a) final inverse-standardized sample-relative Euclidean error normalized by exact implicit differentiation, (b) paired Phi-CMR-minus-Tikhonov differences across seeds, (c) the fraction of training measurements with an active collective lift, and (d) measured equilibrium-solve plus backward time relative to implicit differentiation. Shading in panel (a) denotes error ratios at most 1.05.
Figure 11: Cross-benchmark transient evidence with (a) seven-step free physical-time rollouts for all eight backward methods on 12 dynamic datasets, (b) finite causal-response error relative to the static implicit adjoint, and (c) original DEQ state error relative to equilibrium, where panels (b,c) use implicit, CMR, and Phi-CMR models at solver horizons . Shading in panel (a) denotes error ratios at most 1.05.
| Family | Baseline | Capacity | Ratio | Paired delta 95% CI | Seeds improved |
|---|---|---|---|---|---|
| Darcy 2D | 1.1119 | 0.6483 | 0.583 | 5/5 | |
| Diffusion–Reaction 2D | 1.0170 | 0.9931 | 0.977 | 5/5 | |
| PDEArena Navier–Stokes | 0.6405 | 0.4496 | 0.702 | 5/5 | |
| PDEArena Shallow Water | 0.3629 | 0.3287 | 0.906 | 5/5 | |
| PDEArena Maxwell 3D | 0.9674 | 0.6244 | 0.645 | 5/5 | |
| LagrangeBench TGV | 0.2204 | 0.1252 | 0.573 | 5/5 |
Table 10: Phi-CMR inverse-standardized sample-relative Euclidean error in the forward-capacity extension, with capacity-to-baseline ratios and paired 95% intervals for the capacity-minus-baseline difference.
Figure 12: Held-out prediction geometry over five seeds with (a) representative-field relative error, (b) centered spatial correlation, and (c) prediction-to-target amplitude ratio, where points are individual seeds, short horizontal marks are means, and Diffusion–Reaction uses the separate transient task.
Figure 13: Higher-capacity Darcy 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Panels (a–e) use the median-error example, while panel (f) aggregates all five Darcy regimes.
Figure 14: Transient-aligned Diffusion–Reaction 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first physical channel for the separately evaluated task.
Figure 15: Higher-capacity PDEArena Navier–Stokes 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first vorticity/state channel.
Figure 16: Higher-capacity PDEArena Shallow Water 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first pressure/height channel.
Figure 17: Higher-capacity PDEArena Maxwell 3D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels show the central slice of the first electric-displacement component after vector-aware Conv3D reconstruction.
Figure 18: Higher-capacity LagrangeBench Taylor–Green vortex 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Panels (a–e) retain the native particle coordinates.
Figure 19: PDEArena Navier–Stokes 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first vorticity/state channel.
Figure 20: PDEArena Shallow Water 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first pressure/height channel.
Figure 21: PDEArena Maxwell 3D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels show the central slice of the first electric-displacement component.
Figure 22: PDEArena Kuramoto–Sivashinsky 1D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the scalar state.
Figure 23: DynaBench Advection with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the advected scalar.
Figure 24: DynaBench Burgers with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the conserved scalar.
Figure 25: DynaBench Gas Dynamics with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display density, while panel (f) aggregates all predicted variables.
Figure 26: DynaBench Kuramoto–Sivashinsky with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the scalar state for the deterministic example nearest the Phi-CMR median error (offset 3 of 16).
Figure 27: DynaBench Reaction–Diffusion with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first reaction–diffusion channel.
Figure 28: DynaBench Wave with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first wave-state channel.
Figure 29: PDEGym Poisson–Gauss with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Panels (a–e) show the source-to-solution operator example.
Figure 30: CFDBench cylinder geometry with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display velocity for a held-out geometry case.
Figure 31: LagrangeBench Taylor–Green vortex 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Panels (a–e) retain the native particle coordinates, and panel (d) is position error.
| Physical family | Mathematical setting | Displayed quantity |
|---|---|---|
| Advection 1D | linear transport | state |
| Burgers 1D | nonlinear conservation law | state |
| Reaction–Diffusion 1D | coupled reaction and diffusion | state |
| Diffusion–Sorption 1D | reactive transport | concentration |
| Diffusion–Reaction 2D | coupled parabolic system | first field |
| Shallow Water 2D | hyperbolic free-surface flow | height |
Table 11: Physical systems and displayed quantities in the native SILVA per-family analysis. Darcy is evaluated as an elliptic coefficient-to-solution map, while the remaining entries are one-step dynamical predictions.
Figure 32: Advection 1D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the scalar state.
Figure 33: Burgers 1D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the scalar state.
Figure 34: Reaction–Diffusion 1D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first state channel.
Figure 35: Diffusion–Sorption 1D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display concentration.
Figure 36: Diffusion–Reaction 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first physical channel.
Figure 37: Shallow Water 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display fluid height.
Figure 38: Compressible CFD 1D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display density, while panel (f) aggregates density, pressure, and velocity.
Figure 39: Incompressible Navier–Stokes 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display horizontal velocity.
Figure 40: The Well turbulent radiative layer 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Field panels display the first physical channel from the held-out split.
Figure 41: Darcy 2D with (a) input, (b) target, (c) Phi-CMR prediction, (d) absolute error, (e) Tikhonov-minus-Phi-CMR error, (f) five-seed all-method relative- , (g) original and effective collective denominators with active lifts, and (h) source-weighted singular response. Panels (a–e) use a fixed coefficient-to-solution example, while panel (f) aggregates five Darcy regimes.