Func-JEPA: Learning a Sobolev Space of Functions as Latent Vectors
Organizations: Independent Researcher
Abstract
Pretraining an encoder once and reusing it across tasks is a widespread approach in vision and language. However, neural symbolic mathematics still trains a dedicated sequence-to-sequence model for each task, so that no representation of a function is shared. In this work, we propose the Function Joint-Embedding Predictive Architecture (Func-JEPA, abbreviated f-JEPA): an encoder of mathematical functions, pretrained on their symbolic form, whose latent space imitates the Sobolev space H^1 and in which differentiation is a learned latent operator. The norm of a latent is computed exactly from the function, whereas its direction is learned to match Sobolev angles. The frozen encoder is then reused by lightweight modules for decoding, composition, integration, and regression from samples. The representation and four downstream tasks are evaluated within and outside the pretraining distribution. f-JEPA recovers exact formulas from noisy samples more often than the compared symbolic-regression methods, whereas computer algebra and task-specific models remain stronger on standard integration and differential-equation benchmarks.
Figures & tables
| Metric | f-JEPA | |
| Geometry | Correlation of with | 0.993 |
| Correlation of with | 0.998 | |
| Mean absolute cosine error | 0.017 | |
| Linearity | Scale error vs. ( / / ) | 0.012 / 0.014 / 0.050 |
| Combinations vs. | 0.071 | |
| Random sums vs. | 0.207 |
| Geometry | Property probes (NMSE) | Derivative probe | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Representation | Dim. | Value cos. | Dist. rank | kNN@10 | Trust. | NCR | Upward. | Avg. | Osc. | Top-5 | Align. |
| SNIP, symbolic encoder [ 34 ] | 512 | 0.002 | -0.090 | 0.080 | 0.570 | 0.680 | 0.716 | 0.805 | 0.084 | 0.338 | 0.936 |
| Lample and Charton encoder [ 28 ] | 1024 | 0.012 | 0.133 | 0.087 | 0.601 | 0.319 | 0.217 | 0.210 | 0.044 | 0.979 | 0.908 |
| f-JEPA | 256 | 0.946 | 1.000 | 0.900 | 0.994 | 0.235 | 0.389 | 0.583 | 0.390 | 0.929 | 0.997 |
| Method | |||||||
|---|---|---|---|---|---|---|---|
| Identity | 0.099 | 0.034 | 0.017 | 0.023 | 0.016 | 0.038 | 0.005 |
| Ridge regression | 0.032 | 0.021 | 0.041 | 0.019 | 0.096 | 0.196 | 0.118 |
| Linear part of only | 0.095 | 0.000 | 0.032 | 0.201 | 0.149 | 0.682 | 0.000 |
| Zero-shot | – | – | – | – | – | – | 0.444 |
| , direction only | 0.578 | 0.508 | 0.481 | 0.253 | 0.703 | 0.675 | 0.660 |
| 0.690 | 0.647 | 0.584 | 0.568 | 0.922 | 0.919 | 0.796 |
| Solve rate | Definite integral | |||
| Method | Test | Hard | Test | Hard |
| Latent shortlists, decoding (beam 16) and constant refitting | 0.773 | 0.167 | 0.957 | 0.900 |
| + exact combination of decoded pieces and pool functions | 0.787 | 0.200 | 0.977 | 0.900 |
| + splitting sums into terms | 0.813 | 0.200 | 0.977 | 0.900 |
| + residual peeling in latent space | 0.817 | 0.200 | 0.983 | 0.900 |
| + latent tabular integration by parts | 0.827 | 0.433 | 0.983 | 0.967 |
| Exact / within threshold | Classic (of 24) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | NMSE | Extrap. | Nodes | Rel. err. | |||||
| E2E | 0.00 / 0.02 | 0.00 / 0.00 | 0.00 / 0.00 | 0.12 | 6.2e-01 | 20 | 1.7e-01 | 0 / 0 | 0 / 0 |
| NeSymReS | 0.06 / 0.14 | 0.00 / 0.12 | 0.00 / 0.08 | 0.22 | 8.0e-01 | 12 | 2.7e-01 | 3 / 4 | 0 / 2 |
| SNIP | 0.04 / 0.58 | 0.00 / 0.30 | 0.00 / 0.10 | 0.74 | 4.0e-04 | 22 | 6.5e-03 | 0 / 10 | 0 / 2 |
| MMSR | 0.00 / 0.12 | 0.00 / 0.08 | 0.00 / 0.02 | 0.28 | 4.1e-01 | 15 | 3.2e-01 | 3 / 5 | 1 / 3 |
| PySR | 0.06 / 0.74 | 0.00 / 0.58 | 0.00 / 0.18 | 0.94 | 4.0e-07 | 22 | 5.2e-03 | 9 / 19 | 1 / 4 |
| Best learned vs. best network-free control: wins / ties / losses | 5 / 1 / 6 | 11 / 0 / 1 | 9 / 1 / 2 |
| Median ratio (control / learned) | 0.96 | 1.43 | 1.40 |
| Composed atoms vs. latent atoms only: wins / ties / losses | 7 / 5 / 0 | 10 / 1 / 1 | 10 / 2 / 0 |
| Median ratio (latent / composed) | 1.15 | 1.55 | 2.00 |
| Best learned vs. Padé approximant with coefficients: wins | 12 / 12 | 12 / 12 | 12 / 12 |
| Median error ratio (learned / Padé) | 0.037 | 0.011 | 0.004 |
| Linear ODEs | Logistic-type ODEs | |||||
|---|---|---|---|---|---|---|
| Method | ||||||
| Strong form (SINDy-like, no network) | 0.520 | 0.200 | 0.073 | 0.853 | 0.513 | 0.200 |
| Weak form (no network) | 0.673 | 0.233 | 0.053 | 0.473 | 0.380 | 0.127 |
| Samples formula exact derivatives | 0.487 | 0.440 | 0.440 | 0.633 | 0.647 | 0.600 |
| SINDy, PySINDy [ 17 ] † | 0.073 | 0.000 | 0.000 | 0.200 | 0.060 | 0.013 |
| Weak SINDy, PySINDy [ 35 ] † | 0.167 | 0.047 | 0.013 | 0.580 | 0.433 | 0.080 |
| Lample and Charton [ 28 ] | |||||
| Method | FWD | BWD | IBP | MIT Bee | Rubi [ 39 ] |
| Approximate antiderivatives (only f-JEPA produces them) | |||||
| f-JEPA, relative error of below | 0.700 | 0.530 | 0.760 | 0.500 | 0.365 |
| f-JEPA, within of the true antiderivative ( ) | 0.660 | 0.485 | 0.735 | 0.511 | 0.375 |
| Exact antiderivatives verified by SymPy | |||||
| f-JEPA | 0.250 | 0.240 | 0.195 | 0.359 | 0.165 |
| ODE1 | ODE2 | |||||
|---|---|---|---|---|---|---|
| Method | Verified | Res. | Res. | Verified | Res. | Res. |
| f-JEPA (numerical IVP samples-to-formula) | 0.185 | 0.240 | 0.305 | 0.160 | 0.220 | 0.280 |
| f-JEPA + fine-tuned decoder + ODE-aware search | 0.250 | 0.295 | 0.365 | 0.230 | 0.280 | 0.350 |
| Lample and Charton, rerun on these IVPs | 0.945 | 0.960 | 0.965 | 0.915 | 0.950 | 0.950 |
| SymPy dsolve (10 s) | 0.620 | 0.620 | 0.620 | 0.320 | 0.335 | 0.335 |
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Share | Family | Share | Family | Share |
|---|---|---|---|---|---|
| linear ODE solutions | 19.3 | Chebyshev series | 5.7 | logistic | 2.7 |
| polynomials of a component | 13.1 | exponential | 4.7 | damped oscillation | 2.7 |
| polynomial | 7.5 | Gaussian | 4.2 | type | 2.5 |
| Fourier sum | 6.4 | Lorentzian | 3.5 | 2.4 | |
| polynomial trigonometric | 5.9 | 3.5 | random expression trees | 4.2 | |
| harmonic | 5.3 | rational | 3.5 |
| Hyperparameter | Value |
|---|---|
| Architecture | BERT, 10 layers, hidden size 512, 8 heads, feed-forward 2176, |
| Parameters | 33M (encoder), 1.6M ( ) |
| Initialisation | truncated normal (std 0.02), residual outputs scaled by |
| Optimizer | AdamW, learning rate ( for ), 700 warm-up steps, cosine |
| Steps / batch size | 40,000 / 384 |
| Batch composition | 15% equivalence groups, 25% sweeps, plus 15% synthetic rows (half combinations, half scaled copies) |
| Benchmark | Kept | Scanned | Main rejection reasons (counts) |
|---|---|---|---|
| L&C integration FWD | 1000 | 2764 | not finite/real on [-pi, pi] (1096); outside norm range (283); operator outside vocabulary (212) |
| L&C integration BWD | 1000 | 8203 | not finite/real on [-pi, pi] (3463); too long (1278); operator outside vocabulary (1038) |
| L&C integration IBP | 1000 | 2098 | not finite/real on [-pi, pi] (580); outside norm range (283); operator outside vocabulary (143) |
| MIT Integration Bee | 92 | 322 | not finite/real on [-pi, pi] (163); outside norm range (35); pole / discontinuity in [-pi, pi] (18) |
| Rubi (indefinite) | 1000 | 58104 | not univariate in x (51687); not finite/real on [-pi, pi] (3570); outside norm range (923) |
| L&C ODE1 (as IVPs) | 530 | 6523 | not finite/real on [-pi, pi] (3317); operator outside vocabulary (964); pole / discontinuity in [-pi, pi] (517) |
| Metric | ||
|---|---|---|
| cosine correlation | 0.991 | 0.996 |
| inner-product correlation | 0.999 | 1.000 |
| effective rank of | 54.7 | 45.4 |
| scale error, | 0.015 | 0.012 |
| scale error, | 0.054 | 0.037 |
| linearity error (combinations) | 0.070 | 0.070 |
| Prediction | Top-1 | Rel. error | ||
|---|---|---|---|---|
| vs. | 0.820 | 0.070 | 0.317 | 0.671 |
| vs. | 0.458 | 0.267 | 0.075 | 0.264 |
| vs. | 0.501 | 0.232 | 0.178 | 0.339 |
| Integrator | 0.670 | 0.176 | 0.002 | 0.172 |
| 0.690 | 0.246 | 0.000 | 0.042 | |
| 0.647 | 0.275 | 0.000 | 0.008 |
| True function | Decoded function | Error |
|---|---|---|
| ✓ | ||
| ✓ | ||
| ✓ | ||
| 4.0e-01 | ||
| ✓ | ||
| ✓ |
| All inputs | Input not decoded | |||
|---|---|---|---|---|
| Operator | Learned | Symbolic path | Learned | Symbolic path |
| ( ) | 0.820 / 0.960 | 0.901 / 0.963 | 0.802 / 0.953 | 0.783 / 0.917 |
| 0.675 / 0.924 | 0.850 / 0.909 | 0.585 / 0.866 | 0.543 / 0.723 | |
| 0.640 / 0.889 | 0.849 / 0.905 | 0.528 / 0.792 | 0.502 / 0.686 | |
| 0.591 / 0.893 | 0.773 / 0.886 | 0.509 / 0.831 | 0.498 / 0.747 | |
| 0.573 / 0.958 | 0.836 / 0.928 | 0.509 / 0.949 | 0.637 / 0.841 | |
| Method | |||
|---|---|---|---|
| Nearest function in the training pool | 0.110 | 0.110 | 0.110 |
| Value encoder + decoding (beam 8) + constant refitting | 0.640 | 0.470 | 0.210 |
| + beam 16 and refitting of every candidate | 0.730 | 0.630 | 0.447 |
| + exact combination with polynomial atoms | 0.763 | 0.630 | 0.447 |
| + residual peeling | 0.800 | 0.630 | 0.447 |
| True latent instead of (upper bound) | 0.870 | – | – |
| Method | Exact | Exact | Exact | ||||||
|---|---|---|---|---|---|---|---|---|---|
| E2E transformer [ 23 ] | 0.00 | 0.00 | 0.39 | 0.00 | 0.00 | 0.44 | 0.00 | 0.00 | 0.37 |
| PySR [ 14 ] | 0.03 | 0.63 | 0.98 | 0.00 | 0.66 | 0.98 | 0.00 | 0.02 | 0.94 |
| f-JEPA | 0.77 | 0.93 | 0.97 | 0.67 | 0.86 | 0.98 | 0.55 | 0.57 | 0.99 |
| True latent instead of (upper bound) | 0.84 | 0.93 | 0.97 | – | – | – | – | – | – |
| Method | Exact | Exact | ||||
|---|---|---|---|---|---|---|
| E2E transformer [ 23 ] | 0.000 | 0.010 | 0.450 | 0.000 | 0.000 | 0.440 |
| PySR [ 14 ] | 0.170 | 0.680 | 0.890 | 0.000 | 0.010 | 0.880 |
| f-JEPA | 0.100 | 0.250 | 0.610 | 0.070 | 0.100 | 0.640 |
| ODE | Solutions | Residual | p@20 | p@20 (GD) | Exact | New |
|---|---|---|---|---|---|---|
| 47 | 0.081 | 0.85 | 0.70 | 8 | 0 | |
| 5 | 0.069 | 0.25 | 0.25 | 17 | 9 | |
| 39 | 0.009 | 0.40 | 0.45 | 6 | 3 | |
| 20 | 0.049 | 0.80 | 0.55 | 7 | 0 | |
| 5 | 0.013 | 0.20 | 0.25 | 1 | 0 |
| ODE | Solutions | Latent p@20 | Recall@1% | Verified top 20 | New |
|---|---|---|---|---|---|
| 244 | 0.00 | 0.89 | 20/20 | 2 | |
| 295 | 0.05 | 0.96 | 20/20 | 1 | |
| 7,522 | 1.00 | 0.91 | 20/20 | 2 | |
| 10,567 | 1.00 | 0.99 | 20/20 | 0 | |
| 7,418 | 0.00 | 0.86 | 20/20 | 2 | |
| 2,835 | 0.90 | 0.96 | 20/20 | 3 |
| ODE | Control | Latent atoms | + composed | Best learned | Padé |
|---|---|---|---|---|---|
| 1.4e-2 | 4.0e-2 | 1.7e-2 | 9.5e-3 | 1.0e0 | |
| 8.4e-3 | 2.6e-2 | 5.0e-3 | 5.0e-3 | 1.3e-1 | |
| 3.3e-2 | 4.3e-2 | 4.1e-2 | 2.4e-2 | 3.5e-1 | |
| 1.1e-2 | 3.4e-2 | 1.5e-2 | 7.7e-3 | 6.5e-1 | |
| 2.8e-3 | 3.8e-3 | 2.4e-3 | 2.4e-3 | 5.9e-3 | |
| 1.2e-2 | 1.7e-2 | 7.8e-3 | 5.4e-3 | 3.9e-1 |
| Method | ODEs | Median residual | Median IC error |
|---|---|---|---|
| Random atoms (control) | 12 | 1.5e-01 | 1.0e-03 |
| Value-space neighbours (control) | 12 | 7.2e-02 | 2.0e-04 |
| Value-space peeling (control) | 12 | 1.0e-01 | 3.2e-04 |
| Latent neighbours | 12 | 1.1e-01 | 2.4e-04 |
| Latent ODE shortlist | 11 | 1.0e-01 | 2.9e-04 |
| Latent peeling | 12 | 3.9e-02 | 9.4e-05 |
| Integrand | |||
|---|---|---|---|
| 7.5e-2 | 1.8e-2 | 2.9e-3 | |
| 6.7e-1 | 2.9e-1 | 1.1e-1 | |
| 3.9e-1 | 4.9e-2 | 1.9e-2 | |
| 1.4e-1 | 7.0e-3 | 1.2e-3 | |
| 1.1e-1 | 4.3e-2 | 1.2e-2 | |
| 1.2e-1 | 1.7e-2 | 6.4e-3 |
| Integrand | Antiderivative found by f-JEPA |
|---|---|
| Method | Solve rate | Definite integral | Not solved by SymPy |
| Latent shortlists, decoding and constant refitting | 0.101 | 0.544 | 0 |
| + exact combination, term splitting and residual peeling | 0.121 | 0.664 | 0 |
| + latent tabular integration by parts | 0.174 | 0.658 | 3 |
| + substitution and trigonometric reduction | 0.443 | 0.711 | 8 |
| Lample and Charton, FWD+BWD+IBP, beam 10 [ 28 ] | 0.993 | – | 42 |
| SymPy integrate (reference, 10 s limit) | 0.718 | – | – |
| Exact equation | Extrapolation NMSE | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | |||||||||||
| f-JEPA: samples formula exact derivatives | 0.80 | 0.87 | 0.89 | 0.67 | 0.78 | 0.67 | 0.15 | 0.07 | 0.07 | 2.4e-03 | 1.9e-03 |
| SINDy (PySINDy) | 0.96 | 0.93 | 0.48 | 0.80 | 0.67 | 0.20 | 0.07 | 0.02 | 0.00 | 1.0e-05 | 2.0e-01 |
| Weak SINDy (PySINDy) | 0.93 | 0.96 | 0.89 | 0.83 | 0.87 | 0.70 | 0.07 | 0.04 | 0.00 | 5.8e-06 | 5.7e-03 |
| Estimated derivatives + PySR | 0.98 | 0.54 | 0.35 | 0.98 | 0.52 | 0.17 | 0.00 | 0.00 | 0.00 | 2.2e-08 | 1.1e+01 |
| ODEFormer (pretrained) | 0.85 | 0.85 | 0.83 | 0.67 | 0.67 | 0.61 | 0.00 | 0.00 | 0.00 | 2.7e-03 | 2.3e-03 |
| ODE1 | ODE2 | |
|---|---|---|
| Median prefix length of the reference solution (pretraining: 18) | 19 | 19 |
| Reference beyond the 99th percentile of pretraining lengths | 0.025 | 0.000 |
| Reference with irrational constants | 0.355 | 0.245 |
| f-JEPA verified | 0.000 | 0.020 |
| f-JEPA numerically exact, not provable (rel. error ) | 0.394 | 0.408 |
| Reference with rational constants only | 0.645 | 0.755 |
| Component | Parameters | Steps | Time |
|---|---|---|---|
| Encoder and | 35.1M | 40k | 6.6 h (0.6 s/step) |
| Decoder | 17.7M | 105k | 3.5 h (0.11 s/step) |
| Composition bank (7 operators) | 52.9M | 40k | 1 h |
| Integrator | 7.6M | 60k | 13 min (0.013 s/step) |
| Value encoder (fine-tuning) | 7.4M | 20k | 39 min |
| Task | Method | Seconds per item | Parameters |
|---|---|---|---|
| Symbolic regression | f-JEPA (value encoder and decoder) | 0.44 | 25.1M |
| NeSymReS [ 7 ] | 1.6 | 26.4M | |
| E2E transformer [ 23 ] | 2.0 | 93.5M | |
| Operon [ 9 ] (20 s budget) | 20.0 | – | |
| PySR [ 14 ] (20 s budget) | 20.2 | – | |
| SNIP [ 34 ] | 23.1 | 166.7M |
Representation Learning
Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.
See all 865 papers