Meta-LSQ: A Closed-Form Sobolev Inner Solver for Few-Shot Meta-Learning
Organizations: Lawrence Berkeley National Laboratory
Abstract
Gradient-based meta-learners such as model-agnostic meta-learning (MAML) adapt to a new task with an inner loop of gradient descent that has to be differentiated through, and they meta-learn every network weight. Meta-LSQ instead fits each task with one regularised least-squares solve over a fixed random Fourier basis and meta-learns only three scalars, a bandwidth, a ridge and a derivative weight. Since the derivative of a sinusoid is again a sinusoid, the same solve can also take derivative observations, and the fitted model has exact closed-form derivatives of any order. On 10-shot sinusoid regression Meta-LSQ trains in 14 s and ends with a 5.5 times lower error than a first-order MAML baseline that trains 36 times longer, and a Gaussian-process control suggests that most of this gain comes from sharing one kernel across tasks. In simulated beam tuning at DESY's ARES facility with model derivatives it is slightly more accurate than an ALPaCA-style model and well ahead of gradient descent. Over DESY's full range of target beams, again in simulation and from measurements alone, its best setting has a median error of 7.3 m, against 14.4 for the published final beams of their reinforcement-learning agent, which has the lower median on their 300 trials of small targets. On real sensor-calibration data a learned prior mean puts it ahead of MAML on two of four targets, but an exact Gaussian process with the same change and an ALPaCA-style model stay more accurate on most targets, and it breaks down on non-smooth targets. On noiseless targets outside the span of the basis, in-sample criteria such as generalised cross-validation choose too small a ridge.
Figures & tables
| method | learn. | train s | ms/t | test MSE |
| MAML std. (8k) | 1 761 | 51 | ||
| MAML strong (10k) | 8 513 | 130 | ||
| ” (30k) | 399 | |||
| ” (60k) | 512 | |||
| MAML Sob. (8k) | 1 761 | 78 | ||
| ALPaCA-style (8k) | 1 721 | 12 | n/a |
| method | s | s | s |
|---|---|---|---|
| MLP-MAML | |||
| Meta-LSQ | |||
| Meta-LSQ-Sob. | |||
| floor |
| target | Meta-LSQ | GP | ALPaCA | MAML | lin. | quad. |
|---|---|---|---|---|---|---|
| CO | ||||||
| NO 2 | ||||||
| benzene ∗ | ||||||
| NO x |
| method | uses | median | mean | to |
|---|---|---|---|---|
| RL agent [ 24 ] | values | ( ) | ||
| BO [ 24 ] | values | – ( ) | ||
| ALPaCA-style | values | – ( ) | ||
| Meta-LSQ | values | – ( ) | ||
| Gradient descent | deriv. | – ( ) | ||
| ALPaCA-style | deriv. | ( ) |
| size (mm) | share | agent | BO | ALPaCA | Meta-LSQ |
|---|---|---|---|---|---|
| below | |||||
| – | |||||
| – | |||||
| – | |||||
| whole grid |
| MLP-MAML | freq. | |||||||
|---|---|---|---|---|---|---|---|---|
| family | mean | med | mean | med | mean | med | mean | med |
| sin | e-5 | e-3 | ||||||
| tanh | ||||||||
| triangle | ||||||||
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| floor | CO | NO 2 | benzene | NO x |
|---|---|---|---|---|
| / | / | / | / | |
| / | / | / | / | |
| / | / | / | / | |
| / | / | / | / | |
| / | / | / | / | |
| / | / | / | / |
| method | CO | NO 2 | benzene | NO x |
|---|---|---|---|---|
| Meta-LSQ, first design | e-3 e-3 | |||
| Meta-LSQ, prior mean and trend | e-4 e-4 | |||
| GP RBF, first design | e-3 e-3 | |||
| GP RBF, prior mean and trend | e-5 e-5 | |||
| ALPaCA-style, prior mean | e-4 e-4 | |||
| MLP-MAML | e-4 e-4 |
| ridge | median | p90 | p99 | max | top-1% |
|---|---|---|---|---|---|
| e5 | e6 | ||||
| e4 | |||||
| family | policy | mean | median | p99 |
|---|---|---|---|---|
| sin | tuned | e-5 | ||
| floor | e-3 | |||
| leave-out CV | e-5 | |||
| GCV | e-7 | |||
| tanh | tuned | |||
| floor |
Meta-Learning
Meta-learning trains models across tasks so they can adapt quickly to new problems with few examples, limited feedback, or no parameter updates.
See all 174 papers