In-context learning is a remarkable property of transformers and has recently received a lot of interest. In many studies of in-context learning, it has been shown that transformers are capable of implementing solvers for linear and non-linear regression problems, in which the most of them implement gradient descent algorithm. However, it is still unclear whether those implementations have actually been acquired through training. In this paper, we constructed a transformer with masked linear self-attention, which in-context learns the least squares estimate in a simple regression task. The point here is that the closed form (analytical) solution is obtained by using layer normalization rather than an approximate solution based on the gradient descent algorithm. Then, we showed an experimental example, in which, by taking the non-uniqueness of the transformer representation into account, our implementation for a closed form solution is used in the transformer trained with the l1 regularization when the target output is the least squares estimate.
Figures & tables
Figure 1: Structure of transformer
No.
Average test error
Correlation
1
0.000058
0.966911
2
0.000269
0.417908
3
0.000307
0.535359
4
0.000101
0.662808
5
0.000109
0.988581
6
0.000095
0.354986
Table 1: Average squared test error and correlation between reciprocals of standard deviation in layer normalization and variance of in-context input samples
Figure 2: Absolute output weight values
Figure 3: Examples of fitting curves after training
t(um∣Dm)
t0(um∣Dm)
t1(um∣Dm)
R2
0.999974
0.997697
0.957023
α0
-0.00205
-0.018546
0.018853
α1
-0.000005
0.000670
-0.001094
α2
-0.016581
0.374289
-0.387986
α3
1.000339
0.046484
0.951257
α4
1.000394
1.003497
-0.003356
Table 2: R2 and estimated coefficients in regression analysis of t(um∣Dm) , t0(um∣Dm) and t1(um∣Dm) .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
η
test error( T=1 )
test error( T=4 )
0.4
0.080785
0.000425
0.45
0.050393
0.000130
0.5
0.042642
0.000061
0.55
0.057532
0.000309
0.6
0.095063
0.002125
Appendix
Table 3: Average squared test error of gradient descent
In-context learning is a remarkable property of transformers and has recently received a lot of interest. In many studies of in-context learning, it has been shown that transformers are capable of implementing solver for linear and non-linear regression problems, in which the most of them implement gradient descent algorithm. However, it is still unclear whether those implementations have actually been acquired through training. In this paper, we construct a transformer with linear self-attention, which in-context learns the least squares estimate in a simple regression task. The point here is that the closed form (analytical) solution is approximately obtained by using layer normalization rather than an approximate solution based on gradient descent algorithm. Then, we show an experimental example, in which our implementation is mainly used in the transformer trained with l1 regularization when the target output is the least squares estimate.
Katsuyuki Hagiwara
Faculty of Education, Mie University · Kurima-Machiya-cho, Tsu, 514-8507, Japan
Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, we study ICL in the nonlinear regression setting. Through the interaction mechanism in attention, we explicitly construct transformer networks to realize nonlinear features, such as polynomial or spline bases, which span a wide class of functions. Based on this construction, we establish a framework to analyze end-to-end in-context nonlinear regression with the constructed features. Our theory provides finite-sample generalization error bounds in terms of context length and training set size. We numerically validate the theory on synthetic regression tasks.
Alexander Hsu, Zhaiming Shen, Wenjing Liao +1
Department of Mathematics, Purdue University · School of Mathematics, Georgia Institute of Technology
Recent work has demonstrated that transformers and linear attention models can perform in-context learning (ICL) on simple function classes, such as linear regression. In this paper, we empirically study how these two attention mechanisms differ in their ICL behavior on the canonical linear-regression task of Garg et al. We evaluate learning quality (MSE), convergence, and generalization behavior of each architecture. We also analyze how increasing model depth affects ICL performance. Our results illustrate both the similarities and limitations of linear attention relative to quadratic attention in this setting.