Covariance-Aware Transformers for Quadratic Programming and Decision Making

Kutay Tire; Yufan Zhang; Ege Onur Taga; Samet Oymak

cs.LGFeb 16, 2026

Covariance-Aware Transformers for Quadratic Programming and Decision Making

Authors: Kutay Tire, Yufan Zhang, Ege Onur Taga, Samet Oymak

Organizations: The University of Texas at Austin · University of Michigan

Abstract

We explore the use of transformers for solving quadratic programs and how this capability benefits decision-making problems that involve covariance matrices. We first show that the linear attention mechanism can provably solve unconstrained QPs by tokenizing the matrix variables (e.g.~ $A$ of the objective $\frac{1}{2}x^\top Ax+b^\top x$ ) row-by-row and emulating gradient descent iterations. Furthermore, by incorporating MLPs, a transformer block can solve (i) $\ell_1$ -penalized QPs by emulating iterative soft-thresholding and (ii) $\ell_1$ -constrained QPs when equipped with an additional feedback loop. Our theory motivates us to introduce Time2Decide: a generic method that enhances a time series foundation model (TSFM) by explicitly feeding the covariance matrix between the variates. We empirically find that Time2Decide uniformly outperforms the base TSFM model for the classical portfolio optimization problem that admits an $\ell_1$ -constrained QP formulation. Remarkably, Time2Decide also outperforms the classical "Predict-then-Optimize (PtO)" procedure, where we first forecast the returns and then explicitly solve a constrained QP, in suitable settings. Our results demonstrate that transformers benefit from explicit use of second-order statistics, and this can enable them to effectively solve complex decision-making problems, like portfolio construction, in one forward pass.

Figures & tables

Figure 1: Overview of the portfolio decision-making pipelines at decision time $t$ . (a) Predict-then-Optimize (PtO): historical multivariate returns are tokenized into patches and fed to TimePFN to predict next-step returns $\hat{r}$ . A QP solver then takes these predictions to compute the allocation $s_{t}$ under simplex and $\ell_{1}$ rebalancing constraints (b) Time2Decide: unifies both stages in a single forward pass. We first derive learned covariance tokens $(\sigma_{1},\ldots,\sigma_{m})$ from $\hat{\Sigma}$ by applying an MLP and prepend them to the return and allocation patch tokens. TimePFN with joint self-attention processes this full sequence to directly output both $\hat{r}$ and $\hat{s}$ . The final feasibility projection enforces the portfolio constraints to yield $s_{t}$ , enabling an end-to-end, covariance-aware decision without a separate optimization step.

Optimization Program	Architecture	Emulated Algorithm
Unconstrained QP	1 Linear Attention Block	Gradient Descent
Linearly Constrained QP	2 Sequential Attention Blocks	Arrow-Hurwicz Primal-Dual Method
$\ell_{1}$ -Regularized QP	1 Linear Attention Block + FFN	ISTA (Proximal Gradient Descent)
$\ell_{1}$ -Constrained QP	1 Linear Attention Block + FFN with Threshold Loop	Projected Gradient Descent

Table 1: Summary of transformer-based optimization algorithms.

Panel of Figure 2 — Figure 2: Convergence on a semilog- $y$ scale. Each panel compares a standard algorithm (orange) with our transformer-based construction (blue): (a) GD for the unconstrained case (U), (b) ISTA for the regularized case (R), and (c) PGD for the constrained case (C). The curves nearly overlap across layers (depth $=$ iterations), showing that the transformer mirrors the reference methods.


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.275	0.334	0.468	0.635	0.331	0.401	0.561	0.632
2	0.578	0.658	0.711	0.779	0.693	0.817	0.865	0.881
4	0.863	0.894	0.898	0.917	0.934	0.958	0.962	0.965

Table 2:

R^{2}

on the QP function-approximation task (

n{=}5

m{=}3

) across layers and attention heads.

Appendix figures & tables49 assets

Supplementary material from the paper’s appendix.

Appendix

	Gradient Descent (U)			ISTA (R)			Projected GD (C)
$n$	Class.	Transf.	Overhead	Class.	Transf.	Overhead	Class.	Transf.	Overhead
16	0.040	0.050	1.25 $\times$	0.052	0.070	1.35 $\times$	0.108	0.364	3.37 $\times$
32	0.286	0.310	1.08 $\times$	0.305	0.341	1.12 $\times$	0.416	0.696	1.67 $\times$
64	0.327	0.341	1.04 $\times$	0.358	0.389	1.09 $\times$	0.700	0.970	1.39 $\times$
128	0.595	0.642	1.08 $\times$	0.658	0.719	1.09 $\times$	1.452	1.653	1.14 $\times$

Appendix

Table 4: Per-step timing (ms) for unconstrained QP (U),

\ell_{1}

-regularized QP (R), and

\ell_{1}

-constrained QP (C). These problems depend only on the variable dimension

n

$n$	$m$	Classical (ms)	Transformer (ms)	Overhead
16	8	0.136	0.169	1.25 $\times$
16	16	0.178	0.210	1.18 $\times$
32	8	0.543	0.580	1.07 $\times$
32	16	1.241	1.263	1.02 $\times$
32	32	1.316	1.336	1.02 $\times$
64	16	1.339	1.388	1.04 $\times$

Appendix

Table 5: Per-step timing (ms) for linearly constrained QP (LC) using Arrow–Hurwicz iterations. This problem depends on both the variable dimension

n

and the number of constraints

m


Problem Dimensions		Model Architecture
$n$	$m$	LinearTransformer	SoftmaxTransformer	MLP	LSTM
5	3	0.9738	0.9354	0.6715	0.7045
7	3	0.9600	0.8914	0.5271	0.4734
7	6	0.9355	0.8713	0.5071	0.4880
10	3	0.9054	0.8423	0.3566	0.2189

Appendix

Table 7:

R^{2}

Comparison: Transformer architectures vs. MLP and LSTM baselines.


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.226	0.253	0.315	0.472	0.239	0.299	0.377	0.478
2	0.485	0.544	0.482	0.485	0.652	0.755	0.784	0.811
4	0.782	0.829	0.844	0.814	0.928	0.948	0.942	0.937

Appendix

Table 8:

R^{2}

on the QP Function-Approximation Task (

n{=}7

m{=}3


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.260	0.286	0.335	0.484	0.285	0.329	0.394	0.488
2	0.452	0.462	0.511	0.535	0.721	0.729	0.749	0.766
4	0.751	0.807	0.797	0.763	0.897	0.920	0.925	0.915

Appendix

Table 9:

R^{2}

on the QP Function-Approximation Task (

n{=}7

m{=}6


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.186	0.195	0.213	0.242	0.191	0.221	0.261	0.298
2	0.336	0.361	0.357	0.357	0.645	0.674	0.669	0.656
4	0.666	0.657	0.670	0.661	0.867	0.893	0.896	0.879

Appendix

Table 10:

R^{2}

on the QP Function-Approximation Task (

n{=}10

m{=}3


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.207	0.215	0.230	0.258	0.212	0.242	0.270	0.320
2	0.341	0.342	0.363	0.364	0.554	0.665	0.667	0.641
4	0.603	0.663	0.651	0.642	0.816	0.868	0.852	0.856

Appendix

Table 11:

R^{2}

on the QP Function-Approximation Task (

n{=}10

m{=}6


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.247	0.256	0.273	0.309	0.251	0.279	0.317	0.371
2	0.382	0.397	0.505	0.485	0.645	0.683	0.679	0.667
4	0.626	0.659	0.755	0.746	0.850	0.880	0.892	0.882

Appendix

Table 12:

R^{2}

on the QP Function-Approximation Task (

n{=}10

m{=}9


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.198	0.208	0.213	0.226	0.210	0.219	0.236	0.259
2	0.306	0.325	0.338	0.333	0.608	0.636	0.639	0.628
4	0.636	0.706	0.739	0.753	0.861	0.895	0.889	0.876

Appendix

Table 13:

R^{2}

on the QP Function-Approximation Task (

n{=}15

m{=}10


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.150	0.156	0.155	0.158	0.161	0.158	0.157	0.155
2	0.208	0.201	0.199	0.196	0.462	0.561	0.540	0.537
4	0.351	0.360	0.370	0.297	0.821	0.828	0.754	0.793

Appendix

Table 14:

R^{2}

on the QP Function-Approximation Task (

n{=}20

m{=}8


	SoftmaxTransformer				LinearTransformer
Number of Layers	Number of Heads				Number of Heads
	1 gy	2 gy	4 gy	8 gy	1 gy	2 gy	4 gy	8 gy
1	0.211	0.220	0.222	0.229	0.218	0.226	0.224	0.228
2	0.305	0.342	0.332	0.342	0.616	0.637	0.634	0.532
4	0.599	0.671	0.720	0.732	0.887	0.877	0.849	0.868

Appendix

Table 15:

R^{2}

on the QP Function-Approximation Task (

n{=}20

m{=}16

Panel of Figure 3 — Figure 3: Error distributions for the best LinearTransformer configuration ( $n=10,m=3$ , layers=8, heads=2).

Panel of Figure 4 — Figure 4: Error distributions for the best SoftmaxTransformer configuration ( $n=10,m=3$ , layers=16, heads=8).

Panel of Figure 5 — Figure 5: Error distributions for the best LinearTransformer configuration ( $n=10,m=6$ , layers=8, heads=1).

Panel of Figure 6 — Figure 6: Error distributions for the best SoftmaxTransformer configuration ( $n=10,m=6$ , layers=16, heads=8).

Panel of Figure 7 — Figure 7: Error distributions for the best LinearTransformer configuration ( $n=10,m=9$ , layers=8, heads=2).

Panel of Figure 8 — Figure 8: Error distributions for the best SoftmaxTransformer configuration ( $n=10,m=9$ , layers=16, heads=8).


$\kappa$ Range	LinearTransformer		SoftmaxTransformer
$\kappa$ Range	R 2	Drop	R 2	Drop
No $\kappa$	0.973832	–	0.935439	–
1.2–2	0.971755	-0.21%	0.930962	-0.48%
2–5	0.968110	-0.59%	0.924842	-1.13%
5–10	0.960139	-1.41%	0.909774	-2.74%

Appendix

Table 16: R 2 Performance under distribution shift in condition number

\kappa

Panel of Figure 9 — Figure 9: Empirical verification of predict-then-optimize (PtO) suboptimality compared to end-to-end learning. (a) The optimal shrinkage factor $\rho=\tau^{2}/(\tau^{2}+s^{2})$ increases toward 1 as sample size $n$ grows; the shaded region shows the shrinkage effect that E2E exploits while PtO uses $\rho=1$ . (b) Regret for both methods; E2E consistently achieves lower regret than PtO across all sample sizes. (c) Decision error $|w-w_{\mathrm{oracle}}|$ as a function of observed signal strength $|\hat{\delta}|$ ; PtO error grows linearly with signal strength while E2E remains stable due to shrinkage. Parameters: $\tau^{2}=0.01$ , $\sigma^{2}=0.5$ , $\lambda=0.2$ .

Model	Layers	Heads	Hidden	R²
LinearTransformer	4	2	16	0.9935
LinearTransformer	4	4	16	0.9892

Appendix

Table 18: LinearTransformer covariance learning performance (R²) with input projection layers.

Model	Layers	Heads	R²
BasicLinearTransformer	4	4	0.2827
BasicLinearTransformer	8	4	0.2473
BasicLinearTransformer	4	2	0.2234
BasicLinearTransformer	8	2	0.1953

Appendix

Table 19: BasicLinearTransformer covariance learning performance (R²) without input projection layers.

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0425	0.0425	0.0425	0.0425	0.0425	0.0425	0.0425	0.0425	0.0425
Pretrained	0.0447	0.0447	0.0447	0.0447	0.0447	0.0447	0.0447	0.0447	0.0447
SFT	0.0408	0.0462	0.0559	0.0566	0.0538	0.0626	0.0610	0.0626	0.0624
Predict-then-Optimize	0.0245	0.0245	0.0245	0.0245	0.0245	0.0245	0.0245	0.0245	0.0245
Time2Decide	0.0635	0.0618	0.0521	0.0526	0.0482	0.0477	0.0439	0.0367	0.0346

Appendix

Table 20: MSE results for

\gamma=0.5

across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0425	0.0425	0.0425	0.0425	0.0425	0.0425	0.0425
Pretrained	0.0447	0.0447	0.0447	0.0447	0.0447	0.0447	0.0447
SFT	0.0622	0.0621	0.0627	0.0621	0.0610	0.0618	0.0614
Predict-then-Optimize	0.0245	0.0245	0.0245	0.0245	0.0245	0.0245	0.0245
Time2Decide	0.0340	0.0339	0.0326	0.0325	0.0325	0.0323	0.0323

Appendix

Table 21: MSE results for

\gamma=0.5

across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0444	0.0444	0.0444	0.0444	0.0444	0.0444	0.0444	0.0444	0.0444
Pretrained	0.0458	0.0458	0.0458	0.0458	0.0458	0.0458	0.0458	0.0458	0.0458
SFT	0.0411	0.0499	0.0554	0.0548	0.0527	0.0635	0.0643	0.0648	0.0639
Predict-then-Optimize	0.0294	0.0294	0.0294	0.0294	0.0294	0.0294	0.0294	0.0294	0.0294
Time2Decide	0.0646	0.0607	0.0501	0.0495	0.0510	0.0476	0.0417	0.0378	0.0359

Appendix

Table 22: MSE results for

\gamma=0.75

across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0444	0.0444	0.0444	0.0444	0.0444	0.0444	0.0444
Pretrained	0.0458	0.0458	0.0458	0.0458	0.0458	0.0458	0.0458
SFT	0.0637	0.0632	0.0635	0.0634	0.0633	0.0626	0.0624
Predict-then-Optimize	0.0294	0.0294	0.0294	0.0294	0.0294	0.0294	0.0294
Time2Decide	0.0352	0.0351	0.0350	0.0348	0.0348	0.0347	0.0346

Appendix

Table 23: MSE results for

\gamma=0.75

across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479
Pretrained	0.0500	0.0500	0.0500	0.0500	0.0500	0.0500	0.0500	0.0500	0.0500
SFT	0.0413	0.0504	0.0542	0.0553	0.0528	0.0681	0.0688	0.0690	0.0680
Predict-then-Optimize	0.0360	0.0360	0.0360	0.0360	0.0360	0.0360	0.0360	0.0360	0.0360
Time2Decide	0.0629	0.0561	0.0504	0.0566	0.0503	0.0478	0.0436	0.0414	0.0387

Appendix

Table 24: MSE results for

\gamma=1.0

across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479
Pretrained	0.0500	0.0500	0.0500	0.0500	0.0500	0.0500	0.0500
SFT	0.0683	0.0681	0.0669	0.0680	0.0675	0.0672	0.0673
Predict-then-Optimize	0.0360	0.0360	0.0360	0.0360	0.0360	0.0360	0.0360
Time2Decide	0.0386	0.0391	0.0387	0.0387	0.0388	0.0387	0.0387

Appendix

Table 25: MSE results for

\gamma=1.0

across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485
Pretrained	0.0514	0.0514	0.0514	0.0514	0.0514	0.0514	0.0514	0.0514	0.0514
SFT	0.0446	0.0560	0.0550	0.0523	0.0523	0.0683	0.0678	0.0685	0.0672
Predict-then-Optimize	0.0376	0.0376	0.0376	0.0376	0.0376	0.0376	0.0376	0.0376	0.0376
Time2Decide	0.0639	0.0501	0.0489	0.0498	0.0486	0.0452	0.0416	0.0400	0.0390

Appendix

Table 26: MSE results for

\gamma=1.25

across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485
Pretrained	0.0514	0.0514	0.0514	0.0514	0.0514	0.0514	0.0514
SFT	0.0667	0.0681	0.0678	0.0675	0.0679	0.0678	0.0676
Predict-then-Optimize	0.0376	0.0376	0.0376	0.0376	0.0376	0.0376	0.0376
Time2Decide	0.0391	0.0393	0.0394	0.0395	0.0395	0.0395	0.0395

Appendix

Table 27: MSE results for

\gamma=1.25

across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0505	0.0505	0.0505	0.0505	0.0505	0.0505	0.0505	0.0505	0.0505
Pretrained	0.0523	0.0523	0.0523	0.0523	0.0523	0.0523	0.0523	0.0523	0.0523
SFT	0.0445	0.0557	0.0529	0.0532	0.0689	0.0698	0.0693	0.0692	0.0698
Predict-then-Optimize	0.0413	0.0413	0.0413	0.0413	0.0413	0.0413	0.0413	0.0413	0.0413
Time2Decide	0.0574	0.0490	0.0472	0.0457	0.0461	0.0435	0.0423	0.0414	0.0410

Appendix

Table 28: MSE results for

\gamma=1.5

across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0505	0.0505	0.0505	0.0505	0.0505	0.0505	0.0505
Pretrained	0.0523	0.0523	0.0523	0.0523	0.0523	0.0523	0.0523
SFT	0.0683	0.0686	0.0690	0.0679	0.0677	0.0680	0.0681
Predict-then-Optimize	0.0413	0.0413	0.0413	0.0413	0.0413	0.0413	0.0413
Time2Decide	0.0412	0.0415	0.0418	0.0420	0.0420	0.0420	0.0419

Appendix

Table 29: MSE results for

\gamma=1.5

across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0539	0.0539	0.0539	0.0539	0.0539	0.0539	0.0539	0.0539	0.0539
Pretrained	0.0567	0.0567	0.0567	0.0567	0.0567	0.0567	0.0567	0.0567	0.0567
SFT	0.0583	0.0515	0.0544	0.0541	0.0732	0.0724	0.0723	0.0727	0.0721
Predict-then-Optimize	0.0472	0.0472	0.0472	0.0472	0.0472	0.0472	0.0472	0.0472	0.0472
Time2Decide	0.0626	0.0538	0.0486	0.0474	0.0472	0.0457	0.0448	0.0445	0.0442

Appendix

Table 30: MSE results for

\gamma=1.75

across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0539	0.0539	0.0539	0.0539	0.0539	0.0539	0.0539
Pretrained	0.0567	0.0567	0.0567	0.0567	0.0567	0.0567	0.0567
SFT	0.0711	0.0716	0.0718	0.0720	0.0721	0.0725	0.0726
Predict-then-Optimize	0.0472	0.0472	0.0472	0.0472	0.0472	0.0472	0.0472
Time2Decide	0.0444	0.0448	0.0452	0.0453	0.0454	0.0455	0.0455

Appendix

Table 31: MSE results for

\gamma=1.75

across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0586	0.0586	0.0586	0.0586	0.0586	0.0586	0.0586	0.0586	0.0586
Pretrained	0.0604	0.0604	0.0604	0.0604	0.0604	0.0604	0.0604	0.0604	0.0604
SFT	0.0620	0.0620	0.0567	0.0565	0.0769	0.0754	0.0771	0.0770	0.0771
Predict-then-Optimize	0.0553	0.0553	0.0553	0.0553	0.0553	0.0553	0.0553	0.0553	0.0553
Time2Decide	0.0618	0.0575	0.0524	0.0526	0.0506	0.0499	0.0490	0.0488	0.0488

Appendix

Table 32: MSE results for

\gamma=2.0

across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0586	0.0586	0.0586	0.0586	0.0586	0.0586	0.0586
Pretrained	0.0604	0.0604	0.0604	0.0604	0.0604	0.0604	0.0604
SFT	0.0766	0.0768	0.0769	0.0767	0.0763	0.0772	0.0771
Predict-then-Optimize	0.0553	0.0553	0.0553	0.0553	0.0553	0.0553	0.0553
Time2Decide	0.0491	0.0503	0.0502	0.0506	0.0506	0.0505	0.0507

Appendix

Table 33: MSE results for

\gamma=2.0

across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0336	0.0336	0.0336	0.0336	0.0336	0.0336	0.0336	0.0336	0.0336
Pretrained	0.0409	0.0409	0.0409	0.0409	0.0409	0.0409	0.0409	0.0409	0.0409
SFT	0.0358	0.0329	0.0324	0.0354	0.0373	0.0395	0.0383	0.0431	0.0433
Predict-then-Optimize	0.0224	0.0224	0.0224	0.0224	0.0224	0.0224	0.0224	0.0224	0.0224
Time2Decide	0.0423	0.0232	0.0353	0.0377	0.0421	0.0371	0.0596	0.0666	0.0594

Appendix

Table 34: MSE results for

\gamma=0.5

with noisy data across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0336	0.0336	0.0336	0.0336	0.0336	0.0336	0.0336
Pretrained	0.0409	0.0409	0.0409	0.0409	0.0409	0.0409	0.0409
SFT	0.0451	0.0465	0.0499	0.0506	0.0490	0.0471	0.0462
Predict-then-Optimize	0.0224	0.0224	0.0224	0.0224	0.0224	0.0224	0.0224
Time2Decide	0.0540	0.0507	0.0446	0.0409	0.0361	0.0332	0.0310

Appendix

Table 35: MSE results for

\gamma=0.5

with noisy data across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0356	0.0356	0.0356	0.0356	0.0356	0.0356	0.0356	0.0356	0.0356
Pretrained	0.0428	0.0428	0.0428	0.0428	0.0428	0.0428	0.0428	0.0428	0.0428
SFT	0.0349	0.0347	0.0357	0.0393	0.0467	0.0402	0.0449	0.0419	0.0400
Predict-then-Optimize	0.0300	0.0300	0.0300	0.0300	0.0300	0.0300	0.0300	0.0300	0.0300
Time2Decide	0.0302	0.0277	0.0245	0.0303	0.0343	0.0388	0.0400	0.0578	0.0549

Appendix

Table 36: MSE results for

\gamma=0.75

with noisy data across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0356	0.0356	0.0356	0.0356	0.0356	0.0356	0.0356
Pretrained	0.0428	0.0428	0.0428	0.0428	0.0428	0.0428	0.0428
SFT	0.0410	0.0429	0.0451	0.0467	0.0461	0.0465	0.0461
Predict-then-Optimize	0.0300	0.0300	0.0300	0.0300	0.0300	0.0300	0.0300
Time2Decide	0.0516	0.0461	0.0421	0.0385	0.0347	0.0330	0.0330

Appendix

Table 37: MSE results for

\gamma=0.75

with noisy data across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0403	0.0403	0.0403	0.0403	0.0403	0.0403	0.0403	0.0403	0.0403
Pretrained	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485
SFT	0.0422	0.0408	0.0426	0.0458	0.0440	0.0500	0.0485	0.0432	0.0427
Predict-then-Optimize	0.0407	0.0407	0.0407	0.0407	0.0407	0.0407	0.0407	0.0407	0.0407
Time2Decide	0.0413	0.0452	0.0390	0.0347	0.0371	0.0422	0.0392	0.0460	0.0487

Appendix

Table 38: MSE results for

\gamma=1.0

with noisy data across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0403	0.0403	0.0403	0.0403	0.0403	0.0403	0.0403
Pretrained	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485	0.0485
SFT	0.0419	0.0431	0.0433	0.0455	0.0459	0.0491	0.0492
Predict-then-Optimize	0.0407	0.0407	0.0407	0.0407	0.0407	0.0407	0.0407
Time2Decide	0.0485	0.0474	0.0450	0.0420	0.0400	0.0387	0.0373

Appendix

Table 39: MSE results for

\gamma=1.0

with noisy data across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0415	0.0415	0.0415	0.0415	0.0415	0.0415	0.0415	0.0415	0.0415
Pretrained	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479
SFT	0.0448	0.0432	0.0441	0.0460	0.0451	0.0468	0.0454	0.0428	0.0422
Predict-then-Optimize	0.0446	0.0446	0.0446	0.0446	0.0446	0.0446	0.0446	0.0446	0.0446
Time2Decide	0.0420	0.0467	0.0412	0.0377	0.0394	0.0372	0.0394	0.0332	0.0410

Appendix

Table 40: MSE results for

\gamma=1.25

with noisy data across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0415	0.0415	0.0415	0.0415	0.0415	0.0415	0.0415
Pretrained	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479	0.0479
SFT	0.0426	0.0420	0.0437	0.0447	0.0460	0.0476	0.0489
Predict-then-Optimize	0.0446	0.0446	0.0446	0.0446	0.0446	0.0446	0.0446
Time2Decide	0.0408	0.0423	0.0410	0.0412	0.0398	0.0391	0.0381

Appendix

Table 41: MSE results for

\gamma=1.25

with noisy data across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0449	0.0449	0.0449	0.0449	0.0449	0.0449	0.0449	0.0449	0.0449
Pretrained	0.0529	0.0529	0.0529	0.0529	0.0529	0.0529	0.0529	0.0529	0.0529
SFT	0.0454	0.0449	0.0474	0.0480	0.0453	0.0489	0.0471	0.0449	0.0445
Predict-then-Optimize	0.0522	0.0522	0.0522	0.0522	0.0522	0.0522	0.0522	0.0522	0.0522
Time2Decide	0.0550	0.0510	0.0421	0.0408	0.0414	0.0382	0.0344	0.0353	0.0384

Appendix

Table 42: MSE results for

\gamma=1.5

with noisy data across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0449	0.0449	0.0449	0.0449	0.0449	0.0449	0.0449
Pretrained	0.0529	0.0529	0.0529	0.0529	0.0529	0.0529	0.0529
SFT	0.0441	0.0434	0.0446	0.0454	0.0482	0.0523	0.0503
Predict-then-Optimize	0.0522	0.0522	0.0522	0.0522	0.0522	0.0522	0.0522
Time2Decide	0.0391	0.0400	0.0408	0.0404	0.0409	0.0418	0.0411

Appendix

Table 43: MSE results for

\gamma=1.5

with noisy data across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0506	0.0506	0.0506	0.0506	0.0506	0.0506	0.0506	0.0506	0.0506
Pretrained	0.0582	0.0582	0.0582	0.0582	0.0582	0.0582	0.0582	0.0582	0.0582
SFT	0.0549	0.0546	0.0518	0.0515	0.0545	0.0541	0.0529	0.0506	0.0503
Predict-then-Optimize	0.0633	0.0633	0.0633	0.0633	0.0633	0.0633	0.0633	0.0633	0.0633
Time2Decide	0.0408	0.0436	0.0444	0.0431	0.0394	0.0404	0.0402	0.0396	0.0400

Appendix

Table 44: MSE results for

\gamma=1.75

with noisy data across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0506	0.0506	0.0506	0.0506	0.0506	0.0506	0.0506
Pretrained	0.0582	0.0582	0.0582	0.0582	0.0582	0.0582	0.0582
SFT	0.0496	0.0574	0.0575	0.0574	0.0564	0.0576	0.0580
Predict-then-Optimize	0.0633	0.0633	0.0633	0.0633	0.0633	0.0633	0.0633
Time2Decide	0.0414	0.0430	0.0437	0.0437	0.0459	0.0468	0.0473

Appendix

Table 45: MSE results for

\gamma=1.75

with noisy data across different return loss weights (Part 2).

Strategy	$\lambda_{r}$ =0	$\lambda_{r}$ =10	$\lambda_{r}$ =50	$\lambda_{r}$ =100	$\lambda_{r}$ =200	$\lambda_{r}$ =300	$\lambda_{r}$ =500	$\lambda_{r}$ =1000	$\lambda_{r}$ =3000
Uniform	0.0585	0.0585	0.0585	0.0585	0.0585	0.0585	0.0585	0.0585	0.0585
Pretrained	0.0664	0.0664	0.0664	0.0664	0.0664	0.0664	0.0664	0.0664	0.0664
SFT	0.0614	0.0614	0.0578	0.0591	0.0624	0.0620	0.0606	0.0603	0.0562
Predict-then-Optimize	0.0781	0.0781	0.0781	0.0781	0.0781	0.0781	0.0781	0.0781	0.0781
Time2Decide	0.0484	0.0494	0.0490	0.0506	0.0478	0.0485	0.0487	0.0481	0.0476

Appendix

Table 46: MSE results for

\gamma=2.0

with noisy data across different return loss weights (Part 1).

Strategy	$\lambda_{r}$ =5000	$\lambda_{r}$ =10000	$\lambda_{r}$ =20000	$\lambda_{r}$ =40000	$\lambda_{r}$ =80000	$\lambda_{r}$ =160000	$\lambda_{r}$ =320000
Uniform	0.0585	0.0585	0.0585	0.0585	0.0585	0.0585	0.0585
Pretrained	0.0664	0.0664	0.0664	0.0664	0.0664	0.0664	0.0664
SFT	0.0661	0.0653	0.0643	0.0647	0.0652	0.0652	0.0653
Predict-then-Optimize	0.0781	0.0781	0.0781	0.0781	0.0781	0.0781	0.0781
Time2Decide	0.0481	0.0487	0.0503	0.0516	0.0539	0.0557	0.0557

Appendix

Table 47: MSE results for

\gamma=2.0

with noisy data across different return loss weights (Part 2).

Quadratic Programming

Quadratic programming optimizes quadratic objectives under constraints, supporting applications from robotics and control to finance and machine learning.

See all 19 papers

An Optimal Control Approach To Transformer Training
Mar 10, 2026Kağan Akman, Naci Saldi, Serdar YükselTransformer Transformer Training Dynamics
Department of Mathematics, Bilkent University, Ankara, 06800, Turkey · Department of Mathematics and Statistics, Queen’s University, Kingston K7L 3N6, ON, Canada
Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making
Jul 21, 2026Yuyang Shen, Shan Dai, Daimin ChenNon-Stationary RL Sequential Decision Making
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data · University of International Business and Economics
Decision-focused Sparse Tangent Portfolio Optimization
Jul 1, 2026Haeun Jeon, Seunghoon Choi, Hyunglip Bae +2Decision-Focused Learning Portfolio Optimization
Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · Department of Big Data Convergence, and Bigdata Research Lab, Chonnam National University, Gwangju, Republic of Korea · Department of Industrial Engineering, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea

Mar 10, 2026J/K move · Enter open · S save

An Optimal Control Approach To Transformer Training

Kağan Akman, Naci Saldi, Serdar Yüksel

Department of Mathematics, Bilkent University, Ankara, 06800, Turkey · Department of Mathematics and Statistics, Queen’s University, Kingston K7L 3N6, ON, Canada

Transformer Transformer Training Dynamics Optimal Control

Open paper AskPDF arXiv

In this paper, we develop a rigorous optimal control-theoretic approach to Transformer training that respects key structural constraints such as (i) realized-input-independence during execution, (ii) the ensemble control nature of the problem, and (iii) positional dependence. We model the Transformer architecture as a discrete-time controlled particle system with shared actions, exhibiting noise-free controlled McKean--Vlasov dynamics, and we show that lifting the dynamics to probability measures produces a fully-observed Markov decision process (MDP). Positional encodings are incorporated into the state space to preserve the sequence order under lifting. Using the dynamic programming principle, we establish the existence of globally optimal policies under mild assumptions of compactness. We further prove that closed-loop policies in the lifted problem is equivalent to an initial-distribution dependent open-loop policy, which are realized-input-independent and compatible with standard Transformer training. To train a Transformer, we propose a triply quantized training procedure for the lifted MDP by quantizing the state space, the space of probability measures, and the action space, and show that any optimal policy for the triply quantized model is near-optimal for the original training problem. Finally, we establish stability and empirical consistency properties of the lifted model by showing that the value function is continuous with respect to the perturbations of the initial empirical measures and convergence of policies as the data size increases. This approach provides a globally optimal and robust theoretical alternative to gradient-based training without requiring smoothness or convexity.

Abstract

Quadratic Programming

Explore similar work

An Optimal Control Approach To Transformer Training

Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making

Decision-focused Sparse Tangent Portfolio Optimization