Jacobian Descent for Multi-Objective Optimization

Pierre Quinton; Valérian Rey

cs.LGJun 23, 2024

Jacobian Descent for Multi-Objective Optimization

Authors: Pierre Quinton, Valérian Rey

Organizations: Independent

Abstract

Many optimization problems require balancing multiple conflicting objectives. To generalize gradient descent, which is limited to single-objective optimization, we introduce Jacobian descent (JD) and its stochastic variants. This algorithm iteratively updates parameters using the Jacobian matrix of a vector-valued objective function, in which each row is the gradient of an individual objective. While several methods to combine gradients already exist in the literature, they generally struggle when the objectives conflict. In contrast, we propose projecting gradients to fully resolve conflict. We prove convergence to the Pareto front in the smooth convex case with this approach, a stronger guarantee than the usual convergence to a weakly Pareto stationary point. Our method also enables instance-wise risk minimization (IWRM), a novel learning paradigm in which the loss of each training example is considered a separate objective. Early experimentation on image classification shows promising results for IWRM. Lastly, we provide an efficient implementation of JD using the Gramian of the Jacobian matrix to drastically reduce memory requirements.

Figures & tables

Panel of Figure 1 — Figure 1: Aggregation of $J=[{\bm{g}}_{1}\;{\bm{g}}_{2}]^{\top}\in\mathbb{R}^{2\times 2}$ by four different aggregators. The dual cone of $\{{\bm{g}}_{1},{\bm{g}}_{2}\}$ is represented in green. (a) $\mathcal{A}_{\text{UPGrad}}$ projects ${\bm{g}}_{1}$ and ${\bm{g}}_{2}$ onto the dual cone and averages the results. (b) The mean $\mathcal{A}_{\text{Mean}}(J)=\frac{1}{2}({\bm{g}}_{1}+{\bm{g}}_{2})$ conflicts with ${\bm{g}}_{1}$ . $\mathcal{A}_{\text{DualProj}}$ projects this mean onto the dual cone, so it lies on its boundary. $\mathcal{A}_{\text{MGDA}}(J)$ is almost orthogonal to ${\bm{g}}_{2}$ because of its larger norm.

Ref.	Aggregator	Non- conflicting	Linear under scaling	Weighted
—	$\mathcal{A}_{\text{Mean}}$	✗	✓	✓
Désidéri (2012)	$\mathcal{A}_{\text{MGDA}}$	✓	✗	✓
Lopez-Paz & Ranzato (2017)	$\mathcal{A}_{\text{DualProj}}$	✓	✗	✓
Yu et al. (2020)	$\mathcal{A}_{\text{PCGrad}}$	✗	✓	✓
Chen et al. (2020)	$\mathcal{A}_{\text{GradDrop}}$	✗	✗	✗
Liu et al. (2021b)	$\mathcal{A}_{\text{IMTL-G}}$	✗	✗	✓

Table 1: Properties satisfied for any number of objectives. Proofs are provided in Appendix D .

Panel of Figure 2 — Figure 2: Optimization metrics obtained with IWRM with 1024 training examples and a batch size of 32, averaged over 8 independent runs. The shaded area around each curve shows the estimated standard error of the mean over the 8 runs. Curves are smoothed for readability. Best viewed in color.

Aggregator	SVHN	CIFAR-10	EuroSAT	MNIST	F-MNIST	K-MNIST
$\mathcal{A}_{\text{Mean}}$	69.6	47.7	68.5	96.3	80.7	76.9
$\mathcal{A}_{\text{UPGrad}}$	69.7	48.1	73.0	96.0	80.8	77.1
$\mathcal{A}_{\text{MGDA}}$	41.7	32.9	62.5	85.2	73.4	53.7
$\mathcal{A}_{\text{Aligned-MTL}}$	61.3	47.3	65.3	88.3	78.5	67.9
$\mathcal{A}_{\text{PCGrad}}$	49.1	48.1	59.3	85.9	69.3	54.7
$\mathcal{A}_{\text{DualProj}}$	69.3	47.2	72.7	96.1	80.8	76.4

Table 2: Test set accuracies (%) obtained by IWRM with different aggregators across datasets. The corresponding box plots are shown in Figure 12 .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Optimization trajectories of various aggregators when optimizing ${\bm{f}}:\begin{bmatrix}x_{1}&x_{2}\end{bmatrix}^{\top}\mapsto\begin{bmatrix}x_{1}^{2}&x_{2}^{2}\end{bmatrix}^{\top}$ with JD. Colored dots represent the initial points. The trajectories start in red and evolve towards yellow.

Panel of Figure 4 — Figure 4: Optimization trajectories of various aggregators when optimizing the convex quadratic function ${\bm{g}}$ of ( 14 ) with JD. Colored dots represent initial parameter values.

Panel of Figure 5 — Figure 5: Optimization trajectories of various aggregators when optimizing the convex quadratic function ${\bm{h}}$ of ( 17 ) with JD. Colored dots represent initial parameter values. The trajectories start in red and evolve towards yellow.

Panel of Figure 6 — Figure 6: SVHN results.

Panel of Figure 7 — Figure 7: CIFAR-10 results.

Panel of Figure 8 — Figure 8: EuroSAT results.

Panel of Figure 9 — Figure 9: MNIST results.

Panel of Figure 10 — Figure 10: Fashion-MNIST results.

Panel of Figure 11 — Figure 11: Kuzushiji-MNIST results.

Figure 12: Box plots of the test set accuracies (%) obtained by IWRM with different aggregators across datasets, over 8 independent runs.

Panel of Figure 13 — Figure 13: CIFAR-10 results with different batch sizes (BS). The number of epochs is always 20, so the number of iterations varies.

Panel of Figure 14 — Figure 14: Results with the Adam optimizer.

Conv2d (3 input channels, 16 output channels, 1 group), ELU

Conv2d (16 input channels, 32 output channels, 16 groups)

MaxPool2d (stride of 2

\times

2, kernel size of 2

\times

2), ELU

Conv2d (32 input channels, 32 output channels, 32 groups)

MaxPool2d (stride of 3

\times

3, kernel size of 3

\times

3), ELU , Flatten

Linear (512 input features, 64 output features), ELU

Linear (64 input features, 10 outputs)

Appendix

Table 3: Architecture used for SVHN

Conv2d (3 input channels, 32 output channels, 1 group), ELU

Conv2d (32 input channels, 64 output channels, 32 groups)

MaxPool2d (stride of 2

\times

2, kernel size of 2

\times

2), ELU

Conv2d (64 input channels, 64 output channels, 64 groups)

MaxPool2d (stride of 3

\times

3, kernel size of 3

\times

3), ELU , Flatten

Linear (1024 input features, 128 output features), ELU

Linear (128 input features, 10 outputs)

Appendix

Table 4: Architecture used for CIFAR-10

Conv2d (3 input channels, 32 output channels, 1 group)

MaxPool2d (stride of 2

\times

2, kernel size of 2

\times

2), ELU

Conv2d (32 input channels, 64 output channels, 32 groups)

MaxPool2d (stride of 2

\times

2, kernel size of 2

\times

2), ELU

Conv2d (64 input channels, 64 output channels, 64 groups)

MaxPool2d (stride of 3

\times

3, kernel size of 3

\times

3), ELU , Flatten

Linear (1024 input features, 128 output features), ELU

Appendix

Table 5: Architecture used for EuroSAT

Conv2d (1 input channel, 32 output channels, 1 group), ELU

Conv2d (32 input channels, 64 output channels, 1 group)

MaxPool2d (stride of 2

\times

2, kernel size of 2

\times

2), ELU

Conv2d (64 input channels, 64 output channels, 1 group)

MaxPool2d (stride of 3

\times

3, kernel size of 3

\times

3), ELU , Flatten

Linear (576 input features, 128 output features), ELU

Linear (128 input features, 10 outputs)

Appendix

Table 6: Architecture used for MNIST, Fashion-MNIST and Kuzushiji-MNIST

Dataset	Epochs	Iterations	GPU-Hours
SVHN	25	800	17
CIFAR-10	20	640	15
EuroSAT	30	960	32
MNIST	8	256	6
Fashion-MNIST	25	800	17
Kuzushiji-MNIST	10	320	8

Appendix

Table 7: Numbers of epochs, iterations, and GPU-hours for each dataset

Objective	Method	SVHN	CIFAR-10	EuroSAT	MNIST
ERM	SGD	32.0 MB	44.6 MB	72.6 MB	54.2 MB
IWRM	SSJD	315.4 MB	619.3 MB	1447.5 MB	786.0 MB
IWRM	SSJD (Gramian-based)	39.8 MB	75.3 MB	104.2 MB	65.6 MB

Appendix

Table 8: Peak GPU memory used during one epoch of training on the ERM objective with SGD, on the IWRM objective with SSJD, and on the IWRM objective with Gramian-based SSJD (as described in Section 5 ). The batch size is always 32.

Objective	Method	SVHN	CIFAR-10	EuroSAT	MNIST
ERM	SGD	0.83	0.44	0.76	0.43
IWRM	SSJD– $\mathcal{A}_{\text{UPGrad}}$	1.63	1.78	2.86	1.74
IWRM	SSJD– $\mathcal{A}_{\text{UPGrad}}$ (Gramian-based)	1.15	1.12	1.51	1.11
IWRM	SSJD– $\mathcal{A}_{\text{Mean}}$	1.36	1.67	2.71	1.62
IWRM	SSJD– $\mathcal{A}_{\text{Mean}}$ (Gramian-based)	1.01	1.00	1.32	0.97
IWRM	SSJD– $\mathcal{A}_{\text{MGDA}}$	3.92	3.95	5.36	4.05

Appendix

Table 9: Time required in seconds for one epoch of training on the ERM objective with SGD, on the IWRM objective with SSJD, and on the IWRM objective with Gramian-based SSJD (as described in Section 5 ), for different aggregators, on an NVIDIA L4 GPU. The batch size is always 32.

Multi-Objective Optimization

Multi-objective optimization studies how to find and choose solutions that balance competing goals, constraints, and preferences.

See all 266 papers

A Unified Framework for Gradient Aggregation in Multi-Objective Optimization
May 28, 2026Zeou Hu, Kelvin Ho, Yaoliang YuGradient Descent Stochastic Optimization Convergence
1Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · 3Vector Institute · 2Computer Science, The Chinese University of Hong Kong
Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates
Oct 9, 2026El Mahdi Chayti, Alireza Mirrokni, Martin JaggiLMO Methods Multi-Objective Optimization
Machine Learning and Optimization Laboratory (MLO), EPFL
Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction
Jul 29, 2026Hiroki Hamaguchi, Yuya Hikima, Hiroshi Sawada +1Stochastic Optimization Convergence Adaptive Gradient Methods
Graduate School of Information Science and Technology, The University of Tokyo · Department of Industrial Engineering and Economics, Institute of Science Tokyo · Communication Science Laboratories, NTT, Inc. +1

May 28, 2026J/K move · Enter open · S save

A Unified Framework for Gradient Aggregation in Multi-Objective Optimization

Zeou Hu, Kelvin Ho, Yaoliang Yu

1Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · 3Vector Institute · 2Computer Science, The Chinese University of Hong Kong

Gradient Descent Stochastic Optimization Convergence Multi-Objective Optimization

Open paper AskPDF arXiv

Many machine learning problems involve multiple inherent trade-offs that are best addressed by gradient-based multi-objective optimization (MOO) algorithms. Existing methods are often proposed with various motivations, analyzed case by case, and differ algorithmically in how the component gradients are aggregated at each step. In this work, we develop a unifying framework for gradient aggregation in MOO, establishing (optimal) rates of convergence to Pareto stationarity, the standard measure of performance in MOO. Central to our analysis is a sufficient alignment condition, from which we derive a theorem showing that non-conflicting directions, when chosen within the convex hull of gradients, form a fundamental sufficient condition for convergence. We further show that feasibility can be ensured through projection onto the dual cone, broadening the scope of methods that admit convergence guarantees. In parallel, we present a primal optimization perspective of gradient aggregation that encompasses established algorithms, clarifies their theoretical relationships, and enables the design of new variants. As an illustration, we introduce capped MGDA, derived from a CVaR-based formulation, and demonstrate its robustness in adversarial federated learning. Finally, we validate our theory through experiments on synthetic problems and practical benchmarks.

Abstract

Multi-Objective Optimization

Explore similar work

A Unified Framework for Gradient Aggregation in Multi-Objective Optimization

Geometry-Aware Multi-Objective Optimization: Norm-Constrained Oracles with Entropic Weight Updates

Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction