Task Gradient

Task gradient research focuses on optimizing the training of models that learn multiple tasks simultaneously (multi-task learning) or sequentially (continual learning). Current efforts center on addressing the challenges posed by conflicting task gradients, which hinder performance, through techniques like gradient projection, weighting, and rotation, aiming for balanced optimization across all tasks. These advancements improve model generalization and efficiency, particularly in applications like molecular property prediction, image recognition, and meta-learning, where handling multiple, potentially conflicting objectives is crucial. The ultimate goal is to develop more robust and efficient multi-task learning algorithms that avoid negative transfer and maximize knowledge transfer between tasks.

Papers

September 3, 2024

Task Weighting through Gradient Projection for Multitask Learning
Christian Bohn, Ido Freeman, Hasan Tercan, Tobias Meisen
Multitask Learning Task Specific Weight Gradient Projection Task Gradient

July 15, 2024

Understanding the Dependence of Perception Model Competency on Regions in an Image
Sara Pohland, Claire Tomlin
Deep Neural Network Region Specific Perception Model Activity Dependence Task Gradient Low Quality Model

January 29, 2024

Enhancing Molecular Property Prediction with Auxiliary Learning and Task-Specific Adaptation
Vishal Dey, Xia Ning
Graph Neural Network Molecular Property Prediction Task Adaptation Auxiliary Learning Molecular Property Prediction Task Task Gradient

November 1, 2023

Implicit biases in multitask and continual learning from a backward error analysis perspective
Benoit Dherin
Continual LEArning Stochastic Gradient Descent Multi Task Implicit Bias Implicit Regularization Task Gradient Flatness Aware

August 23, 2023

Dual-Balancing for Multi-Task Learning
Baijiong Lin, Weisen Jiang, Feiyang Ye, Yu Zhang, Pengguang Chen, Ying-Cong Chen, Shu Liu, James T. Kwok
Multi Task Learning Balancing Strategy Task Loss Gradient Step Task Gradient

June 6, 2023

FAMO: Fast Adaptive Multitask Optimization
Bo Liu, Yihao Feng, Peter Stone, Qiang Liu
Gradient Descent Multitask Learning Multi Task Optimization Task Loss Task Gradient

May 31, 2023

Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNs
Yi Sun, Xin Xu, Jian Li, Xiaochang Hu, Yifei Shi, Ling-Li Zeng
Task Inference Gradient Conflict Task Gradient

March 12, 2023

RotoGBML: Towards Out-of-Distribution Generalization for Gradient-Based Meta-Learning
Min Zhang, Zifeng Zhuang, Zhitao Wang, Donglin Wang, Wenbin Li
Distribution Generalization Meta Training Gradient Based Meta Learning Generalization Problem Invariant Causal Feature Task Gradient

March 10, 2023

Gradient Coordination for Quantifying and Maximizing Knowledge Transference in Multi-Task Learning
Xuanhua Yang, Jianxin Zhao, Shaoguo Liu, Liang Wang, Bo Zheng
Multi Task Learning Knowledge Transfer Task Transfer Task Gradient

January 31, 2023

GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning
Xin Dong, Ruize Wu, Chao Xiong, Hai Li, Lei Cheng, Yong He, Shiyou Qian, Jian Cao, Linjian Mo
Multi Task Learning Multi Task Gradient Update Mixed Task Task Gradient

January 30, 2023

ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning
Junguang Jiang, Baixu Chen, Junwei Pan, Ximei Wang, Liu Dapeng, Jie Jiang, Mingsheng Long
Auxiliary Task Negative Transfer Human Like Generalization Task Gradient Generalization Benefit

November 24, 2022

Improving Multi-task Learning via Seeking Task-based Flat Regions
Hoang Phan, Lam Tran, Ngoc N. Tran, Nhat Ho, Dinh Phung, Trung Le
Multi Task Learning Flat Minimum Single Task Learning Task Gradient

Task Gradient

Papers

Task Weighting through Gradient Projection for Multitask Learning

Understanding the Dependence of Perception Model Competency on Regions in an Image

Enhancing Molecular Property Prediction with Auxiliary Learning and Task-Specific Adaptation

Implicit biases in multitask and continual learning from a backward error analysis perspective

Dual-Balancing for Multi-Task Learning

FAMO: Fast Adaptive Multitask Optimization

Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNs

RotoGBML: Towards Out-of-Distribution Generalization for Gradient-Based Meta-Learning

Gradient Coordination for Quantifying and Maximizing Knowledge Transference in Multi-Task Learning

GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning

ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning

Improving Multi-task Learning via Seeking Task-based Flat Regions