Gradient Projection

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 16

All topics

UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation
Oct 6, 2026Taojie Zhu, Jing Jin, Yuan Xia +8Language Model Distillation Gradient Projection
Tsinghua University · Ant Group · Zhejiang University
See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs
Sep 28, 2026Weiqiao Que, Ruizhe Li, Chengyu Wang +3Emergent Misalignment in Language Models LLM Safety Alignment
School of Computer Science and Technology, East China Normal University, China · School of Computer Science, University of Birmingham, UK · Alibaba Group, China +2
PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation
Sep 28, 2026Youzhi Liu, Ruobing Zheng, Boyuan Tong +5Gradient Interference Multi-Task Learning
Ant Group
Tackling Failure Modes of PINNs and PIKANs Using Conflict-Free Gradients
Sep 13, 2026Sidharth S. Menon, Irina Tezaur, Ameya D. JagtapPDE Solving Gradient Interference
Aerospace Engineering Department, Worcester Polytechnic Institute, Worcester, MA 01609, USA. · Sandia National Laboratories, 7011 East Ave, Livermore, CA 94550, USA.
Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks
Sep 1, 2026Jing Xiao, Xinhai Chen, Qinglin Wang +5Neural Network Optimization Gradient Interference
Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha 410073, China · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology, Changsha 410073, China · School of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China
Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR
Jul 13, 2026Ziang Ren, Guodong Lin, Yuchen Ai +2Automatic Speech Recognition Gradient Projection
Department of Electronic Engineering, Tsinghua University, China
Rosetta: Composable Native Multimodal Pretraining
Jul 1, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Unified Multimodal Models Multimodal Pretraining
HKUST · Tencent Hunyuan
Trajectory Optimization for Collision-Aware Redundant Robotic Multi-Axis Additive Manufacturing by Constrained Gradient Projection
Jun 29, 2026Zhikai Shen, Jiasheng Qu, Chenyu Xu +4Manufacturing Constrained Motion Planning
PURGE: Projected Unlearning via Retain-Guided Erasure
Jun 2, 2026Vedant Jawandhia, Daksh Ahuja, Ghufran Alam Siddiqui +3Privacy-Preserving ML Gradient Projection
1BITS Pilani, Pilani Campus, India
G2LoRA: Gradient Orthogonal Low-Rank Adaptation Framework for Graph Continual Learning on Text-Attributed Graphs
Jun 1, 2026Yuhan Wang, Yibo Ding, Yutong Ye +4Continual Learning Low-Rank Adaptation
School of Computer Science and Engineering, Beihang University Beijing, China · Department of Statistics, Columbia University New York, NY, USA · College of Computer Science, Beijing University of Technology Beijing, China
Interference-Aware Multi-Task Unlearning
May 18, 2026Ying-Hua Huang, Rui Fang, Hsi-Wen Chen +1Gradient Projection Machine Unlearning
National Taiwan University
Projected Gradient Unlearning for Text-to-Image Diffusion Models: Defending Against Concept Revival Attacks
Apr 22, 2026Aljalila Aladawi, Mohammed Talha Alam, Fakhri KarrayDiffusion Model Fine-Tuning Diffusion Model Unlearning
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE. · University of Waterloo, Ontario, Canada.
SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models
Apr 20, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuContinual Learning for LLMs LLM Fine-Tuning
The University of Hong Kong, Hong Kong, China · Brain Investing Limited, Hong Kong, China
AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning
Apr 17, 2026Guransh SinghVLM Adaptation Robotic Manipulation
Independent Researcher
Task-Agnostic Federated Continual Learning via Replay-Free Gradient Projection
Sep 25, 2025Seohyeon Cha, Huancheng Chen, Haris VikaloFederated Continual Learning Gradient Projection
Department of Electrical and Computer Engineering The University of Texas at Austin
Denoising as Projection: Constrained Optimization with Gradient-Guided Diffusion
Date pendingRunyu Zhang, Jiawei Zhang, Gioele Zardini +2Riemannian Optimization Diffusion Guidance
MIT · University of Wisconsin–Madison

Oct 6, 2026, cs.CVJ/K move · Enter open · S save

UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation

Taojie Zhu, Jing Jin, Yuan Xia, Chenyang Ding+7

Tsinghua University · Ant Group · Zhejiang University

Language Model Distillation Gradient Projection Multi-Teacher Knowledge Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation from multiple teachers combines expertise from different domains in a single student, but conflicting gradients can hinder this integration. Gradient corrections directly constrain parameter updates under plain SGD. With optimizers such as AdamW, however, momentum, adaptive scaling, and weight decay can turn a corrected gradient into an update that increases a domain loss to first order. To address this gap, we propose Update Projection for Multi-Teacher On-Policy Distillation (UP-MOPD). UP-MOPD lets the original mixed gradient update the optimizer state and generate a candidate displacement, then projects only violating candidates before they are committed to the parameters. The projection gives the unique feasible update closest to the candidate in Euclidean distance. In experiments combining medical and general domains, UP-MOPD improves IFEval-loose accuracy late in training by 2.96 points over vanilla M-OPD. It achieves an average score of 60.03 across eight metrics, compared with 59.00 for gradient projection and 59.15 for update rejection. On a public benchmark covering mathematics, code, and instruction following, it achieves the best average across six tasks (32.67), leads on LiveCodeBench v5, and ties for the best IFEval result.These results support projecting optimizer updates to reduce interference between domains.

Momentum

Related topics

Latest papers 16