Relational Knowledge Distillation

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 19

All topics

Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking
Oct 5, 2026Zhicheng Ding, Xinyu Chu, Qing TianVisual Object Tracking Autonomous Driving Perception
Department of Computer Science, Bowling Green State University, OH 43402, USA · Department of Computer Science, University of Alabama at Birmingham, AL 35294, USA
Gromov-Wasserstein Distillation for Inductive Multi-View Embedding
Sep 30, 2026Rafael Pereira Eufrazio, Eduardo Fernandes Montesuma, Charles Casimiro CavalcanteRepresentation Learning Multi-View Learning
Instituto Federal do Ceará (IFCE) Canindé, Ceará, Brazil · Federal University of Ceará (UFC) Fortaleza, Ceará, Brazil · Sigma Nova Paris, France
Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition
Sep 14, 2026Yanjiang Shi, Peng Zhao, Nan Qi +1Video Understanding Human Activity Recognition
School of Computer Science and Technology, and with the National Engineering Laboratory for Big Data Analytics (NEL-BDA), Xi’an Jiaotong University, Xi’an 710049, China
ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models
Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Language Model Pretraining Autoregressive Language Modeling
VJ Technologies, Da Nang, Vietnam · AJ Technologies, Nagoya, Japan · Vietnam National University, Ho Chi Minh City, Vietnam +1
Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models
Aug 11, 2026Kiet T. Nguyen, Hanbo Shim, Jinwoo Kim +1Visual Spatial Reasoning VLM Distillation
KAIST
Flow-Map Distillation on Relation Manifolds for Image Restoration
Aug 6, 2026Zihao He, Songhua LiuFlow Matching Flow Map Learning
School of Artificial Intelligence · Shanghai Jiao Tong University · Shanghai City, China
AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition
Jul 28, 2026Yuqi Li, Yi-Cheng Lin, Xianglong Wang +5Relational Knowledge Distillation Speech Emotion Recognition
Department of Electrical Engineering, The City College of New York · National Taiwan University · Wyze Inc. +1
Privileged Lesion-Context Relational Distillation for Mask-Free Skin Lesion Classification
Jul 21, 2026Abu Mukaddim Rahi, Md Mithun Hossain, Md Zulficar Hasan Joy +2Skin Lesion Classification Medical Image Classification
Department of IT in Artificial Intelligence, Victoria University, Footscray, VIC, Australia · BUBT Resarch Graduate School, Bangladesh University of Business and Technology, Dhaka, Bangladesh · Department of Artificial Intelligence, Daffodil Institute of Information Technology, Dhaka, Bangladesh +2
Displacement Preserving Relational Distillation for Robust Medical Segmentation
Jul 6, 2026Zhicheng Ding, Xinyu Chu, Jung Im Choi +5Model Compression Relational Knowledge Distillation
Bowling Green State University, Bowling Green, OH, USA · University of Alabama at Birmingham, Birmingham, AL, USA · The University of Findlay, Findlay, OH, USA +3
ContiStain: Cross-Domain Relation-Preserving Distillation for Continual Multi-Domain Virtual IHC Staining
Jul 4, 2026Fuqiang Chen, Yifeng Wang, Hongpeng Wang +1Computational Pathology Virtual Staining
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China · School of Computer Science and Technology, Tsinghua University, Beijing, China
PHF: Privileged Hidden Flow for On-Policy Self-Distillation
Jun 28, 2026Yuhan Li, Mingxu Zhang, Dazhong Shen +1On-Policy Self-Distillation Language Model Distillation
1The Hong Kong University of Science and Technology (Guangzhou) · 2Nanjing University of Aeronautics and Astronautics · 3The 63rd Research Institute, National University of Defense Technology, Nanjing
Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models
Jun 3, 2026Tran Dinh Tien, Zhiqiang ShenVLM Adaptation VLM Distillation
Department of Machine Learning Mohamed bin Zayed University of Artificial Intelligence
SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
May 31, 2026Aditya Makineni, Qing TianCross-Architecture Knowledge Distillation Relational Knowledge Distillation
Department of Computer Science University of Alabama at Birmingham Birmingham, US
Lightweight SAR Ship Detection via Contrastive Distillation
May 27, 2026Surendar Devasundaram, Banafsheh Saber Latibari, Abhijit MahalanobisEfficient Object Detection Relational Knowledge Distillation
University of Arizona · Department of Electrical and Computer Engineering · Tucson, Arizona, USA
Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning
May 9, 2026Zhen-Hao Xie, Yan Wang, Hao Sun +3Class-Incremental Learning Continual Learning
School of Artificial Intelligence and the State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China
SARA: Semantically Adaptive Relational Alignment for Video Diffusion Models
May 8, 2026Jiesong Lian, Zixiang Zhou, Ruizhe Zhong +6Diffusion Model Alignment Video Diffusion Models
1Huazhong University of Science and Technology · 2Tencent Hunyuan · 3Shanghai Jiao Tong University +1
MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation
May 2, 2026Pham Khanh Chi, Quoc Phong Dao, Thuat Nguyen +3Representation Learning Language Model Distillation
1Hanoi University of Science and Technology · 2Monash University · University of Oregon
Relational Representation Distillation
Jul 16, 2024Nikos Giakoumoglou, Tania StathakiContrastive Learning Relational Knowledge Distillation
Imperial College London London, UK, SW7 2AZ
CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillation
Date pendingJeannie Chung, Hanna Jang, Ingyeong Yang +2Cross-Modal Learning Cross-Modal Knowledge Distillation
Ewha Womans University, Seoul 03760, South Korea

Oct 5, 2026, cs.CVJ/K move · Enter open · S save

Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking

Zhicheng Ding, Xinyu Chu, Qing Tian

Department of Computer Science, Bowling Green State University, OH 43402, USA · Department of Computer Science, University of Alabama at Birmingham, AL 35294, USA

Visual Object Tracking Autonomous Driving Perception Relational Knowledge Distillation Feature-Based Knowledge Distillation

Open paper AskPDF arXiv

In autonomous driving perception, visual object tracking systems must satisfy stringent latency and power constraints while remaining robust in complex and dynamic environments. Although transformer-based trackers achieve state-of-the-art accuracy, their substantial computational and memory overheads hinder deployment on real-time, resource-constrained platforms. To move toward this goal, we propose Difference Feature Map Knowledge Distillation (DFM-KD), a novel relational distillation framework tailored for transformer-based visual object tracking. Unlike conventional feature distillation methods that minimize point-wise discrepancies (e.g., mean squared error) between teacher and student feature representations, DFM-KD transfers knowledge through inter-sample feature differences, explicitly aligning the relational structure of the feature space. By distilling how the teacher models appearance variation and consistency across samples, rather than enforcing similarity in absolute activations, DFM-KD enables the student to better capture the structural dynamics of visual changes within a batch. As a result, the distilled model exhibits enhanced feature robustness and improved tracking performance. Extensive experiments demonstrate that DFM-KD consistently outperforms conventional feature-level distillation methods in both tracking precision and success rates.

Momentum

Related topics

Latest papers 19