Second Ranked Logits

Second-ranked logits, the probabilities assigned to the second-most likely class by a model, are becoming a focus in machine learning research. Current work explores their use in improving knowledge distillation, enhancing adversarial attack strategies, and addressing challenges in federated learning and long-tailed recognition by leveraging information beyond the top prediction. This research aims to improve model robustness, calibration, and efficiency, impacting areas such as model compression, out-of-distribution detection, and the security of large language models.

Papers

January 18, 2024

Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access
Saibo Geng, Berkay Döner, Chris Wendler, Martin Josifoski, Robert West
Large Language Model Text Generation Second Ranked Logits Black Box Large Language Model Constrained Decoding Language Model Output Black Box LLM

October 4, 2023

Learning to Scale Logits for Temperature-Conditional GFlowNets
Minsu Kim, Joohwan Ko, Taeyoung Yun, Dinghuai Zhang, Ling Pan, Woochang Kim, Jinkyoo Park, Emmanuel Bengio, Yoshua Bengio
LeArning Abstract Energy Based GFlowNets Controllability Condition Second Ranked Logits Goal Conditioned GFlowNets

September 8, 2023

UER: A Heuristic Bias Addressing Approach for Online Continual Learning
Huiwei Lin, Shanshan Feng, Baoquan Zhang, Hongliang Qiao, Xutao Li, Yunming Ye
Constructive Approach Anti Forgetting Online Continual Learning Experience Replay Dot Product Second Ranked Logits

September 6, 2023

SWAP: Exploiting Second-Ranked Logits for Adversarial Attacks on Time Series
Chang George Dong, Liangwei Nathan Zheng, Weitong Chen, Wei Emma Zhang, Lin Yue
Adversarial Attack Time Series Time Series Classification Adversarial Method Swap Distance Minimization Second Ranked Logits

August 20, 2023

Rethinking Client Drift in Federated Learning: A Logit Perspective
Yunlu Yan, Chun-Mei Feng, Mang Ye, Wangmeng Zuo, Ping Li, Rick Siow Mong Goh, Lei Zhu, C. L. Philip Chen
Unlearned Model Second Ranked Logits Client Drift Logit Vector

August 1, 2023

NormKD: Normalized Logits for Knowledge Distillation
Zhihao Chi, Tu Zheng, Hengjia Li, Zheng Yang, Boxi Wu, Binbin Lin, Deng Cai
Knowledge Distillation Supervised ImageNet Hyper Parameter Second Ranked Logits

June 15, 2023

Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models
Qinhong Zhou, Zonghan Yang, Peng Li, Yang Liu
Pre Trained Language Model Language Understanding Large Pre Trained Language Model Decision Relevant Information Second Ranked Logits Conventional Knowledge Distillation

May 23, 2023

Dual Focal Loss for Calibration
Linwei Tao, Minjing Dong, Chang Xu
Calibration Performance Focal Loss Second Ranked Logits Overconfident Prediction Calibration Network

May 2, 2023

Boosting Adversarial Transferability via Fusing Logits of Top-1 Decomposed Feature
Juanjuan Weng, Zhiming Luo, Dazhen Lin, Shaozi Li, Zhun Zhong
Adversarial Attack Singular Value Decomposition Second Ranked Logits Adversarial Transferability Feature Decomposition Feature Space Hijacking Attack

February 15, 2023

Uncertainty-Estimation with Normalized Logits for Out-of-Distribution Detection
Mouxiao Huang, Yu Qiao
Distribution Detection Uncertainty Estimation Distribution Data Second Ranked Logits Uncertainty Score Robust Learning Method

November 4, 2022

Logits are predictive of network type
Ali Borji
Network Programming Deep Network Second Ranked Logits Logit Vector

October 27, 2022

Exploiting Features and Logits in Heterogeneous Federated Learning
Yun-Hin Chan, Edith C. -H. Ngai
Feature Wise Heterogeneous Federated Learning Edge Intelligence Second Ranked Logits

September 1, 2022

Federated Learning with Label Distribution Skew via Logits Calibration
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Chao Wu
Cross Entropy Loss Label Distribution Second Ranked Logits Pairwise Ranking Softmax Cross Entropy

May 28, 2022

Teaching Models to Express Their Uncertainty in Words
Stephanie Lin, Jacob Hilton, Owain Evans
Natural Language High Uncertainty Anticipation Epistemic Uncertainty Word List Pre Trained Representation Second Ranked Logits Teaching Model

May 19, 2022

Bypassing Logits Bias in Online Class-Incremental Learning with a Generative Framework
Gehui Shen, Shibo Jie, Ziheng Li, Zhi-Hong Deng
Continual LEArning Online Continual Learning Generative Framework Memory Replay Second Ranked Logits Metric Learning Loss Online Class Incremental Learning

May 13, 2022

l-Leaks: Membership Inference Attacks with Logits
Shuhao Li, Yajie Wang, Yuanzhang Li, Yu-an Tan
Machine Learning Membership Inference Attack Second Ranked Logits Leakage Attack Shadow Model

March 18, 2022

A Closer Look at Knowledge Distillation with Features, Logits, and Gradients
Yen-Chang Hsu, James Smith, Yilin Shen, Zsolt Kira, Hongxia Jin
Knowledge Distillation Transfer Learning Knowledge Transfer Feature Wise Glance Annotation Second Ranked Logits

November 12, 2021

Speeding Up Entmax
Maxat Tezekbayev, Vassilina Nikoulina, Matthias Gallé, Zhenisbek Assylbekov
Text Generation Softmax Function Second Ranked Logits