Token-Level Loss Weighting

Follow this topic Ask about this topic

Latest papers

10 papers3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

PORT: Preference Optimization via Robust Token-Level Reweighting
Nov 9, 2026Ding Zhu, Xiukun Wei, Tian Xie +3Preference Learning for LLMs LLM Alignment
Department of Computer Science and Engineering, The Ohio State University
TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization
Nov 8, 2026Suxin Ji, Hungtao WanDirect Preference Optimization Safety-Preserving Fine-Tuning of Language Models
University of Pennsylvania · Independent Researcher
Self-Privileged Fine-Tuning
Nov 3, 2026Yangrong Hu, Xueyu Zhou, Yue Wu +1Token-Level Loss Weighting Supervised Fine-Tuning
Department of Data Science and Artificial Intelligence The Hong Kong Polytechnic University · Department of Applied Mathematics The Hong Kong Polytechnic University
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation
Oct 20, 2026Guangyu Liu, Siyuan Gan, Shilong Zhou +3Token-Level Loss Weighting Language Model Distillation
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features
Sep 27, 2026Cunchun Li, Haonan He, Yifan Gao +4Supervised Fine-Tuning Adaptive Loss Weighting
Shanghai AI Laboratory · University of Science and Technology of China · Fudan University +2
A Unified Per-Token Gating Family for On-Policy Distillation: FKL/RKL Mixing with Multi-Channel and Bias Coefficients
Sep 10, 2026Suwan Wu, Yumeng Lin, Pengcheng Yuan +1Adaptive Loss Weighting Language Model Distillation
Xiaohongshu Inc. · Tianjin University
Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning
Sep 9, 2026Yaning Jia, Chunhui Zhang, Wenxuan Xu +3Numerical Reasoning in Language Models Supervised Fine-Tuning
Dartmouth College · Carnegie Mellon University
TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
Jul 31, 2026Guanzhi Deng, Haibo Wang, Kuan Wu +5Mixture-of-Experts Language Models Mixture of Experts
City University of Hong Kong, Hong Kong, China · Carnegie Mellon University, Pittsburgh, USA · University of Waterloo, Ontario, Canada +1
Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
May 20, 2026Meng Shen, Minghao Wu, Deepu RajanVision-Language Models Hallucination in Language Models
Nanyang Technological University · Monash University
InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
May 14, 2026Mahdi Sabbaghi, George Pappas, Adel Javanmard +1Supervised Fine-Tuning Language Model Generalization
University of Pennsylvania · University of Southern California

Nov 9, 2026J/K move · Enter open · S save

PORT: Preference Optimization via Robust Token-Level Reweighting

Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu+2

Department of Computer Science and Engineering, The Ohio State University

Preference Learning for LLMs LLM Alignment Token-Level Loss Weighting

Open paper AskPDF arXiv

Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity. Existing robust preference optimization methods primarily operate at the sequence level, implicitly treating entire responses as uniformly correct or incorrect. However, rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. In this work, we propose Preference Optimization via Robust Token-level reweighting (PORT), a fine-grained framework for robust alignment under noisy preferences. PORT performs token-level reweighting using the empirical cumulative distribution function (CDF) of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization that selectively suppresses corrupted tokens without additional backward-pass computation. We provide theoretical analysis showing that PORT reduces the gradient bias from noisy preference labels by minimizing an upper bound on the bias term. Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines. The code is available at https://github.com/osu-srml/PORT