Paper ID: 2408.12799 • Published Aug 23, 2024
Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora
JoonHo Lee, JuYoun Son, Juree Seok, Wooseok Jang, Yeong-Dae Kwon
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
Inconsistent annotations in training corpora, particularly within preference
learning datasets, pose challenges in developing advanced language models.
These inconsistencies often arise from variability among annotators and
inherent multi-dimensional nature of the preferences. To address these issues,
we introduce a self-curation method that preprocesses annotated datasets by
leveraging proxy models trained directly on them. Our method enhances
preference learning by automatically detecting and selecting consistent
annotations. We validate the proposed approach through extensive
instruction-following tasks, demonstrating performance improvements of up to
33\% across various learning algorithms and proxy capabilities. This work
offers a straightforward and reliable solution to address preference
inconsistencies without relying on heuristics, serving as an initial step
toward the development of more advanced preference learning methodologies. Code
is available at this https URL .