Self Alignment

Self-alignment in large language models (LLMs) focuses on improving model behavior and aligning it with desired characteristics, such as adherence to cultural values or factual accuracy, without extensive human supervision. Current research explores various methods, including iterative self-enhancement paradigms, meta-rewarding techniques where the model judges its own responses, and resolving internal preference contradictions within the model. These advancements aim to reduce the reliance on costly human annotation and improve the reliability and safety of LLMs, impacting both the development of more robust AI systems and their practical applications across diverse fields.

Papers

November 13, 2024

Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models
Somanshu Singla, Zhen Wang, Tianyang Liu, Abdullah Ashfaq, Zhiting Hu, Eric P. Xing
Language Model Prompt Optimization Self Alignment Alignment Tuning

November 6, 2024

Self-Consistency Preference Optimization
Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, Jane Yu
Complex Reasoning Task Preference Optimization Self Consistency Answer Correctness Self Alignment

October 31, 2024

SelfCodeAlign: Self-Alignment for Code Generation
Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Zachary Mueller, Harm de Vries, Leandro von Werra, Arjun Guha, Lingming Zhang
Large Language Model Code Generation Instruction Tuning Self Alignment Instruction Response Pair

October 18, 2024

Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
Chenhang Cui, An Zhang, Yiyang Zhou, Zhaorun Chen, Gelei Deng, Huaxiu Yao, Tat-Seng Chua
Fine Grained Large Vision Language Model Vision Language Alignment Modality Alignment Self Alignment Preference Model Next Token

October 12, 2024

SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins
Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Bodapati, Aram Galstyan
Reinforcement Learning Alignment Problem Direct Preference Optimization Self Feedback Self Alignment Implicit Reward Alignment Algorithm

August 29, 2024

Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning
Rochelle Choenni, Ekaterina Shutova
Large Language Model Context Learning Multilingual Large Language Model Alignment Performance Self Alignment Cultural Value Value Survey

August 15, 2024

I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm
Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, Wenhao Huang, Jiajun Zhang
Medical LLM Scratch Project Better Alignment Electric SHEEP Self Alignment Self Enhancement

July 28, 2024

Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, Sainbayar Sukhbaatar
Iterative Training Self Alignment LLM a a Judge Language Reward Self Improving

June 13, 2024

ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions
Xu Zhang, Xunjian Yin, Xiaojun Wan
Large Language Model Language Model Direct Preference Optimization Preference Learning Self Alignment

June 5, 2024

Is Free Self-Alignment Possible?
Dyah Adila, Changho Shin, Yijing Zhang, Frederic Sala
Language Model Preference Pair Self Alignment

May 12, 2024

Learning Reward for Robot Skills Using Large Language Models via Self-Alignment
Yuwei Zeng, Yao Mu, Lin Shao
Reward Function Reward Learning Robot Skill Self Alignment

May 2, 2024

A New Self-Alignment Method without Solving Wahba Problem for SINS in Autonomous Vehicles
Hongliang Zhang, Yilan Zhou, Lei Wang, Tengchao Huang
Autonomous Vehicle Right Problem Vehicle Perception Self Alignment Initial Alignment

May 1, 2024

Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment
Zhili Liu, Yunhao Gou, Kai Chen, Lanqing Hong, Jiahui Gao, Fei Mi, Yu Zhang, Zhenguo Li, Xin Jiang, Qun Liu, James T. Kwok
Large Language Model Chain of Thought Top Two Answer Model Synergy Alignment Approach Self Alignment Input Mixture Span Answer

February 28, 2024

Learning or Self-aligning? Rethinking Instruction Fine-tuning
Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, Le Sun
Large Language Model LeArning Abstract Instruction Fine Tuning Self Alignment

February 23, 2024

Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations
Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, Tat-Seng Chua
Self Alignment Unanswerable Question Unknown Question

February 14, 2024

Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation
Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, Helen Meng
Large Language Model Fine Tuning Mitigating Hallucination Factual Claim Self Feedback Self Alignment Self Generated Response

February 12, 2024

Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
Haoyu Wang, Guozheng Ma, Ziqiao Meng, Zeyu Qin, Li Shen, Zhong Zhang, Bingzhe Wu, Liu Liu, Yatao Bian, Tingyang Xu, Xueqian Wang, Peilin Zhao
Large Language Model Medical LLM Human Annotation Bootstrapping End to End Model Alignment Self Alignment Aligned Model

February 8, 2024

Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation
Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, Siheng Chen
Large Language Model Human Value Self Alignment

January 23, 2024

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment
Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou
Multi Turn Dialogue Character Persona Role Play Self Alignment First Order Superposition

October 24, 2023

The Mason-Alberta Phonetic Segmenter: A forced alignment system based on deep neural networks and interpolation
Matthew C. Kelley, Scott James Perry, Benjamin V. Tucker
Deep Neural Network Speech Data Interpolation Regime Self Alignment Acoustic Modeling Forced Alignment Phoneme Segmentation

Self Alignment

Papers

Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models

Self-Consistency Preference Optimization

SelfCodeAlign: Self-Alignment for Code Generation

Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins

Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning

I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm

Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge

ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions

Is Free Self-Alignment Possible?

Learning Reward for Robot Skills Using Large Language Models via Self-Alignment

A New Self-Alignment Method without Solving Wahba Problem for SINS in Autonomous Vehicles

Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment

Learning or Self-aligning? Rethinking Instruction Fine-tuning

Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations

Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation

Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping

Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

The Mason-Alberta Phonetic Segmenter: A forced alignment system based on deep neural networks and interpolation