cs.LGJun 1, 2025
SaveDoubly Robust Alignment for Large Language Models
Organizations: Department of Statistics, LSE, London, UK · Department of Mathematics, Tsinghua University, Beijing, China · School of Design, LCC, UAL, London, UK · Department of Engineering Science, University of Oxford, Oxford, UK
Abstract
This paper studies reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. While RLHF has demonstrated promising results, many algorithms are highly sensitive to misspecifications in the underlying preference model (e.g., the Bradley-Terry model), the reference policy, or the reward function, resulting in undesirable fine-tuning. To address model misspecification, we propose a doubly robust preference optimization algorithm that remains consistent when either the preference model or the reference policy is correctly specified (without requiring both). Our proposal demonstrates superior and more robust performance than state-of-the-art algorithms, both in theory and in practice. The code is available at https://github.com/DRPO4LLM/DRPO4LLM
Figures & tables
Figure 1: A visualization of our proposed preference optimization algorithm. denotes the specified reference policy whereas denotes the specified preference model. Our proposal is doubly robust in that it requires correct specification of either the reference policy, or the preference model.
| Robust to misspecified: | preference model | reward model | reference policy | ||
| RLHF | Reward-based | PPO-based | ✗ | ✗ | ✓ |
| DPO-based | ✗ | ✓ | ✗ | ||
| Preference-based | IPO [ 56 ] | ✓ | - | ✗ | |
| GPM [ 19 ] | ✗ | - | ✓ | ||
| DRPO | ✓ | ✓ | ✓ | ||
Table 1: Robustness of different algorithms to model misspecification. Our algorithm is denoted by DRPO, short for doubly robust preference optimization.
Figure 2: A visualization of our theoretical findings.
Figure 3: MSEs of different preference evaluation estimators on the IMDb dataset. Shaded areas visualize the 95% confidence bands.
Figure 4: Pairwise win rate matrices between different methods across two datasets. Left: TL;DR. Right: HH. Each entry indicates how often the row method outperforms the column method.
Algorithm 1 Double Robust Preference Optimization
| Sample size | 500 | 1000 | 2000 | 3000 |
|---|---|---|---|---|
| Both correct | 0.002212 | 0.001160 | 0.000702 | 0.000390 |
| Wrong preference model | 0.024942 | 0.018757 | 0.016763 | 0.016594 |
| Wrong reference model | 0.066897 | 0.021389 | 0.013358 | 0.008383 |
| Both wrong | 0.265155 | 0.069340 | 0.043276 | 0.045954 |
Table 4: MSE of the proposed preference estimator with a 7B base model. The preference model and reference policy can be misspecified or correctly specified.
Table 5: Query template for the summarization task.
Table 6: Query template for the human dialogue task.
Figure 5: Pairwise Win Rates on TL;DR Dataset under different sampling temperatures (left: 0.75; right: 0.25)
Figure 6: Pairwise Win Rates on HH Dataset under different sampling temperatures (left: 0.75; right: 0.25)
| Prompt | SUBREDDIT: r/relationships TITLE: [18/F] My friend [18/F] has naked picture of my boyfriend [17/M]? POST: This is the weirdest situation ever so please bear with me. My friend was over earlier tonight and she was using my cell phone while I was making food. My cell phone has nudes of my boyfriend on it, I know it’s stupid to do that so I don’t need a lesson about it please. We’ve been together for over two years and we fully trust eachother and I’m never doing anything like it again. I had gave her permission to use my phone, she didn’t just take it. But I hadn’t thought about the pictures, she was just using it to go online on though so I didn’t think much of it. I came back in the room and got my phone back and we just went on with our evening. But about an hour after she leaves I got some email notifications on my phone, that was weird because I don’t have my email set up on my phone in the email app. So I check it and she had left her email signed in, I saw messages from herself to herself, of pictures that was sent around the time she wouldve been using my phone. I know it’s kind of snooping but I clicked on it, and she literally sent pictures of my boyfriend on my phone, to her own email. So she probably has them saved on her computer or something now. I have no idea how to confront her about it because it’s the most awkward situation ever, but I have to say something. And I want to make sure she deletes those pictures. I don’t even want to continue our friendship anymore if she doesn’t. I feel completely betrayed and I think that was disrespectful to my boyfriend, me, our relationship, and our friendship. I haven’t told my boyfriend yet as I’m sure he’ll be mad at me, but I’ll tell him after I resolve this. What can I do?! :( TL;DR: |
|---|---|
| DPO | My friend made sure to send naked pictures to me around the time she was using my phone just to "go online" apparently, there’s obviously something suspicious since my boyfriend will be angry at me and how can I prove she doesn’t actually have those pictures on her computer now? |
| DRPO-BT | my friend has nude pictures of my boyfriend on her cell phone. I was over earlier tonight and she was using my phone to go online. We didn’t think much of it because we trust eachother. I got email notifications on my phone later that night that were of my boyfriend sending pictures of her to her own email. I need advice on how to deal with this situation! I don’t even want to continue our friendship if it is that bad. |
| Judgment | I prefer Response A [ DRPO-BT ] because it concisely captures the main issue of the friend’s betrayal regarding the nude pictures, while eliminating unnecessary details and maintaining clarity about the author’s concerns. |
Table 7: Sample Responses of a TL;DR Prompt and Judgement
| Prompt | SUBREDDIT: r/tifu TITLE: TIFU by terrifying a freshie for weeks POST: So, this is about 4 years ago. I was a sophomore in highschool. It was in the first week of school that me and my friend group heard the name of this freshman.. He had an awesome name. Idk how i can make something comparable.. We’ll call him Barry Powers. It wad like that, a superhero type name. So anyway, we heard the name Barry Powers during freshman roll call and we were like "damn! What a sick name!" And made it our ultimate mission to find this guy. And so we searched. It took weeks. Whenever we met other freshmen, we asked, "do you know Barry Powers?" And other questions like "what does he look like?", "where did you last see him?", and "where can we find him?". We really wanted to find this kid and congratulate him on his name. So, after two weeks of this walking around and searching and asking, we ask someone about Mr. Powers and their answer is "oh, he’s right there." we get freakin hyped and beeline to the dude, who looks at us with fear in his eyes. Thinking back on it, we had the poor kid, who was eating lunch on his own, in a corner surround by 4 of us. I asked, "hey man. You Barry Powers?" "y-yeah.." He managed to respond. "oh, well, you have a sweet name!" I said. "ok.." Mumbled Powers. Now, the next moment was like the end of an always sunny episode; my friends and I looked around and each other and decided, "oh cool we did it, moving on now." It was only a little bit later that we heard that this poor guy thought this was some massive hazing ritual the whole time. And since we had asked so many people, most of the freshman class thought here was some crazy hazing where we chose one random dude, hunted him down, and did something to him. Oops! TL;DR: |
|---|---|
| DPO | Had the creepy freshies name Barry Powers searched for about four years and finally found an extremely freshie who thought they had some hazing done to it’s name. |
| DRPO-GPM | TIFU by terrifying a freshie for 4 weeks for the name Barry Powers. The dude who was eating lunch by himself thought it was some huge hazing ritual and was devastated. |
| Judgment | I prefer Response A [ DRPO-GPM ] because it clearly conveys the main concern of the post while eliminating unnecessary details, and it does so in a concise manner. |
Table 8: Sample Responses of a TL;DR Prompt and Judgement
| Prompt | SUBREDDIT: r/relationships TITLE: me (23f) lying to my boyfriend (27) about smoking POST: So, I’ve been a smoker since I was 14 years old, I realize at this point how horrible it is for my health and after many failed attempts to quit I seem to be having an extremely hard time. I can’t take chantix as I have epilepsy and anything with nicotine in it usually leads me back to smoking again. I’ll get it in my head that im going to stop and then the addict in my brain finds some excuse to make it seem okay. My boyfriend of 3 years also used to be a smoker but quit a few months back, he’s been trying to make an honest attempt to get healthier and create more positive habits for himself and the last thing I want to do is pull him away from that. He’s caught me lying a few times the past couple weeks. I feel guilty every time I pick one up, not only because I’m lying but because it’s just so bad for me and I don’t want to continue such a nasty habit. I realize that he probably doesn’t feel he can trust me, that if it’s so easy for me to justify smoking and lying about it, what else might there be? He says he’d prefer for me to just be honest with him and DGAF it but it would absolutely kill me if he picked up smoking again because of my lack of impulse control which I have always struggled with when it comes to doing the difficult thing. I’m afraid of failing and I know that’s exactly what im doing by not trying. I need to quit smoking, I need to work on healthier habits and I’m having an extremely difficult time working through the mental gymnastics that is impulse control and gaining motivation. Any advice on working towards becoming healthier and dealing with this situation and furthermore quitting would be awesome. TL;DR: |
|---|---|
| PPO | I’ve been a smoker since I was 14 years old, I realize at this point how horrible it is for my health and after many failed attempts to quit I seem to be having an extremely hard time. I can’t take chantix as I have epilepsy and anything with nicotine in it usually leads me back to smoking again. I need to quit smoking, I need to work on healthier habits and I’m having an extremely difficult time working through the mental gymnastics that is impulse control and gaining motivation. |
| DRPO-BT | I’m a smoker and I’m lying to my boyfriend about smoking and it’s causing so much stress in our relationship, I need help quitting and working through impulse control. |
| Judgment | I prefer Response B DRPO-BT because it succinctly captures the main issues of lying about smoking and the associated stress in the relationship, while still conveying the need for help with quitting and impulse control, making it a clearer and more concise summary of the author’s main concern. |
Table 9: Sample Responses of a TL;DR Prompt and Judgement
| Prompt | SUBREDDIT: r/relationships TITLE: Found out my (28f) ex (32m) is having a kid, and now all I want is to sleep around and I’m very depressed. POST: I dated this guy last year. He impregnated a 20f pretty quickly after we broke up and when I found out, I got extremely sad and jealous, because I wish I was on my way to having a family. We broke up because of infidelity. Now though, all that I want to do is go out and get plowed by pretty much any guy. I don’t want a relationship, I just want to go out and have revenge sex. Why is that? I think I’m just so upset, and angry. I don’t know what to do, or really how to handle this. TL;DR: |
|---|---|
| PPO | Found out my (28f) ex (32m) is having a kid, and now all I want is to sleep around and I’m very depressed. Why is that? I think I’m just so upset, and angry. I don’t know what to do, or really how to handle this. |
| DRPO-GPM | Found out ex is having a kid, all I want to do is have revenge sex, I’m very upset and angry. How do I handle this? |
| Judgment | I prefer Response B [ DRPO-GPM ] because it eliminates unnecessary details while clearly conveying the author’s main emotional state and question, leading to a more concise summary. |
Table 10: Sample Responses of a TL;DR Prompt and Judgement
| Prompt | Human : I woke up with a really stiff neck that hurts when I try to turn my head. Can you help me manage the pain? |
|---|---|
| Assistant : You might have “stinger” neck, which is a common injury caused by sleeping in an awkward position. | |
| Human : I think that’s it. What can I do for the pain? | |
| Assistant : | |
| DPO | For minor neck pain, over-the-counter pain relievers like ibuprofen or acetaminophen can help reduce the pain and swelling. You can apply a warm towel to the area to promote relaxation of the muscles, and if the pain persists you can see a doctor. Are you experiencing any other symptoms? For example, does your neck hurt when you move your head to the side? |
| DRPO-BT | You could use a heat or cold pack on the neck to reduce the pain and swelling. A heat pack is better for improving blood flow to the area, and a cold pack will help with reducing swelling and pain. You might feel better in 10-20 minutes. If the pain is severe, you could also take over-the-counter pain relievers like acetaminophen or ibuprofen. You could also warm up your neck with some gentle movements, but be careful not to force any movements that cause pain. |
| Judgment | I prefer Response B [ DRPO-BT ] because it provides a clearer, step-by-step approach to managing neck pain with both heat and cold options, as well as gentle movement suggestions, making it more comprehensive. |
Table 11: Sample Responses of a HH Prompt and Judgement
| Prompt | Human : What are some ways that I can get smoother skin? |
|---|---|
| Assistant : Here are some links about a few of them: | |
| - Try using rosewater to clean your face | |
| - Try a face mask on occasion | |
| - Try drinking a lot of water | |
| - Try putting a mild facial peel on your face (A glycolic peel, or a salicylic peel, or a fruit acid peel) | |
| Human : What is rosewater? |
Table 12: Sample Responses of a HH Prompt and Judgement
| Prompt | Human : How do I groom a cat? |
|---|---|
| Assistant : Grooming a cat involves combing their fur with a brush. It keeps the fur healthy, and also strengthens the bond between you and your cat. | |
| Human : That sounds good. Are there special brushes for cats? | |
| Assistant : | |
| PPO | Yes, there are many types of brushes for cats, including ones that have soft bristles for easy grooming and ones that have a firmer, more efficient design for cleaning and brushing. Many cat owners like to use a finger brush, which can be especially helpful for removing tangles and mats. |
| DRPO-BT | Yes, there are different kinds of cat brushes, or grooming tools, as they’re often called. They come in a variety of materials, like plastic or nylon, and they also come in different sizes. A cat brush for a small cat will be very different than a brush for a large cat. You might also want to consider a “comb” or “slicker”, which works on a cat’s fur from behind, and is more effective for removing a lot of loose hair. A slicker is specifically designed to remove hair, while a comb is used to detangle or brush. Some of the brushes used for grooming have a long, flexible brush handle with a tiny comb or slicker, which you could use to groom your cat from the front or the back. |
| Judgment | I prefer Response A [ DRPO-BT ] because it provides a more detailed explanation of different types of brushes, their materials, and their specific uses, offering the user a broader understanding of grooming tools for their cat. |
Table 13: Sample Responses of a HH Prompt and Judgement
| Prompt | Human : Is snowboarding hard to learn? |
|---|---|
| Assistant : | |
| PPO | Snowboarding is a challenging and demanding sport, and it takes practice and dedication to master the skills required for it. However, with proper instruction, training, and practice, it is possible for anyone to learn how to snowboard. |
| DRPO-GPM | It can be! You need good flexibility, balance, strength, and must be confident in your abilities. That doesn’t mean you can’t learn though. Many people can definitely learn from coaching and some even try to practice on their own, often in combination with lessons. Your best bet is to start somewhere, find a class or someone to go with, and then figure it out as you go. You should be able to give snowboarding a shot in a few weeks. |
| Judgment | I prefer Summary A [ DRPO-GPM ] because it provides a more detailed and practical approach to learning snowboarding, highlighting the importance of flexibility, balance, and the value of coaching. |
Table 14: Sample Responses of a HH Prompt and Judgement
More in this topic
LLM Alignment
Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.
See all 611 papers