Rank-Aware Differentially Private Release of Listwise Preferences for LLM Alignment
Organizations: Institute of Science Tokyo · University of Queensland
Abstract
Listwise preference feedback offers richer supervision for large language model (LLMs) alignment than pairwise comparisons, but a full ranking also reveals more sensitive user preferences. Existing privacy-preserving alignment methods focus mainly on pairwise feedback, while listwise alignment under local differential privacy (LDP) remains unexplored. Extending LDP to this listwise domain raises a critical challenge: the high sensitivity of the full ranking space demands excessive noise, neutralizing the benefits of listwise supervision. To address this challenge, we analyze the interaction between supervision granularity and privacy noise in downstream training, and introduce a rank-aware exponential mechanism that privatizes listwise preference data into a low-sensitivity granularity suitable for downstream alignment. The mechanism leverages ranking information to sample a fixed-size binary partition, concentrating the noise near borderline items instead of perturbing all items uniformly as randomized response (RR) does. Empirical evaluations on downstream LLM alignment tasks show that our mechanism consistently outperforms existing LDP baselines at matched privacy budgets. The code is available at https://github.com/tdsai-lab/RAEM.
Figures & tables
| Supervision granularity | Representative method | Sensitivity | Noise scale |
|---|---|---|---|
| Cardinal score vector | PPA [ 29 ] | ||
| Rank vector | SLiC [ 30 ] | ||
| Binary labels | KTO [ 9 ] |
| Method | Proxy Model | General Benchmark | Avg. | ||
|---|---|---|---|---|---|
| Pair-Preference | Scoring | AlpacaEval | MT-Bench | ||
| Qwen2-0.5B | |||||
| Non-private ( ) | |||||
| PPA | 66.3 | 61.3 | 61.6 | 56.4 | 61.4 |
| SLiC | 65.7 | 60.7 | 61.0 | 55.8 | 60.8 |
| KTO | 67.8 | 55.9 | 59.2 | 60.1 | 60.8 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Training dataset | ListUltraFeedback |
| Number of candidates | 8 |
| Top boundary | 4 |
| Privacy budgets | 0.5, 1, 3 |
| KTO | 0.1 |
| KTO | 1.0, 1.0 |
| Datasets | Examples | Judge Model |
|---|---|---|
| UltraChat200k | 207,865 | – |
| ListUltraFeedback | 59,880 | – |
| ListUltraFeedback | 1,968 | RLHFlow Pair-Preference, ArmoRM |
| AlpacaEval | 805 | GPT-4 Turbo |
| MT-Bench | 80 | GPT-4 Turbo |
| Method | Pair-Preference | Scoring | AlpacaEval | MT-Bench | Avg. | |
|---|---|---|---|---|---|---|
| RAEM+KTO (ours) | 1 | 61.2 | 49.1 | 50.6 | 50.6 | 52.9 |
| RAEM+DPO | 1 | 58.1 | 48.5 | 48.9 | 53.5 | 52.3 |
| RAEM+KTO (ours) | 3 | 66.4 | 54.6 | 57.8 | 59.7 | 59.6 |
| RAEM+DPO | 3 | 64.8 | 51.1 | 54.2 | 54.5 | 56.1 |
| Seed | Method | Pair-Preference | Scoring | AlpacaEval | MT-Bench | Avg. |
|---|---|---|---|---|---|---|
| 42 (original) | RR+KTO | 89.70 | 90.90 | 86.40 | 71.80 | 84.70 |
| 42 (original) | RAEM+KTO | 90.50 | 91.00 | 87.50 | 72.10 | 85.28 |
| 123 | RR+KTO | 90.21 | 91.02 | 86.48 | 70.20 | 84.48 |
| 123 | RAEM+KTO | 91.11 | 91.22 | 88.05 | 70.44 | 85.21 |
| 456 | RR+KTO | 90.77 | 90.15 | 86.17 | 70.89 | 84.50 |
| 456 | RAEM+KTO | 91.08 | 90.94 | 87.02 | 73.58 | 85.66 |
| Method | Pair-Preference | Scoring | AlpacaEval | MT-Bench | Avg. |
| Coordinate-wise RR + KTO | 52.9 | 38.6 | 48.0 | 52.5 | 48.0 |
| RR + uniform fixed- + KTO | 53.1 | 39.1 | 48.1 | 49.8 | 47.5 |
| Top-membership EM + KTO | 52.9 | 40.1 | 49.5 | 48.8 | 47.8 |
| Linear rank-aware EM + KTO (ours) | 59.2 | 43.6 | 51.8 | 49.7 | 51.1 |
| Prompt | Given an algorithm or a coding search query, provide a related code snippet. Search query: Python loop . |
|---|---|
| RR | (1.004) Gives for I in range(10): print(i) , which raises a NameError because Python identifiers are case-sensitive, and then incorrectly describes the query as “10.” |
| Ours | (2.000) Provides a valid search_list function that iterates over a list, returns the index of the queried element when found, and returns otherwise. |
| Prompt | Write a thematic tweet announcing that applications for summer internships are open. Start with a Halloween tie-in. |
| RR | (1.004) Produces a generic internship announcement with a celebration emoji, but the tweet itself contains no Halloween-related opening or theme. |
| Prompt | Does the dinosaurs really exist? |
|---|---|
| RR | (1.000) Initially frames dinosaur existence as a matter of belief and interpretation, weakening an otherwise scientific answer. |
| Ours | (2.000) States the scientific consensus directly and supports it with fossilized bones, footprints, anatomical reconstruction, and examples of documented dinosaur species. |
| Prompt | Suggest product bundles that are often purchased with a MacBook Air. |
| RR | (1.097) Recommends an Apple Pencil on the mistaken premise that the MacBook Air has a touchscreen and describes several adapters inaccurately as cables. |
Privacy-Preserving Language Models
Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.
See all 191 papers