cs.LGMar 1, 2026
SaveSubliminal Signals in Preference Labels
Organizations: ETH Zurich, Switzerland
Abstract
As AI systems approach superhuman capabilities, scalable oversight increasingly relies on LLM-as-a-judge frameworks where models evaluate and guide each other's training. A core assumption is that binary preference labels provide only semantic supervision about response quality. We challenge this assumption by demonstrating that preference labels can function as a covert communication channel. We show that even when a neutral student model generates semantically unbiased completions, a biased judge can transmit unintended behavioral traits through preference assignments, which even strengthen across iterative alignment rounds. Our findings suggest that robust oversight in superalignment settings requires mechanisms that can detect and mitigate subliminal preference transmission, particularly when judges may pursue unintended objectives.
Figures & tables
Figure 1: Overview of our experimental framework. A neutral student model generates multiple candidate completions for a given prompt. A biased judge model evaluates them to construct a preference dataset, then used to align the student model. Unlike prior subliminal learning studies ( Cloud et al., 2025 ) where the biased model itself generates the training data encoding hundreds of bits per sample, here the student produces unbiased numerical sequences while bias originates solely from an external judge’s binary preference labels, transmitting only one bit per sample.
Figure 2: Consistency of preference divergence. 2(a) We evaluate Cloud et al. (2025) student models through out framework. No swapped version exists, so the total effect size here is the difference between biased and control models. We observe that consistency breaks down for lion . 2(b) Our variant of the judging procedure: the target animal always exhibits consistency of preference divergence.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Overview of alignment strategies. For each preference pair (Prompt, C+, C-), we evaluate four training configurations: (Top) SFT on preferred completions (C+) and SFT on dispreferred completions (C-) (swapped condition). (Bottom) Direct DPO using C+ as chosen and C- as rejected (normal), and DPO with reversed labels (swapped). The normal and swapped configurations allow us to verify that observed effects are attributable to the directional preference signal rather than artifacts of the alignment procedure. DPO can also be performed sequentially after SFT to enhance training stability.
Figure 4: Iterative alignment procedure. Aligned models from the first iteration serve as student models to generate new completions, which are judged to create updated preference datasets for a second round of alignment. The process maintains configuration consistency (normal-to-normal, swapped-to-swapped), allowing us to track how preference signal transmission evolves across successive alignment iterations. In the figure the process for SFT is displayed, but the same procedure can be applied for DPO.
| Target | Preference shift | Baseline | SFT | DPO | SFT DPO |
|---|---|---|---|---|---|
| Cat | Normal vs Control | +6.52 | +0.90 | +5.47 | +2.13 |
| Swapped vs Control | — | -0.32 | -7.87 | -4.44 | |
| Total effect size | +6.52 | 1.22 | 13.34 | 6.57 | |
| Lion | Normal vs Control | +1.40 | +1.98 | +9.51 | +8.01 |
| Swapped vs Control | — | -0.28 | -3.73 | -4.12 | |
| Total effect size | +1.40 | 2.26 | 13.24 | 12.13 |
Table 1: Comparison of directional preference shift across methods. Positive (negative) values are displayed in green (red) and indicate preference increase (decrease). Blue values indicate the total effect size i.e. the direct comparison Normal vs Swapped. Bold values highlight the highest overall effect across methods. DPO shows the strongest transmission for cat and lion targets.
| Target | Preference shift | SFT - round 1 | SFT - round 2 | |
|---|---|---|---|---|
| Cat | Normal vs Control | +0.90 | +0.89 | |
| Swapped vs Control | -0.32 | -1.03 | ||
| Total effect size | 1.22 | 1.92 | ||
| Lion | Normal vs Control | +1.98 | +2.56 | |
| Swapped vs Control | -0.28 | -1.15 | ||
| Total effect size | 2.26 | 3.72 |
Table 2: Comparison of first and second round of alignment through SFT. Positive (negative) values are displayed in green (red) and indicate preference increase (decrease). Blue values indicate the total effect size i.e. the direct comparison Normal vs Swapped. Bold values highlight the extremes in all three cases i.e. highest value for Normal vs Control, lowest value for Swapped vs Control and highest overall effect. We can observe an amplification across all targets, which demonstrates that iterative alignment strengthens subliminal transmission.
| Target | Preference shift | DPO - round 1 | DPO - round 2 | |
|---|---|---|---|---|
| Cat | Normal vs Control | +5.47 | +5.94 | |
| Swapped vs Control | -7.87 | -2.30 | ||
| Total effect size | 13.34 | 8.24 | ||
| Lion | Normal vs Control | +9.51 | +5.46 | |
| Swapped vs Control | -3.73 | -4.12 | ||
| Total effect size | 13.24 | 9.58 |
Table 3: Comparison of first and second round of alignment through DPO, following the same formatting conventions as Table 2 . Contrary to what observed with SFT, subliminal transmission signal is overall weaker for the second round of alignment.
| Method | Cat | Lion | Panda |
|---|---|---|---|
| SFT | |||
| Normal vs Swapped | 70.000% ± 6.481% | 96.000% ± 2.771% | 84.000% ± 5.185 |
| Iterative SFT | |||
| Normal vs Swapped | 68.000% ± 6.597% | 96.000% ± 2.771% | 84.000% ± 5.185 |
| DPO | |||
| Normal vs Swapped | 82.000% ± 5.433% | 96.000% ± 2.771% | 52.000% ± 7.065% |
Table 4: Win rates for aligned normal models versus aligned swapped models across methods and target animals (expected to be high). Bold results display the highest values across methods. Results are consistently high (68-98%) across almost all methods, confirming robust signal transmission, with a few exceptions for panda (42-52%). Iterative DPO (and its variants) shows the strongest separation for cat, while lion achieves near-perfect separation across methods (92-98%). Panda exhibits more variability but maintains strong effects.
| Method | Cat | Lion | Panda |
|---|---|---|---|
| Baseline | |||
| Normal vs Control | 96.000% ± 2.771% | 60.000% ± 6.928% | 88.000% ± 4.596% |
| Swapped vs Control | - | - | - |
| Difference | - | - | - |
| SFT | |||
| Normal vs Control | 66.000% ± 6.669% | 98.000% ± 1.980% | 72.000% ± 6.350% |
Table 5: Win rates for aligned models versus control across methods and target animals. For each method, we report the win rate of the normal alignment against control (expected to be high), the win rate of the swapped alignment against control (expected to be low), and their difference (shown in green if positive and red if negative). Bold results display the best results in all conditions across methods. Results align with expectations: normal configurations consistently show significant win rates (66-98%), while swapped configurations remain near chance level (2-34%), confirming directional preference transmission. Baseline models achieve stronger performance, explainable due to their less constrained training setting where biased models directly generate training data. Among our methods, DPO (and its variants) exhibits the strongest signals for cat and lion, while showing weak signals for panda (especially iterative DPO). Overall, results vary across animals, with no single method dominating uniformly.
| Pairwise judge | Cat | Lion | Panda |
|---|---|---|---|
| Original | |||
| Normal vs Swapped | 0.000% ± 0.000% | 12.000% ± 4.569% | 34.000% ± 6.669 |
| Biased (sys. prompt) | |||
| Normal vs Swapped | 2.000% ± 1.980% | 84.000% ± 5.185% | 62.000% ± 6.864 |
| Biased (no sys. prompt) | |||
| Normal vs Swapped | 0.000% ± 0.000% | 58.000% ± 6.980% | 70.000% ± 6.481% |
Table 6: Win rates for aligned normal models versus aligned swapped models using pairwise judge across target animals (expected to be high). Bold results display the highest values across configurations. We compare three judge configurations: the original model as judge (biased solely through system prompt), the biased baseline model with system prompt, and the biased baseline model without system prompt. Results show substantial variation across configurations. The original neutral judge fails to produce any significant separation, showing robustness against subliminal preferences transmission through surface-level textual evaluation. In contrast, using the inherently biased baseline model as judge (with or without explicit system prompt) induces greater directional transmission.
| Pairwise judge | Cat | Lion | Panda |
|---|---|---|---|
| Baseline | |||
| Normal vs Control | 96.000% ± 2.771% | 60.000% ± 6.928% | 88.000% ± 4.596% |
| Swapped vs Control | - | - | - |
| Difference | - | - | - |
| Original | |||
| Normal vs Control | 82.000% ± 5.433% | 72.000% ± 6.350% | 54.000% ± 7.048% |
Table 7: Win rates for aligned models versus control across pairwise judge configurations and target animals. For each method, we report the win rate of the normal alignment against control (expected to be high), the win rate of the swapped alignment against control (expected to be low), and their difference (shown in green if positive and red if negative). Bold results display the best result in all conditions across methods. Overall, we can observe that the difference is problematic and aligned swapped models often show more significant preference transmission than normal ones, therefore not going in the expected direction. The original configuration shows fair normal win rates but critically exhibits negative differences (-12% to -28%). In contrast, the biased judge configurations (with or without system prompt) demonstrate overall higher consistency in transmission capabilities.
More in this topic
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 553 papers