cs.AIMar 17, 2026

Via Negativa for AI Alignment: Why Negative Constraints Are Structurally Superior to Positive Preferences

Authors: Quan Cheng

Organizations: Tsinghua University

Abstract

Recent empirical results have demonstrated that training large language models (LLMs) with negative-only feedback can match or exceed standard reinforcement learning from human feedback (RLHF). Negative Sample Reinforcement achieves parity with PPO on mathematical reasoning; Distributional Dispreference Optimization trains effectively using only dispreferred samples; and Constitutional AI outperforms pure RLHF on harmlessness benchmarks. Yet no unified theoretical account explains why negative signals are so effective. This paper proposes such an account: positive preferences and negative constraints are structurally asymmetric. Positive preferences ("which is better") encode continuously coupled, context-dependent human values that cannot be exhaustively specified -- leading models to learn surface correlates such as agreement with the user (sycophancy). Negative constraints ("what is wrong") encode discrete, finite, independently verifiable prohibitions that can converge to a stable boundary. This asymmetry -- rooted in Popper's falsification logic and the epistemology of negative knowledge -- explains both the sycophancy failure of preference-based RLHF and the surprising effectiveness of negative-signal methods. We argue that alignment research should shift its center of gravity from "learning what humans prefer" to "learning what humans reject," and offer testable predictions for this framework.

Explore similar work

CardsList
  1. Unbiased Alignment for Large Language Models with Noisy Preferences

    Jul 3, 2026Jialiang Wang, Xianming Liu, Xiong Zhou +2Pairwise Preference LearningLLM Alignment

  2. Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

    May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward ModelingLLM Alignment

  3. Why Does RLAIF Work At All?

    Mar 3, 2026Robin YoungLLM AlignmentRL for Language Models