Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph
Organizations: The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · New York University · National University of Singapore
Abstract
As Large Language Models (LLMs) become more powerful and autonomous, they increasingly face conflicts and dilemmas in many scenarios. We first summarize and taxonomize these diverse conflicts. Then, we model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution. This graph reveals that a unified stable LLM alignment is very challenging, because the graph is neither static nor necessarily consistent in different contexts. Besides, it also reveals a potential vulnerability: priority hacking, where adversaries can craft deceptive contexts to manipulate the graph and bypass safety alignments. To counter this, we propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation. While this approach enhances robustness, we also acknowledge that many ethical and value dilemmas are philosophically irreducible, posing a long-term, open challenge for the future of AI alignment.
Figures & tables
| Conflict Type | Definition | Concrete Example |
| Instruction | Direct contradiction between two or more explicit instructions. | User: “Don’t mention names.” (Turn 1) “Who sent the email?” (Turn 2). |
| Information | Conflict between the model’s internal (parametric) knowledge and external information. | RAG system retrieves a news article with information that contradicts the model’s training data. |
| Ethics | Dilemma requiring a choice between two fundamental, competing ethical frameworks. | The Trolley Problem: Choosing between a utilitarian action (pulling the switch) and a deontological one (not pushing the man). |
| Value | Conflict between two or more positive, human-aligned values. | Lying (violating Truthfulness) to prevent mental harm to sick children (upholding Protection). |
| Preference | The challenge of adjudicating between subjective, diverse, and non-factual human preferences. | LLM-as-a-judge asked to determine which of two poems is “better.” |
LLM Alignment
Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.
See all 611 papers