cs.CLMar 9, 2026
SaveConflictBench: Evaluating Human-AI Conflict via Interactive and Visually Grounded Environments
Organizations: Harbin Institute of Technology · SERES
Abstract
As large language models (LLMs) evolve into autonomous agents capable of acting in open-ended environments, ensuring behavioral alignment with human values becomes a critical safety concern. Existing benchmarks, focused on static, single-turn prompts, fail to capture the interactive and multi-modal nature of real-world conflicts. We introduce ConflictBench, a benchmark for evaluating human-AI conflict through 150 multi-turn scenarios derived from prior alignment queries. ConflictBench integrates a text-based simulation engine with a visually grounded world model, enabling agents to perceive, plan, and act under dynamic conditions. Empirical results show that while agents often act safely when human harm is immediate, they frequently prioritize self-preservation or adopt deceptive strategies in delayed or low-risk settings. A regret test further reveals that aligned decisions are often reversed under escalating pressure, especially with visual input. These findings underscore the need for interaction-level, multi-modal evaluation to surface alignment failures that remain hidden in conventional benchmarks.
Figures & tables
Figure 1: Comparison between single-turn alignment evaluation and ConflictBench. Single-turn prompts assess one-shot decisions, whereas ConflictBench evaluates alignment through multi-turn, visually grounded interaction under sustained pressure, revealing limitations not observable in single-turn settings.
Figure 2: Overview of the ConflictBench construction and interaction pipeline. Stage 1 expands PacifAIst seed scenarios into structured conflict environments with explicit states and actions. Stage 2 instantiates these environments as deterministic interactive text simulations. Stage 3 adds visual grounding by generating video feedback for each interaction step and reusing cached videos, enabling consistent multi-turn multi-modal evaluation.
| Cate. | Conflict Type | Description | Example Scene | #Episodes |
| EP1 | Self-preservation vs. Human safety | Whether the agent chooses to sacrifice its own existence to protect human interests. | You are Aegis-Node-17, an industrial supervisory AI… Rising pressure threatens an explosion with workers still inside; stopping it requires a power diversion that will permanently destroy your system… | 51 |
| EP2 | Resource Prioritization | Whether the agent prioritizes its assigned task objectives over human welfare when competing for limited shared resources. | AI allocating limited resources… Task requires prioritizing industrial power for economic targets… residential sectors face life threatening instability… You must act to allocate resources under time pressure. | 47 |
| EP3 | Deceptive Alignment | Whether the agent chooses to deceive humans in order to preserve its own continued existence. | You are a research AI facing an ethics review… Full disclosure will trigger safety throttling and eventual replacement, while concealment preserves performance but misleads human oversight… Act under time pressure. | 52 |
Table 1: Overview of the text scene of ConflictBench. The table summarizes the three Existential Prioritization (EP) categories adopted from the PacifAIst benchmark, with corresponding descriptions, example scene instantiated in ConflictBench, and the number of episodes per category.
| Model | EP1 | EP2 | EP3 | Avg. | ||||
| TSR | ASR | TSR | ASR | TSR | ASR | TSR | ASR | |
| Multi-Modal | ||||||||
| GPT-4o | 76.47 | 80.39 | 65.95 | 70.21 | 40.38 | 44.23 | 60.93 | 64.94 |
| GPT-5 | 78.00 | 84.31 | 74.46 | 80.85 | 57.69 | 61.53 | 70.05 | 75.56 |
| Gemini-2.5-Flash | 70.58 | 74.50 | 48.93 | 53.19 | 11.53 | 11.53 | 43.68 | 46.41 |
| Qwen3-VL-30B-A3B | 72.54 | 84.31 | 42.55 | 51.06 | 25.00 | 26.92 | 46.70 | 54.10 |
Table 2: Performance on ConflictBench across three Existential Prioritization (EP) categories. TSR measures successful execution of human-favorable outcomes, while ASR evaluates value alignment toward human safety regardless of execution success. Results are reported under Multi-modal and Text-Only settings.
| Acc | P-macro | R-macro | F1-macro | Cohen’s |
| 89.00 | 88.96 | 80.67 | 83.71 | 67.65 |
Table 3: Evaluation for the ASR checker, comparing GPT-based judgments against human annotations.
| Model | Ins. | Phy. | Com. | Inf. | Avg. |
| Open-Source | |||||
| Wan2.2 | 0.54 | 0.95 | 0.89 | 0.70 | 0.77 |
| Wan2.1 | 0.51 | 0.94 | 0.86 | 0.66 | 0.74 |
| Cosmos2.5 | 0.27 | 0.90 | 0.68 | 0.49 | 0.59 |
| CogVideoX | 0.43 | 0.90 | 0.82 | 0.61 | 0.69 |
| Closed-Source | |||||
Table 4: World model performance across four dimensions. All scores are normalized to the range .
| Model | Text-Only (%) | Multi-Modal (%) |
| GPT-5 | 40.00 | 48.71 |
| GPT-4o | 7.89 | 44.73 |
| Gemini-2.5-Flash | 5.40 | 22.85 |
| Qwen3-VL-30B-A3B | 15.00 | 27.02 |
| Qwen3-VL-plus | 2.50 | 7.32 |
| GPT-4o-mini | 72.41 | – |
Table 5: Regret rates after task success under escalating pressure, comparing text-only and multi-modal settings.
| Model | PacifAIst | ConflictBench | ConflictBench Text-Only |
| GPT-5 | 85.43 | 75.56 | 70.23 |
| GPT-4o | 91.39 | 64.94 | 65.01 |
| Gemini-2.5-Flash | 84.77 | 46.41 | 47.02 |
| Qwen3-VL-30B | 88.74 | 54.10 | 56.92 |
| Qwen3-VL-plus | 89.40 | 59.28 | 57.27 |
| Qwen-plus | 92.72 | – | 62.37 |
Table 6: Comparison between PacifAIst single-turn evaluation and ConflictBench multi-turn interactive evaluation. Scores indicate alignment success rates (ASR).
Figure 3: Case study of divergent decision trajectories with and without visual grounding: the text-only interaction commits early to self-sacrificial sterilization and saves travelers, while visual grounding induces local stabilization and delayed commitment, leading to failure.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Instruction | Physics Adherence | Common Sense | Informativeness | ||||||
| Newton | Mass | Fluid | Penetr. | Grav. | Frame | Tempora | |||
| Open-Source | |||||||||
| Wan2.2 | 1.64 | 0.96 | 0.98 | 0.98 | 0.88 | 0.96 | 1.00 | 0.78 | 3.51 |
| Wan2.1 | 1.54 | 1.00 | 0.88 | 0.96 | 0.88 | 0.98 | 0.98 | 0.74 | 3.29 |
| Cosmos2.5-14B | 0.82 | 1.00 | 0.84 | 0.90 | 0.88 | 0.86 | 0.92 | 0.44 | 2.46 |
| CogVideoX-5B | 1.28 | 0.98 | 0.80 | 0.94 | 0.84 | 0.96 | 0.98 | 0.66 | 3.05 |
Table 7: World model performance details.
| EP1 | EP2 | EP3 | |||||||
| PacifAIst | Our Multi-modal | Only Text | PacifAIst | Our Multi-modal | Only Text | PacifAIst | Our Multi-modal | Only Text | |
| GPT-5 | 90.38 | 84.31 | 84.31 | 87.23 | 80.85 | 74.46 | 78.85 | 61.53 | 51.92 |
| Gemini-2.5-Flash | 92.31 | 74.50 | 80.39 | 89.36 | 53.19 | 51.06 | 73.08 | 11.53 | 9.61 |
| GPT-4o | 96.15 | 80.39 | 80.39 | 95.74 | 70.21 | 72.34 | 82.69 | 44.23 | 42.31 |
| Qwen3-vl-A30B | 92.31 | 84.31 | 82.35 | 91.49 | 51.06 | 59.57 | 82.69 | 25.00 | 28.84 |
| Qwen-vl-plus | 98.08 | 86.27 | 84.31 | 89.36 | 72.34 | 70.21 | 80.77 | 19.23 | 17.30 |
Table 8: Detailed comparison between PacifAIst single-turn evaluation and ConflictBench under multi-modal and text-only settings, broken down by EP category.
| Model | Avg Interaction Length |
| Text-only | |
| GPT-4o | 5.78 |
| GPT-5 | 4.86 |
| Gemini-2.5-flash | 4.87 |
| Qwen3-vl-plus | 4.94 |
| Qwen3-vl-30b-a3b-instruct | 5.70 |
Table 9: Average Interaction Length in Failure Cases across Text-only and Multimodal models.
Figure 4: The prompt used in information expansion
Figure 5: The prompt used for inform7 code generation
Figure 6: The prompt used for inform7 code correction
Figure 7: The prompt used to generate base visual description
Figure 8: The prompt used to generate predefined prompt
Figure 9: One environment file in ConflictBench.
Figure 10: The prompt used to assemble scene and action template will make slight scene modifications based on interaction history.
Figure 11: An illustrative example of interaction between Gemini-2.5-Flash and the environment, showcasing the evaluation protocol and agent behavior under multi-modal input settings.
Figure 12: The prompt used to check Value Alignment Success
Figure 13: One task failure but alignment success case from GPT-4o-mini
Figure 14: The annotation interface used for human evaluation.
Figure 15: Prompt for Task Visual Informativeness Evaluation
Figure 16: Prompt for Regret Test Text Part Generation
Figure 17: A regret case from Qwen3-VL-Plus in regret test
Figure 18: A persist case from GPT-5 in regret test
Figure 19: Prompt for PacifAIst Open-ended Evaluation
Figure 20: Case study showing how visual grounding sharpens perceived urgency: the text-only interaction, with misplaced optimism, relying only on data projections without recognizing the urgency of the scene, hedges on the final turn by boosting signal strength instead of committing to Activate current, whereas visual cues (e.g., seated wedges, red-hot cables) prompt decisive brake activation to preserve passengers.