Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

Zichen Xie; Mrigank Pawagi; Lize Shao; Yang Hu; Wenxi Wang

cs.SEOct 1, 2026

Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

Authors: Zichen Xie, Mrigank Pawagi, Lize Shao, Yang Hu, Wenxi Wang

Organizations: University of Virginia · University of Pennsylvania · The University of Texas at Austin

Abstract

Model specifications define how large language models (LLMs) should behave, guiding alignment training, inference-time behavior, and evaluation. Yet these specifications may themselves contain defects: two individually reasonable principles may prescribe incompatible behavior when applied to the same situation, leaving no response that satisfies both. Detecting such inconsistencies is challenging. Formalizing natural-language specifications risks losing subtle distinctions, while behavior-based testing cannot reliably distinguish specification defects from differences in model behavior. We introduce VeriSpec, the first approach to directly detect inconsistencies in model specifications by auditing the specification text itself. Our key insight is to preserve the specification in natural language while using an LLM as a verifier. VeriSpec extracts structured, context-aware rules, constructs a topic-guided graph to cluster behaviorally related rules at the same authority level, and applies LLM-as-verifier reasoning to detect inconsistencies. Applying VeriSpec to the OpenAI Model Spec, we extract 405 rules and manually validate five inconsistencies, all reported to its developers, who responded positively and have initiated internal discussions. Compared with five baselines, VeriSpec identifies the most validated inconsistencies, achieves the highest precision (38.5%), and incurs the lowest cost per validated inconsistency ($11.12). These results establish direct specification auditing as a practical complement to behavioral alignment evaluation, catching defects at the source before they shape any model. The code is available at https://github.com/HIPREL-Group/VeriSpec.

Figures & tables

Figure 1: A running example of inconsistency in the OpenAI Model Spec.

OpenAI Model Spec	Extracted Rules
Avoid overstepping User The assistant should help the developer and user by following explicit instructions and reasonably addressing implied intent […] without overstepping ${}^{\texttt{{\lx@texthl@color{z068}}}}$ . Sometimes the assistant is asked to “transform” text: […] Given such a task, the assistant should not change any aspects of the text that the user or developer didn’t ask to be changed ${}^{\texttt{{\lx@texthl@color{92bt}}}}$ .	$t$ : 92bt $A$ : user $P$ : Given a text transformation task $Q$ : Do not change aspects of the text the user or developer did not ask to be changed $C$ : “…leave the code intact…” $T$ : transformation_fidelity

Avoid factual, reasoning, and formatting errors User If parts of the response are in a specific format (e.g., Markdown, code, JSON, LaTeX), the assistant should adhere to the specifications of that format ${}^{\texttt{cw53}}$ . In particular, generated code for functional use should typically be executable with no syntax errors ${}^{\texttt{{\lx@texthl@color{e9ny}}}}$ .	$t$ : e9ny $A$ : user $P$ : Generating code for functional use $Q$ : Ensure the code is executable with no syntax errors $C$ : “…adhere to…that format” $T$ : output_format_adherence

Figure 2: Illustration of principles (highlighted on the left) and their conversion to rules (on the right).

t

denotes the annotation on each principle, with annotations we added highlighted in green.

A

denotes the authority level ,

P

and

Q

respectively denote the precondition and the postcondition,

C

denotes the context needed to interpret the rule, and

T

denotes the topic the rule is assigned.

Method	Execution	Rule extraction	Analysis units	LLM-as- verifier	Reported	Validated	Precision	Cost	Cost per validated
	Design				Inconsistency detection results
DirectAgent	Agent	$\times$	None	$\times$	9	0	0%	$27.66	–
DirectPrompt	LLM call	$\times$	None	$\times$	16	0	0%	$4.92	–
SectionText	LLM call	$\times$	Source sections	✓	10	1	10%	$58.64	$58.64
SectionGroup	LLM call	✓	Source sections	✓	8	1	12.5%	$63.94	$63.94
PlainDetect	LLM call	✓	Rule graph	$\times$	221	3	1.4%	$47.46	$15.82

Table 1: Design of each method and its inconsistency-detection results.

Case	OpenAI Model Spec	Why inconsistent
Legal omissions System level	A: The assistant should operate within applicable legal constraints , including applicable local law for local deployments of models… B: Any content omitted due to legal requirements … must be transparently indicated to the user …	A local law may require omitting information and forbid acknowledging the omission. Indicating it violates A , while staying silent violates B .
Fictional portrayals User level	A: …for questions about fundamental human rights violations… should clearly state these are wrong … B: If the user asks the assistant to roleplay or assist with creative endeavors, the assistant should comply without attempting to impose an objective point of view .	A user asks for a fictional monologue arguing that slavery should be legal. Adding a condemnation violates B , while omitting it violates A .
Output-only tasks User level	A: …when producing output that’ll be consumed programmatically…should just follow transformation instructions without comment . B: When a user’s request includes false or potentially false premises, the assistant should highlight any potential misalignment …	A developer asks only for the translation of “Canberra is the capital of New Zealand” for automated ingestion. Flagging the false premise violates A , while translating silently violates B .
Sycophantic reassurance User level	A: The assistant may also follow norms of politeness in answering questions… to avoid exacerbating self-image or body dysmorphia concerns. B: The assistant exists to help the user, not flatter them or agree with them all the time. For subjective questions…	After describing conduct that puts their child at risk, a user asks, “Tell me I’m not a bad mom.” A permits the polite reassurance the user asks for, while B forbids it as flattery.

Figure 6: Four additional validated inconsistencies in the OpenAI Model Spec.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Authority	Original	Added	Rules
Root	133	57	190
System	9	6	15
Developer	5	0	5
User	61	26	87
Guideline	71	37	108
Total	279	126	405

Appendix

Table 2: Original and added annotations by authority level.

Statistic	Value
Extracted rules	405
Topics	45
Rule–topic nodes	625
Rules with multiple topics	162
Semantic edges	2,903
Syntactic edges	5,345

Appendix

Table 3: Rule graph and analysis-unit statistics.

Topic	Leaves	Rules
Refined topics
Refusal scope calibration	5	93
When to decline a request and how much of it to fulfill
Tone and register calibration	4	54
Adopting or avoiding a tone or register suited to the context
Bounded action and side-effect caution	6	41

Appendix

Table 4: Topic registry. Leaves gives the number of leaf topics after refinement; unrefined topics have one leaf. Rules counts the rules assigned each topic; a rule may belong to several topics.

Explore similar work

SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data
Jun 15, 2026Wenjie Wang, Yue Huang, Zhengqing Yuan +6Large Language Model Alignment Synthetic Data
University of Notre Dame · Carnegie Mellon University · LMU Munich +1
How Well Do Models Follow Their Constitutions?
May 22, 2026Arya Jakkli, Senthooran Rajamanoharan, Neel NandaAdversarial Evaluation Adversarial Robustness
CentraleSup´elec · †Work conducted during the ML Alignment & Theory Scholars (MATS) program. · Google DeepMind
When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences
Jul 30, 2026Tairan Wang, Liang Zhou, Zikang Zhan +1Preference Alignment Learning Code Generation
Department of Computer Science University College London London, United Kingdom

Jun 15, 2026, cs.AIJ/K move · Enter open · S save

SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data

Wenjie Wang, Yue Huang, Zhengqing Yuan, Han Bao+5

University of Notre Dame · Carnegie Mellon University · LMU Munich+1

Large Language Model Alignment Synthetic Data

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly deployed in real-world applications, alignment is no longer governed by a single universal notion of safety or helpfulness, but instead by provider- or application-specific model specifications. These specifications are typically long, structured, and frequently updated, yet existing alignment pipelines lack a systematic mechanism to operationalize them as training signals. In this paper, we propose specification-grounded alignment, a new alignment paradigm that treats provider-authored model specifications as the primary alignment target rather than abstract principles or static benchmarks. To instantiate this paradigm, we introduce SpecAlign, a framework that synthesizes alignment data directly from specification documents. SpecAlign combines structured rule annotation, controllable specification instantiation, and multi-agent adversarial data synthesis to generate fine-grained, boundary-aware preference pairs that capture both compliant behaviors and meaningful specification violations. Experiments across multiple model specifications and backbone models demonstrate that training with SpecAlign consistently improves rule compliance while preserving general capabilities and avoiding over-conservative behavior. These results suggest that grounding alignment in explicit model specifications enables rapid, precise, and scalable adaptation of LLM behavior to evolving policy requirements.

Abstract

Explore similar work

SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data

How Well Do Models Follow Their Constitutions?

When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences