RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

Bohan Yu; Shi-Yang Li; Pengfei Cao; Jun Zhao; Kang Liu

cs.CLAug 27, 2026

RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

Authors: Bohan Yu, Shi-Yang Li, Pengfei Cao, Jun Zhao, Kang Liu

Organizations: School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Abstract

Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.

Figures & tables

Figure 1: Progressive rule construction: an example of transforming a WikiHow Koupaee and Wang (2018) Meta Rule into a complex rule through five-round semantic enhancement.

Dataset	Corpus-Derived Rules	Semantic Rule Typing	Rule-Centered Scenario QA with Process-Level Scoring	Multi-Source / Cross-Domain Composition
Instruction-Following Benchmarks
IFEval Zhou et al. (2023)	✗	✗	✗	✗
WizardLM Xu et al. (2024)	✗	✗	✗	✗
InfoBench Qin et al. (2024)	✗	✗	✗	✗
FollowBench Jiang et al. (2024)	✗	✗	✗	✗
ComplexBench Wen et al. (2024)	✗	✗	✗	✗

Table 1: Left: comparison of RuleWeaver with instruction-following and logical reasoning benchmarks across core evaluation dimensions. ✓ : support;

\ominus

: partial support; ✗ : unsupported. Right: distribution of the 200 complex rule groups and 96 scenario QA instances. Detailed rule-source and QA-type distributions are provided in Appendices A.1 and A.2 , respectively, with question-type descriptions in Appendix B.4 .

Figure 2: Overview of RuleWeaver: from real-world corpora to progressively augmented complex rules, rule-centered scenario QA, and process-level scoring.

Model	Same-Source			Cross-Source
Model	Recall	Precision	Rubric	Recall	Precision	Rubric
Claude-Opus-4.6	70.42	70.41	48.06	62.50 $\downarrow$ 7.92	73.74 $\uparrow$ 3.33	50.27 $\uparrow$ 2.21
Claude-Sonnet-4.6	71.67	63.15	43.96	59.58 $\downarrow$ 12.09	54.74 $\downarrow$ 8.41	36.56 $\downarrow$ 7.40
GPT-5.4	61.67	71.31	48.21	48.33 $\downarrow$ 13.34	78.64 $\uparrow$ 7.33	42.58 $\downarrow$ 5.63
GPT-5.5	69.17	74.58	53.83	52.08 $\downarrow$ 17.09	69.48 $\downarrow$ 5.10	46.40 $\downarrow$ 7.43
Gemini-3.1-Pro-Preview	60.83	58.44	31.77	45.83 $\downarrow$ 15.00	72.95 $\uparrow$ 14.51	27.81 $\downarrow$ 3.96

Table 2: Left: Main results under the 200 complex-rule setting. Rule recall and precision are percentages, rubric score is on a 0–100 scale, arrows indicate cross-source changes relative to same-source scores, and bold marks the best result in each column. Right: Radar summaries of semantic correct-application performance (top) and seven question-type performance (bottom).

Rubric Dimension	Max	Mean	Norm.
Question Understanding	10	5.28	52.8%
Issue Decomposition	10	2.84	28.4%
Citation Format Compliance	8	6.62	82.8%
Rule-Grounded Reasoning	17	6.44	37.9%
Dependency Chain Alignment	18	2.55	14.2%
Exception/Conflict Handling	12	3.65	30.4%

Table 3: Average rubric-dimension scores under the 200-rule setting. “Norm.” denotes the mean score normalized by each dimension’s maximum points.

Panel of Figure 3 — Figure 3: Top : gray points show individual QA instances, and colored markers show mean rubric scores with standard-error bars across scenario complexity levels. Bottom : performance under only-gold, 100-rule, and 200-rule settings for three selected models.

Error Type	All		Full Recall
Error Type	Count	Rate	Count	Rate
Rule selection	762	72.2%	–	–
Cited-rule application	580	54.9%	191	65.6%
Rule interaction	734	69.5%	166	57.0%
Multi-step integration	916	86.7%	218	74.9%
Final synthesis	678	64.2%	193	66.3%

Table 4: Error-type prevalence over all model–QA answers and the subset with full rule recall.

GPT-5.5 Audit	Additional Cross-Model Audits
Judge–human agreement Spearman’s $\rho$ : 0.808 Pearson’s $r$ : 0.795	Audit outputs	Judge–human $\rho$	Judge–human $r$	Human–human $\rho$	Human–human $r$
	Claude-Opus-4.6	0.755	0.733	0.977	0.969
	Claude-Sonnet-4.6	0.842	0.826	0.945	0.952
	DeepSeek-V4-Pro	0.795	0.635	0.947	0.901
	GPT-5.4	0.834	0.762	0.935	0.902

Table 5: Agreement on total rubric scores. Each audit contains 48 answers.

\rho

: Spearman;

r

: Pearson.

Judge	Setting	Spearman $\rho$	Pearson $r$
Grok-4.3	Same-source	0.882	0.878
	Cross-source	0.918	0.860
	Overall	0.918	0.893
Gemini-3-Flash	Same-source	0.845	0.818
	Cross-source	0.873	0.793
	Overall	0.882	0.846

Table 6: Correlation of alternative-judge and DeepSeek-V4-Flash model-level mean rubric scores.

Source Corpus	Rule Domain	Groups	Variants	ABSTRACT	ADDITIVE	NEGATE	EXCEPTION	CONFLICT	IRONCLAD
GovReport	Policy reports	50	200	19	16	15	50	50	50
WikiHow	Procedural guides	50	200	21	16	13	50	50	50
CUAD	Contractual clauses	50	200	15	16	19	50	50	50
BookSum	Narrative contexts	50	200	23	15	12	50	50	50
Total	–	200	800	78	63	59	200	200	200

Table 7: Statistics of the complex final rule set.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Source Meta Rule	Augmented Information	Augmentation Definition
ABSTRACT	If a third-party claim arises from an intellectual property infringement, then the indemnifying party should defend the indemnified party.	The condition is expanded with the broader alternative “or any other similar legal wrong.”	This abstracts the concrete trigger, intellectual property infringement , into a higher-level legal category, requiring models to match the rule beyond surface lexical overlap.
ADDITIVE	If an item is controlled on the applicable export control list, then the exporter should submit a license application to the relevant regulatory authority.	Added condition: “the item is not explicitly pre-approved for export under a special government-to-government agreement.” Added outcome: “not proceed with the export without first obtaining a binding advisory opinion.”	The new information does not reverse or suspend the export-control rule; it supplements the original scope with an additional eligibility condition and an additional compliance requirement.
NEGATE	If a site scores above 28.5 on the applicable hazard assessment metric, then the site is eligible for placement on the relevant national priority list.	The rule becomes: if the site “does not score above 28.5,” then it is “not eligible for placement.”	Both the IF condition and THEN outcome are reversed, testing whether a model tracks the changed semantic direction instead of applying the original eligibility rule.
EXCEPTION	If a party wishes to assign the Agreement, then that party should obtain prior written consent from the other party.	Added exception: “except where the assignment is to an affiliate.”	The added affiliate case restricts the general consent requirement by specifying a special circumstance in which the rule should not be directly applied.
CONFLICT	If the service provider uses commercially reasonable efforts to correct a Program Error, then failure to correct it within the specified period shall not be deemed a breach.	Added competing condition: if the failure results from “willful misconduct or fraud,” then the failure “shall be deemed a breach” and triggers penalties.	The added rule creates an incompatible outcome with the original no-breach conclusion, requiring conflict detection and resolution when both conditions are relevant.
IRONCLAD	If a federal agency intends to dispose of property, then the agency should assess the need for environmental cleanup.	Added mandatory prohibition: the agency “must not transfer the property to any non-federal entity before the assessment is complete.”	The new item is framed as a strict, non-optional constraint that must be preserved even when other outcomes, waivers, or procedural alternatives are considered.

Appendix

Table 9: Representative examples of the six semantic augmentation types selected from the organized final rule set.

Source	Meta Rule	IF Condition	THEN Outcome
BookSum	If a person is excommunicated from the Church, then they are banished from entering Rome.	A person is excommunicated from the Church	They are banished from entering Rome
CUAD	If the manufacturer intends to change a manufacturing process, then the manufacturer should obtain written approval from the client via a formal change order.	The manufacturer intends to change a manufacturing process	The manufacturer should obtain written approval from the client via a formal change order
GovReport	If the postal service provider proposes to eliminate a standard delivery day, then the provider should request an advisory opinion from the relevant regulatory oversight body.	The postal service provider proposes to eliminate a standard delivery day	The provider should request an advisory opinion from the relevant regulatory oversight body
WikiHow	If a tank lacks a bubbler, then the water becomes acidic.	A tank lacks a bubbler	The water becomes acidic

Appendix

Table 10: Examples of Meta Rules from Four Source Datasets.

Panel of Figure 4 — Figure 4: Complete five-round progressive augmentation cases, with final semantic enhancements of EXCEPTION , CONFLICT , and IRONCLAD , respectively.

Question Type	Expected Output	Reasoning Focus
Multi-Issue Synthesis	A synthesis of multiple parallel and relevant points that jointly resolve the question.	Integrating several sub-issues instead of producing a single isolated conclusion.
Definitive Conclusion	One unique, explicit conclusion derived from the scenario, question, and applicable rules.	Checking all relevant rule conditions and deriving a determinate final answer.
Event Prediction	The rule-governed next event or consequence if the current scenario proceeds.	Projecting consequences beyond the immediately stated facts while remaining grounded in the rules.
Priority Arbitration	The controlling rule-governed path and the resulting choice or conclusion.	Resolving rule interactions involving conflicts, competing implications, or priority ordering.
Special-Case Judgment	The outcome for a boundary case or special circumstance.	Determining how a qualifying circumstance changes rule applicability or the final outcome.
What-If Revision	The revised rule-governed outcome after a specified change to scenario conditions.	Updating the reasoning chain under a counterfactual modification rather than reusing the original conclusion.

Appendix

Table 11: Scenario QA question types used in RuleWeaver.

Rubric Dimension	Max	What It Measures
Question Understanding	10	Whether the answer addresses the requested scenario question and identifies the correct target outcome or decision.
Issue Decomposition	10	Whether the answer breaks the scenario into the necessary sub-issues and rule-application steps before reaching the final conclusion.
Citation Format Compliance	8	Whether cited rules use canonical rule identifiers, appear at the point of application, and avoid invented or malformed rule references.
Rule-Grounded Reasoning	17	Whether cited rules are correctly applied to concrete scenario facts and support the stated intermediate or final conclusions.
Dependency Chain Alignment	18	Whether the answer follows the required rule logic chain and preserves dependencies among intermediate conclusions.
Exception/Conflict Handling	12	Whether the answer detects and resolves exceptions, blockers, conflicts, or priority relations that affect rule applicability.

Appendix

Table 12: Scenario QA answer-judgment rubric dimensions. The maximum scores sum to 100 points.

Figure 5: Model-level normalized rubric-dimension scores under the 200-rule setting. Each cell reports the mean percentage score for one model and one rubric dimension.

	Same-Source			Cross-Source
Model	Mean	95% CI	$\Delta$	Mean	95% CI	$\Delta$
GPT-5.5	53.88	[44.42, 63.33]	+0.05	46.49	[38.21, 54.77]	+0.09
Claude-Opus-4.6	48.21	[38.65, 57.77]	+0.15	50.43	[40.81, 60.04]	+0.16
GPT-5.4	48.39	[38.40, 58.38]	+0.18	42.67	[33.83, 51.50]	+0.09
Claude-Sonnet-4.6	44.11	[33.90, 54.31]	+0.15	36.69	[27.38, 46.00]	+0.13
Qwen3.5-Plus	38.03	[29.00, 47.06]	+0.28	34.47	[26.65, 42.29]	+0.32

Appendix

Table 13: Bootstrap mean rubric scores and 95% CIs.

\Delta

is the bootstrap mean minus the corresponding point estimate in Table 2 .

Same-source comparison	Diff.	95% CI
GPT-5.5 – Doubao-Seed-2.0-Pro	26.36	[16.62, 36.10]
GPT-5.5 – MiniMax-M2.7	24.08	[12.75, 35.40]
GPT-5.5 – Gemini-3.1-Pro-Preview	21.96	[11.15, 32.77]
GPT-5.5 – Kimi-K2.6	21.46	[10.81, 32.10]
GPT-5.5 – DeepSeek-V4-Pro	19.19	[8.38, 30.00]
GPT-5.5 – GLM-5	16.48	[7.42, 25.54]

Appendix

Table 14: Paired bootstrap rubric-score differences (leader minus comparator) and 95% CIs, based on 20,000 resamples of the 48 aligned instances in each setting.

Model	ABSTRACT	ADDITIVE	NEGATE	EXCEPTION	CONFLICT	IRONCLAD
Claude-Opus-4.6	50.55	48.02	46.08	40.24	56.18	51.52
Claude-Sonnet-4.6	39.45	38.24	39.06	42.68	41.57	39.39
Deepseek-V4-Pro	37.53	32.58	31.50	36.59	41.57	37.88
Qwen3.5-Plus	41.23	37.25	34.39	32.93	40.45	42.42
GPT-5.4	44.11	41.93	40.85	36.59	49.44	46.97
GPT-5.5	46.99	43.34	45.12	40.24	51.69	51.52

Appendix

Table 15: Semantic correct-application rates by enhancement type under the 200-rule setting. Scores are percentages, and bold values mark the best model for each semantic type.

Model	Multi-Issue Synthesis	Definitive Conclusion	Event Prediction	Priority Arbitration	Special-Case Judgment	What-If Revision	Action Prescription
Claude-Opus-4.6	45.24	55.64	45.00	52.43	51.73	50.17	43.00
Claude-Sonnet-4.6	39.18	40.14	40.21	24.79	48.53	44.33	46.70
GPT-5.4	38.94	53.36	48.93	42.79	53.33	47.17	29.90
GPT-5.5	40.94	54.86	57.50	38.79	60.73	52.67	45.60
Gemini-3.1-Pro-Preview	27.18	52.64	33.79	13.21	30.07	26.50	23.40
Deepseek-V4-Pro	30.06	39.79	34.07	24.64	27.93	35.17	29.60

Appendix

Table 16: Rubric scores by scenario question type under the 200-rule setting. Scores are on a 0–100 scale, and bold values mark the best model for each question type.

Group	Metric	Spearman $\rho$
Complexity	Reasoning Depth	$-0.29$
	Dependency Load	$-0.22$
	Branching	$-0.01$
Rule Selection	Rule Recall	$0.20$
Rule Selection	Rule Precision	$0.66$

Appendix

Table 17: Spearman rank correlations between QA-level factors and mean rubric score. Rule recall and precision are averaged over the 11 evaluated models for each QA.

Figure 6: Recall missing. The model misses a necessary aggregation rule and therefore treats determinate obligations as merely contingent. Red text highlights the erroneous part of the model output.

Figure 7: Rule misuse. The model identifies the relevant premise but reverses the exception condition, producing the opposite legal consequence. Red text highlights the erroneous part of the model output.

Figure 8: Dependency incorrect. The model applies rules locally but fails to propagate an intermediate conclusion through the required rule-dependency chain. Red text highlights the erroneous part of the model output.

Figure 9: A zero-score case for Question Understanding .

Figure 10: A zero-score case for Issue Decomposition .

Figure 11: A zero-score case for Citation Format Compliance .

Figure 12: A zero-score case for Rule-Grounded Reasoning .

Figure 13: A zero-score case for Dependency-Chain Alignment .

Figure 14: A zero-score case for Exception/Conflict Handling .

Figure 15: A zero-score case for Intermediate Conclusions .

Figure 16: A zero-score case for Final Answer Consistency .

Figure 17: A zero-score case for No External Facts .

Figure 18: Meta rule extraction prompt.

Figure 19: IF-THEN filtering prompt.

Figure 20: Meta rule quality filtering prompt.

Figure 21: Rule augmentation prompt.

Figure 22: Dependency plan construction prompt.

Figure 23: Sub-scenario generation prompt.

Figure 24: Final synthesis prompt (part 1).

Figure 25: Final synthesis prompt (part 2).

Figure 26: Scenario question answering quality judgment prompt (part 1).

Figure 27: Scenario question answering quality judgment prompt (part 2).

Figure 28: Question answering template.

Figure 29: Answer judge template.

Explore similar work

Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models
Aug 24, 2026Bohan Yu, Pengfei Cao, Chen Han +7Knowledge Augmentation for Language Models LLM Evaluation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +3
PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces
Sep 17, 2026Pyrros Koussios, Benjamin Jäger, John Hua Yao +3LLM Evaluation Temporal Reasoning in Language Models
ETH Zürich · Stanford University
DeonticBench: A Benchmark for Reasoning over Rules
Apr 6, 2026Guangyao Dou, Luis Brena, Akhil Deo +4Logical Reasoning Legal Reasoning
Johns Hopkins University · Télécom Paris, Institut Polytechnique de Paris

Aug 24, 2026J/K move · Enter open · S save

Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models

Bohan Yu, Pengfei Cao, Chen Han, Chenxi Zhou+6

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences+3

Knowledge Augmentation for Language Models LLM Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) excel at text understanding and generation, yet still struggle to reliably understand and apply externally provided procedural rules at scale. To evaluate this capability, we introduce RuleWorld, a large-scale benchmark that reformulates rules as globally reusable abstract units rather than instance-specific facts. In RuleWorld, several scenarios, including single-rule, parallel multi-rule, and multi-hop reasoning, are settled for comprehensive evaluation. We further propose DynaRule, an end-to-end framework that injects the given rules into the KV cache and turns retrieval into an internal, learnable, step-wise process. Specifically, DynaRule employs Stacked Step-Level Attention Training with a special <search> token to enable dynamic rule re-attention and updating during inference. In this way, the model can re-attend to the most relevant rules at each step, dynamically replacing outdated ones to support more stable multi-step reasoning. Experiments on RuleWorld show that existing LLMs face challenges under large rule pools, while DynaRule improves average QA accuracy by up to 19 points and achieves over 85% Recall@1 at 10K rules, outperforming strong baselines by large margins. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/Beyond-Factual-Knowledge.

Abstract

Explore similar work

Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models

PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces

DeonticBench: A Benchmark for Reasoning over Rules