Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models

Bohan Yu; Pengfei Cao; Chen Han; Chenxi Zhou; Zhiheng Zhang; Zhiyang Xie; Wenhao Teng; Xiangwen Liao

cs.CLAug 24, 2026

Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models

Authors: Bohan Yu, Pengfei Cao, Chen Han, Chenxi Zhou, Zhiheng Zhang, Zhiyang Xie, Wenhao Teng, Xiangwen Liao, +2 more

Organizations: School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, Chinese Academy of Sciences · Department of Gastrointestinal Surgery, Fujian Provincial Cancer Hospital · College of Computer and Data Science, Fuzhou University

Abstract

Large language models (LLMs) excel at text understanding and generation, yet still struggle to reliably understand and apply externally provided procedural rules at scale. To evaluate this capability, we introduce RuleWorld, a large-scale benchmark that reformulates rules as globally reusable abstract units rather than instance-specific facts. In RuleWorld, several scenarios, including single-rule, parallel multi-rule, and multi-hop reasoning, are settled for comprehensive evaluation. We further propose DynaRule, an end-to-end framework that injects the given rules into the KV cache and turns retrieval into an internal, learnable, step-wise process. Specifically, DynaRule employs Stacked Step-Level Attention Training with a special <search> token to enable dynamic rule re-attention and updating during inference. In this way, the model can re-attend to the most relevant rules at each step, dynamically replacing outdated ones to support more stable multi-step reasoning. Experiments on RuleWorld show that existing LLMs face challenges under large rule pools, while DynaRule improves average QA accuracy by up to 19 points and achieves over 85% Recall@1 at 10K rules, outperforming strong baselines by large margins. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/Beyond-Factual-Knowledge.

Figures & tables

Figure 1: Illustration of rule examples and RuleWorld QA types. Rules from different types may interact, and the benchmark includes three QA types: single-rule, parallel multi-rule, and multi-hop reasoning.

Panel of Table 1 — Table 1: Left : Comparison of rule-guided reasoning benchmarks across five dimensions. ✓ = Yes, ✗ = No, = Partial. Right : Distribution of 3.37M QA instances over 11 sub-tasks; the outer ring shows seven rule sub-types, with thick separators marking major types. Suffixes denote gold rule counts or hops.

Figure 2: Overview of Stacked Step-Level Attention Training ( left ) and dynamic rule attention at inference ( right ). Training builds a shared candidate pool $\tilde{\mathcal{C}}^{(l)}$ from sequence-level relevance between $\tilde{\mathbf{Q}}^{l}$ and injected rule keys $\tilde{\mathbf{K}}^{l}$ , and learns step-level retrieval with question and <search> tokens. At inference, top- $K$ rules are selected at prefill and updated through re-attention triggered by <search> tokens. Values are omitted for clarity.

Method	First-order Logic						Natural Language						Avg.
Method	Single	PM (2–4)	PM (5–8)	MH-2	MH-3	MH-4	Single	PM (2–4)	PM (5–8)	MH-2	MH-3	MH-4	Avg.
Rule Num = 100
Prompting	0.8200	0.7389	0.4917	0.5000	0.0600	0.0400	0.8200	0.8194	0.5684	0.3400	0.1000	0.1200	0.4515
KBLaM	0.9800	0.9939	0.9871	1.0000	0.9000	0.9400	1.0000	0.9917	0.9863	0.9600	0.4800	0.7200	0.9116
SR-KI	1.0000	0.9883	0.9750	0.9800	0.9200	0.9000	1.0000	0.9750	0.9583	0.9600	0.5400	0.5600	0.8964
DynaRule	1.0000	0.9917	0.9863	0.9600	0.9300	0.9500	1.0000	0.9967	0.9825	0.9200	0.9600	0.7000	0.9481

Table 2: Exact match accuracy on RuleWorld with Qwen2.5-7B-Instruct. Single : Single-Rule QA. PM : Parallel Multi-Rule QA, where columns 2–4 and 5–8 report mean exact match over the corresponding rule counts. MH : Multi-Hop Rule QA, where the numeric suffix denotes hop count.

\text{RAG}_{\text{dense}}

and

\text{RAG}_{\text{bm25}}

use Qwen3-Embedding-8B and BM25 retrievers, respectively;

\text{RAG}_{\text{hybrid}}

uses RRF 5 with a fusion constant of 60. Detailed results for additional rule sizes are reported in Tables 8 and 9 .

Figure 3: Confidence layer (CL) identification (FOL/NL), with shading showing attention entropy standard deviation ( left ). Per-layer retention analysis with correct rules injected into a single target layer ( right ).

Panel of Figure 4 — Figure 4: Retrieval performance (FOL) across QA types and reasoning steps, with step-wise attention visualization.

Method	First-order Logic			Natural Language			Avg.
Method	S-Rule	PM-Rule	MH-Rule	S-Rule	PM-Rule	MH-Rule	Avg.
Qwen2.5-14B-Instruct	0.7200	0.4486	0.1933	0.6200	0.4031	0.1933	0.4297
Qwen2.5-32B-Instruct	0.8400	0.5038	0.2067	0.7400	0.5357	0.2200	0.5077
Qwen2.5-72B-Instruct	0.7200	0.5945	0.2533	0.7600	0.6698	0.1867	0.5307
Llama-3.1-70B-Instruct	0.8600	0.5571	0.2067	0.9600	0.5633	0.1467	0.5490
Llama-3.3-70B-Instruct	0.8600	0.4555	0.1667	0.9600	0.4686	0.1867	0.5162

Table 3: Partial prompting comparison results on RuleWorld under 1000 rules. S-Rule : Single-Rule QA; PM-Rule : Parallel Multi-Rule QA; MH-Rule : Multi-Hop Rule QA. Detailed results are provided in Appendix C.2 .

Panel of Figure 5 — Figure 5: Rule generalization, iterative retrieval, and gold-rule application analyses.

Method	First-order Logic			Natural Language
Method	Recall@100	Recall@10	Recall@1	Recall@100	Recall@10	Recall@1
Rule Num = 100
Dense	-	0.5325	0.4448	-	0.6646	0.5414
BM25	-	0.6901	0.4375	-	0.5869	0.5292
Hybrid	-	0.6364	0.4793	-	0.5972	0.5128
KBLaM	-	0.5151	0.4122	-	0.5278	0.4025

Table 4: Retrieval performance on FOL and NL rules, with extended results in Table 10 .

Panel of Figure 6 — Figure 6: Effectiveness and efficiency evaluation.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Major Type	Relation Type	All Rules	Train/Eval Rules
Attribute	Entity2Attr	1,877,380	74,888
Attribute	AttrChange2Attr	4,021	3,772
Action	Action2Env	100	100
	Action2Attr	200	200
	Action2State	200	200
Environment	Env2State	100	100

Appendix

Table 5: Count distribution of full rule set and the subset used for training and evaluation.

QA Type	Sub-task Type	All QA Count	Train QA Count
Single-Rule QA	single-rule	50,000	25,000
Parallel Multi-Rule QA	multi-rule-2	154,167	6,702
	multi-rule-3	381,116	8,696
	multi-rule-4	839,755	12,384
	multi-rule-5	886,455	10,924
	multi-rule-6	618,577	6,824

Appendix

Table 6: Count distributions for full and training QA instances across RuleWorld QA types.

Figure 7: Illustration of the construction pipelines for the RuleWorld rule set ( left ) and the RuleWorld QA benchmark ( right ), including single-rule, parallel multi-rule, and multi-hop QA generation. The modular process integrates curated vocabularies, logic constraints, programmatic instantiation, and task-specific generation workflows. In the rule set, Attr, Env, Act, and State denote the four major types Attribute , Environment , Action , and State , and the directed arrows between them indicate their interaction relations.

Major Type	Relation Type	FOL	NL
Attribute	Entity2Attr	Tiny_cat(A) $\Rightarrow$ Has(Strong_horn, 4)	If A is a tiny cat, it has 4 strong horns.
Attribute	AttrChange2Attr	Lose_Strong_fin(A, 1) $\Rightarrow$ Drop_Mineral_fur(A, 2)	If A loses 1 strong fin, A will drop 2 mineral furs.
Action	Action2Env	Chase(A, B) $\Rightarrow$ Enter(B, Bridge)	If A chases B, B will enter bridge.
	Action2Attr	Chase(A, B) $\Rightarrow$ Drop_Floral_paw(B, 1)	If A chases B, B will drop 1 floral paw.
	Action2Attr	Chase(A, B) $\Rightarrow$ Get_Stormy_ember(A, 2)	If A chases B, A will get 2 stormy embers.
	Action2State	Scratch(A, B) $\Rightarrow$ Completely_numb(A)	If A scratches B, A will be completely numb.

Appendix

Table 7: Examples of abstract procedural rules represented in FOL and NL formats.

Panel of Figure 8 — Figure 8: Extended confidence layer (CL) identification under FOL and NL rule representations. Shaded regions denote entropy standard deviation.

Figure 9: Retrieval performance in terms of Recall@100, Recall@10, and Recall@1 across different reasoning steps and numbers of injected rules under both FOL and NL representations.

Method	Single-Rule QA	Parallel Multi-Rule QA							Multi-Hop Rule QA			Avg.
Method	single-rule	multi-rule-2	multi-rule-3	multi-rule-4	multi-rule-5	multi-rule-6	multi-rule-7	multi-rule-8	multi-hop-2	multi-hop-3	multi-hop-4	Avg.
Rule Num = 100
Prompting	0.8200	0.7000	0.7367	0.7800	0.6983	0.5183	0.4650	0.2850	0.5000	0.0600	0.0400	0.5094
KBLaM	0.9800	1.0000	0.9817	1.0000	1.0000	0.9833	0.9750	0.9900	1.0000	0.9000	0.9400	0.9773
SR-KI	1.0000	1.0000	0.9950	0.9700	0.9950	0.9550	0.9750	0.9750	0.9800	0.9200	0.9000	0.9695
DynaRule	1.0000	1.0000	0.9950	0.9800	0.9950	0.9850	0.9700	0.9950	0.9600	0.9300	0.9500	0.9782

Appendix

Table 8: Exact match accuracy on RuleWorld ( FOL ), using Qwen2.5-7B-Instruct as the base model.

\text{RAG}_{\text{dense}}

and

\text{RAG}_{\text{bm25}}

use Qwen3-Embedding-8B and BM25 retrievers, respectively;

\text{RAG}_{\text{hybrid}}

uses RRF with the fusion constant set to 60 to fuse BM25 and dense retrieval results; suffixes denote the gold parallel rule count or hop number.

Method	Single-Rule QA	Parallel Multi-Rule QA							Multi-Hop Rule QA			Avg.
Method	single-rule	multi-rule-2	multi-rule-3	multi-rule-4	multi-rule-5	multi-rule-6	multi-rule-7	multi-rule-8	multi-hop-2	multi-hop-3	multi-hop-4	Avg.
Rule Num = 100
Prompting	0.8200	0.8600	0.7833	0.8150	0.7617	0.5217	0.5350	0.4550	0.3400	0.1000	0.1200	0.5556
KBLaM	1.0000	1.0000	0.9950	0.9800	0.9950	0.9850	0.9700	0.9950	0.9600	0.4800	0.7200	0.9164
SR-KI	1.0000	0.9800	1.0000	0.9450	0.9950	0.9583	0.9200	0.9600	0.9600	0.5400	0.5600	0.8926
DynaRule	1.0000	1.0000	0.9950	0.9950	0.9900	0.9800	0.9750	0.9850	0.9200	0.9600	0.7000	0.9545

Appendix

Table 9: Exact match accuracy on RuleWorld ( NL ), using Qwen2.5-7B-Instruct as the base model.

\text{RAG}_{\text{dense}}

and

\text{RAG}_{\text{bm25}}

use Qwen3-Embedding-8B and BM25 retrievers, respectively;

\text{RAG}_{\text{hybrid}}

uses RRF with the fusion constant set to 60 to fuse BM25 and dense retrieval results; suffixes denote the gold parallel rule count or hop number.

Method	First-order Logic			Natural Language
Method	Recall@100	Recall@10	Recall@1	Recall@100	Recall@10	Recall@1
Rule Num = 100
Dense	-	0.5325	0.4448	-	0.6646	0.5414
BM25	-	0.6901	0.4375	-	0.5869	0.5292
Hybrid	-	0.6364	0.4793	-	0.5972	0.5128
KBLaM	-	0.5151	0.4122	-	0.5278	0.4025

Appendix

Table 10: Retrieval performance on FOL and NL rules. Dense uses Qwen3-Embedding-8B; Hybrid uses RRF (fusion constant 60) to combine BM25 and dense retrieval; KBLaM, SR-KI, and DynaRule use confidence-layer retrieval. For Recall@1,

K

is set to the number of gold rules per instance or step.

Panel of Figure 10 — Figure 10: Heatmap of attention weights across reasoning steps. Required rules are ordered for clarity.

Method	100	500	1K	2K	5K	10K
Prompting	1.6 / $\pm$ 1.9	2.4 / $\pm$ 3.0	2.0 / $\pm$ 2.5	4.1 / $\pm$ 5.1	–	–
$\text{RAG}_{\text{dense}}$	3.1 / $\pm$ 3.8	3.6 / $\pm$ 4.5	1.5 / $\pm$ 1.9	1.7 / $\pm$ 2.1	2.3 / $\pm$ 2.9	0.8 / $\pm$ 1.0
$\text{RAG}_{\text{bm25}}$	3.1 / $\pm$ 3.8	1.1 / $\pm$ 1.4	1.0 / $\pm$ 1.2	1.5 / $\pm$ 1.9	2.7 / $\pm$ 3.4	3.2 / $\pm$ 4.0
$\text{RAG}_{\text{hybrid}}$	2.0 / $\pm$ 2.5	1.1 / $\pm$ 1.4	2.3 / $\pm$ 2.9	1.4 / $\pm$ 1.7	3.5 / $\pm$ 4.3	2.3 / $\pm$ 2.9
KBLaM	2.1 / $\pm$ 2.6	1.8 / $\pm$ 2.2	2.9 / $\pm$ 3.6	3.6 / $\pm$ 4.4	0.7 / $\pm$ 0.9	0.0 / $\pm$ 0.0
SR-KI	2.0 / $\pm$ 2.5	1.7 / $\pm$ 2.2	0.6 / $\pm$ 0.8	1.2 / $\pm$ 1.4	1.8 / $\pm$ 2.3	3.0 / $\pm$ 3.7

Appendix

Table 11: Seed-level variation for exact match accuracy, reported as standard deviation /

t

-based 95% confidence-interval half-width (percentage points).

Method	100	500	1K	2K	5K	10K
Dense	1.0 / $\pm$ 1.2	1.7 / $\pm$ 2.1	1.6 / $\pm$ 2.0	1.6 / $\pm$ 1.9	1.5 / $\pm$ 1.8	1.3 / $\pm$ 1.6
BM25	0.8 / $\pm$ 1.1	1.1 / $\pm$ 1.4	1.2 / $\pm$ 1.5	1.8 / $\pm$ 2.2	1.5 / $\pm$ 1.9	2.0 / $\pm$ 2.5
Hybrid	0.7 / $\pm$ 0.8	1.0 / $\pm$ 1.3	0.6 / $\pm$ 0.8	0.6 / $\pm$ 0.7	1.0 / $\pm$ 1.2	1.4 / $\pm$ 1.7
KBLaM	1.1 / $\pm$ 1.3	1.3 / $\pm$ 1.6	1.3 / $\pm$ 1.6	1.4 / $\pm$ 1.7	1.3 / $\pm$ 1.6	4.1 / $\pm$ 5.1
SR-KI	0.5 / $\pm$ 0.6	0.7 / $\pm$ 0.8	0.6 / $\pm$ 0.8	1.0 / $\pm$ 1.3	1.4 / $\pm$ 1.8	1.3 / $\pm$ 1.6
DynaRule	0.4 / $\pm$ 0.4	0.3 / $\pm$ 0.4	0.6 / $\pm$ 0.7	0.8 / $\pm$ 1.0	1.3 / $\pm$ 1.6	1.1 / $\pm$ 1.3

Appendix

Table 12: Seed-level variation for retrieval, reported as standard deviation /

t

-based 95% confidence-interval half-width (percentage points).

Figure 11: Comparison of retrieval performance across QA types under FOL (top) and NL (bottom) rule representations. Dense uses Qwen3-Embedding-8B; Hybrid uses RRF with a fusion constant of 60; KBLaM, SR-KI, and DynaRule use confidence-layer retrieval. Recall@1 sets

K

to the number of gold rules per instance or step.

Figure 12: Case example of rule omission (missing rule application).

Figure 13: Case example of incorrect rule application.

Figure 14: Case example of action-type misalignment.

Figure 15: Basic question templates for Attribute-type rules.

Figure 16: Basic question templates for Action-type rules.

Figure 17: Basic question templates for Environment-type rules.

Figure 18: Basic question templates for State-type rules.

Figure 19: Template for combining questions to form Parallel Multi-Rule QA.

Figure 20: Question-answering template for DynaRule training.

Figure 21: Question-answering template for prompting evaluation.

Figure 22: Examples of Single-Rule, Parallel Multi-Rule, and Multi-Hop Rule QA under FOL representation.

Figure 23: Examples of Single-Rule, Parallel Multi-Rule, and Multi-Hop Rule QA under NL representation.

Method	Single-Rule QA	Parallel Multi-Rule QA							Multi-Hop Rule QA			Avg.
Method	single-rule	multi-rule-2	multi-rule-3	multi-rule-4	multi-rule-5	multi-rule-6	multi-rule-7	multi-rule-8	multi-hop-2	multi-hop-3	multi-hop-4	Avg.
FOL Representation
Rule Num = 100
Qwen2.5-14B-Instruct	0.9200	0.8300	0.8400	0.7633	0.7483	0.6450	0.6950	0.6250	0.6400	0.1600	0.1000	0.6333
Qwen2.5-32B-Instruct	0.9600	0.9900	0.9000	0.8433	0.8267	0.7433	0.6950	0.6700	0.6000	0.2600	0.1400	0.6935
Qwen2.5-72B-Instruct	0.9200	0.9700	0.9433	0.9250	0.8783	0.8117	0.8950	0.8850	0.7400	0.6000	0.4800	0.8226

Appendix

Table 13: Prompting comparison results on the RuleWorld benchmark under the FOL and NL rule representations, compared with DynaRule.

Explore similar work

RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
Aug 27, 2026Bohan Yu, Shi-Yang Li, Pengfei Cao +2LLM Evaluation Reasoning Evaluation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Procedural Knowledge at Scale Improves Reasoning
Apr 1, 2026Di Wu, Devendra Singh Sachan, Wen-tau Yih +1LLM Reasoning Knowledge Augmentation for Language Models
University of California, Los Angeles · Meta FAIR
RLIE: Rule Generation with Logistic Regression, Iterative Refinement, and Evaluation for Large Language Models
Oct 22, 2025Yang Yang, Hua XU, Zhangyi Hu +1Logistic Regression Neuro-Symbolic Reasoning
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511400, China · Institute of Deep Perception Technology, JITRI, Wuxi 214000, China

Aug 27, 2026J/K move · Enter open · S save

RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

Bohan Yu, Shi-Yang Li, Pengfei Cao, Jun Zhao+1

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

LLM Evaluation Reasoning Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.

Abstract

Explore similar work

RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

Procedural Knowledge at Scale Improves Reasoning

RLIE: Rule Generation with Logistic Regression, Iterative Refinement, and Evaluation for Large Language Models