RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios

Chenhao Yuan; Bohan Yu; Yinhao Xu; Chao Yin; Chenxi Zhou; Zhaoxuan Wang; Shuwen Xu; Shaoping Huang

cs.AIOct 14, 2026

RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios

Authors: Chenhao Yuan, Bohan Yu, Yinhao Xu, Chao Yin, Chenxi Zhou, Zhaoxuan Wang, Shuwen Xu, Shaoping Huang,

Organizations: The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China

Abstract

Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introduces RuleTrek, a multi-domain benchmark for agentic rule reasoning in complex scenarios. Its rule-grounded construction pipeline extracts structured rule graphs, generates and incrementally composes rule-specific sub-scenarios while ensuring consistency and solvability. To systematically evaluate agentic rule reasoning in complex scenarios, RuleTrek assesses three capabilities in interactive workspaces: Rule Identification (RI), Rule Deduction (RD), and End-to-End Rule Reasoning (E2E). Experiments on 11 frontier LLMs show that the highest RI and RD scores are only 62.47% and 60.43%, respectively, while E2E scores range from 26.97% to 55.29%. These results demonstrate that current LLM agents face substantial challenges in rule reasoning over complex scenarios. We make our code and dataset publicly available at RuleTrek.

Figures & tables

Figure 1: Comparison of rule reasoning in prior work and complex real-world scenarios. Prior work typically presents scenarios using a few factual statements, whereas RuleTrek requires agents to integrate complex scenario information distributed across heterogeneous files, identify applicable rules from complex rule documents, and apply them to derive correct decisions.

Rule-Reasoning Benchmarks
Benchmark	Rule App.	Native Rules	Dist. Scenario	Multi-Domain	Workspace
RuleTaker ( Clark et al., 2020 )	$\ominus$	✗	✗	✗	✗
PolicyQA ( Ahmad et al., 2020 )	✗	$\ominus$	✗	✗	✗
LegalBench ( Guha et al., 2023 )	$\ominus$	$\ominus$	✗	$\ominus$	✗
RuleArena ( Zhou et al., 2025 )	✓	$\ominus$	✗	✓	✗
RuleWeaver ( Yu et al., 2026b )	✓	✗	✗	✓	✗

Table 1: Left: Comparison of RuleTrek with representative rule-reasoning and workspace benchmarks. Rule App. indicates whether tasks require applying domain rules to concrete cases; Native Rules whether rules retain their original representations; Dist. Scenario whether scenario evidence is distributed across multiple artifacts; Multi-Domain whether the benchmark covers multiple application domains; and Workspace whether agents operate in an interactive environment. ✓ , ✗ , and

\ominus

denote full, absent, and partial support, respectively. Right: Distribution of RuleTrek instances across six domains.

Figure 2: Overview of the RuleTrek construction pipeline. Stage 1 extracts and verifies rule graphs from real-world rule repositories. Stage 2 generates rule-specific sub-scenarios requiring the application of corresponding rules. Stage 3 incrementally composes sub-scenarios according to rule dependencies, verifies consistency and solvability at each step, and generates questions from the final scenarios.

Model	Avg.			Tax			Game			Procurement			Cybersecurity			Software Policy			PostgreSQL
Model	RI	RD	E2E	RI	RD	E2E	RI	RD	E2E	RI	RD	E2E	RI	RD	E2E	RI	RD	E2E	RI	RD	E2E
GPT-6.1-Sol	62.47	60.43	55.29	65.44	86.42	85.89	67.76	69.20	71.99	57.78	43.14	26.92	64.81	56.78	47.82	67.39	68.00	59.31	47.28	25.17	25.49
Grok-4.7	61.86	56.12	51.34	65.76	87.14	85.95	67.59	68.75	65.53	56.48	30.86	20.40	68.57	48.11	48.15	60.03	61.82	57.01	48.33	25.99	15.02
DeepSeek-V4.1-Flash	59.21	56.39	50.06	64.27	92.30	77.78	64.60	66.57	62.93	45.83	28.00	23.62	72.60	42.80	41.98	56.69	65.52	55.91	47.93	29.41	26.46
DeepSeek-V4-Pro	58.25	51.81	49.07	64.78	88.13	82.40	68.42	65.91	67.93	48.30	23.47	25.16	66.50	43.18	40.83	50.59	49.82	46.62	45.43	25.59	14.37
GLM-5.3-Flash	56.53	51.06	47.16	58.94	79.90	81.00	60.23	66.78	63.75	47.06	21.29	19.63	72.00	42.97	36.85	51.13	59.98	53.14	47.43	22.25	12.34

Table 2: Main results (%) across six domains and three tasks: Rule Identification (RI), Rule Deduction (RD), and End-to-End Decision (E2E). Green and red cells indicate the best and second-best scores, respectively; best scores are bold.

Domain	$N$	Nodes	Edges	Depth
Cybersecurity	38	7.05	6.18	1.55
Game	45	5.56	4.91	1.82
PostgreSQL	30	13.90	13.07	4.10
Procurement	42	8.74	7.55	2.43
Software	37	7.68	7.84	2.78
Tax	48	12.21	13.15	3.67

Table 3: Evaluation-set statistics.

N

: number of instances; Nodes/Edges: mean operative rule-graph size; Depth: mean longest directed path length in edges. All means are computed over instances.

Instance	Missed or Misidentified Rules
Cyber-007	Fine-grained firewall and remote-access controls.
Tax-007	Data-loading and tax-year initialization rules.
Game-042	Distinct rules for different travel paces.

Table 4: Examples of frequently missed or misidentified rules in Rule Identification (RI) across models.

Figure 3: Rule-graph complexity and E2E performance. Tasks are grouped by (a) rule-node count and (b) rule-graph depth. Lines show mean scores across 11 models; shaded bands and error bars indicate the minimum–maximum range.

n

denotes the number of tasks per bin.

Figure 4: Comparison of mean RD and E2E scores across seven models. Annotations show

\Delta=\mathrm{RD}-\mathrm{E2E}

Figure 5: Relationship between mean agent steps per task and mean E2E scores across seven models. Each point represents one model.

(a) Human agreement
Task	Pair	Spearman $\rho$	Pearson $r$
RI	H–J	0.822	0.821
	H–H	0.951	0.939
RD	H–J	0.857	0.798
	H–H	0.933	0.919
E2E	H–J	0.846	0.837

Table 5: Scoring reliability across three tasks: human agreement and cross-family judge robustness.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Criterion	$\boldsymbol{\kappa}$	Initial Pass
Answerability	0.91	91.7%
Answer uniqueness	0.88	86.3%
Rule alignment	0.89	82.5%
Scenario consistency	0.93	90.4%
Difficulty validity	0.85	79.6%
Average	0.89	86.1%

Appendix

Table 6: Initial human verification results for benchmark quality.

\kappa

denotes Cohen’s agreement between two annotators, and Initial Pass denotes the proportion of instances satisfying each criterion before revision. All failed instances are revised and re-evaluated in subsequent quality control rounds.

Round	Reviewed Instances	Pass Rate
Initial review	184 / 240	76.7%
After revision	228 / 240	95.0%
Final validation	240 / 240	100%

Appendix

Table 7: Multi-round human quality control process. Instances failing any criterion are revised and re-evaluated until all benchmark instances satisfy the requirements.

Figure 6: Relationship between mean output tokens per task and E2E performance across evaluated models. Each point represents one evaluated model.

Domain	Scenario complexity	Identification challenge	Representative case
Cybersecurity	Cross-file evidence dependencies among policies, system configurations, network records, and exception documents.	Models need to associate distributed evidence with condition-specific controls rather than retrieving only general security requirements.	High-impact baseline scenarios require linking network flows, remote-access records, and exceptions to specific NIST control enhancements.
Tax	Multi-stage computation involving hidden prerequisites, intermediate states, and dependent calculations.	Models need to recover upstream rules that initialize states or provide required inputs before applying downstream formulas.	Tax scenarios require connecting receipts, asset information, and tax-year states before applying depreciation or income calculations.
Game	Fine-grained rule interactions with similar descriptions but different conditional effects.	Models need to distinguish rules with subtle applicability boundaries instead of merging semantically similar rules.	Travel scenarios contain Fast, Normal, and Slow pace rules whose effects depend on specific conditions.
PostgreSQL	Stateful operational scenarios involving configuration settings, execution states, and recovery procedures.	Models need to identify rules that determine when operations become valid under specific system states.	Replication and recovery tasks require reasoning over slot configuration, synchronization state, and failover conditions.
Procurement	Hierarchical rule structures with dependencies between objects, extensions, and field-level constraints.	Models need to determine higher-level scopes before applying fine-grained field rules.	OCDS scenarios require identifying applicable extensions and object scopes before checking individual field-level constraints.
Software Policy	Parameterized policies involving templates, versions, resource types, and configurable thresholds.	Models need to jointly identify policy logic, parameters, and matching conditions.	Policy violation scenarios depend on API versions, resource kinds, and configurable thresholds that determine rule applicability.

Appendix

Table 8: Representative rule identification challenges caused by complex scenario structures across domains.

Figure 7: Selected E2E trajectories on game-025 . GPT-6.1-Sol correctly updates the protection state after the first trigger, while Qwen3.8-Flash retains an outdated state and produces an incorrect outcome.

Figure 8: Selected Qwen3.8-Flash trajectories on tax-003 . The RD setting exposes the prerequisite netting rule through the rule graph, while the E2E setting requires retrieving the rule from original sources and misses this dependency.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,194 papers

RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
Aug 27, 2026Bohan Yu, Shi-Yang Li, Pengfei Cao +2LLM Evaluation Reasoning Evaluation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models
Aug 24, 2026Bohan Yu, Pengfei Cao, Chen Han +7Knowledge Augmentation for Language Models LLM Evaluation
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +3
T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
Jun 9, 2026Genta Indra Winata, Amartya Chakraborty, Yuzhen Lin +12Customer Support Automation LLM Agent Evaluation
AI Foundations, Capital One

Aug 27, 2026J/K move · Enter open · S save

RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

Bohan Yu, Shi-Yang Li, Pengfei Cao, Jun Zhao+1

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

LLM Evaluation Reasoning Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.

Abstract

LLM Agent Evaluation

Explore similar work

RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models

T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains