ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems

Noel Thomas

cs.AIJan 5, 2026

ChaosBench-Logic: A Benchmark for Logical and Symbolic Reasoning on Chaotic Dynamical Systems

Authors: Noel Thomas

Organizations: Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

Abstract

Large language models (LLMs) excel at natural language tasks but remain brittle in domains requiring precise logical and symbolic reasoning. Chaotic dynamical systems provide an especially demanding test because chaos is deterministic yet often misinterpreted as randomness or complexity. We introduce ChaosBench-Logic, a benchmark that evaluates LLM reasoning across 30 diverse dynamical systems using a unified first-order logic (FOL) ontology. Each system is annotated with truth assignments for 11 semantic predicates, and 621 questions are generated across seven reasoning categories, including multi-hop implications, cross-system analogies, counterfactual reasoning, bias probes, and multi-turn dialogues. We define metrics for logical accuracy, implication consistency, dialogue coherence, and contradiction, and we release an open-source evaluation pipeline. Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91-94% per-item accuracy, yet still score 0% on compositional items and exhibit fragile global coherence. Dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zero-shot). ChaosBench-Logic provides a rigorous testbed for diagnosing such failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs.

Figures & tables

Class	Examples	Chaos?
Chaotic ODEs	Lorenz-63, Lorenz-84, Lorenz-96, Rössler, Chen, HR neuron	yes
Non-chaotic ODEs	Simple harmonic oscillator (SHM), Van der Pol, Brusselator, FitzHugh–Nagumo	no
Maps	Logistic, Hénon, Ikeda, Baker, Arnold Cat, Standard Map, Circle Map	mixed
PDEs	Kuramoto–Sivashinsky (chaotic), Sine–Gordon (integrable)	mixed
Stochastic	Ornstein–Uhlenbeck	random, non-chaotic

Table 1: Overview of dynamical systems included in ChaosBench-Logic . Each system is annotated with 11 semantic predicates and participates in first-order logic reasoning.

Family	What it measures (typical failure)
Atomic QA	Correctly recall system-level truth values (confident hallucination).
Multi-hop	Apply directionality of implications (converse fallacy).
Analogy	Transfer only the shared structure (surface-form analogy bias).
Counterfactual	Reason about regime change vs invariance (overgeneralize chaos).
Bias probes	Resist misleading cues (“chaos = random”, “PDE = chaotic”).
Dialogues	Maintain consistent belief state over turns (belief drift).

Table 2: What each task family tests and why it is non-trivial for LLMs.

Model	Mode	Acc (%)	DlgAcc (%)
GPT-4	Zeroshot	94.0	69.4
GPT-4	CoT	88.2	53.1
Claude 3.5 Sonnet	Zeroshot	91.6	67.3
Gemini 2.5 Flash	Zeroshot	91.9	71.4
LLaMA-3 70B	Zeroshot	91.6	75.5
LLaMA-3 70B	CoT	89.5	65.3

Table 3: Overall performance on ChaosBench-Logic . Acc is per-item accuracy; DlgAcc is the fraction of dialogues where all turns are correct.

Task family	Best model	Acc (%)
Atomic logical QA	GPT-4 (Zeroshot)	96.1
Bias probes	GPT-4 (Zeroshot)	93.0
Counterfactuals	GPT-4 (Zeroshot)	97.4
Multi-hop reasoning	GPT-4 (Zeroshot)	97.1
Multi-turn (per-turn)	LLaMA-3 (Zeroshot)	93.9
Dialogue (dialogue-level)	LLaMA-3 (Zeroshot)	75.5

Table 4: Task-specific best models on ChaosBench-Logic . Accuracies are over items with a valid normalised answer.

Model	$\Delta$ Acc (pp)	$\Delta$ DlgAcc (pp)	Trend
GPT-4	$-5.8$	$-16.3$	worse
LLaMA-3 70B	$-2.1$	$-10.2$	worse

Table 5: CoT impact on overall and dialogue-level performance.

Symbolic Reasoning in Language Models

See all 18 papers

May 23, 2026

ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

Standard accuracy on binary reasoning benchmarks hides critical failure modes: prior collapse, inconsistency under paraphrase, and inability to reason about parameter-dependent dynamics. We present ChaosBench-Logic v2, a 40,886-question benchmark over 165 dynamical systems with 27 FOL predicates and 78 axiom edges, together with CARE (Calibration- and Adversarial-Robust Evaluation), a protocol that surfaces these pathologies. Evaluating 14 models, we find that regime-transition reasoning remains near random (MCC = 0.05) even for frontier models, whereas FOL deduction with given premises reaches MCC = 0.52. Per-family decomposition shows that the proprietary-model advantage concentrates on cross-indicator (+0.40) and consistency tasks, while open-source Qwen 2.5-32B dominates indicator diagnostics (0.91 vs. 0.45). Two models exhibit negative MCC on bifurcation questions, confirmed as systematic anti-correlation via confusion-matrix analysis.

Noel Thomas

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

LLM Evaluation Language Model Robustness LLM Reasoning Large Reasoning Models

Ask PDF

Jun 22, 2026

HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

Logical reasoning is essential for reliable AI, yet existing benchmarks are largely first-order-logic-centric, focusing on object-level deduction over fixed predicates. This misses many realistic scenarios where models must reason over rules, predicates, functions, constraints, and decision procedures themselves. We introduce HOLMES (Higher-Order Logic Meets real-world Explainable Symbolic reasoning), the first real-world benchmark for higher-order symbolic reasoning in LLMs, containing 1379 instances. Built on higher-order logic, HOLMES pairs natural-language problems with HOL formalizations, ground-truth answers, verifiable reasoning traces, and fine-grained controllable reasoning factors across law and finance. Experiments show that current LLMs still struggle on HOLMES, with an average accuracy of only 50.64% and the best model reaching 59.54%. Our analyses further reveal that high final-answer accuracy can mask shortcut reasoning in conflict-resolution settings, while performance drops sharply under scope-conditioned and compositional reasoning. These findings identify higher-order symbolic reasoning as a key bottleneck for building reliable and verifiable LLMs. The project code and dataset are publicly available at https://github.com/wuyucheng2002/HOLMES.

Yucheng Wu, Jundong Xu, Mingzhen Ju +4

State Key Laboratory of Multimedia Information Processing, Peking University · School of Computer Science, Peking University · School of Computing, National University of Singapore +3

Logical Reasoning LLM Evaluation Symbolic Reasoning LLM Reasoning Symbolic Reasoning in Language Models

Ask PDF

May 19, 2026

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening

Evaluating large language models (LLMs) on natural-language logical reasoning is essential because rule-governed tasks require conclusions to follow strictly from stated premises. Many existing logical-reasoning benchmarks are generated by templating natural-language items from sampled formulas, provide only coarse or unaudited formal annotations, and are now quickly saturated by frontier reasoning models. We present LLMEval-Logic, a Chinese logical reasoning benchmark built from realistic situational scenarios. Its pipeline forward-authors and expert-audits natural-language items together with their reference formalizations, verifies annotated answers with Z3, constructs expert rubrics for natural-to-formal grading, and hardens selected items through a closed-loop adversarial workflow. The benchmark is released in two paired subsets: a 246-item Base subset shipped with 1,400 expert-developed rubric atoms, and a 190-item Hard subset with 938 multi-step sub-questions over closed model spaces. Evaluating 14 frontier LLMs on LLMEval-Logic reveals substantial gaps in current models: the best model reaches only 37.5% Hard Item Accuracy, and even with reference symbols the highest joint Z3+Rubric formalization score among evaluated models reaches only 60.16%. Our benchmark is publicly available at https://github.com/llmeval/LLMEval-Logic.

Ming Zhang, Qiyuan Peng, Yinxi Wei +13

Institute of Trustworthy Embodied Artificial Intelligence, Fudan University · 2Hunyuan Team, Tencent · School of Philosophy, Fudan University

Logical Reasoning LLM Evaluation LLM Reasoning

Ask PDF

Abstract

Symbolic Reasoning in Language Models

Explore similar work

ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening