Thought without systematicity? Evaluating reasoning models on rule induction tasks

Simon Schug; Brenden M. Lake

cs.CLSep 12, 2026

Thought without systematicity? Evaluating reasoning models on rule induction tasks

Authors: Simon Schug, Brenden M. Lake

Organizations: Princeton University

Abstract

A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations of that concept. Do reasoning models robustly exhibit such systematicity? If so, we would expect consistent performance on structurally equivalent variants of the same task. Here, we extend established rule induction tasks from cognitive science to assess the systematicity of thought in current reasoning models. Each task family has compositional structure that we use to create structurally equivalent task variations through task isomorphisms such as recombination and substitution. We find that despite being able to correctly solve a task, models often fail on structurally equivalent variants of the same task. These findings suggest that many model behaviors lack systematicity, rendering it difficult to robustly establish the cognitive abilities of reasoning models beyond the particular contexts they were evaluated in.

Figures & tables

Figure 1: Systematicity of reasoning models on rule induction tasks. We evaluate reasoning models on compositional families of rule induction tasks. For each particular task within a family, we generate

K=5

structurally equivalent task variants per type of task invariance. For each set of task variations, we report whether any of the variations was solved correctly, the fraction of variations solved correctly and whether all variations were solved correctly and report the average over task variation sets within a task family. We find that almost all models are perfectly systematic on the Boolean category learning task family, but see a notable systematicity gap between the ability to solve any task variation and solving all task variations on the remaining task families.

Figure 2: Evaluating systematicity in rule induction tasks with compositional structure. (A) Compositional task families allow to construct large numbers of tasks with related structure. The shown task is a simplified version of the grammar-based instruction-learning task presented in Section 4.1 . (B) Each rule induction task requires inferring hidden rules on a set of input

\rightarrow

output support examples to solve novel query examples. (C) The compositional task structure allows creating systematic variations of structurally equivalent tasks. (D) Reasoning models can be independently evaluated across structurally equivalent task variants to assess their systematicity.

Panel of Figure 3 — Figure 3: Relation-ship between the original and transformed hidden rule.

Figure 4: We use bullet charts to compactly visualize the three systematicity metrics.

Panel of Figure 5 — Figure 5: Example Hamming graph used to construct the support set from objects at the decision boundary (dashed line) of a given rule.

Figure 6: Reasoning effort does not consistently improve systematicity. Varying the reasoning effort for different Gemini reasoning models across task families does not consistently improve systematicity.

Figure 7: Systematicity gap persists with greedy decoding. When setting the decoding sampling temperature to zero during inference, reasoning model task performance remains comparable and the systematicity gap persists.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Metric	Task family
		Integer list		Grammar-based instruction			Raven	Boolean concept
		Substitution	Reordering	Recomposition	Reordering	Substitution	Rebinding	Substitution	Reordering
Claude Opus 4.7	$\rho_{\text{total}}$	0.20	0.17	0.12	0.17	0.20	0.62	-	-
	$\rho_{\text{within}}$	0.16	0.17	0.18	0.17	0.16	0.08	-	-
	$\rho_{\text{between}}$	0.04	0.00	0.00	0.01	0.04	0.54	-	-
GPT-5.5	$\rho_{\text{total}}$	0.19	0.15	0.17	0.17	-	0.19	-	-

Appendix

Table 1: Variance decomposition across tasks and models. Average estimated variance components for each model and task family grouped by the specific task variation type. Since our estimators are noisy,

\rho_{\text{within}}

can be larger than

\rho_{\text{total}}

, in which case we clip

\rho_{\text{between}}

to zero.

Figure 8: Grammar-based instruction-learning. (A) The systematicity gap persists across variation type. (B) Showing 8 rather than 4 support examples in addition to the curriculum examples increases the average number of tasks where any variation is solved but does not resolve the systematicity gap.

Figure 9: Program induction over integer sequences. Providing hints in the task instructions improves performance for Gemini 3.1 Flash and Gemini 3.1 Pro and almost eliminates the systematicity gap in the latter.

Figure 10: Symbolic Raven’s progressive matrices. Increasing the number of features notably increases difficulty and leads to a corresponding drop in the average number of tasks where any variation can be solved.

Reasoning Consistency in Language Models

Reasoning consistency in language models measures whether conclusions and reasoning remain reliable across equivalent prompts, samples, steps, and contexts.

See all 115 papers

What Do Current Systematic Generalization Tasks Miss? A Reasoning-Centered Analysis
Sep 16, 2026Chengwen Qi, Deheng Ye, Yatao BianLogical Reasoning Inductive Inference
National University of Singapore · Nanyang Technological University
The Reasoning Error About Reasoning: Why Different Types of Reasoning Require Different Representational Structures
Mar 23, 2026Yiling WuStructured Reasoning Logical Reasoning
BridgeM, Inc. · Department of Philosophy, University of Massachusetts Amherst
Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
Aug 5, 2026Ian B. de Haan, Peter van der Putten, Max van DuijnLLM Evaluation Theory of Mind
LIACS, Leiden University, The Netherlands

Sep 16, 2026J/K move · Enter open · S save

What Do Current Systematic Generalization Tasks Miss? A Reasoning-Centered Analysis

Chengwen Qi, Deheng Ye, Yatao Bian

National University of Singapore · Nanyang Technological University

Logical Reasoning Inductive Inference Abductive Reasoning Compositional Generalization

Open paper AskPDF arXiv

Systematic generalization, the ability to solve novel problems by recombining known atomic elements, is central to human intelligence but difficult to study rigorously under controlled settings. Existing studies therefore rely on simplifications such as elemental composition, productivity-based tests, and action-explicit goals, which make systematic generalization easier to study but omit some essential aspects of this capability. To characterize what these simplifications miss, we adopt a reasoning-centered lens and introduce TranSGrid, a testbed that brings deductive, inductive, and abductive reasoning together within a unified task. Experiments with seven Transformer models on 4,800 TranSGrid instances show that all models perform much worse on TranSGrid than on a held-out test set: the largest model solves 79.6% of the test set, but only 55.3% of TranSGrid and 15.8% of the hardest subset. The gap remains within the training length range, showing that productivity alone is not sufficient to evaluate systematic generalization. Additionally, we reintroduce the other two simplifications into TranSGrid: one variant limits interactions among action effects to approximate elemental composition (reducing the inductive demand); the other makes goals action-explicit (reducing the abductive one). In both, solve rates return to roughly the test set level, showing that either simplification alone is enough to reduce TranSGrid to an ordinary held-out test set. Together, our results show that existing tasks reduce either or both of the inductive and abductive demands of systematic generalization, and that comprehensively measuring this capability requires a task that involves all three forms of reasoning.

Abstract

Reasoning Consistency in Language Models

Explore similar work

What Do Current Systematic Generalization Tasks Miss? A Reasoning-Centered Analysis

The Reasoning Error About Reasoning: Why Different Types of Reasoning Require Different Representational Structures

Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning