Evaluating Relational Reasoning in LLMs with REL

Lukas Fesser; Yasha Ektefaie; Ada Fang; Sham M. Kakade; Marinka Zitnik

cs.AIApr 14, 2026

Evaluating Relational Reasoning in LLMs with REL

Authors: Lukas Fesser, Yasha Ektefaie, Ada Fang, Sham M. Kakade, Marinka Zitnik

Organizations: Harvard University · Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard

Abstract

Relational reasoning is the ability to infer relations that jointly bind multiple entities, attributes, or variables. This ability is central to scientific reasoning, but existing evaluations of relational reasoning in large language models often focus on structured inputs such as tables, graphs, or synthetic tasks, and do not isolate the difficulty introduced by higher-arity relational binding. We study this problem through the lens of Relational Complexity (RC), which we define as the minimum number of independent entities or operands that must be simultaneously bound to apply a relation. RC provides a principled way to vary reasoning difficulty while controlling for confounders such as input size, vocabulary, and representational choices. Building on RC, we introduce REL, a generative benchmark framework spanning algebra, chemistry, and biology that varies RC within each domain. Across frontier LLMs, performance degrades consistently and monotonically as RC increases, even when the total number of entities is held fixed. This failure mode persists with increased test-time compute and in-context learning, suggesting a limitation tied to the arity of the required relational binding rather than to insufficient inference steps or lack of exposure to examples. Our results identify a regime of higher-arity reasoning in which current models struggle, and motivate re-examining benchmarks through the lens of relational complexity.

Figures & tables

Figure 2: REL evaluates relational reasoning across algebraic, biological, and chemical domains.

Figure 3: Three examples of Raven’s Progressive Matrices with increasing relational complexity. The answers are shown in bold.

Figure 4: From provided parameters, we generate a phylogenetic tree, alignment, inject shared motifs, and ask the model to use this alignment and tree to identify the taxa that are in homoplasy.

Figure 5: Model performance on REL -A tasks. RPMs at the top, with RPTs below. The models are given 8 answer choices, so trivial accuracy is 12.5%. All three model perform well on tasks with low RC ( REL -A1 and REL -A2, top two rows), but struggle once RC increases: on REL -A3 and REL -A4, where RC increases with input size, performance drops by as much as 80%. RPTs ( REL -A5, REL -A6, and REL -A7), which always have a higher RC, independent of input size, are challenging to impossible for all three models.

Figure 6: Performance decreases with increasing RC controlled by increasing the number of homoplastic taxa increases in REL -B1.

Figure 7: Top: Schema of variables in multivariate regression. Bottom: Explained variance of performance on REL -B1 across five measures of complexity. RC, which is number of homoplastic taxa, explains the most variance.

Figure 8: Task completion rate decreases as from C1 (RC=2 and OC easy) to C2 (RC=2 and OC medium) to C3 (RC= $N_{\text{isomers}},N_{\text{observed}}$ and OC hard). This observation holds across different number of molecules in the task.

Figure 9: Task completion rate on REL -C2 evaluated with Is Substructure. For this task, RC is fixed at 2 in Left: increasing $N_{\text{molecules}}$ does not have an effect on performance until $N_{\text{molecules}}=50$ , in Right: increasing the molecule size increases OC and leads to decreased IsSubstructure rate.

Figure 10: Task completion rate on REL -C3 evaluated with recall, RC depends on both $N_{\text{observed}}$ and $N_{\text{missing}}$ . Left: As RC increases with $N_{\text{observed}}$ recall decreases. Right: As RC increases with $N_{\text{missing}}$ recall decreases.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Accuracy as a function of the number of homoplastic taxa, motif ratio, sequence length, average pairwise distance between taxa, and prompt length for REL -B1.

Dataset	RC-2	RC-3	RC-4	RC-5	RC-6
GFP	81.4%	27.5%	46.3%	13.2%	11.8%
HA	81.4%	34.3%	47.1%	16.2%	8.0%
HA2	83.3%	35.3%	40.4%	14.0%	6.6%
GB1	78.4%	15.7%	–	–	–
TrpB	76.5%	21.6%	–	–	–
Chance	33%	25%	25%	25%	25%

Appendix

Table A.1 : Performance across relational complexity (RC) levels for each dataset in REL -B2 task for GPT-5.2.

$N_{\text{molecules}}$	5	10	15	20	25	30	35	40	45	50
# Questions	100	100	100	100	100	100	100	100	100	100
Avg. Atoms	5.75	5.78	5.94	6.10	6.20	6.10	6.34	6.21	6.15	6.32
Avg. DBE	4.31	4.54	4.61	4.68	4.78	4.80	4.79	5.35	5.31	5.47

Appendix

Table A.2: Details for REL -C1. Below double bond equivalent (DBE) measures the degree of unsaturation in the formula.

$N_{\text{molecules}}$	5	10	15	20	25	30	35	40	45	50
# Questions	120	120	120	120	120	120	120	76	53	47
Avg. atoms	30.74	30.75	27.28	28.18	27.25	27.39	26.46	27.42	27.42	28.45
Avg. elements	3.32	3.34	3.29	3.34	3.30	3.25	3.14	3.02	2.96	2.83
Avg. DBE	24.55	24.59	22.11	22.55	22.09	22.12	21.79	23.08	23.27	24.08
Avg. rings	3.03	2.97	2.98	3.00	3.09	3.07	3.13	3.32	3.42	3.81

Appendix

Table A.3: REL -C2. Number of questions by number of molecules. The number of questions decreases with the number of molecules as it becomes more unlikely to sample a set of molecules with a largest common motif of at least 8 atoms.

Motif (SMILES)	Count
COc1ccccc1	180
CCc1ccccc1	114
O=SHc1ccccc1	50
CCOc1ccccc1	17
CC12C=CC(=O)C=C1CCC1C2CCC2(C)C(C=O)CCC12	13
CC(C)NCC(O)COc1ccccc1	9

Appendix

Table A.4 : Most frequent molecular substructure motifs and their occurrence counts in REL -C2.

Figure A.2 : Sample of 20 molecules used in REL -C2.

$N_{\text{molecules}}$	5	10	15	20	25	30	35	40	45	50
# Questions	100	100	100	100	100	100	100	100	100	100
# Missing Isomers	36.41	32.92	29.93	30.39	24.13	27.28	24.17	33.24	28.16	26.44
Avg. atoms	5.91	5.83	5.84	6.22	6.23	6.19	6.42	6.18	6.22	6.31
Avg. DBE	4.56	4.69	4.73	4.74	4.59	4.78	4.66	5.39	5.34	5.40

Appendix

Table A.5: REL -C3. Number of missing isomers as a function of the number of molecules.

Model	Count	$N_{\text{observed}}$	Recall	Precision	F1
Claude Opus 4.5	300	$<20$	0.341 (0.013)	0.569 (0.016)	0.386 (0.011)
	400	20–40	0.313 (0.011)	0.390 (0.013)	0.299 (0.008)
	300	$\geq 40$	0.253 (0.009)	0.398 (0.015)	0.289 (0.009)
Gemini 3 Pro	300	$<20$	0.417 (0.014)	0.472 (0.015)	0.403 (0.012)
	400	20–40	0.362 (0.012)	0.275 (0.011)	0.260 (0.009)
	300	$\geq 40$	0.259 (0.010)	0.256 (0.012)	0.234 (0.010)

Appendix

Table A.6 : Performance on REL -C3 across increasing RC which is determined by

N_{\text{observed}}

. Metrics reported are mean (s.e.) across the questions. For GPT 5.4 we provide the model access to RDKit.

Figure A.3 : Proportion correct as the task completion rate on REL -C4 stratified by RC which is given by $N_{\text{molecules}}$ . Performance of GPT-5.4 is shown.

Model	Rule Type (RC)	3x3	9x9	15x15	30x30
Llama 3.1-70b	Progression (2)	0.912	0.864	0.856	0.816
Llama 3.1-70b	Permutation (n)	0.648	0.552	0.296	0.144
Qwen3-4B	Progression (2)	0.616	0.568	0.528	0.504
Qwen3-4B	Permutation (n)	0.552	0.216	0.184	0.104

Appendix

Table A.7 : Performance on open source models for 500 questions from REL-A2 and 500 questions from REL-A3.

Figure A.4 : Models are evaluated with one-shot in-context learning on 10% of questions from REL -C1,C2,C3.

Digits	GPT REL -A5 ( $3\times 3\times 3$ )	Claude REL -A5 ( $3\times 3\times 3$ )	GPT REL -A4 ( $9\times 9$ )	Claude REL -A4 ( $9\times 9$ )
4096 Tokens	$0.648$	$0.616$	$0.152$	$0.152$
8192 Tokens	$0.664$	$0.624$	$0.168$	$0.144$
16384 Tokens	$0.680$	$0.648$	$0.176$	$0.168$

Appendix

Table A.8: Accuracy by number of thinking tokens on small examples of REL -A4 and REL -A5. Gains are small, but consistent.

Figure A.5 : Models are evaluated with different levels of test-time compute on 10% of questions from REL -C1,C2,C3.

Digits	GPT REL-A3 ( $9\times 9$ )	Claude REL-A3 ( $9\times 9$ )	GPT REL-A4 ( $3\times 3$ )	Claude REL-A4 ( $3\times 3$ )
5 Digits	$0.912$	$0.752$	$0.248$	$0.208$
10 Digits	$0.872$	$0.688$	$0.184$	$0.192$
20 Digits	$0.360$	$0.336$	$0.136$	$0.112$

Appendix

Table A.9: Accuracy by number of digits per entry on small examples of REL -A3 and REL -A4. Only for extremely large input values does accuracy plummet.

Motif Ratio	Gemini	GPT	Claude
0–5%	2.6%	0.8%	0.0%
5–10%	9.2%	10.2%	1.8%
10–12.5%	13.5%	18.6%	2.1%
12.5–15%	22.6%	29.0%	12.9%
15–17.5%	19.6%	17.6%	17.6%
17.5–20%	34.2%	23.7%	28.9%

Appendix

Table A.10 : Accuracy by motif ratio (motif size / sequence length). Performance collapses as operand complexity increases in small motif ratios.

	Relational Complexity (RC)
Method	4	10	20	25
Baseline	45%	15%	5%	5%
Structured	65%	25%	10%	5%
$\Delta$	+20	+10	+5	0

Appendix

Table A.11: Accuracy (%) on REL -B1 under increasing relational complexity (RC). Structured prompting improves performance at low RC but yields diminishing gains as RC increases.

Relational Reasoning

Relational reasoning studies how AI systems represent and infer relationships among objects, entities, events, and facts.

See all 34 papers

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains
Jul 20, 2026Liam SwayneLLM Evaluation LLM Reasoning
Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners
Jun 23, 2026Anirban Das, Joanne Boisson, Irtaza Khalid +2Language Model-Guided Search Heuristics Relational Reasoning
Cardiff University · University of Pennsylvania
Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints
Apr 15, 2026Md. Fahad Ullah Utsho, Mohd. Ruhul Ameen, Akif Islam +2LLM Evaluation Large Reasoning Models
Department of Information and Communication Engineering, University of Rajshahi, Bangladesh · College of Engineering and Computer Sciences, Marshall University, Huntington, WV, USA · Department of Computer Science and Engineering, University of Rajshahi, Bangladesh

Jul 20, 2026J/K move · Enter open · S save

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

Liam Swayne

LLM Evaluation LLM Reasoning Language Model Reasoning Evaluation

Open paper AskPDF arXiv

Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.

Abstract

Relational Reasoning

Explore similar work

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints