Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge

Zohreh Abbasi Liasi; Batoul Hammoud; Anas Madkoor; Ehsaneddin Asgari

cs.CLNov 4, 2026

Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge

Authors: Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor, Ehsaneddin Asgari

Organizations: Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University · Universitat Pompeu Fabra

Abstract

Retrieval-augmented generation (RAG) assumes that language models can reason over relations stated in their context rather than fall back on associations memorized during pretraining. We study this evidence-conditioned relational reasoning in isolation, with the facts supplied directly in the prompt rather than retrieved. Rel-Bloom is a controlled multilingual diagnostic of 450 multiple-choice questions built from 30 Wikidata relations; the same source relations are instantiated at five Bloom-inspired levels (Remember, Understand, Apply, Analyze, Evaluate) and under three veracity conditions (Factual, Counterfactual, Neutralized). Across 43 models, six languages, and two multilingual settings (226,800 predictions), three findings stand out. First, accuracy drops sharply from Apply to Analyze in every evaluated open-weight model series, reaching 47.2 percentage points for the Large Granite bucket, but not for either Gemini model. Second, Evaluate items remain a bottleneck even for models near ceiling on the other task types: Gemini~3.1 Pro averages 99.5% on the other four levels but 60.7% on Evaluate. Third, item- and relation-level difficulty rankings are consistently preserved across languages, indicating a difficulty structure partly shared across languages within this benchmark. Across knowledge conditions, Counterfactual contexts are generally harder than Factual ones, while removing entity identities can improve higher-level performance in some model regimes. Greater model capability therefore does not by itself ensure reliable reasoning over supplied relational evidence. Relevant resources are available at: https://huggingface.co/spaces/QCRI/non-parametric-memory-relational-reasoning.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of Rel-Bloom . The same 30 curated Wikidata source relations (10 per relation usage frequency tier) are each instantiated at all five Bloom-inspired levels (shown as a pyramid) and under all three veracity conditions, yielding $30\times 5\times 3=450$ English items; this full crossing allows level and condition effects to be compared on matched relations. Items are then evaluated in six languages under two multilingual settings. The 43 models comprise 41 open-weight models from seven model families, analyzed as eight model series because Qwen2.5 Base and Qwen2.5 Instruct are treated separately, plus two Gemini models.

Bloom level	Operationalization in Rel-Bloom
Remember	Extract the value explicitly stated in the fact.
Understand	Select a paraphrase preserving the relation semantics; distractors express different relations or attributes.
Apply	Apply an abstract relation schema to a new statement with the same subject–relation–object structure and meaning.
Analyze	Discriminate the target relation from near-miss alternatives (semantically related or structurally similar but logically incorrect).
Evaluate	Select the strongest justification grounded in the defining semantic criterion of the relation; distractors are plausible but non-definitional.

Table 1: Bloom-level operationalization in Rel-Bloom .

Variant	Construction rule	What it tests
Factual	Verbalize the extracted triple $(h,p,t)$ as a sentence-level fact.	Baseline relation understanding with real entity surface forms.
Counterfactual	Edit exactly one argument of $(h,p,t)$ , preserving coarse entity type and relation context.	Evidence-following under conflict with parametric priors.
Neutralized	Replace entity mentions with typed placeholders (e.g., [PERSON 1] , [SPORT 1] ) while preserving the question template.	Relation abstraction with reduced entity-prior shortcuts.

Table 2: Veracity variants in Rel-Bloom , generated with GPT-5 and manually verified.

Theme	ID	Statement
Scale	H1	Diminishing returns: Small $\to$ Medium gain exceeds Medium $\to$ Large.
Scale	H2	Monotonic scaling: Medium $>$ Small and Large $>$ Medium.
Bloom level	H3	Apply $>$ Analyze accuracy; a “cliff” is marked when the lower 95% CI on the drop exceeds 10 pp.
Bloom level	H8	Evaluate underperforms the mean of the other four Bloom levels.
Veracity	H4	Ordering: Neutralized $>$ Factual $>$ Counterfactual.
	H5	Factual–Counterfactual gap is larger at Analyze/Evaluate than at Remember/Understand.

Table 3: The ten hypotheses tested in Rel-Bloom , grouped by theme.

		Open-weight series (of 8)		Gemini (of 2)
ID	Hypothesis	Fully transl.	Hybrid	Fully transl.
Per-bucket tests: series significant in all 3 buckets (in $\geq$ 1 bucket)
H9	Item-level cross-language stability	8 (8)	8 (8)	2
H10	Relation-level cross-language stability	8 (8)	8 (8)	2
H3	Apply $>$ Analyze	7 (8)	6 (8)	0
H8	Evaluate bottleneck	6 (8)	5 (8)	2

Table 4: Summary of hypothesis support (Holm-corrected; full matrices in Tables 6 – 7 ). Units differ between blocks and are stated in the block headers and cells. The Gemini column counts supported models (fully translated setting only).

Entity	Wrong answer	Correct relational answer
Apollo 11	Returned safely to Earth	Recovered by USS Hornet
Bohemian Rhapsody	A six-minute song	Tonality is B-flat major
Amanita muscaria	Red cap with white spots	Spore print color is white

Table 5: Representative errors consistent with memory competition. In the Factual condition, the model selects a salient entity-associated answer rather than the relation required by the question.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Series	H1	H2	H3	H4	H5	H6	H7	H8	H9	H10
Qwen2.5 Base		S,M,L	S,M $\dagger$ ,L* $\dagger$	S,M,L	S,M	M	S,M,L*	S,M,L*	S,M,L*	S,M,L*
Qwen2.5 Instruct		S,M,L	S,M $\dagger$ ,L* $\dagger$	S,M,L*	S	x	S,M,L*	S,M,L*	S,M,L*	S,M,L*
Qwen3		S,M,L	S,M $\dagger$ ,L* $\dagger$	S,M,L	S,M*,L	M,L	S*,M,L	S,M,L*	S,M,L*	S,M,L*
Qwen3.5		S,M,L	S* $\dagger$ ,M* $\dagger$ ,L*	S,M	S,L	x	S*,M	S,M,L*	S,M,L*	S,M,L*
Gemma		S,M,L	S* $\dagger$ ,M* $\dagger$ ,L*	S,M	S,M,L	x	S*,M	S,M,L*	S,M,L*	S,M,L*
LLaMA	x	S,M,L	S,M $\dagger$ ,L* $\dagger$	L	x	S	S,M,L	S,M,L	S,M,L*	S,M,L*

Appendix

Table 6: Hypothesis support by model series in the fully translated setting . Cells are shaded to convey strength of support at a glance: all present buckets significant and an H3 cliff (

\dagger

); all present buckets significant (starred); partial significance (some buckets starred, some directional only); direction observed but no statistical support; no support ( x ); not applicable (—). Reading the cell contents: S, M, and L indicate that the required direction is observed in the Small, Medium, and Large model buckets, respectively; starred labels indicate statistical support under the hypothesis-specific procedure; Holm correction is applied where applicable.

\dagger

marks an H3 cliff whose lower 95% relation-cluster bootstrap bound exceeds 10 percentage points. H1 is a single series-level test rather than a per-bucket test. Gemini models are reported individually because their parameter counts are undisclosed and are therefore not evaluated under H1 or H2; for Gemini,

*

denotes statistical support and

\checkmark

denotes the required direction without statistical support.

\ddagger

marks a value at the H6 five-percentage-point boundary under the rounded output.

Series	H1	H2	H3	H4	H5	H6	H7	H8	H9	H10
Qwen2.5 Base		S,M,L	S* $\dagger$ ,M* $\dagger$ ,L* $\dagger$	S,M,L	x	M,L	S,M,L*	S,M,L*	S,M,L*	S,M,L*
Qwen2.5 Instruct		S,M,L	S,M $\dagger$ ,L* $\dagger$	S,M,L*	x	M	S,M,L*	S,M,L*	S,M,L*	S,M,L*
Qwen3		S,M,L	S,M $\dagger$ ,L* $\dagger$	S,M,L	S,M,L	M,L	S,M,L*	S,M,L*	S,M,L*	S,M,L*
Qwen3.5		S,M,L	S* $\dagger$ ,M* $\dagger$ ,L*	S,M*,L	S,L*	x	S*,M,L	S,M,L*	S,M,L*	S,M,L*
Gemma		S,M,L	S* $\dagger$ ,M* $\dagger$ ,L*	S*,M,L	S,M,L*	L	S*,M,L	S,M,L*	S,M,L*	S,M,L*
LLaMA	x	S,M,L	S,M* $\dagger$ ,L* $\dagger$	L*	L	S,M	S,M,L*	S,M,L	S,M,L*	S,M,L*

Appendix

Table 7: Hypothesis support by model series in the hybrid cross-lingual setting . Cell shading follows the same scale as Table 6 (deeper darkblue

=

stronger support; rose

=

no support). Gemini models were not evaluated in this setting.

Figure A1: English: model accuracy by Bloom level and veracity condition in the fully translated setting.

Figure A2: Arabic: model accuracy by Bloom level and veracity condition in the fully translated setting.

Figure A3: Chinese: model accuracy by Bloom level and veracity condition in the fully translated setting.

Figure A4: Hindi: model accuracy by Bloom level and veracity condition in the fully translated setting.

Figure A5: Persian: model accuracy by Bloom level and veracity condition in the fully translated setting.

Figure A6: Spanish: model accuracy by Bloom level and veracity condition in the fully translated setting.

Language Model Reasoning Evaluation

See all 128 papers

Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation
Oct 2, 2026Oliver Hauck, Mario Sanz-Guerrero, Katharina von der WenseCross-Lingual Reasoning in Language Models Retrieval-Augmented Reasoning
Johannes Gutenberg University Mainz, Germany
Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning
Jun 11, 2026Zilin Xiao, Qi Ma, Chun-cheng Jason Chen +4Analogical Reasoning RL for Language Model Reasoning
Meta Superintelligence Labs · Rice University · Work partially done during internship at Meta
LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
Mar 6, 2026Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2RAG Evaluation LLM Evaluation
neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

Oct 2, 2026J/K move · Enter open · S save

Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation

Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense

Johannes Gutenberg University Mainz, Germany

Cross-Lingual Reasoning in Language Models Retrieval-Augmented Reasoning Agentic RAG Knowledge-Intensive QA

Open paper AskPDF arXiv

Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning language matches the language of the prompt -- but only for a setting where the model reasons over a short prompt. Here, we ask whether the same holds for retrieval-augmented generation (RAG), where the model must read and integrate a large amount of retrieved evidence in the target language. To study this, we build a fully monolingual German RAG question-answering testbed over the fictional world of the tabletop role-playing game The Dark Eye, a domain that is richly documented in German but too niche for the model to answer from memory, so that it has to rely on retrieval. Varying the forced reasoning language of an agentic RAG system on this testbed, we find that aligning the reasoning language with the language of the query and the retrieved documents helps. Forced German reasoning outperforms forced French, although the model benchmarks higher in French, so the benefit comes from alignment and not from language proficiency. The advantage grows when the retrieved context is richer and structure-aware. However, forced German only reaches the level of the model's native, unconstrained English reasoning without surpassing it, showing that native multilingual reasoning is needed. We publicly release the testbed and QA benchmark.

Abstract

Language Model Reasoning Evaluation

Explore similar work

Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation

Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation