Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning language matches the language of the prompt -- but only for a setting where the model reasons over a short prompt. Here, we ask whether the same holds for retrieval-augmented generation (RAG), where the model must read and integrate a large amount of retrieved evidence in the target language. To study this, we build a fully monolingual German RAG question-answering testbed over the fictional world of the tabletop role-playing game The Dark Eye, a domain that is richly documented in German but too niche for the model to answer from memory, so that it has to rely on retrieval. Varying the forced reasoning language of an agentic RAG system on this testbed, we find that aligning the reasoning language with the language of the query and the retrieved documents helps. Forced German reasoning outperforms forced French, although the model benchmarks higher in French, so the benefit comes from alignment and not from language proficiency. The advantage grows when the retrieved context is richer and structure-aware. However, forced German only reaches the level of the model's native, unconstrained English reasoning without surpassing it, showing that native multilingual reasoning is needed. We publicly release the testbed and QA benchmark.
Figures & tables
Figure 1: Our setting. An agentic RAG system answers German questions using retrieved German sourcebook passages; we control the language the model reasons in. In this real example, English reasoning drops the umlaut of the river Südask , while German reasoning keeps it.
Figure 3: Architecture of our agentic RAG system. Given a German query, the LLM reasons and then either issues a retrieval call or produces the final answer. Retrieval embeds the query, runs similarity search over the hierarchical German knowledge base, and reranks the hits to the top-3 chunks, which are appended to the context; the loop repeats until the model answers, always in German. The one control we vary is the language the model is forced to reason in (German, English, French, or none).
SectionChunks
SimpleChunks
Book
C
TC
C~
TC~
G01
738
499
736
501
G05
665
549
730
501
G08
843
466
784
501
Total
2246
501
2250
501
Table 1: Knowledge-base statistics. C is the number of chunks and TC the mean chunk size in tokens. SimpleChunks uses a fixed chunk size TC~ equal to the mean SectionChunks chunk size, so both hold the same 1.13M tokens in total at comparable granularity.
Reasoning
1
2
3
4
5
Avg.
NoRAG (no retrieval)
small model
478
49
40
5
13
1.335
large model
490
36
36
11
12
1.323
SimpleChunks (fixed-size)
German
115
36
130
103
201
3.409
English
111
45
118
82
229
3.467
Table 2: Single-passage set (585 questions), LLM-as-a-Judge ratings (1–5). Columns 1–5 give how many outputs received each rating; Avg. is the mean. All questions and answers are in German; Reasoning is the language the model is forced to reason in. Pairwise significance tests are in Appendix D .
Figure 4: Average ratings on the single-passage set. Retrieval (orange, green) is far above no-retrieval (blue). Among reasoning settings on the structure-aware SectionChunks base (green), forced German rises above forced French and reaches forced and unconstrained English, while disabling reasoning is highest.
Figure 5: Full rating distribution per configuration on the single-passage set (Table 2 as shares). Colour encodes the judge rating from 1 (orange, hallucinated) to 5 (blue, correct). Retrieval moves the mass from rating 1 toward rating 5; for the retrieval settings rating 2 is the rarest.
Reasoning
RA
RB
RC
Avg.
NoRAG (large)
1.20
1.13
1.23
1.189
no reasoning
2.87
2.90
2.57
2.778
French
2.87
2.70
2.77
2.778
German
2.87
3.03
2.77
2.889
English
3.30
2.83
2.87
3.000
unconstrained
3.13
3.17
2.73
3.011
Table 3: Multi-hop set (30 questions), human ratings (1–5) from three expert reviewers RA,RB,RC . All use the SectionChunks knowledge base.
Human rating
Judge
1
2
3
4
5
1
12
1
2
2
0
2
0
1
0
0
0
3
0
1
4
4
0
4
0
1
0
2
2
5
0
0
0
4
14
Table 4: LLM-as-a-Judge vs. human ratings on 50 sampled answers (five per configuration). Exact agreement ( diagonal ) is 66%; the larger disagreements sit in the top row, where the judge assigns 1 to an answer the human rates 3–4.
Figure 6: Mean answer length by judge rating, pooled over the eight retrieval configurations on the single-passage set (4,680 graded answers). Higher-rated answers are shorter (Spearman ρ=−0.39 , p<0.001 ); colour encodes the rating. This is direct evidence for the judge’s length bias.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Comparison
Δ
W/T/L
p
pHolm
no reason. > French
0.41
214/263/108
8.2⋅10−9
8.2⋅10−8 ***
no reason. > English
0.23
195/265/125
.0008
.007 **
unconstr. > French
0.23
193/254/138
.0013
.010 *
German > French
0.21
182/265/138
.0027
.019 *
no reason. > German
0.20
190/270/125
.0033
.020 *
no reason. > unconstr.
0.19
185/273/127
.0080
.040 *
Appendix
Table 5: Pairwise Wilcoxon signed-rank tests on the single-passage set (585 paired ratings), for the five SectionChunks reasoning settings. Δ is the difference in mean rating; W/T/L counts questions where the first setting wins, ties, or loses; pHolm is corrected across the ten pairs. *** p<.001 , ** p<.01 , * p<.05 .
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · MYbank, AntGroup