Natural language explanations produced by large language models (LLMs) are often persuasive, but not necessarily scrutable: users cannot easily verify whether the claims in an explanation are supported by evidence. In XAI, this motivates a focus on faithfulness and traceability, i.e., the extent to which an explanation's claims can be grounded in, and traced back to, an explicit source. We study these desiderata in retrieval-augmented generation (RAG) for programming education, where textbooks provide authoritative evidence. We benchmark six LLMs on 90 Stack Overflow questions grounded in three programming textbooks and quantify source faithfulness via source adherence metrics. We find that non Retrieval-Augmented Generation (RAG) models have median source adherence of 0%, while baseline RAG systems still exhibit low median adherence (22-40%, depending on the model). Motivated by Achinstein's illocutionary theory of explanation, we introduce illocutionary macro-planning as a descriptive design principle for source-faithful explanations and instantiate it with chain-of-illocution prompting (CoI), which expands a query into implicit explanatory questions that drive retrieval. Across models, CoI yields statistically significant gains (up to 63%) in source adherence, although absolute adherence remains moderate and the gains are weak or non-significant for some models. A user study with 165 retained participants (220 recruited) indicates that these gains do not harm satisfaction, relevance, or perceived correctness.
Figures & tables
Tag
Textbook
#Pages
#Words
Year
Copyright
java
[ 15 ]
773
350’935
2022
CC BY-NC-SA 4.0
python
[ 13 ]
244
68’202
2015
CC BY-NC 3.0
pharo
[ 14 ]
376
90’572
2018
CC BY-SA 3.0
Table 1: Statistics of Selected Textbooks.
Tag
Question Size Avg.
Body Size Avg.
Approved Answer Size Avg.
java
8.67
53.3
287.47
python
9.37
51.83
265.37
pharo
9.57
95.1
181.57
Table 2: Statistics of the selected Stack Overflow questions. Lengths are approximate whitespace-token counts computed on the raw exported Stack Overflow fields.
Model
Know. Cut-off
Size
Open Source
Max. Tokens
gpt-3.5-turbo
2021-09
-
No
16k
gpt-4o
2023-10
-
No
128k
Llama 3:8b-instruct
2023-12
8B
Yes
8k
Llama 3:70b-instruct
2023-12
70B
Yes
8k
mistral:7b-instruct
2021-08
7B
Yes
32k
mixtral:8x7b-instruct
2023-12
45B
Yes
32k
Table 3: Statistics of selected LLM .
Figure 1: Source faithfulness across six LLM , with one-sided p-values (significant in bold), Cohen’s dz effect sizes, and 95% confidence intervals (CI).
Figure 2: Chain-of-illocution pipeline. Black arrows denote the standard RAG flow from the user question to retrieval, prompting, and final explanation. Green arrows denote the CoI additions: an illocution planner that proposes implicit explanatory questions, per-question retrieval from the knowledge source, and the concatenation of question–context pairs (Q1,C1),…,(Qn,Cn) into the final prompt. The multiple green arrows between the retriever and the knowledge source indicate that evidence is fetched separately for multiple implicit questions, not only for the user’s original question.
Figure 3: Study interface with an explanation generated via chain-of-illocution.
Large language models (LLMs) achieve strong performance and have revolutionized NLP, but their lack of explainability keeps them treated as black boxes, limiting their use in domains that demand transparency and trust. A promising direction to address this issue is post-hoc text-based explanations, which aim to explain model decisions in natural language. Prior work has focused on generating convincing rationales that appear to be subjectively faithful, but it remains unclear whether these explanations are epistemically faithful, whether they reflect the internal evidence the model actually relied on for its decision. In this paper, we first assess the epistemic faithfulness of LLM-generated explanations via counterfactuals and show that they are often unfaithful. We then introduce a training-free method that enhances faithfulness by guiding explanation generation through attention-level interventions, informed by token-level heatmaps extracted via a faithful attribution method. This method significantly improves epistemic faithfulness across multiple models, benchmarks, and prompts.
Bar Alon, Itamar Zimerman, Lior Wolf
Blavatnik School of Computer Science, Tel Aviv University
Retrieval-Augmented Generation (RAG) aims to enhance the trustworthiness of Large Language Models (LLMs) by grounding their outputs in external documents, often using inline citations for verifiability. However, the faithfulness of these citations -- whether the model genuinely uses a source to generate an answer -- remains a critical, unverified assumption. This paper offers the first mechanistic account of how a large language model decides whether to attach an inline citation while answering a factoid question. Using the Llama-3.1-8B-Instruct model in a controlled experimental environment based on the PopQA dataset, we employ an activation patching approach. We map the underlying mechanism responsible for citation, discovering that it is not a single, localized component but a distributed, multi-stage "attributional ensemble" of attention heads and MLP layers. We show that amplifying or attenuating only those critical heads and MLPs repairs over 90% of missed citations and eliminates 69% of spurious ones on PopQA without harming answer accuracy. Although gains on the multi-document HotpotQA benchmark are modest, the same component set still moves citation rates in the intended direction, indicating that the underlying mechanism is not dataset-specific. The results reveal a potential disconnect between the model's apparent reasoning and its internal computational pathway, suggesting that inline citations can create a false sense of security.
Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning language matches the language of the prompt -- but only for a setting where the model reasons over a short prompt. Here, we ask whether the same holds for retrieval-augmented generation (RAG), where the model must read and integrate a large amount of retrieved evidence in the target language. To study this, we build a fully monolingual German RAG question-answering testbed over the fictional world of the tabletop role-playing game The Dark Eye, a domain that is richly documented in German but too niche for the model to answer from memory, so that it has to rely on retrieval. Varying the forced reasoning language of an agentic RAG system on this testbed, we find that aligning the reasoning language with the language of the query and the retrieved documents helps. Forced German reasoning outperforms forced French, although the model benchmarks higher in French, so the benefit comes from alignment and not from language proficiency. The advantage grows when the retrieved context is richer and structure-aware. However, forced German only reaches the level of the model's native, unconstrained English reasoning without surpassing it, showing that native multilingual reasoning is needed. We publicly release the testbed and QA benchmark.
Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense