Progressive Training for Explainable Citation-Grounded Dialogue: Reducing Hallucination to Zero in English-Hindi LLMs

Vedant Pandya

cs.CLMar 19, 2026

Progressive Training for Explainable Citation-Grounded Dialogue: Reducing Hallucination to Zero in English-Hindi LLMs

Authors: Vedant Pandya

Organizations: School of Artificial Intelligence and Data Engineering (SAIDE), Indian Institute of Technology Jodhpur

Abstract

Knowledge-grounded dialogue systems aim to generate informative, contextually relevant responses by conditioning on external knowledge sources. However, most existing approaches focus exclusively on English, lack explicit citation mechanisms for verifying factual claims, and offer limited transparency into model decision-making. We present XKD-Dial, a progressive four-stage training pipeline for explainable, knowledge-grounded dialogue generation in a bilingual (English-Hindi) setting, comprising: (1) multilingual adaptation, (2) English dialogue SFT with citation grounding, (3) bilingual dialogue SFT, and (4) GRPO alignment with citation-aware rewards. We evaluate six models spanning encoder-decoder (250M-3B) and decoder-only (1B-7B) architectures at every pipeline stage. Our key contributions are: (i) three post-hoc explainability analyses - cross-attention alignment, Integrated Gradients attribution, and occlusion-based causal grounding - applied systematically across the training trajectory to reveal how citation behaviour is learned, not only whether it is learned; (ii) citation-grounded SFT reduces hallucination to 0.0% for encoder-decoder models from Stage 2 onward; (iii) the progressive pipeline prevents catastrophic forgetting while improving Hindi capabilities; (iv) smaller models match larger models on English after SFT; and (v) GRPO provides marginal improvement over well-designed SFT for structured citation tasks. We evaluate across six automatic metrics (BLEU, ROUGE, BERTScore, FactScore, Citation-F1, and hallucination rate).

Figures & tables

System	Cite	Hindi	RL	XAI
Wizard of Wikipedia	×	×	×	×
FaithDial	×	×	×	×
BlenderBot 2.0	×	×	×	×
RAG	×	×	×	×
Atlas	×	×	×	×
InstructGPT	×	×	✓	×

Table 1: Comparison with related work across key dimensions. XKD-Dial is the first to address all four dimensions simultaneously.

Model	Type	Params	Layers
Flan-T5-Base	Enc-Dec	250M	12+12
Flan-T5-Large	Enc-Dec	780M	24+24
Flan-T5-XL	Enc-Dec	3B	24+24
LLaMA-3.2-1B	Dec-Only	1B	16
Gemma-2-2B	Dec-Only	2B	26
Mistral-7B	Dec-Only	7B	32

Table 2: Model architectures used in our study.

Sym.	Component	$w$
$r_{\text{fact}}$	Factual consistency (NLI)	5.0
$r_{\text{ent}}$	Entity overlap	3.0
$r_{\text{attr}}$	Citation attribution	1.5
$r_{\text{flu}}$	Fluency proxy	1.0
$r_{\text{len}}$	Length penalty	$-$ 0.1
$r_{\text{hal}}$	Hallucination penalty	$-$ 10.0

Table 3: GRPO reward function components with weights.

Split	Total	EN	HI	EN%
Train	135,000	71,957	63,043	53.3
Val	7,500	4,014	3,486	53.5
Test	7,500	4,029	3,471	53.7

Table 4: Dataset statistics. All splits maintain a balanced English–Hindi distribution.

Param.	Base	Large	XL	Mistr.
Stages 1–3: Supervised Fine-Tuning
LR	5e-5	3e-5	3e-5	1e-5
Batch	8	4	2	2
Grad.Acc.	4	8	16	16
Eff.Batch	64	64	64	64
Epochs	3	3	3	3

Table 5: Training hyperparameters. Effective batch size is held constant at 64 via gradient accumulation.

Model	Stage	BLEU	ROUGE-1	ROUGE-L	FactScore	Cit-F1	Halluc.	BERTScore
Encoder-Decoder Models
Flan-T5-Base (250M)	Baseline	0.004	0.201	0.201	0.059	0.673	0.005	0.551
	Stage 1	0.005 $\sim$	0.238 $\uparrow$	0.237 $\uparrow$	0.056 $\sim$	0.738 $\uparrow$	0.028 $\uparrow$	0.611 $\uparrow$
	Stage 2	0.094 $\uparrow$	0.412 $\uparrow$	0.388 $\uparrow$	0.106 $\uparrow$	0.859 $\uparrow$	0.000 $\downarrow$	0.739 $\uparrow$
	Stage 3	0.092 $\sim$	0.507 $\uparrow$	0.483 $\uparrow$	0.096 $\sim$	0.902 $\uparrow$	0.000 $\sim$	0.766 $\uparrow$
	Stage 4	0.092 $\sim$	0.507 $\sim$	0.483 $\sim$	0.098 $\sim$	0.902 $\sim$	0.000 $\sim$	0.766 $\sim$

Table 6: Overall evaluation results across all stages. Best result per model is bolded . “

\dagger

” indicates generation collapse (empty outputs). Hallucination rate of 0.0 from Stage 2 onward is highlighted.

Model	Stage	BLEU	ROUGE-1	ROUGE-L	FactScore	Cit-F1	Halluc.	BERTScore
Encoder-Decoder Models
Base (250M)	Baseline	0.007	0.100	0.100	0.039	0.960	0.005	0.836
	Stage 1	0.007 $\sim$	0.101 $\sim$	0.101 $\sim$	0.040 $\sim$	0.956 $\sim$	0.004 $\sim$	0.835 $\sim$
	Stage 2	0.172 $\uparrow$	0.354 $\uparrow$	0.308 $\uparrow$	0.121 $\uparrow$	0.980 $\uparrow$	0.000 $\downarrow$	0.889 $\uparrow$
	Stage 3	0.170 $\sim$	0.349 $\sim$	0.304 $\sim$	0.120 $\sim$	0.980 $\sim$	0.000 $\sim$	0.889 $\sim$
	Stage 4	0.170 $\sim$	0.349 $\sim$	0.304 $\sim$	0.124 $\sim$	0.980 $\sim$	0.000 $\sim$	0.889 $\sim$

Table 7: English evaluation results. After Stage 2 SFT, Base (250M) and Large (780M) converge to identical performance across all metrics. Citation F1 reaches 0.980 for both models.

Model	Stage	BLEU	ROUGE-1	ROUGE-L	FactScore	Cit-F1	Halluc.	BERTScore
Encoder-Decoder Models
Base (250M)	Baseline	0.000	0.318	0.318	0.082	0.340	0.005	0.221
	Stage 1	0.003 $\sim$	0.396 $\uparrow$	0.395 $\uparrow$	0.075 $\sim$	0.485 $\uparrow$	0.056 $\uparrow$	0.351 $\uparrow$
	Stage 2	0.003 $\sim$	0.481 $\uparrow$	0.481 $\uparrow$	0.088 $\uparrow$	0.718 $\uparrow$	0.000 $\downarrow$	0.565 $\uparrow$
	Stage 3	0.001 $\sim$	0.691 $\uparrow$	0.691 $\uparrow$	0.069 $\downarrow$	0.812 $\uparrow$	0.000 $\sim$	0.624 $\uparrow$
	Stage 4	0.001 $\sim$	0.690 $\sim$	0.690 $\sim$	0.069 $\sim$	0.811 $\sim$	0.000 $\sim$	0.623 $\sim$

Table 8: Hindi evaluation results. Stage 1 builds the foundation, Stage 2 transfers citation skills, and Stage 3 provides the largest Hindi improvement. Note: Hindi BLEU is near-zero due to morphological variation (see Section 6 ).

Figure 3: Per-language BLEU progression across training stages. English BLEU (left) shows convergence of all models after Stage 2 SFT, while Hindi BLEU (right) remains near-zero for encoder-decoder models due to morphological variation (Section 6.3 ). Gemma-2-2B achieves the highest Hindi BLEU (0.155) after Stage 3, followed by Mistral-7B (0.070).

	Metric	Stage 3	$\Delta$
Encoder-Decoder
Base	Cit-F1	0.902	0.000
	Halluc.	0.000	0.000
	BERTScore	0.766	0.000
	FactScore	0.096	$+$ 0.002
Large	Cit-F1	0.896	$-$ 0.001

Table 9: Change from Stage 3 to Stage 4 (GRPO). Deltas are negligible for encoder-decoder models. Mistral shows marginal gains in Citation F1 and FactScore.

	Model	Best	Final	$\Delta$
Enc-Dec	Base (250M)	1.040	0.648	$-$ 0.391 $\downarrow$
	Large (780M)	0.868	0.157	$-$ 0.711 $\downarrow$
	XL (3B)	1.782	0.482	$-$ 1.300 $\downarrow$
Dec-Only	LLaMA (1B)	3.08	2.51	$-$ 0.57 $\downarrow$
	Gemma (2B)	3.40	3.40	0.00
	Mistral (7B)	2.889	2.433	$-$ 0.457 $\downarrow$

Table 10: GRPO reward trajectory during Stage 4 training. Encoder-decoder and decoder-only models (LLaMA, Mistral) show reward decline from best to final, suggesting training instability. Gemma-2-2B is the exception: its best reward equals its final reward, indicating reward improvement continued to the last step.

	Model	Stage	Steps	Best Loss
Encoder-Decoder	Base	Stage 1	3,906	1.552
		Stage 2	7,500	1.916
		Stage 3	4,000	0.945
	Large	Stage 1	3,906	1.439
		Stage 2	8,973	1.776
		Stage 3	4,000	0.912

Table 11: Training summary showing total steps and best validation loss per stage. Mistral-7B consistently achieves the lowest loss across all stages.

Model	Base	S1	S2	S3	S4
Flan-T5-Base	0.017	0.022	0.035	0.019	0.019
Flan-T5-Large	0.030	0.032	0.037	0.022	0.022
Flan-T5-XL	0.003	0.004	0.000 †	0.026	0.026
Gemma-2-2B	N/A (decoder-only)
LLaMA-3.2-1B	N/A (decoder-only)
Mistral-7B	N/A (decoder-only)

Table 12: Mean cross-attention alignment with cited knowledge tokens (encoder-decoder models only). Decoder-only models show 0.000 by architecture. Stage 2 collapse for XL is marked with

\dagger

Figure 6: Representative cross-attention heatmap for Flan-T5-Base (Stage 3). Output tokens (y-axis) attend to input knowledge tokens (x-axis). Brighter cells indicate higher attention weight. Citation-relevant tokens receive disproportionately high attention.

	Baseline		Stage 2		Stage 4
Model	Entr.	Conc.	Entr.	Conc.	Entr.	Conc.
Base	4.36	0.378	4.44	0.343	4.50	0.326
Large	4.45	0.327	4.48	0.322	4.49	0.320
XL	4.38	0.360	NaN	NaN	4.53	0.306

Table 13: Gradient saliency entropy and concentration for Flan-T5 encoder-decoder models across training stages. Higher entropy = more diffuse attribution; lower concentration = less dominated by top tokens. XL Stage 2 NaN confirms generation collapse.

Model	Baseline	Stage 1	Stage 3	Stage 4
Flan-T5-Base	0.647	0.691	0.889	0.889
Flan-T5-Large	0.656	0.630	0.909	0.909
Flan-T5-XL	0.960	0.964	0.808	0.808
LLaMA-3.2-1B	0.696	0.544	0.722	0.632
Gemma-2-2B ‡	0.733	0.000	0.000	0.000
Mistral-7B †	0.767	0.797	0.000	0.000

Table 14: Occlusion causal grounding: fraction of citations that disappear when source is removed. Higher = stronger causal grounding. Both Mistral-7B and Gemma-2-2B drop to 0.000 post-training, indicating decoder-only models learn citation format without source-conditioned grounding.

L	Query (excerpt)	Expected Output	LLaMA-3.2-1B Output	Cit.
EN	Can you bring your own liquor to Pizza Hut Cherry Hinton?	As noted in [1], No, outside beverages are not allowed.	No, you cannot bring your own liquor at Pizza Hut Cherry Hinton. Do you have any other questions? Would you like to make a reservation? $\ldots$	0
EN	How long did California Love stay on Billboard Hot 100?	Based on [1], yeah, stayed on the billboard hot 100’s for two weeks straight!	It was one of 2Pac’s most popular singles, reaching number one on the Billboard Hot 100 for two weeks. It was a double A-side single $\ldots$	0
EN	Restaurant booking confirmation	According to [1], booking was successful. Ref: ENOALFAV .	Your reservation is all set. The reference number is 1ZQD0Z1A . $\ldots$	0
HI	[ Query about wheelchair accessibility at Prezzo ]	[2] \devanagarifont में कहा गया है कि हां, प्रेज़ो में व्हीलचेयर उपलब्ध है।	[2] \devanagarifont के आधार पर, हां, प्रेज़ो में व्हीलचेयर उपलब्ध है इसलिए उन्हें आपकी सहायता करने में सक्षम होना चाहिए।	[2] ✓
HI	[ Query about the color black ]	[1] \devanagarifont में उल्लेख है कि काले रंग के बारे में, यह पूरी तरह से दिखाई देने वाले प्रकाश को अवशोषित करता है।	[1] \devanagarifont में कहा गया है कि मुझे नहीं पता था कि यह था, काले रंग के बारे में…	[1] ✓

Table 15: Representative LLaMA-3.2-1B Stage 3 outputs vs. expected references. EN outputs contain zero citation markers despite the model having access to the numbered knowledge passage. HI outputs correctly produce [N] markers.

Figure 17: KL divergence (policy vs. reference) for Mistral-7B during GRPO. The negative KL indicates the policy diverges from the reference, stabilizing around $-$ 2.0 after 300 steps despite the KL penalty ( $\beta=0.04$ ).

Grounded Text Generation

Grounded text generation produces language tied to verifiable evidence, structured data, or perceptual context to limit unsupported claims.

See all 31 papers

Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations
Apr 25, 2026Bhaskar Singh, Shobhit Banga, Mahima Manik +1Speech Language Models Full-Duplex Spoken Dialogue Systems
JoshTalks · Independent
Lost in Interpretation: The Plausibility-Faithfulness Trade-off in Cross-Lingual Explanations
May 19, 2026Somnath Banerjee, Pranav Jha, Rima Hazra +1Language Model Auditing Cross-Lingual Reasoning in Language Models
Indian Institute of Technology Kharagpur · TCG Crest · National University of Singapore
KARLA: Knowledge-base Augmented Retrieval for Language Models
Jun 25, 2026Francois Crespin, Fabian M. Suchanek, Nils HolzenbergerRetrieval-Augmented Generation LLM Grounding
IP Paris, LTCI · Télécom Paris · Institut Polytechnique de Paris, France

Apr 25, 2026J/K move · Enter open · S save

Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations

Bhaskar Singh, Shobhit Banga, Mahima Manik, Pranav Sharma

JoshTalks · Independent

Speech Language Models Full-Duplex Spoken Dialogue Systems Turn-Taking in Dialogue

Open paper AskPDF arXiv

Full-duplex spoken dialogue systems can model natural conversational behaviours such as interruptions, overlaps, and backchannels, yet such systems remain largely unexplored for Indian languages. We present the first open, reproducible full-duplex spoken dialogue system for Hindi by adapting Moshi, a state-of-the-art duplex speech architecture, using a custom Hindi tokeniser and training on 26,000 hours of real spontaneous conversations collected from 14,695 speakers with separate speaker channels, enabling direct learning of turn-taking and overlap patterns from natural interactions. To support Hindi text generation, we replace the original English tokeniser and reinitialise text-vocabulary-dependent parameters while retaining the pre-trained audio components. We propose a two-stage training recipe -- large-scale pre-training followed by fine-tuning on 1,000 hours of conversational data. Evaluation through the prompted dialogue continuation paradigm with both automatic metrics and human judgments demonstrates that the resulting model generates natural and meaningful full-duplex conversational behaviour in Hindi. This work serves as a first step toward real-time duplex spoken dialogue systems for Hindi and other Indian languages.

Abstract

Grounded Text Generation

Explore similar work

Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations

Lost in Interpretation: The Plausibility-Faithfulness Trade-off in Cross-Lingual Explanations

KARLA: Knowledge-base Augmented Retrieval for Language Models