Traditional retrieval-augmented generation (RAG) frameworks process documents without attending to their hierarchical structure, leading to poor performance, especially in low-resource languages such as Bengali. To address this, we propose a structure-aware RAG framework that models Bengali textbooks as hierarchical graphs and uses a contrastively trained graph neural network to retrieve a small set of relevant passages. These passages provide focused context for a large language model, enabling topic-specific multiple-choice question (MCQ) generation and in-domain answer prediction. Experimental results demonstrate that our framework outperforms strong dense retrieval baselines across retrieval metrics, produces more relevant MCQs, and achieves superior answer prediction accuracy.
Figures & tables
Figure 1: A simple schematic illustration of our textbook’s hierarchical graph. Seq = Sequential edge, Hier = Hierarchical edge
Figure 2: End-to-end BengaliMCQ framework
Figure 3: Excerpt of two test samples where passages highlighted in yellow color refer to query-relevant passages identified GNN but missed by DPR
Baseline
HR@5
Recall@5
nDCG@5
MRR@5
HR@10
Recall@10
nDCG@10
MRR@10
BM25
0.539
0.506
0.506
0.506
0.600
0.600
0.453
0.434
Translation Baseline (DPR)
0.753
0.710
0.618
0.608
0.821
0.782
0.642
0.617
Sparse Retrieval ( BGE-M3 )
0.795
0.760
0.676
0.666
0.852
0.820
0.696
0.674
DPR ( Harrier-OSS-v1 )
0.519
0.487
0.418
0.411
0.575
0.545
0.438
0.419
DPR ( Harrier-OSS Fine-tuned)
0.836
0.802
0.703
0.689
0.901
0.873
0.728
0.698
DPR ( BGE-M3 Fine-tuned)
0.842
0.803
0.707
0.696
0.899
0.866
0.729
0.703
Table 1: Retrieval performance comparison across different methods. Bold highlights the best scores, underlined values indicate second-best.
Baseline
IWF
QF
TR
DS
PPL
Div
Answ. Accuracy (%)
Aver. Context Length (Tokens)
Zero-shot
0.39
14.53
0.46
0.71
7.60
0.44
57.21
43,591
Few-shot
0.31
14.40
0.46
0.77
7.83
0.45
68.35
43,591
CoT
0.22
14.50
0.48
0.78
7.98
0.58
71.87
43,591
DPR + Reranker
0.27
13.96
0.59
0.75
7.88
0.52
82.80
2,825
GAT + Reranker
0.24
14.81
0.67
0.78
7.57
0.52
91.41
1,651
Table 2: Comparison of MCQ generation and answer prediction performance across different baselines. All metrics are reported as averages across all books for each baseline.
Dimension
Baseline
HR@5
Recall@5
nDCG@5
MRR@5
HR@10
Recall@10
nDCG@10
MRR@10
Edge Type
No Hier
0.013
0.012
0.007
0.006
0.016
0.014
0.008
0.007
No Seq
0.865
0.824
0.724
0.712
0.920
0.887
0.745
0.719
No Sem
0.865
0.826
0.727
0.716
0.920
0.886
0.748
0.723
All Edges
0.868
0.829
0.728
0.716
0.920
0.889
0.749
0.723
Passage Initialization
MoS
0.857
0.818
0.711
0.697
0.915
0.882
0.734
0.704
Zero
0.868
0.829
0.728
0.716
0.920
0.889
0.749
0.723
Table 3: Ablation study of the proposed GAT-based retrieval framework. The “All Edges”, “Zero” and “Using Temp” rows report the same reference configuration and are therefore identical.
Domain
N
Mean
% ≥3
Literature
60
3.17
76.7
Biology
40
3.58
92.5
Social Science ∗
40
3.13
80.0
Overall
140
3.27
82.1
∗ Averaged across both experts ( κ=0.55 ).
Table 4: Human evaluation of topic relevance (1 = not relevant, 2 = somewhat relevant, 3 = relevant, and 4 = highly relevant).