KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

Mohammad Amanlou; Erfan Shafiee Moghaddam; Yasaman Amou Jafari; Mahdi Noori; Farhan Farsi; Behnam Bahrak

cs.CLFeb 23, 2026

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

Authors: Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari, Mahdi Noori, Farhan Farsi, Behnam Bahrak

Organizations: University of Tehran · Independent Researcher · Amirkabir University of Technology · TEIAS Institute

Abstract

With the rise of large language models (LLMs), they have become instrumental in applications such as Retrieval-Augmented Generation (RAG). Yet evaluating these systems remains bottlenecked by the time and cost of building specialized assessment datasets. We introduce KNIGHT, an LLM-based, knowledge-graph-driven framework for generating multiple-choice question (MCQ) datasets from external sources. KNIGHT constructs a topic-specific knowledge graph, a structured and parsimonious summary of entities and relations, that can be reused to generate instructor-controlled difficulty levels, including multi-hop questions, without repeatedly re-feeding the full source text. This knowledge graph acts as a compressed, reusable state, making question generation a cheap read over the graph. We instantiate KNIGHT on Wikipedia/Wikidata while keeping the framework domain- and ontology-agnostic. As a case study, KNIGHT produces six MCQ datasets in History, Biology, and Mathematics. We evaluate quality on five criteria: fluency, unambiguity (single correct answer), topic relevance, option uniqueness, and answerability given the provided sources (as a proxy for hallucination). Results show that KNIGHT enables token- and cost-efficient generation from a reusable graph representation, achieves high quality across these criteria, and yields model rankings aligned with MMLU-style benchmarks, while supporting topic-specific and difficulty-controlled evaluation.

Figures & tables

Figure 1: KNIGHT High-level pipeline. Given a prompt/topic and depth, KNIGHT retrieves evidence, builds a focused KG, generates MCQs, and filters them to produce the final dataset.

Figure 2: KNIGHT architecture. (Left) A topic/prompt-driven RAG pipeline retrieves evidence, extracts triples, and curates a compact KG under depth budget $d_{\max}$ ; (Right) multi-hop paths are sampled to generate questions/distractors and validated for evidence-grounded answerability to form the final MCQA dataset.

	Grammar Accuracy $\uparrow$					Fluency-automatic $\uparrow$					Fluency-human $\uparrow$
Topic	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT
History	0.9994	0.9993	0.9992	0.9990	0.9989	0.9498	0.9582	0.9536	0.9549	0.9581	4.8/5	4.7/5	4.7/5	4.7/5	4.8/5
Biology	0.9992	0.9994	0.9995	0.9989	0.9998	0.9702	0.9653	0.9681	0.9591	0.9626	4.9/5	4.9/5	4.8/5	4.8/5	4.9/5
Math	0.9978	0.9986	0.9981	0.9983	0.9991	0.9711	0.9671	0.9658	0.9602	0.9685	4.9/5	4.8/5	4.7/5	4.8/5	4.7/5

Table 1: Linguistic quality aggregated over Levels. Systems: Plain GPT-4o-mini, GPT-4o-mini+RAG, GPT-4o-mini+RAG+KG, GPT-4o-mini+RAG+Validator, and KNIGHT .

	REPEATED $\downarrow$					SINGLE_KEY $\downarrow$					OPTION_UNIQUENESS $\downarrow$					ANSWERABLE $\downarrow$
Split	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT
History (L1)	20	19	15	2	0	10	4	6	3	2	8	5	5	5	3	26	10	8	10	6
Biology (L1)	19	17	16	3	1	16	5	6	3	1	4	3	5	5	2	19	8	7	9	4
Math (L1)	13	14	14	1	0	11	4	6	4	2	5	4	4	4	2	20	8	6	8	5
History (L3)	14	11	9	1	1	15	6	7	7	2	7	5	5	4	3	24	13	9	12	6
Biology (L3)	15	13	12	1	1	14	5	6	5	2	6	4	4	3	1	21	10	7	8	4

Table 2: Human audit flags per 100 items (lower is better). All systems use GPT-4o-mini : Plain, RAG, RAG+KG, RAG+Val, and KNIGHT .

	Topic Relevance Score (Entailment) $\uparrow$					Topic Relevance (LLM) $\uparrow$					Human TOPIC Flags $\downarrow$					Off-topic Rate (LLM $\cap$ Entailment) $\downarrow$
Split	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT
History (L1)	0.9432	0.9938	0.8080	0.9945	0.8214	0.8318	0.9116	0.7301	0.9218	0.7692	7	5	17	4	8	6%	4%	19.3%	3%	10.6%
Biology (L1)	0.9156	0.9885	0.7962	0.9888	0.9053	0.8611	0.9542	0.7260	0.9619	0.7979	4	3	15	3	6	3%	1%	18.0%	1%	5.5%
Math (L1)	0.9219	0.9983	0.8103	0.9989	0.8852	0.8322	0.9092	0.8211	0.9514	0.8418	8	5	14	5	7	7%	4%	14.7%	4%	7.3%
History (L3)	0.9086	0.9975	0.5765	0.9981	0.8974	0.8115	0.8884	0.5291	0.9384	0.8124	9	6	42	5	9	8%	6%	34.4%	5%	8.1%
Biology (L3)	0.9309	0.9975	0.5971	0.9977	0.8832	0.8575	0.9205	0.5430	0.9381	0.8966	8	5	39	4	3	5%	3%	22.4%	3%	3.4%

Table 3: Topic relevance (entailment and LLM;

\uparrow

), expert TOPIC flags (

\downarrow

), and off-topic rate as the intersection of automated checks (LLM

\cap

Entailment;

\downarrow

) across systems (all with GPT-4o-mini ).

	Mean Entropy $H$ $\uparrow$					Std. Dev. of $H$ $\uparrow$					Probe Acc. (%) $\downarrow$
Split	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT	Plain	RAG	RAG+KG	RAG+Val	KNIGHT
History (L1)	0.0	0.0106	0.0122	0.0098	0.0134	0.0	0.0058	0.0692	0.0046	0.0855	100.00	99.00	88.00	98.00	86.83
Biology (L1)	0.0	0.0038	0.0096	0.0046	0.0189	0.0	0.0007	0.0897	0.0004	0.0803	100.00	98.00	89.00	99.00	86.21
Math (L1)	0.0	0.0021	0.0256	0.0039	0.0231	0.0	0.0004	0.0991	0.0005	0.1084	100.00	100.00	84.00	98.00	84.29
History (L3)	0.0	0.0	0.0435	0.0011	0.0489	0.0	0.0	0.1846	0.0002	0.1703	100.00	99.00	69.00	99.00	66.29
Biology (L3)	0.0	0.0017	0.0191	0.0039	0.0278	0.0	0.0003	0.1156	0.0003	0.1144	100.00	99.00	71.00	99.00	66.48

Table 4: Distractor competition via predictive entropy. Mean and standard deviation of predictive entropy (

H

) (higher

\uparrow

= more competitive distractors) and probe accuracy (lower

\downarrow

= harder items) using LLaMA 3.2-3B-Instruct as a fixed probe.

	History		Biology		Math		KNIGHT Avg.	MMLU	ARC	CSQA	RACE	MedMCQA	OBQA	Bench Avg.
Model	L1	L3	L1	L3	L1	L3
GPT-4o [ 63 ]	92.95	86.39	95.98	87.11	95.30	86.41	90.52 1st	79.09	86.31	70.28	67.87	57.85	67.21	71.45 1st
Mistral Large [ 64 ]	92.19	84.16	95.18	86.84	95.07	84.10	89.59 2nd	68.76	72.32	55.35	70.17	43.44	58.66	61.45 2nd
Llama3-70B-Instruct [ 65 ]	92.12	85.15	94.12	86.08	95.03	84.67	89.53 3rd	59.67	67.09	55.49	58.21	41.67	40.94	53.85 3rd
Claude 3 Haiku [ 66 ]	91.95	81.64	95.05	83.40	93.97	81.60	87.70	57.35	63.89	55.87	57.20	40.57	42.32	52.87
Qwen1.5 (1.8B) [ 67 ]	76.76	72.45	79.88	72.89	75.84	71.43	74.88	9.99	15.84	31.13	34.91	4.70	20.37	19.49

Table 5: Benchmark utility of KNIGHT . Accuracy (%) of multiple models on KNIGHT , alongside standard MCQ benchmarks. KNIGHT Avg. averages over domains and levels; Bench Avg. averages over external suites. Superscripts denote rank by the corresponding average.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Distribution of question lengths for each dataset (histograms), demonstrating an approximately normal shape.

Dataset	GRAMMAR	SINGLE_KEY	OPTION_UNIQUENESS	ANSWERABLE	TOPIC
Hist-1	1	2	3	6	9
Bio-1	0	1	2	4	7
Math-1	2	2	2	5	8
Hist-3	2	2	3	6	10
Bio-3	1	2	1	4	4
Math-3	2	3	2	6	4

Appendix

Table 6: Aggregated expert-raised flags indicating potential quality violations by dataset and criterion. Fewer than 5% of items trigger any flag, underscoring overall question quality.

Split (Entailment)	Welch $t$ ( $p$ )	MWU ( $p$ )	BM ( $p$ )	Hedges’ $g$ Cliff’s $\delta$	Holm $p$
History (L1)	$-0.98$ ( $0.33$ )	$p=0.41$	$p=0.37$	$-0.10\;\|\;-0.06$	$0.44$
Biology (L1)	$-1.12$ ( $0.26$ )	$p=0.49$	$p=0.44$	$-0.09\;\|\;-0.05$	$0.52$
Math (L1)	$-1.35$ ( $0.18$ )	$p=0.38$	$p=0.31$	$-0.12\;\|\;-0.07$	$\mathbf{0.18}$
History (L3)	$-1.28$ ( $0.20$ )	$p=0.46$	$p=0.40$	$-0.11\;\|\;-0.06$	$0.39$
Biology (L3)	$-0.89$ ( $0.37$ )	$p=0.52$	$p=0.48$	$-0.08\;\|\;-0.04$	$0.60$
Math (L3)	$-1.41$ ( $0.16$ )	$p=0.35$	$p=0.29$	$-0.13\;\|\;-0.07$	$0.21$

Appendix

Table 7: Per-item topicality comparison (MNLI-based entailment). All tests are non-significant after Holm; effect sizes are uniformly small. The minimum Holm-adjusted

p

across splits is

\approx 0.18

Split (LLM)	Welch $t$ ( $p$ )	MWU ( $p$ )	BM ( $p$ )	Hedges’ $g$ Cliff’s $\delta$	Holm $p$
History (L1)	$-1.05$ ( $0.29$ )	$p=0.43$	$p=0.39$	$-0.11\;\|\;-0.06$	$0.46$
Biology (L1)	$-0.92$ ( $0.36$ )	$p=0.47$	$p=0.41$	$-0.10\;\|\;-0.05$	$0.50$
Math (L1)	$-1.22$ ( $0.22$ )	$p=0.39$	$p=0.33$	$-0.12\;\|\;-0.07$	$0.22$
History (L3)	$-1.18$ ( $0.24$ )	$p=0.45$	$p=0.40$	$-0.11\;\|\;-0.06$	$0.41$
Biology (L3)	$-0.71$ ( $0.48$ )	$p=0.54$	$p=0.50$	$-0.08\;\|\;-0.04$	$0.65$
Math (L3)	$-1.36$ ( $0.17$ )	$p=0.36$	$p=0.30$	$-0.13\;\|\;-0.07$	$0.20$

Appendix

Table 8: Per-item topicality comparison (LLM-based topicality score). All tests are non-significant after Holm; effect sizes are small.

Figure 4: Entropy distributions by topic and difficulty level visualized using boxen plots with swarm overlays. Difficulty level 3 datasets consistently show higher entropy and wider distributions, reflecting greater model uncertainty.

Figure 5: Knowledge-graph traversal from Hafez to Shiraz , 7th century , Iran , and >90 million , generating MCQ templates of increasing difficulty. The purple path denotes a 1-hop (Level 1) question.

Multiple-Choice Question Generation

See all 8 papers

Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models
Apr 12, 2026Mehmet Can Şakiroğlu, H. Altay Güvenir, Kamer KayaMultiple-Choice Question Generation Automated Test Generation
Computer Engineering Department, Bilkent University, Ankara, Turkey · Faculty of Engineering and Natural Sciences, Sabanci University · VERIM, Center of Excellence in Data Analytics, Sabanci University
BioGraphletQA: Knowledge-Anchored Generation of Complex QA Datasets
Apr 28, 2026Richard A. A. Jonker, Bárbara Maria Ribeiro de Abreu Martins, Sérgio MatosBiomedical QA Synthetic Benchmark Generation
IEETA, DETI, LASI, University of Aveiro, Aveiro, Portugal · USF Atlântico Norte, Portugal
Knowledge-Graph Based Augmentation versus Retrieval Augmented Generation for Cultural-Related Question Answering
Sep 16, 2026Pablo Poulenard, Yannis Karmim, Valentin BarrièreRetrieval-Augmented Generation Graph Retrieval-Augmented Generation
DCC, Universidad de Chile, Santiago, Chile · École Polytechnique, Palaiseau, France · Inria, Almanach, Paris, France +2

Apr 12, 2026J/K move · Enter open · S save

Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models

Mehmet Can Şakiroğlu, H. Altay Güvenir, Kamer Kaya

Computer Engineering Department, Bilkent University, Ankara, Turkey · Faculty of Engineering and Natural Sciences, Sabanci University · VERIM, Center of Excellence in Data Analytics, Sabanci University

Multiple-Choice Question Generation Automated Test Generation Item Difficulty Estimation Educational Assessment

Open paper AskPDF arXiv

Generating multiple-choice questions (MCQs) with difficulty estimation remains challenging in automated MCQ-generation systems used in adaptive, AI-assisted education. This study proposes a novel methodology for generating MCQs with difficulty estimation from the input documents by utilizing knowledge graphs (KGs) and large language models (LLMs). Our approach uses an LLM to construct a KG from input documents, from which MCQs are then systematically generated. Each MCQ is generated by selecting a node from the KG as the key, sampling a related triple or quintuple -- optionally augmented with an extra triple -- and prompting an LLM to generate a corresponding stem from these graph components. Distractors are then selected from the KG. For each MCQ, nine difficulty signals are computed and combined into a unified difficulty score using a data-driven approach. Experimental results demonstrate that our method generates high-quality MCQs whose difficulty estimation is interpretable and aligns with human perceptions. Our approach improves automated MCQ generation by integrating structured knowledge representations with LLMs and a data-driven difficulty estimation model.

Abstract

Multiple-Choice Question Generation

Explore similar work

Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models

BioGraphletQA: Knowledge-Anchored Generation of Complex QA Datasets

Knowledge-Graph Based Augmentation versus Retrieval Augmented Generation for Cultural-Related Question Answering