Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models

Mehmet Can Şakiroğlu; H. Altay Güvenir; Kamer Kaya

cs.CLApr 12, 2026

Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models

Authors: Mehmet Can Şakiroğlu, H. Altay Güvenir, Kamer Kaya

Organizations: Computer Engineering Department, Bilkent University, Ankara, Turkey · Faculty of Engineering and Natural Sciences, Sabanci University · VERIM, Center of Excellence in Data Analytics, Sabanci University

Abstract

Generating multiple-choice questions (MCQs) with difficulty estimation remains challenging in automated MCQ-generation systems used in adaptive, AI-assisted education. This study proposes a novel methodology for generating MCQs with difficulty estimation from the input documents by utilizing knowledge graphs (KGs) and large language models (LLMs). Our approach uses an LLM to construct a KG from input documents, from which MCQs are then systematically generated. Each MCQ is generated by selecting a node from the KG as the key, sampling a related triple or quintuple -- optionally augmented with an extra triple -- and prompting an LLM to generate a corresponding stem from these graph components. Distractors are then selected from the KG. For each MCQ, nine difficulty signals are computed and combined into a unified difficulty score using a data-driven approach. Experimental results demonstrate that our method generates high-quality MCQs whose difficulty estimation is interpretable and aligns with human perceptions. Our approach improves automated MCQ generation by integrating structured knowledge representations with LLMs and a data-driven difficulty estimation model.

Figures & tables

Figure 1: An example multiple-choice question.

Figure 2: An illustrative example of a knowledge graph representing entities, their types, and the semantic relationships between them.

	Input	QG Model	Mul.-hop QG	MCQ Gen.	Difficulty Modeling	Human Calib. in Diff. Mod.
Reddy et al. [2017]	KG	RNN-based (seq2seq)	✗	✗	-	-
Elsahar et al. [2018]	KG triple and related texts	Encoder-decoder with copy mechanism	✗	✗	-	-
Vachev et al. [2022]	Text	Transformer-based (T5)	✗	✓	-	-
Chen et al. [2023]	KG	Bidirectional graph2seq model with copy mechanism	✓	✗	-	-
Li et al. [2023]	Text/auto-generated entity co-occurrence graph	GPT-2, answer-aware GAT, and multi-head attention generation module	✓	✗	-	-
Zeng et al. [2024]	Text	Relation-aware transformer with reinforcement learning	✓	✗	-	-

Table 1: Comparison of the presented literature, including the proposed study.

Figure 3: Overview of the proposed framework.

Figure 4: The overview of the knowledge graph construction process from input documents using an LLM and integration into a Neo4j graph database.

Figure 5: The overview of the MCQ generation process: The pipeline samples an associated subgraph from the KG given the key, prompts an LLM to generate a question stem from it, retrieves proper graph-based distractors, computes difficulty signals of the generated MCQ after validation, and saves the resulting MCQ with its computed difficulty signals.

Figure 6: Illustrative examples of each subgraph type alongside potential question stems that could be generated from them. The green node represents the key in each case.

Figure 7: Illustration of single-hop and double-hop reasoning for MCQ generation. The green node represents the key.

Figure 8: Illustration of MCQ generation with and without an extra triple. The green node represents the key.

Figure 9: Visualization of candidate distractor depths. The green node represents the key.

Signal	Description
Reasoning	A binary indicator denoting whether the MCQ was derived from a triple or a quintuple.
Extra Triple	A binary indicator representing the use of an extra triple for additional context.
Distractor Depth	The average graph distance between the key and each distractor.
Node Embedding Similarity	The average cosine similarity between the key and distractors based on their graph node embeddings.
Text Embedding Similarity	The average cosine similarity between the distractors and the question stem, divided by the cosine similarity between the key and the stem.
Degree Centrality	The average degree centrality of the entities involved in the question stem, reflecting their graph-level prominence.

Table 2: Overview of MCQ-generation signals and their descriptions.

Figure 10: Histogram of the incorrect answer rates of generated MCQs.

Model	RMSE	MAE	$R^{2}$	Spearman’s $\boldsymbol{\rho}$
Linear Regression	0.14	0.12	0.46	62.4%
Random Forest Regressor	0.14	0.12	0.41	64.2%
Gradient Boosting Regressor	0.15	0.13	0.34	46.4%
XGBoost Regressor	0.13	0.11	0.52	64.3%

Table 3: Performance comparison of regression models on MCQ difficulty estimation. (The target variable, MCQ difficulty, is a continuous value in the range

[0,1]

Panel of Figure 11 — Figure 11: Predicted vs actual difficulty for each regression model.

Figure 12: Feature importance scores from XGBoost Regressor.

Figure 13: SHAP summary plot for the XGBoost Regressor. Each point corresponds to a SHAP value for a feature and an instance.

Figure 14: Single-feature ablation study results.

Model	RMSE	MAE	$R^{2}$	Spearman’s $\boldsymbol{\rho}$
XGBRegressor - All signals	0.13	0.11	0.52	64.3%
XGBRegressor - Best	0.12	0.10	0.58	66.1%

Table 4: Comparison of baseline and improved XGBoost Regressor configurations.

Panel of Figure 15 — Figure 15: Extended ablation study results for 2-feature and 3-feature exclusions. (a) Excluded: Reasoning and DegreeCentrality . No improvement. (b) Excluded: Reasoning and AboveLargestGapCount . *Best-performing exp . (c) Excluded: DegreeCentrality and AboveLargestGapCount . No improvement. (d) Excluded: Reasoning , DegreeCentrality , and AboveLargestGapCount . No improvement.

Multiple-Choice Question Generation

See all 8 papers

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration
Feb 23, 2026Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari +3Multiple-Choice Question Generation LLM Evaluation
University of Tehran · Independent Researcher · Amirkabir University of Technology +1
Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation
Jun 8, 2026Gabriel Ortega, Abelino Jiménez, Séverin Lions +1Educational Assessment Item Difficulty Estimation
Centro de Investigación Avanzada en Educación (CIAE) Instituto de Estudios Avanzados en Educación (IE) Universidad de Chile. · Departamento de Evaluación Medición y Registro Educacional (DEMRE) Universidad de Chile. · Centro de Modelamiento Matemático (CMM) Universidad de Chile. +1
What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language
Oct 1, 2026Peng Cui, Qiaoyuan Zheng, Rudolf Debelak +1Item Difficulty Estimation
Department of Computer Science ETH Zurich · Machine Learning and Optimization Lab EPFL

Feb 23, 2026J/K move · Enter open · S save

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari, Mahdi Noori+2

University of Tehran · Independent Researcher · Amirkabir University of Technology+1

Multiple-Choice Question Generation LLM Evaluation RAG Evaluation

Open paper AskPDF arXiv

With the rise of large language models (LLMs), they have become instrumental in applications such as Retrieval-Augmented Generation (RAG). Yet evaluating these systems remains bottlenecked by the time and cost of building specialized assessment datasets. We introduce KNIGHT, an LLM-based, knowledge-graph-driven framework for generating multiple-choice question (MCQ) datasets from external sources. KNIGHT constructs a topic-specific knowledge graph, a structured and parsimonious summary of entities and relations, that can be reused to generate instructor-controlled difficulty levels, including multi-hop questions, without repeatedly re-feeding the full source text. This knowledge graph acts as a compressed, reusable state, making question generation a cheap read over the graph. We instantiate KNIGHT on Wikipedia/Wikidata while keeping the framework domain- and ontology-agnostic. As a case study, KNIGHT produces six MCQ datasets in History, Biology, and Mathematics. We evaluate quality on five criteria: fluency, unambiguity (single correct answer), topic relevance, option uniqueness, and answerability given the provided sources (as a proxy for hallucination). Results show that KNIGHT enables token- and cost-efficient generation from a reusable graph representation, achieves high quality across these criteria, and yields model rankings aligned with MMLU-style benchmarks, while supporting topic-specific and difficulty-controlled evaluation.

Abstract

Multiple-Choice Question Generation

Explore similar work

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language