Self-hosted Lecture-to-Quiz: Local LLM MCQ Generation with Deterministic Quality Control

Seine A. Shintani

cs.CYFeb 20, 2026

Self-hosted Lecture-to-Quiz: Local LLM MCQ Generation with Deterministic Quality Control

Authors: Seine A. Shintani

Organizations: Department of Biomedical Science, College of Life and Health Sciences, Chubu University; Center for AI, Mathematical and Data Sciences, Chubu University; Kasugai, Aichi 487-8501, Japan

Abstract

We present an end-to-end self-hosted (API-free) pipeline, where API-free means that lecture content is not sent to any external LLM service, that converts lecture PDFs into multiple-choice questions (MCQs) using a local LLM plus deterministic quality control (QC). The pipeline is designed for black-box minimization: LLMs may assist drafting, but the final released artifacts are plain-text question banks with an explicit QC trace and without any need to call an LLM at deployment time. We run a seed sweep on three short "dummy lectures" (information theory, thermodynamics, and statistical mechanics), collecting 15 runs x 8 questions = 120 accepted candidates (122 attempts total under bounded retries). All 120 accepted candidates satisfy hard QC checks (JSON schema conformance, a single marked correct option, and numeric/constant equivalence tests); however, the warning layer flags 8/120 items (spanning 8 runs) that expose residual quality risks such as duplicated distractors or missing rounding instructions. We report a warning taxonomy with concrete before->after fixes, and we release the final 24-question set (three lectures x 8 questions) as JSONL/CSV for Google Forms import (e.g., via Apps Script or API tooling) included as ancillary files under anc/. Finally, we position the work through the AI to Learn (AI2L) rubric lens and argue that self-hosted MCQ generation with explicit QC supports privacy, accountability, and Green AI in educational workflows.

Figures & tables

Component	Setting
Execution	Google Colab Pro+ (NVIDIA A100)
Model	Qwen2.5-14B-Instruct (GGUF, Q4_K_M) via llama-cli [ 12 , 13 , 14 , 11 ]
Context window	4096
Decoding	temperature 0.2, top_p 0.9, max_new_tokens 900
Seeds	{0,1,2,3,4} (seed sweep; recorded per run)
Retry budget	retry_max = 3 per question

Table 1: Reproducible run configuration (shared across all runs unless noted).

Metric	Value	Notes
Runs	15	3 lectures $\times$ 5 seeds
Target items	120	8 per run
QC acceptance	120/120	Acceptance rate 1.0
Total attempts	122	Includes retries
Retry rate	1.6%	2/122 attempts (1.7% overhead per item)
Runtime / run	58.55 $\pm$ 2.34 s	Min 55.29, max 63.50

Table 2: Aggregate performance over the 15-run seed sweep (3 lectures

\times

5 seeds; 8 items target per run).

Lecture	Seed	Runtime (s)	s/q	Tries	Retries	Retry (%)	Flags
InfoTheory	0	57.22	7.15	8	0	0.0	Dup
InfoTheory	1	57.02	7.13	8	0	0.0	Rnd
InfoTheory	2	57.81	7.23	8	0	0.0	–
InfoTheory	3	56.86	7.11	8	0	0.0	–
InfoTheory	4	63.50	7.94	9	1	11.1	–
Thermodynamics	0	58.84	7.35	8	0	0.0	–

Table 3: Seed sweep results (target 8 items per run). Retries indicate extra attempts beyond 8 triggered by hard QC failures. Flags are warning-level checks: Rnd = missing rounding instruction, Dup = duplicated numeric constant distractors.

Flag	Count	Share	Typical cause and mitigation
Rnd	7	5.8%	Numeric approximation without explicit rounding instruction; mitigate by adding “approximately” or specifying rounding digits in the stem/template.
Dup	1	0.8%	Two distractors evaluate to the same constant; mitigate via equivalence checks + re-generation or by constraining distractor generation.

Table 4: Warning-level issue breakdown across the seed sweep (120 accepted items).

Warning type	Before (flagged excerpt)	After (minimal fix)
duplicate_options_numeric_const	L01_Q01 (seed 0) included two options that evaluate to the same constant: C $\log_{2}(0.25)=-2$ and E $2\cdot\log_{2}(0.5)=-2$ .	Replace one distractor with a non-equivalent expression (e.g., E $\log_{2}(0.125)=-3$ ) so that all five options are distinct.
rounding_instruction_missing_suspect	L01_Q02 (seed 1) asked for $H(X)$ for $(0.7,0.3)$ and offered a decimal correct option (0.881 bits) without an explicit rounding rule.	Add a rounding contract in the stem (e.g., “Round to three decimal places”) or redesign the item to ask for the symbolic expression $-0.7\log_{2}0.7-0.3\log_{2}0.3$ .

Table 5: Representative QC warning fixes (before

\rightarrow

after).

Set	Def./formula	Property/law	Worked ex.	Mean stem (words)	Mean expl. (words)
L01	2	5	1	18.4	37.8
L02	4	4	0	22.5	48.4
L03	5	3	0	28.4	43.2
Overall	11	12	1	23.1	43.1

Table 6: A lightweight profile of the final 24-item deploy set. “Def./formula” includes definition- and equation-centric items; “Property/law” includes qualitative statements and inequalities; word counts are computed over the English text fields in the released JSONL/CSV.

Run	Item	Warning	Why it was flagged / how it is handled
L01 (seed 0)	Q01	duplicate_options_numeric_const	Two distractors were constant-equivalent: log2(0.25) and 2*log2(0.5) both evaluate to $-2$ . We resolve this either by regenerating a single distractor or by selecting another seed run without the duplication.
L01 (seed 1)	Q02	rounding_instruction_missing_suspect	The keyed answer was reported as 0.881 bits but the stem did not specify whether rounding (and to how many digits) was required. A minimal fix is to add “approximately” or an explicit rounding rule.
L02 (seed 1)	Q06	rounding_instruction_missing_suspect	A conservative heuristic was triggered by a comparison to a numerical constant (“ $\geq 0$ ”) despite the item being qualitative. This illustrates why the warnings are best interpreted as a review queue , not as automatic rejection.

Table 7: Representative QC warning traces observed in the seed sweep. In deployment, we typically resolve warnings by (i) regenerating only the problematic component or (ii) applying a small human edit after verifying correctness.

AI2L pillar	Local inference	QC + rubric	Static export	Remaining gap
Transparency (black-box exclusion)	✓	✓	✓	Drafting model remains opaque
Accountability (human verification)	—	✓	✓	Human final approval still required
Privacy (info protection)	✓	✓	✓	Depends on local material handling
Green AI (sustainability)	✓	—	✓	Authoring still compute-heavy

Table 8: AI2L pillars as a rubric lens for this work (alignment and remaining gaps).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Description
ID	Unique ID within the run (e.g., Q01 ).
SourcePages	Page range supporting the item (traceability).
Question	MCQ stem.
OptionA – OptionE	Five answer choices.
CorrectOption	Correct choice letter (A–E).
Explanation	Short rationale grounded in the excerpt.

Appendix

Table 9: Internal per-item schema used in run-level artifacts ( questions.json / QuestionBank.csv ).

Field	Description
lecture	Lecture identifier (e.g., Lecture01 ).
run	Selected run tag (e.g., RUN4 ).
seed	RNG seed for the run.
pdf_basename	Input PDF filename (basename).
source_pages	Page pointer supporting the item.
id	Stable item ID (e.g., L01_Q01 ).

Appendix

Table 10: Released export schema for the curated deploy set ( final24_entropy_mcq.jsonl / final24_entropy_mcq.csv ).

Multiple-Choice Question Generation

See all 8 papers

Slide Deck Q&A Quality Assurance App: A Multi-Stage Pipeline for Pedagogical Question Generation
May 26, 2026Jim SalsmanLLM-Assisted Annotation Generative AI in Education
TalkNicer, Inc.
Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos
Sep 1, 2026S M Masrur Ahmed, Jaspal SubhlokEvidence Retrieval Evidence-Grounded Generation
University of Houston
KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration
Feb 23, 2026Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari +3Multiple-Choice Question Generation LLM Evaluation
University of Tehran · Independent Researcher · Amirkabir University of Technology +1

May 26, 2026J/K move · Enter open · S save

Slide Deck Q&A Quality Assurance App: A Multi-Stage Pipeline for Pedagogical Question Generation

Jim Salsman

TalkNicer, Inc.

LLM-Assisted Annotation Generative AI in Education Multimodal Generation

Open paper AskPDF arXiv

Generating high-quality, pedagogically useful questions from lecture slide decks is difficult because important instructional content is distributed across both text and visual elements, and because useful questions must be scaffolded across the flow of a presentation rather than generated slide by slide in isolation. This paper describes Slide Deck Q&A Quality Assurance (slidesqaqa), a Flask-based software system that extracts text and rendered images from PDF slides and processes them through a four-stage large language model pipeline comprising window planning, deck synthesis, slide annotation, and reconciliation. The system reasons jointly about slide modality and pedagogical role, allocates bounded question budgets, and revises draft annotations at the deck level to reduce redundancy and improve coverage. The final output is a structured JSON annotation containing deck-level goals, section structure, slide-level summaries, question sets, and evaluation scores. Initial experiments on two technical lecture decks indicate that the pipeline can filter non-instructional slides and produce high-fidelity, pedagogically coherent questions for visually complex content. The working system is at https://slidesqaqa-974767694043.us-west1.run.app The software repository is at https://github.com/blinding2submit/slidesqaqa

Abstract

Multiple-Choice Question Generation

Explore similar work

Slide Deck Q&A Quality Assurance App: A Multi-Stage Pipeline for Pedagogical Question Generation

Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration