cs.CYFeb 20, 2026

Self-hosted Lecture-to-Quiz: Local LLM MCQ Generation with Deterministic Quality Control

Authors: Seine A. Shintani

Organizations: Department of Biomedical Science, College of Life and Health Sciences, Chubu University; Center for AI, Mathematical and Data Sciences, Chubu University; Kasugai, Aichi 487-8501, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present an end-to-end self-hosted (API-free) pipeline, where API-free means that lecture content is not sent to any external LLM service, that converts lecture PDFs into multiple-choice questions (MCQs) using a local LLM plus deterministic quality control (QC). The pipeline is designed for black-box minimization: LLMs may assist drafting, but the final released artifacts are plain-text question banks with an explicit QC trace and without any need to call an LLM at deployment time. We run a seed sweep on three short "dummy lectures" (information theory, thermodynamics, and statistical mechanics), collecting 15 runs x 8 questions = 120 accepted candidates (122 attempts total under bounded retries). All 120 accepted candidates satisfy hard QC checks (JSON schema conformance, a single marked correct option, and numeric/constant equivalence tests); however, the warning layer flags 8/120 items (spanning 8 runs) that expose residual quality risks such as duplicated distractors or missing rounding instructions. We report a warning taxonomy with concrete before->after fixes, and we release the final 24-question set (three lectures x 8 questions) as JSONL/CSV for Google Forms import (e.g., via Apps Script or API tooling) included as ancillary files under anc/. Finally, we position the work through the AI to Learn (AI2L) rubric lens and argue that self-hosted MCQ generation with explicit QC supports privacy, accountability, and Green AI in educational workflows.

Figures & tables

ComponentSetting
ExecutionGoogle Colab Pro+ (NVIDIA A100)
ModelQwen2.5-14B-Instruct (GGUF, Q4_K_M) via llama-cli [ 12 , 13 , 14 , 11 ]
Context window4096
Decodingtemperature 0.2, top_p 0.9, max_new_tokens 900
Seeds{0,1,2,3,4} (seed sweep; recorded per run)
Retry budgetretry_max = 3 per question
Table 1: Reproducible run configuration (shared across all runs unless noted).
MetricValueNotes
Runs153 lectures ×\times 5 seeds
Target items1208 per run
QC acceptance120/120Acceptance rate 1.0
Total attempts122Includes retries
Retry rate1.6%2/122 attempts (1.7% overhead per item)
Runtime / run58.55 ±\pm 2.34 sMin 55.29, max 63.50
Table 2: Aggregate performance over the 15-run seed sweep (3 lectures ×\times 5 seeds; 8 items target per run).
LectureSeedRuntime (s)s/qTriesRetriesRetry (%)Flags
InfoTheory057.227.15800.0Dup
InfoTheory157.027.13800.0Rnd
InfoTheory257.817.23800.0–
InfoTheory356.867.11800.0–
InfoTheory463.507.949111.1–
Thermodynamics058.847.35800.0–
Table 3: Seed sweep results (target 8 items per run). Retries indicate extra attempts beyond 8 triggered by hard QC failures. Flags are warning-level checks: Rnd = missing rounding instruction, Dup = duplicated numeric constant distractors.
FlagCountShareTypical cause and mitigation
Rnd75.8%Numeric approximation without explicit rounding instruction; mitigate by adding “approximately” or specifying rounding digits in the stem/template.
Dup10.8%Two distractors evaluate to the same constant; mitigate via equivalence checks + re-generation or by constraining distractor generation.
Table 4: Warning-level issue breakdown across the seed sweep (120 accepted items).
Warning typeBefore (flagged excerpt)After (minimal fix)
duplicate_options_numeric_constL01_Q01 (seed 0) included two options that evaluate to the same constant: C log⁡2(0.25)=−2\log_{2}(0.25)=-2 and E 2⋅log⁡2(0.5)=−22\cdot\log_{2}(0.5)=-2 .Replace one distractor with a non-equivalent expression (e.g., E log⁡2(0.125)=−3\log_{2}(0.125)=-3 ) so that all five options are distinct.
rounding_instruction_missing_suspectL01_Q02 (seed 1) asked for H(X)H(X) for (0.7,0.3)(0.7,0.3) and offered a decimal correct option (0.881 bits) without an explicit rounding rule.Add a rounding contract in the stem (e.g., “Round to three decimal places”) or redesign the item to ask for the symbolic expression −0.7log⁡20.7−0.3log⁡20.3-0.7\log_{2}0.7-0.3\log_{2}0.3 .
Table 5: Representative QC warning fixes (before →\rightarrow after).
SetDef./formulaProperty/lawWorked ex.Mean stem (words)Mean expl. (words)
L0125118.437.8
L0244022.548.4
L0353028.443.2
Overall1112123.143.1
Table 6: A lightweight profile of the final 24-item deploy set. “Def./formula” includes definition- and equation-centric items; “Property/law” includes qualitative statements and inequalities; word counts are computed over the English text fields in the released JSONL/CSV.
RunItemWarningWhy it was flagged / how it is handled
L01 (seed 0)Q01duplicate_options_numeric_constTwo distractors were constant-equivalent: log2(0.25) and 2*log2(0.5) both evaluate to −2-2 . We resolve this either by regenerating a single distractor or by selecting another seed run without the duplication.
L01 (seed 1)Q02rounding_instruction_missing_suspectThe keyed answer was reported as 0.881 bits but the stem did not specify whether rounding (and to how many digits) was required. A minimal fix is to add “approximately” or an explicit rounding rule.
L02 (seed 1)Q06rounding_instruction_missing_suspectA conservative heuristic was triggered by a comparison to a numerical constant (“ ≥0\geq 0 ”) despite the item being qualitative. This illustrates why the warnings are best interpreted as a review queue , not as automatic rejection.
Table 7: Representative QC warning traces observed in the seed sweep. In deployment, we typically resolve warnings by (i) regenerating only the problematic component or (ii) applying a small human edit after verifying correctness.
AI2L pillarLocal inferenceQC + rubricStatic exportRemaining gap
Transparency (black-box exclusion)✓✓✓Drafting model remains opaque
Accountability (human verification)—✓✓Human final approval still required
Privacy (info protection)✓✓✓Depends on local material handling
Green AI (sustainability)✓—✓Authoring still compute-heavy
Table 8: AI2L pillars as a rubric lens for this work (alignment and remaining gaps).
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

FieldDescription
IDUnique ID within the run (e.g., Q01 ).
SourcePagesPage range supporting the item (traceability).
QuestionMCQ stem.
OptionA – OptionEFive answer choices.
CorrectOptionCorrect choice letter (A–E).
ExplanationShort rationale grounded in the excerpt.
Appendix
Table 9: Internal per-item schema used in run-level artifacts ( questions.json / QuestionBank.csv ).
FieldDescription
lectureLecture identifier (e.g., Lecture01 ).
runSelected run tag (e.g., RUN4 ).
seedRNG seed for the run.
pdf_basenameInput PDF filename (basename).
source_pagesPage pointer supporting the item.
idStable item ID (e.g., L01_Q01 ).
Appendix
Table 10: Released export schema for the curated deploy set ( final24_entropy_mcq.jsonl / final24_entropy_mcq.csv ).
More in this topic

Multiple-Choice Question Generation

See all 8 papers

Explore similar work

CardsList
May 26, 2026

Slide Deck Q&A Quality Assurance App: A Multi-Stage Pipeline for Pedagogical Question Generation

Generating high-quality, pedagogically useful questions from lecture slide decks is difficult because important instructional content is distributed across both text and visual elements, and because useful questions must be scaffolded across the flow of a presentation rather than generated slide by slide in isolation. This paper describes Slide Deck Q&A Quality Assurance (slidesqaqa), a Flask-based software system that extracts text and rendered images from PDF slides and processes them through a four-stage large language model pipeline comprising window planning, deck synthesis, slide annotation, and reconciliation. The system reasons jointly about slide modality and pedagogical role, allocates bounded question budgets, and revises draft annotations at the deck level to reduce redundancy and improve coverage. The final output is a structured JSON annotation containing deck-level goals, section structure, slide-level summaries, question sets, and evaluation scores. Initial experiments on two technical lecture decks indicate that the pipeline can filter non-instructional slides and produce high-fidelity, pedagogically coherent questions for visually complex content. The working system is at https://slidesqaqa-974767694043.us-west1.run.app The software repository is at https://github.com/blinding2submit/slidesqaqa

Jim Salsman

TalkNicer, Inc.

LLM-Assisted AnnotationGenerative AI in EducationMultimodal Generation
AskPDF
Sep 1, 2026

Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos

Recorded lecture videos, often enhanced with search and summarization features, are a standard study resource. However, students cannot easily ask course specific questions or verify answers against an instructor's lecture. We report a semester-long deployment of VideoPoints platform with a retrieval-augmented chatbot that answers from course lecture materials and returns timestamped citations. The chatbot retrieves only from the active course, uses chapter summaries to guide transcript ranking, and returns clickable timestamped citations. Students used it for quick lookups and exam review. Across 833 messages, 70.5% included citations, none crossed a course boundary, and when no lecture evidence matched, the chatbot usually declined rather than answering. Among the users, citations were the most consistently useful feature, while practice-question generation was the strongest unmet request. We also evaluated the design on the real-world test split of EduVidQA, a public multimodal benchmark for lecture-video question answering. Our design improved correct-lecture retrieval by 6.3 percentage points over dense-only retrieval. Together, the results show that effective deployment depends on course isolation, supported citations, and alignment with students' study practices.

S M Masrur Ahmed, Jaspal Subhlok

University of Houston

Evidence RetrievalEvidence-Grounded GenerationRetrieval-Augmented GenerationQuestion AnsweringVideo QASelective QA+2
AskPDF
Feb 23, 2026

KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

With the rise of large language models (LLMs), they have become instrumental in applications such as Retrieval-Augmented Generation (RAG). Yet evaluating these systems remains bottlenecked by the time and cost of building specialized assessment datasets. We introduce KNIGHT, an LLM-based, knowledge-graph-driven framework for generating multiple-choice question (MCQ) datasets from external sources. KNIGHT constructs a topic-specific knowledge graph, a structured and parsimonious summary of entities and relations, that can be reused to generate instructor-controlled difficulty levels, including multi-hop questions, without repeatedly re-feeding the full source text. This knowledge graph acts as a compressed, reusable state, making question generation a cheap read over the graph. We instantiate KNIGHT on Wikipedia/Wikidata while keeping the framework domain- and ontology-agnostic. As a case study, KNIGHT produces six MCQ datasets in History, Biology, and Mathematics. We evaluate quality on five criteria: fluency, unambiguity (single correct answer), topic relevance, option uniqueness, and answerability given the provided sources (as a proxy for hallucination). Results show that KNIGHT enables token- and cost-efficient generation from a reusable graph representation, achieves high quality across these criteria, and yields model rankings aligned with MMLU-style benchmarks, while supporting topic-specific and difficulty-controlled evaluation.

Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari +3

University of Tehran · Independent Researcher · Amirkabir University of Technology +1

Multiple-Choice Question GenerationLLM EvaluationRAG Evaluation
AskPDF