A Framework for Human-AI Q-Matrix Refinement: A NeuralCDM Evaluation
Organizations: AI4STEM Education Center, University of Georgia, Athens, GA, USA
Abstract
Q-matrices are a cornerstone of theory-driven assessment and learning analytics, making item demands and students' underlying knowledge components and misconceptions explicit and actionable. However, Q-matrices are typically crafted by experts, making them time-consuming to build, prone to subjectivity, and difficult to validate empirically. We propose a framework for human-AI Q-matrix refinement in which large language models (LLMs) generate candidate Q-matrices using structured, misconception-aware prompting, and NeuralCDM provides an empirical evaluation layer to compare candidates based on how well they explain student response data. We apply the framework to a thermodynamics assessment dataset and benchmark locally deployed LLMs against cloud-served models. Results show that iteratively refined LLM-generated Q-matrices can exceed expert-baseline model fit (AUC 0.780 vs. 0.717), and that locally deployed models achieve comparable performance to cloud APIs, supporting privacy-preserving deployment.
Figures & tables
| Features | V0 | V1 | V2 | V3 |
|---|---|---|---|---|
| (Baseline) | (Structured) | (Rule-Based) | (Expert-FewShot) | |
| Misconception Library | ✗ | ✓ | ✓ | ✓ |
| Confidence Calibration | ✗ | ✓ | ✓ | ✓ |
| Evaluation Principles | ✗ | ✗ | ✓ | ✓ |
| Few-Shot Examples | ✗ | ✗ | ✗ | ✓ |
| Reasoning Depth | Shallow | Structured | Logical | Expert-Sim. |
| Model | Config | TP | FP | FN | TN | TPR | TNR | Precision | Recall | Micro F1 | Macro F1 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8B | High-only | 14 | 43 | 62 | 765 | 0.1842 | 0.9468 | 0.2456 | 0.1842 | 0.2105 | 0.5801 |
| Qwen3-30B | High-only | 27 | 70 | 49 | 738 | 0.3553 | 0.9134 | 0.2784 | 0.3553 | 0.3121 | 0.6213 |
| Qwen3-8B | High-only | 24 | 58 | 52 | 750 | 0.3158 | 0.9282 | 0.2927 | 0.3158 | 0.3038 | 0.6203 |
| gpt-4o | High-only | 23 | 31 | 53 | 777 | 0.3026 | 0.9616 | 0.4259 | 0.3026 | 0.3538 | 0.6582 |
| gpt-5 | High-only | 41 | 52 | 35 | 756 | 0.5395 | 0.9356 | 0.4409 | 0.5395 | 0.4852 | 0.746 |
| gpt-5.1 | High-only | 36 | 41 | 40 | 767 | 0.4737 | 0.9493 | 0.4675 | 0.4737 | 0.4706 | 0.7198 |
| High Confidence | Medium/High | All Choices | |||||
| Model | Prompt | AUC | RMSE | AUC | RMSE | AUC | RMSE |
| Expert Baseline | 0.717\>\text{\tiny\pm\mathit{.011}} | 0.400\>\text{\tiny\pm\mathit{.002}} | – | – | – | – | |
| GPT-5 (v0) | 0.642\>\text{\tiny\pm\mathit{.026}} | 0.614\>\text{\tiny\pm\mathit{.025}} | – | – | – | – | |
| Open-Weights (Local) | |||||||
| Qwen3-8B | v1 | 0.692\>\text{\tiny\pm\mathit{.014}} | 0.576\>\text{\tiny\pm\mathit{.015}} | 0.707\>\text{\tiny\pm\mathit{.020}} | 0.533\>\text{\tiny\pm\mathit{.015}} | 0.707\>\text{\tiny\pm\mathit{.032}} | 0.515\>\text{\tiny\pm\mathit{.009}} |
| v2 | 0.722\>\text{\tiny\pm\mathit{.005}} | 0.404\>\text{\tiny\pm\mathit{.002}} | 0.723\>\text{\tiny\pm\mathit{.011}} | 0.406\>\text{\tiny\pm\mathit{.004}} | 0.726\>\text{\tiny\pm\mathit{.009}} | 0.406\>\text{\tiny\pm\mathit{.003}} | |