Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction
Organizations: Department of Counseling, Leadership, and Research Methods, University of Arkansas · School of Computing and Information, University of Pittsburgh · Department of Information, Library, and Research Sciences, University of North Carolina at Greensboro
Abstract
Large language models (LLMs) can generate Q-matrices for diagnostic classification models that approximate expert specifications, but evaluations have focused on the final matrices, leaving the basis of individual item-attribute assignments unexamined. This opacity limits the responsible use of AI in measurement. We propose Reasoning-Trace Analysis (RTA), a framework in which an LLM generates attribute-labeled reasoning steps through chain-of-thought (CoT) prompting and these steps are analyzed to reconstruct, compare, and inspect each assignment. RTA is intended to supplement expert review and empirical validation. In a simulation study with perturbed fraction subtraction items, four LLMs recovered the required attributes and withheld attributes the items did not require. In an empirical study with 11 fraction subtraction items, responses from 536 examinees, four LLMs, up to three temperatures, and 100 replications per condition, CoT-derived Q-matrices overlapped substantially with the reference Q-matrix (mean overlap .809) but less than directly generated Q-matrices (.877), and they varied more across replications. Inspection of the traces linked disagreements to item-type coding, labeling rules, and how operations were expressed as steps. Q-matrices with similar overlap differed in fit to examinee responses. RTA provides inspectable evidence that matrix-level agreement and fit do not.
Figures & tables
| No. | Element | Item | Solution | Answer |
|---|---|---|---|---|
| 1 | Whole1 +1 | |||
| 2 | Whole1 -1 | |||
| 3 | Whole2 +1 | |||
| 4 | Whole2 -1 | |||
| 5 | Num1 -1 | |||
| 6 | Num1 +1 |
| Model | Reasoning-step content |
|---|---|
| GPT-4o mini | Step 1 [A1]: Subtract the fractions directly since they have the same denominator (5/12 - 7/12 = -2/12). Step 2 [A3&A4]: Since the result of the fraction subtraction is negative, borrow from the whole number (4 5/12 = 3 17/12). Step 3 [A1]: Subtract the whole numbers and fractions (3 - 2 = 1 and 17/12 - 7/12 = 10/12). Step 4 [A2]: Simplify the fraction (10/12 = 5/6). Step 5 [Other]: Write the final answer. |
| DeepSeek v4 Flash | Step 1 [A4]: Borrow one from the whole number (4 5/12 = 3 17/12). Step 2 [A3&A1]: Separate the whole numbers and fractions, then subtract each part (3 - 2 = 1 and 17/12 - 7/12 = 10/12). Step 3 [A2]: Simplify the result (1 10/12 -> 1 5/6). Step 4 [Other]: Write the final answer. |
| Claude Haiku 4.5 | Step 1 [A4]: Borrow one from the whole number (4 5/12 = 3 17/12). Step 2 [A3&A1]: Separate the whole numbers and fractions, then subtract each part (3 - 2 = 1 and 17/12 - 7/12 = 10/12). Step 3 [A2]: Simplify the fraction (10/12 = 5/6). Step 4 [Other]: Write the final answer. |
| Kimi K2.6 | Step 1 [A4]: Borrowing one from the whole number (4 5/12 = 3 17/12). Step 2 [A3&A1]: Separate the whole numbers and fractions, then subtract each part (3 - 2 = 1 and 17/12 - 7/12 = 10/12). Step 3 [A2]: Simplify the result (1 10/12 -> 1 5/6). Step 4 [Other]: Write the final answer. |
| Factor | Level | Setting |
|---|---|---|
| Model | GPT-4o mini, DeepSeek v4 Flash, Claude Haiku 4.5 | Low reasoning-effort setting where available |
| Kimi K2.6 | Thinking disabled | |
| Temperature | 0, 0.5, 1.0 | Applied to GPT-4o mini, DeepSeek v4 Flash, and Claude Haiku 4.5 |
| 0.6 | Applied to Kimi K2.6 | |
| Workflow | CoT workflow | Item-level prompt with stepwise skill labels |
| Direct workflow | Attribute labels and item content followed by direct 11 by 4 Q-matrix generation |
| Model | Workflow | Temp. | Overall overlap | Exact item match |
|---|---|---|---|---|
| GPT-4o mini | CoT workflow | 0 | .864 | .455 |
| 0.5 | .886 | .545 | ||
| 1 | .886 | .545 | ||
| Direct workflow | 0 | .795 | .364 | |
| 0.5 | .795 | .364 | ||
| 1 | .795 | .364 |
| Model | Workflow | Temp. | A1 | A2 | A3 | A4 |
|---|---|---|---|---|---|---|
| GPT-4o mini | CoT workflow | 0 | 1.000 | .455 | 1.000 | 1.000 |
| 0.5 | 1.000 | .545 | 1.000 | 1.000 | ||
| 1 | 1.000 | .636 | .909 | 1.000 | ||
| Direct workflow | 0 | 1.000 | .818 | .727 | .636 | |
| 0.5 | 1.000 | .727 | .727 | .727 | ||
| 1 | 1.000 | .727 | .727 | .727 |
| Model | Workflow | Temp. | Accuracy | ||
|---|---|---|---|---|---|
| GPT-4o mini | CoT workflow | 0 | .10 | .250 | .738 |
| 0.5 | .04 | .409 | .770 | ||
| 1 | .03 | .455 | .787 | ||
| Direct workflow | 0 | .45 | .205 | - | |
| 0.5 | .14 | .295 | - | ||
| 1 | .06 | .455 | - |