cs.AINov 19, 2026

Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction

Authors: Jihong Zhang, Zijing Jin, Mingfeng Xue

Organizations: Department of Counseling, Leadership, and Research Methods, University of Arkansas · School of Computing and Information, University of Pittsburgh · Department of Information, Library, and Research Sciences, University of North Carolina at Greensboro

Abstract

Large language models (LLMs) can generate Q-matrices for diagnostic classification models that approximate expert specifications, but evaluations have focused on the final matrices, leaving the basis of individual item-attribute assignments unexamined. This opacity limits the responsible use of AI in measurement. We propose Reasoning-Trace Analysis (RTA), a framework in which an LLM generates attribute-labeled reasoning steps through chain-of-thought (CoT) prompting and these steps are analyzed to reconstruct, compare, and inspect each assignment. RTA is intended to supplement expert review and empirical validation. In a simulation study with perturbed fraction subtraction items, four LLMs recovered the required attributes and withheld attributes the items did not require. In an empirical study with 11 fraction subtraction items, responses from 536 examinees, four LLMs, up to three temperatures, and 100 replications per condition, CoT-derived Q-matrices overlapped substantially with the reference Q-matrix (mean overlap .809) but less than directly generated Q-matrices (.877), and they varied more across replications. Inspection of the traces linked disagreements to item-type coding, labeling rules, and how operations were expressed as steps. Q-matrices with similar overlap differed in fit to examinee responses. RTA provides inspectable evidence that matrix-level agreement and fit do not.

Figures & tables

Explore similar work

Mar 30, 2026

A Framework for Human-AI Q-Matrix Refinement: A NeuralCDM Evaluation

Q-matrices are a cornerstone of theory-driven assessment and learning analytics, making item demands and students' underlying knowledge components and misconceptions explicit and actionable. However, Q-matrices are typically crafted by experts, making them time-consuming to build, prone to subjectivity, and difficult to validate empirically. We propose a framework for human-AI Q-matrix refinement in which large language models (LLMs) generate candidate Q-matrices using structured, misconception-aware prompting, and NeuralCDM provides an empirical evaluation layer to compare candidates based on how well they explain student response data. We apply the framework to a thermodynamics assessment dataset and benchmark locally deployed LLMs against cloud-served models. Results show that iteratively refined LLM-generated Q-matrices can exceed expert-baseline model fit (AUC 0.780 vs. 0.717), and that locally deployed models achieve comparable performance to cloud APIs, supporting privacy-preserving deployment.
Aug 18, 2026

Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses

The evaluation of large language models (LLMs) relies heavily on human-designed assessments, implicitly assuming that AI and humans employ similar underlying cognitive constructs. Challenging this assumption, we investigate whether the latent factors governing LLM performance carry the same substantive, human-interpretable meaning as the cognitive constructs governing human learners. Using responses from humans and six LLMs across quantitative reasoning and chemistry assessments, we conducted Exploratory Factor Analysis (EFA) separately for both groups. Subject-Matter Experts (SMEs) then blindly evaluated the resulting factor graphs to ascribe pedagogical meaning to the emerged constructs. SMEs successfully interpreted most of the human-derived factors. Conversely, they could not ascribe meaning to any LLM-derived factors in quantitative reasoning and interpreted only half of the LLM factors in chemistry. By combining data-driven EFA with blind expert interpretation, this framework shows that LLMs frequently operate on statistically opaque mechanisms distinct from human reasoning.
Jan 30, 2026

Probing the Trajectories of Reasoning Traces in Large Language Models

Large language models (LLMs) increasingly solve difficult problems by producing reasoning traces before emitting a final response. However, it remains unclear how accuracy and decision commitment evolve along a reasoning trajectory, and whether intermediate trace segments provide answer-relevant information beyond generic length or stylistic effects. Here, we propose a trajectory-probing protocol for evaluating how a reasoning model's predicted answer evolves along a partial reasoning trace. The protocol (1) generates a model's full reasoning trace, (2) truncates it at fixed token-percentiles, and (3) injects each partial trace back into the model, measuring the model's induced answer distribution. We apply the protocol to seven open-source reasoning models (Qwen3-4B/8B/14B, gpt-oss-20b/120b, DeepSeek-R1-Distill-Llama-8B and Gemma-4-31B-it) across four benchmarks (GPQA Diamond, MMLU-Pro, Omni-MATH-2 and CompMath). The protocol reveals that accuracy and decision commitment generally increase as the percentage of provided reasoning tokens grows, and length, form, and token-identity controls confirm that gains stem from instance-specific semantic content rather than context length or generic reasoning style effects. Furthermore, cross-model probing shows that even stronger models frequently repeat the source model's errors when given its partial reasoning trace, while allowing further reasoning substantially improves recovery. Together, these results provide a reproducible, model-agnostic measurement of reasoning trace dynamics that generalizes across answer formats and can reveal model-specific failure modes invisible to aggregate accuracy.