Making Reasoning Visible: Configuration-Aware Diagnostic Modeling for Adaptive Scaffolding in Structured Reasoning Tasks

Trystan Geoffre; Zheng Yuan; Thierry Geoffre

cs.CLOct 26, 2026

Making Reasoning Visible: Configuration-Aware Diagnostic Modeling for Adaptive Scaffolding in Structured Reasoning Tasks

Authors: Trystan Geoffre, Zheng Yuan, Thierry Geoffre

Organizations: University of Fribourg, Switzerland · University of Sheffield, England · University of Luxembourg, Luxembourg

Abstract

Modeling the latent, multi-step reasoning processes of learners is a core challenge for adaptive tutoring systems, which often rely on simple right/wrong answers rather than understanding how students think. Instantiated here for grammatical reasoning, this study introduces a diagnostic framework that makes students' step-by-step problem-solving visible by analyzing their action sequences in an interactive learning game. The framework computes five interpretable strategy dimensions (e.g., systematic error correction, random guessing), estimates coordination demand from task structure and behavioral patterns, and classifies reasoning into four distinct states. Analyzing 36,234 interactions from 234 students working on 42 morphosyntactic exercises, we found that reasoning strategies vary primarily by context rather than stable traits (intraclass correlation = 0.14), and students show high persistence in unproductive states; specifically, 74.6% of failures driven by high coordination demand (rather than simple knowledge gaps) are followed by another failure without intervention. Our early warning model predicts validation success with strong accuracy (AUC = 0.826 using strategy patterns alone; 0.902 with additional sequential context) while avoiding circular reliance on current error counts. These findings demonstrate that making latent reasoning processes visible enables earlier, more targeted scaffolding that responds to how students are thinking rather than simply whether they succeed. While demonstrated in language learning, the framework provides a generalizable, interpretable architecture for diagnosing complex, structured reasoning in any domain.

Figures & tables

Figure 1: High-level architecture of the diagnostic framework. Raw interaction sequences (slider movements and validations) are processed through four integrated components to produce interpretable reasoning states and early warning predictions that inform adaptive scaffolding decisions.

Number of sliders	Number of exercises
2	3
3	7
4	2
5	1
6	2
7	4

Table 1: Distribution of exercises per number of sliders

Number of solutions	Number of exercises
1	1
2	6
3	4
4	7
5	2
6	8

Table 2: Distribution of exercises per number of solutions

Component	Description	Count
Total events	All logged interactions	36,234
Unique students	Distinct learner identifiers	234
Total sessions	Student-exercise instances	2,144
Validation events	validate actions	10,046
Tirette actions	tirette (drag) actions	26,188
Exercises	Distinct task types	42

Table 3: Basic dataset composition

Feature Category	Description	Coverage
Strategy Dimensions
$S_{\text{verb}}^{\text{(val)}}$	Verb-First targeting	6,257 (62.3%)
$S_{\text{chain}}^{\text{(val)}}$	Chain-Aware progression	8,730 (86.9%)
$S_{\text{rand}}^{\text{(val)}}$	Randomness	8,730 (86.9%)
$S_{\text{focus}}^{\text{(val)}}$	Error-Focus	8,730 (86.9%)
$S_{\text{strat}}^{\text{(val)}}$	Strategic Commitment	8,712 (86.7%)

Table 4: Feature coverage and availability (validation-level)

Strategy	Increasing (%)	Stable (%)	Decreasing (%)
$S_{\text{verb}}$	24.2	62.6	13.2
$S_{\text{chain}}$	9.6	74.5	15.9
$S_{\text{rand}}$	14.9	72.1	13.0
$S_{\text{focus}}$	14.6	67.6	17.8
$S_{\text{strat}}$	17.8	63.4	18.8

Table 5: Per-student strategy evolution trajectories (N = 234 students)

Strategy	Mean slope	Std. dev.
$S_{\text{verb}}$	$+0.010$	0.035
$S_{\text{chain}}$	$-0.002$	0.031
$S_{\text{rand}}$	$-0.001$	0.034
$S_{\text{focus}}$	$+0.001$	0.047
$S_{\text{strat}}$	$-0.004$	0.046

Table 6: Average strategy evolution slope per dimension (per-student linear regression)

Figure 2: Visual guide: Vertical bars represent the four reasoning states; flowing bands represent the proportion of students transitioning from validation $t$ to $t+1$ . The most prominent visual features are the large, curved self-loops (bands exiting and re-entering the same bar). Key findings: The massive red self-loop on High-Demand Failure (74.6%) and the large orange self-loop on Low-Demand Failure (62.3%) visually dominate, confirming that students rarely escape unproductive states without intervention. Conversely, the blue Fragile Success state shows a strong outward flow toward the green Sound Reasoning state (52.4%).

Panel of Figure 3 — Figure 3: Top predictors of validation success. Standardized logistic regression coefficients shown; bar length reflects magnitude, sign indicates direction of effect. Error-Focus ( $S_{\text{focus}}^{\text{(val)}}$ ) is the strongest strategy predictor in both models; Hamming trend emerges as the overall strongest predictor in the full sequential model.

Panel of Figure 4 — Figure 4: Model calibration curves. Predicted probabilities (x-axis) vs. actual validation success rates (y-axis) across decile bins. Dashed line indicates perfect calibration. Strong alignment in the 0.3–0.7 range for both models confirms reliable risk estimates for scaffolding triggers.

Interaction	$\beta$	$p$ (raw)	Effect
Chain $\times$ Rand	$-1.475$	$<0.001$	Antagonistic
Chain $\times$ Strat	$+1.363$	$<0.001$	Synergistic
Rand $\times$ Strat	$+0.930$	$<0.001$	Synergistic
Focus $\times$ Strat	$+0.291$	$<0.001$	Mild synergy
Rand $\times$ Focus	$+0.561$	$<0.001$	Synergistic
Verb $\times$ Chain	$+0.607$	$<0.001$	Synergistic

Table 7: Pairwise strategy interactions in early warning model. All reported effects survive Bonferroni correction for 10 comparisons at

\alpha=0.05

Check	Strategy-Only	Full Sequential
CV AUC	0.826 [0.818, 0.839]	0.902 [0.894, 0.909]
Held-out AUC	0.837 [0.815, 0.858]	0.912 [0.897, 0.926]
Student-level GroupKFold	0.823 $\pm$ 0.013	0.870 $\pm$ 0.013
Cross-exercise (25% holdout)	0.837	0.873
Temporal stability (Early/Late)	0.849 / 0.801	0.880 / 0.857
Noise resilience ( $\sigma$ =0.15)	0.823	0.899

Table 8: Robustness summary: strategy-only vs. full sequential

Finding	Implication for Scaffolding
Strategy use is state-dominant (mean ICC = 0.14); global profiling fails	Adaptive scaffolding justified; static profiling insufficient
High-Demand Failure persistence = 74.6%	Students rarely self-correct from high-demand states; targeted intervention needed
Error-Focus predicts success ( $\beta=+1.25$ ); strategies are distinct (max $r=0.428$ )	Monitor error-targeting; trigger hints when focus drops; concurrent dimensions enable nuanced diagnosis
AUC = 0.826 (strategy-only; full + sequential context = 0.902) without circular features; calibration reliable in decision-relevant range (0.3–0.7)	Reasoning patterns alone enable reliable early warning; probabilities trustworthy for scaffolding triggers
Adaptive transitions = 45.9%; panic transitions = 0.3%	Preserve moderate exploration; intervene only when configurations become persistently unproductive

Table 9: Core empirical contributions and their implications for adaptive scaffolding

Figure 5: Abstract diagnostic pipeline and cross-domain instantiations. Top row: the generalizable architecture from raw actions to interpretable strategy dimensions, CDI estimation, and quadrant-based triggers. Lower panels: instantiations for (a) grammatical reasoning (this paper), (b) algebra equation solving, and (c) programming debugging. In all domains, the framework preserves full interpretability by grounding every diagnostic signal in theoretically defined, auditable reasoning configurations.

User Modeling

User modeling represents people's interests, knowledge, goals, and changing behavior to personalize predictions and AI assistance.

See all 150 papers

Structuring versus Problematizing: How LLM-based Agents Scaffold Learning in Diagnostic Reasoning
Apr 10, 2026Fatma Betül Güreş, Tanya Nazaretsky, Seyed Parsa Neshaei +1Instructional Scaffolding AI in Education
ETH Zürich, EPFL Lausanne, Switzerland · EPFL Lausanne, Switzerland
UPAIR: Diagnosing Reasoning States via Uncertainty-Progress Alignment for Selective Intervention
Jul 19, 2026Cheng Yan, Zhijun Fan, Guangyang Ye +4Adaptive Inference Overthinking in Language Models
University of Science and Technology of China · AWS Agentic AI
Modeling Grammatical Hypothesis Testing in Young Learners: A Sequence-Based Learning Analytics Study of Morphosyntactic Reasoning in an Interactive Game
Mar 2, 2026Thierry Geoffre, Trystan GeoffreChild Language Acquisition Computational Psycholinguistics
University of Luxembourg FHSE - Department of Education and Social Work Esch-sur-Alzette, Luxembourg · Université de Fribourg Institut de Plurilinguisme Fribourg, Switzerland

Apr 10, 2026J/K move · Enter open · S save

Structuring versus Problematizing: How LLM-based Agents Scaffold Learning in Diagnostic Reasoning

Fatma Betül Güreş, Tanya Nazaretsky, Seyed Parsa Neshaei, Tanja Käser

ETH Zürich, EPFL Lausanne, Switzerland · EPFL Lausanne, Switzerland

Instructional Scaffolding AI in Education

Open paper AskPDF arXiv

Supporting students in developing diagnostic reasoning is a key challenge across educational domains. Novices often face cognitive biases such as premature closure and over-reliance on heuristics, and they struggle to transfer diagnostic strategies to new cases. Scenario-based learning (SBL) enhanced by Learning Analytics (LA) and large language models (LLM) offers a promising approach by combining realistic case experiences with personalized scaffolding. Yet, how different scaffolding approaches shape reasoning processes remains insufficiently explored. This study introduces PharmaSim Switch, an SBL environment for pharmacy technician training, extended with an LA- and LLM-powered pharmacist agent that implements pedagogical conversations rooted in two theory-driven scaffolding approaches: structuring and problematizing, as well as a student learning trajectory. In a between-groups experiment, 63 vocational students completed a learning scenario, a near-transfer scenario, and a far-transfer scenario under one of the two scaffolding conditions. Results indicate that both scaffolding approaches were effective in supporting the use of diagnostic strategies. Performance outcomes were primarily influenced by scenario complexity rather than students' prior knowledge or the scaffolding approach used. The structuring approach was associated with more accurate Active and Interactive participation, whereas problematizing elicited more Constructive engagement. These findings underscore the value of combining scaffolding approaches when designing LA- and LLM-based systems to effectively foster diagnostic reasoning.

Abstract

User Modeling

Explore similar work

Structuring versus Problematizing: How LLM-based Agents Scaffold Learning in Diagnostic Reasoning

UPAIR: Diagnosing Reasoning States via Uncertainty-Progress Alignment for Selective Intervention

Modeling Grammatical Hypothesis Testing in Young Learners: A Sequence-Based Learning Analytics Study of Morphosyntactic Reasoning in an Interactive Game