cs.CLNov 16, 2026

What Do Digital Traces Predict? Task Effects, Student Differences, and Critical Online Reasoning

Authors: Maxim Konca, Alexander Mehler

Organizations: Institute of Informatics and Mathematics, Goethe University, Frankfurt am Main, Germany

Abstract

Digital assessment traces are increasingly used to predict student performance, often evaluated by correlating predicted and observed scores pooled across tasks. However, such correlations may reflect between-task differences rather than performance differences among students within tasks. To investigate this, we analyzed up to 527 browsing sessions from 424 university students completing open-Internet Critical Online Reasoning tasks and assessed the cross-task stability of their ratings. We evaluated the same out-of-fold predictions within and across tasks, compared three validation tiers up to leave-one-task-out, and tested incremental predictive contributions of consultation order and relational structure against matched permutation and randomization controls. None of the twelve prespecified primary tests was significant: on held-out tasks, consulted-page features showed no significant association with relative student performance, nor did observed order or relational structure add predictive value. Pooling observations across tasks substantially increased correlations, particularly for factual correctness, whose scores showed limited evidence of stable between-student differences across tasks. However, sentence embeddings yielded modest positive correlations with generic response quality on held-out tasks, without establishing added predictive value from observed order or relational structure. Our findings show that pooled predictive correlations may not reflect within-task student differences, highlighting the need for within-task evaluation and criterion generalizability audits.

Figures & tables

Explore similar work

CardsList
  1. Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces

    Apr 14, 2026Jakob Schwerter, Loreen Sabel, Judith Bose +5Self-Regulated LearningCross-Domain Generalization

  2. Stable and Faithful Explanations for Knowledge Tracing

    Sep 15, 2026Praveena Padi, Arun Morampudi, Ujval Sai Gopal Irrinki +1Knowledge TracingExplainability Evaluation