What Do Digital Traces Predict? Task Effects, Student Differences, and Critical Online Reasoning
Organizations: Institute of Informatics and Mathematics, Goethe University, Frankfurt am Main, Germany
Abstract
Digital assessment traces are increasingly used to predict student performance, often evaluated by correlating predicted and observed scores pooled across tasks. However, such correlations may reflect between-task differences rather than performance differences among students within tasks. To investigate this, we analyzed up to 527 browsing sessions from 424 university students completing open-Internet Critical Online Reasoning tasks and assessed the cross-task stability of their ratings. We evaluated the same out-of-fold predictions within and across tasks, compared three validation tiers up to leave-one-task-out, and tested incremental predictive contributions of consultation order and relational structure against matched permutation and randomization controls. None of the twelve prespecified primary tests was significant: on held-out tasks, consulted-page features showed no significant association with relative student performance, nor did observed order or relational structure add predictive value. Pooling observations across tasks substantially increased correlations, particularly for factual correctness, whose scores showed limited evidence of stable between-student differences across tasks. However, sentence embeddings yielded modest positive correlations with generic response quality on held-out tasks, without establishing added predictive value from observed order or relational structure. Our findings show that pooled predictive correlations may not reflect within-task student differences, highlighting the need for within-task evaluation and criterion generalizability audits.
Figures & tables
| Outcome | Student | Student task | Rater | Rater task | Residual |
|---|---|---|---|---|---|
| DIRQ | 23 [14, 29] | 23 [17, 30] | 9 [0, 60] | 8 [3, 29] | 37 |
| TaSFC | 8 [0, 20] | 64 [53, 69] | 2 [0, 16] | 5 [2, 14] | 21 |
| COH | 19 [11, 24] | 14 [8, 21] | 19 [6, 87] | 3 [1, 10] | 45 |
| Response fulfillment | 6 [0, 12] | 20 [14, 27] | 0 [0, 11] | 15 [8, 35] | 59 |
| Source-reference use | 36 [23, 44] | 43 [34, 51] | 1 [0, 21] | 1 [0, 2] | 20 |
| Primary claim | Outcome | Estimate | 95% CI | ||
|---|---|---|---|---|---|
| DIRQ | 0.079 | [ 0.170, 0.011] | 0.959 | 1.000 | |
| TaSFC | 0.027 | [ 0.121, 0.070] | 0.718 | 1.000 | |
| COH | 0.039 | [ 0.128, 0.045] | 0.788 | 1.000 | |
| DIRQ | 0.043 | [ 0.102, 0.012] | 0.184 | 1.000 | |
| TaSFC | 0.049 | [ 0.138, 0.040] | 0.332 | 1.000 | |
| COH | 0.019 | [ 0.062, 0.026] | 0.431 | 1.000 |
| RQ | Outcome | Component | 95% CI | ||
|---|---|---|---|---|---|
| RQ3 | DIRQ | Order vs. unordered | [ , ] | 0.396 | |
| Order vs. permutations | [ , ] | 0.426 | |||
| RQ3 | TaSFC | Order vs. unordered | [ , ] | 0.008 | |
| Order vs. permutations | [ , ] | 0.268 | |||
| RQ3 | COH | Order vs. unordered | [ , ] | 0.413 | |
| Order vs. permutations | [ , ] | 0.380 |
| Factor | Elements | |
| Tier | T1, T2, T3 | 3 |
| Feature source | Coh-Metrix, embeddings | 2 |
| Page set | , , | 3 |
| Fixed estimator | Random Forest, SVR-RBF, PLS | 3 |
| Specifications | ||
| Fixed | ; ; ; ; | 8 |
| Block | Group | Features |
|---|---|---|
| Session size and revisits (3) | log visits, log unique urls, revisit rate | |
| Topic shares (16) | arts, culture, entertainment and media, conflict, war and peace, crime, law and justice, disaster, accident and emergency incident, economy, business and finance, education, environment, health, human interest, labour, lifestyle and leisure, politics, science and technology, society, sport, topic missing share | |
| Genre shares (10) | Forum, Information/Explanation, Instruction, Legal, News, Opinion/Argumentation, Other, Promotion, Prose/Lyrical, genre missing share | |
| Emotion shares (5) | anger, fear, joy, sadness, emotion missing share | |
| Order geometry (16) | first pc1, first pc2, first pc3, first pc4, last pc1, last pc2, last pc3, last pc4, step rms mean, step rms std, step rms min, step rms max, endpoint rms, path efficiency, turn cosine mean, centroid distance change | |
| Relation summaries (22) | topic pair same density, topic temporal same rate, topic transition entropy, genre pair same density, genre temporal same rate, genre transition entropy, emotion pair same density, emotion temporal same rate, emotion transition entropy, similarity mean, similarity std, similarity min, similarity max, weighted degree mean, weighted degree std, weighted degree max, temporal similarity mean, non temporal similarity mean, temporal similarity gap, temporal above median, laplacian gap, laplacian radius |
| Coh-Metrix | Embeddings | |||
|---|---|---|---|---|
| Representation | ||||
| GRU, unordered | ||||
| Task | TaSFC criteria | of which bonus items | Excluded |
|---|---|---|---|
| FM | 9 | 3 | – |
| GS | 6 | 2 | – |
| HS | 7 | 0 | – |
| TP | 5 | 0 | cq4_1 , cq4_2 |
| Specification | Minimum | Mean | 95th percentile | Maximum | |
|---|---|---|---|---|---|
| All specifications | 6,192 | ||||
| By predictor source, all tiers | |||||
| Written response included | 1,512 | ||||
| Response alone ( ) | 756 | ||||
| Response with documents ( ) | 756 | ||||
| Browsing trace-only specifications | 4,680 | ||||
| Tier 1, observation | Tier 2, student holdout | Tier 3, student and task holdout | |||||
| Outcome | Representation | [95% CI] | [95% CI] | [95% CI] | |||
| Coh-Metrix indices | |||||||
| DIRQ | [-0.11, 0.05] | 1.000 | [-0.13, 0.04] | 1.000 | [-0.17, 0.01] | 1.000 | |
| [0.14, 0.31] | 0.006 | [0.14, 0.31] | 0.006 | [0.23, 0.40] | 0.006 | ||
| [0.09, 0.26] | 0.006 | [0.08, 0.25] | 0.006 | [0.18, 0.36] | 0.006 | ||
| TaSFC | [-0.07, 0.10] | 0.466 | [-0.05, 0.12] | 0.513 | [-0.12, 0.07] | 1.000 | |
| Block | Description | Features | Median | Largest single feature |
|---|---|---|---|---|
| Written response, 17 Coh-Metrix indices | 17 | WRDPRO [.30, .67] | ||
| Consulted pages, same 17 indices | 17 | WRDPRO [.00, .28] | ||
| Session descriptors | 17 | log unique URLs [.11, .39] | ||
| Block-level, cross-validated | ||||
| Dimensions | Share | |||
| Coh-Metrix content | 17 | |||
| Page class | Visits | Median words | IQR | Median sentence length |
|---|---|---|---|---|
| Content pages | 3,383 | 759 | 486–1074 | 12.5 |
| Search-results pages | 1,668 | 220 | 198–296 | 10.5 |
| Task pair | DIRQ | TaSFC | COH | |
|---|---|---|---|---|
| FM–GS | 23 | |||
| FM–HS | 17 | |||
| FM–TP | 23 | |||
| GS–HS | 21 | |||
| GS–TP | 27 | |||
| HS–TP | 25 |
| Outcome | Student | Student task | Rater | Rater task | Residual |
|---|---|---|---|---|---|
| DIRQ | 23 ( 0.001) | 23 ( 0.001) | 9 (0.112) | 8 ( 0.001) | 37 |
| TaSFC | 8 (0.110) | 64 ( 0.001) | 2 (0.139) | 5 ( 0.001) | 21 |
| COH | 19 ( 0.001) | 14 ( 0.001) | 19 (0.001) | 3 ( 0.001) | 45 |
| Response fulfillment | 6 (0.049) | 20 ( 0.001) | 0 (1.000) | 15 ( 0.001) | 59 |
| Source-reference use | 36 ( 0.001) | 43 ( 0.001) | 1 (0.022) | 1 ( 0.001) | 20 |
| Rated criterion | Range across tasks | |
| g5_coherence | +0.25 to +0.50 | |
| g4_2_sourcesref | +0.20 to +0.30 | |
| g4_sources | +0.19 to +0.27 | |
| cq4_values | +0.17 to +0.17 | |
| cq1_evidence | +0.16 to +0.16 | |
| cq3_sum | +0.14 to +0.14 |
| DIRQ | TaSFC | Overall | ||||
| Representation | as scored | standardized | as scored | standardized | as scored | standardized |
| Within-task at Tier 3 | ||||||
| 0.079 | 0.149 | 0.027 | 0.077 | 0.013 | 0.097 | |
| 0.320 | 0.327 | 0.126 | 0.127 | 0.195 | 0.193 | |
| 0.019 | 0.002 | 0.010 | 0.081 | 0.026 | 0.072 | |
| 0.004 | 0.021 | 0.055 | 0.021 | 0.077 | 0.043 | |
| Group | Index | Description |
|---|---|---|
| Connectives | CNCADC | Adversative and contrastive connectives |
| CNCCaus | Causal connectives | |
| CNCNeg | Negative connectives | |
| CNCPos | Positive connectives | |
| Descriptive | DESWLlt | Word length, number of letters, mean |
| Syntactic Pattern Density | DRAP | Adverbial phrase density |
Educational Assessment
Educational assessment measures what learners know and can do, using tests, rubrics, and learning evidence to guide decisions.
See all 159 papers