Online State-Representation Adaptation under Synthetic Interaction-Cue Remapping
Organizations: School of Psychology, Fujian Normal University
Abstract
When cues change what they predict, continuing to update a learned mapping may help an agent choose the appropriate action. We tested this in a one-step synthetic task by comparing a logistic state predictor that continued to learn from state labels with the same predictor frozen at transfer. We evaluated ten conditions in six held-out environments, with 1,296 paired runs per condition and 1,296,000 transfer records in total. Late action accuracy was 93.39% with online updates and 64.63% with frozen coefficients, a paired difference of 28.76 [8.73, 53.19] percentage points (95% environment-bootstrap interval). A direct policy reached 95.65% using the same predictions with a different action threshold. A contextual bandit reached 94.23% and earned more cumulative utility than the adaptive predictor. The bandit used the reward of its chosen action, which in this task reveals the true state when combined with that action. Weights based on development history lowered cumulative utility relative to neutral weights (difference -3.45 [-4.48, -2.43]). The adaptive predictor outperformed its frozen counterpart in late accuracy in each of the six test environments. The other comparisons show how the choice of action threshold and the information available in feedback affect what can be attributed to online adaptation.
Figures & tables
| State | Avoid ( ) | Approach ( ) |
|---|---|---|
| Threat ( ) | ||
| Support ( ) |
| Model condition | Late accuracy (%) | Cumulative utility |
|---|---|---|
| Adaptive state | ||
| Frozen state | ||
| Direct policy | ||
| Full-information value | ||
| Contextual bandit | ||
| Neutral weights |
| Left right | Accuracy difference (pp) | Utility difference |
|---|---|---|
| Adaptive frozen | ||
| Adaptive direct policy | ||
| Adaptive full-information value | ||
| Adaptive contextual bandit | ||
| History neutral weights | ||
| History fixed weights |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Flips | Missing | to | Label flip | Delay | Adaptive | Frozen |
|---|---|---|---|---|---|---|---|
| 200 | 4 | 2 | 0.531 to 0.381 | 0.00 | 0 | 91.53 | 39.63 |
| 201 | 0 | – | 0.593 to 0.593 | 0.05 | 0 | 92.43 | 86.39 |
| 202 | 2 | 4 | 0.526 to 0.526 | 0.00 | 2 | 95.32 | 89.10 |
| 203 | 4 | – | 0.412 to 0.412 | 0.05 | 2 | 94.07 | 13.70 |
| 204 | 0 | 1 | 0.601 to 0.751 | 0.00 | 0 | 93.87 | 85.93 |
| 205 | 2 | – | 0.409 to 0.559 | 0.05 | 0 | 93.12 | 73.06 |
| Model condition | (%) | (%) |
|---|---|---|
| Adaptive state | ||
| Frozen state | ||
| Direct policy | ||
| Full-information value | ||
| Contextual bandit | ||
| Neutral weights |
| Model condition | Early accuracy (%) | Normalized AUC | First threshold crossing |
|---|---|---|---|
| Adaptive state | |||
| Frozen state | |||
| Direct policy | |||
| Full-information value | |||
| Contextual bandit | |||
| Neutral weights |
| Model condition | Utility given threat | Utility given support | Brier score |
|---|---|---|---|
| Adaptive state | |||
| Frozen state | |||
| Direct policy | |||
| Full-information value | – | ||
| Contextual bandit | – | ||
| Neutral weights |
Concept Drift
Concept drift is change over time in the relationship between data and target outcomes, which can erode the accuracy of deployed models.
See all 101 papers