Déjà View: A Shared Working-Memory Model for Visual Tasks
Authors: Mranmay Shetty, Gabriel Kreiman
Organizations: Center for Neuroscience and Cognitive Systems, Italian Institute of Technology, Rovereto, Italy · Harvard Medical School, Boston, USA · Center for Brains, Minds, and Machines, MIT, Cambridge, USA
A working-memory model must remember stimuli and use its memory flexibly to perform different tasks. Here we introduce a model of working memory for images that combines dynamic activity with temporary associations between image features. We train the model on five tasks, including remembering a sample while other images are presented, categorizing distractor images while remembering the sample, and determining whether an image is new or has been presented before. The same trained model performs all tasks, guided by explicit cues about which response is required. We use this flexible memory model to ask how delay, distraction, and image history affect performance. Delays have little effect on sample-memory performance for the delays we tested, while seeing more images leads to more false alarms even when the target image appeared equally recently in all cases. On the sample-memory-plus-categorization task, the probability of correct responses on all questions decreases as more questions are asked, despite robust sample-memory performance. Finally, the model maintains stable performance across 40 brief trials of each task under the tested conditions. Together, these results demonstrate how a single working-memory system can be flexibly used to perform different tasks, and how its errors depend on both previous images and task demands.
Figures & tables
Figure 1: Tasks performed by our working-memory model. On Tasks 1–3, the model must report whether the test image matches the cued sample image. On Task 3, it must also categorize the distractor images presented during the delay. On Tasks 4–5, it must report whether any image repeats (Task 4) or whether each image has been presented before (Task 5). Responses shown are ground truth, not model responses or inputs. The transformed example is an altered version of the same photograph.
Figure 2: Schematic of our working-memory model. Activity ( h ) and associations ( F ) change over time. Context and role cues modify the keys that address the associations. Responses combine activity, retrieved content, image features and cues; category responses also receive normalized image features directly. The model answers from aged memory before writing the current image. Task 4 uses the largest local recognition score and image count for its final answer.
Figure 3: The same pictures under increasing memory demands. (a) Task 1: forty fixed sample–query pairs across blank delays. (b,c) Sample matching with novel nonmatching queries, with sample-response age fixed at 65 time units. (c) Category answers and complete-trial success are scored separately. (d) Familiar distractors used as wrong queries. (e,f) Longer image sequences compared with equal-duration blanks containing only two images; Task 5 shows the last response. Means average three models. Bars are exploratory 95% intervals over twelve photograph groups; (a) is descriptive. Small markers show model means except for category accuracy. Gray regions exceed training support. Lines connect the tested conditions, spaced equally for display.
Figure 4: Longer histories with equally recent targets. The target, query and images between them were identical across lengths. The longer sequence contained 48 additional unique images before the target; neither condition added a blank delay. Means average twelve photograph groups and three models. Open points show individual model means, and bars show descriptive 95% intervals over photograph groups. Length 64 exceeds training support. Appendix Table 4 reports the two length comparisons with intervals adjusted across all nine planned comparisons.
System
T1
T2
T3 joint
T4
T5
Shared h/F
90.71
87.61
44.90
67.03
76.75
Activity only
91.84
91.61
43.15
63.85
64.28
Associations only
91.58
88.27
45.61
69.32
75.89
Transformer
90.62
88.38
40.57
67.19
83.29
Gated DeltaNet
90.62
89.53
43.26
60.42
69.22
Table 1: Performance across the broad task battery (%). Tasks 1, 2, 4 and 5 report balanced accuracy across responses; Task 3 reports the percentage of trials with every response correct. Values average three trained models, using each model’s source-calibrated response rules and resetting memory between trials. Activity-only and associations-only models were trained separately. Equal training-update budgets do not match memory capacity, computation or convergence.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Panel
Trials
Binary
Positive/negative
Main schedule
1
Broad
384
384
192/192
delays 0,16,64,128,256,512
2
Broad
608
608
304/304
loads 0,1,4,8
3
Broad
608
608
304/304
loads 0,1,4,8; category + final
4
Broad
640
640
320/320
lengths 4,8,16,32,64,128; terminal
5
Broad
248
8,960
3,890/5,070
lengths 16,32,64,512
Appendix
Table 2: Response counts per trained model. Tasks 1–4 have one final binary answer per trial; Task 3 adds 2,336 category answers. Task 5 has 1,792 answers in the length panel, 4,096 in 512-image streams and 3,072 in the exposure panel. Exposures are 0, 0.25, 1 or 4 units, with fixed-interval or matched-duration schedules.
Intervening images
Equal-duration blanks
Task
Length
Hit
FPR
BA
Hit
FPR
BA
4
4
100.0
4.2
97.9
100.0
4.2
97.9
4
8
100.0
13.9
93.1
100.0
4.2
97.9
4
16
94.4
26.4
84.0
100.0
4.2
97.9
4
32
86.1
59.7
63.2
100.0
5.6
97.2
4
64
95.8
91.7
52.1
100.0
5.6
97.2
Appendix
Table 3: Class-specific repetition results (%). Task 4 shows its image-free final response; Task 5 shows only the response to the final query image. BA averages hit and correct-rejection rates. Blank conditions match elapsed time but contain only the first and last images. All results are averaged over three models and twelve photograph groups.
Figure 5: Crossed image-history and delay tests. Values are percentages, averaged over twelve groups and three fixed models. Blue shows correct responses; red shows familiar-wrong false positives. Final memory and repetition use balanced accuracy. Category answers occur before the added delay, and are undefined at zero distractors. The nine planned comparisons are reported in Table 4 .
Planned comparison
Difference (pp)
Adjusted interval
Task 1: viewing time × delay
-4.17
[-12.50, 0.00]
Task 2: count × delay, sample BA
+1.39
[0.00, +4.17]
Task 2: count × delay, familiar FPR
-6.94
[-18.06, 0.00]
Task 3: count × delay, sample BA
+1.39
[0.00, +4.17]
Task 3: count × delay, all-correct
0.00
[0.00, 0.00]
Task 4: length × delay
+1.39
[-2.08, +4.86]
Appendix
Table 4: Planned comparisons in the crossed study. Differences are in percentage points; intervals are adjusted across all nine comparisons.
Fixed condition
Carry: first ten
Carry: last ten
Change (pp)
Reset
Task 1: delay 16
94.25
94.17
-0.08
93.33
Task 1: delay 64
92.17
92.33
+0.17
91.67
Task 2: one distractor
92.83
93.17
+0.33
90.83
Task 2: four distractors
92.92
93.42
+0.50
88.33
Task 3: one distractor
82.17
82.08
-0.08
83.33
Task 3: four distractors
44.58
44.58
+0.00
45.00
Appendix
Table 5: Results with memory carried across trials or reset at the start of each trial (%). Each of the forty trials appears at every position, and results average three models. Tasks 1/2/4/5 show balanced accuracy after pooling binary responses; Task 3 shows complete-trial success. First ten refers to trial positions 1–10 and last ten to 31–40. Reset scores are constant across positions. Changes are descriptive, computed as last minus first before rounding.
Figure 6: Same-memory readout comparison. Each history was mirrored, changing only the earlier-presented object. The same query was repeated in one history and novel in the other. Within each history, each query and readout used an independent copy of the same aged memory state, before the query image was written. Horizontal comparisons ( Q ) assess paired ordering: does the repeated image score higher than the novel image? Vertical comparisons ( H ) assess history sensitivity: does the same image score higher after the history in which it repeated? Open circles show individual model means; the diamond shows their mean. The bar shows an exploratory 95% confidence interval over 32 photograph groups. Images are illustrative.