Multimodal models have often scored below text-only baselines in the BabyLM Challenge (Klerings et al., 2024; Takmaz et al., 2025). We test whether additional recurrent computation helps close this gap. Recurrence increases computation by reusing layers without adding parameters or training data. Using a single implementation, we compare standard and looped Transformers under a 100M-word budget, with three seeds per main condition. Each image is represented by one feature vector from a frozen DINOv2 encoder, used as a prefix token. Caption words count toward the budget, so multimodal training also changes the language data. In a pre-registered text-only comparison, a 12-layer stack repeated four times exceeds a standard 48-layer Transformer by 5.4 BLiMP percentage points at matched inference compute. However, multimodal training lowers BLiMP accuracy by about 3.8 points in both architectures, with no clear mitigation from recurrence. An exploratory loop-side no-image control, trained with the multimodal language data and batching schedule, scores 5.7 points below the text-only baseline. Adding the visual prefix raises accuracy by 1.85 points on average, but this contrast is uncertain with three seeds. These results suggest that changes in language data and its presentation are a major source of the drop. Diagnostic tests show that the models use visual information; its contribution to grammatical generalization remains uncertain under this single-vector interface.
Figures & tables
Figure 1: RQ1 compares standard and looped models at matched parameter count or inference compute. RQ2 compares their changes in accuracy from text-only to multimodal training. RQ3 compares the looped multimodal model with a no-image control matched in language data and batching.
Figure 2: Model architecture with a single visual prefix token from frozen DINOv2 features. Looped models add text embeddings after every application of the shared core. The dashed path adds the visual vector before the first Ninj applications in loop_mm .
text
multimodal
standard ( K=1 )
75.21±0.68
71.37±0.29
looped ( K=4 , re-inj.)
76.16±0.45
72.38±0.83
looped, prefix-once
—
72.31±0.63
identity connector ( K=1 )
—
72.48±0.69
identity connector ( K=4 )
—
72.14±0.15
std2x (24 layers)
73.84±0.25
70.96±0.88
Table 1: BLiMP accuracy (%) on the full evaluation set, reported as mean ± standard deviation across three seeds. All multimodal conditions use an initial visual prefix; “re-inj.” also adds the visual vector before the first two core applications.
vs. std_mm
vs. std_mm_id
penalty (standard)
−3.84
−2.74
penalty (looped, re-inj.)
−3.79
−3.79
penalty (looped, prefix-once)
−3.86
−3.86
ΔDiD (re-injected)
+0.06
−1.05
ΔDiD ( matched )
−0.01
−1.12
Table 2: Changes in BLiMP accuracy from text-only to multimodal training and their interactions, in percentage points (three-seed means). Positive ΔDiD indicates a smaller drop for the looped model. The primary contrast is the “matched” row under std_mm , where both multimodal models use only an initial visual prefix (Eq. 3 ).
Task
pen. (std)
pen. (loop)
DiD
DiD m
BLiMP
−3.84
−3.79
+0.06
−0.01
BLiMP Suppl.
−3.66
−4.95
−1.29
−0.08
EWoK
+0.04
−0.93
−0.96
−0.64
COMPS
−0.62
−0.01
+0.61
−0.10
Entity tracking †
−0.42
+1.85
+2.28
+1.88
Table 3: Changes in zero-shot accuracy from text-only to multimodal training, in percentage points (three-seed means). The pen. (loop) and DiD columns use loop_mm , whereas DiD m uses loop_mm_prefix with the same visual-input schedule as std_mm . Both interactions use std_mm as the multimodal standard baseline, and positive values indicate a more favorable change for the looped model. † All conditions score below the 20% chance level on Entity tracking.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Cell
K
Params
Core FLOPs
std_text
1
97,241,856
1×
std_mm
1
98,423,040
1×
loop_text
4
97,241,856
4×
loop_mm
4
98,423,040
4×
Appendix
Table 4: Parameter counts and relative core-stack FLOPs per token ( dmodel=768 , 12 layers, 12 heads, vocab 16,000). Multimodal cells differ from text cells only by the MLP connector ( +1,181,184 parameters); std_mm_id removes it. The frozen vision encoder runs offline and is excluded.
Condition
BLiMP
Suppl.
EWoK
std_text
75.21±0.68
63.37±0.45
53.92±0.85
loop_text
76.16±0.45
63.56±0.68
54.19±0.50
std_mm
71.37±0.29
59.71±0.29
53.96±0.46
loop_mm
72.38±0.83
58.61±1.53
53.26±0.30
std_mm_id
72.48±0.69
58.78±2.31
53.56±0.19
loop_mm_prefix
72.31±0.63
59.82±1.35
53.58±0.76
Appendix
Table 5: Zero-shot accuracy (%) on the final checkpoints, reported as the mean and standard deviation across three seeds. All scores use the official evaluation pipeline, and the BLiMP results match Table 1 .
Figure 3: Interactions across the 13 BLiMP linguistic categories. Filled circles show three-seed means for the primary comparison with matched visual input, and open squares show means for the visual-refresh variant. Small dots show individual seed estimates; whiskers span their minimum and maximum values.
Condition
s42
s43
s44
mean ± sd
std_text
75.23
75.88
74.53
75.21±0.68
loop_text
76.61
75.71
76.17
76.16±0.45
std_mm
71.62
71.05
71.44
71.37±0.29
std_mm_id
71.68
72.93
72.82
72.48±0.69
loop_mm
71.56
73.21
72.36
72.38±0.83
loop_mm_prefix
72.34
71.66
72.92
72.31±0.63
Appendix
Table 6: BLiMP accuracy (%) on the final checkpoints for seeds 42, 43, and 44. The looped model scores below the standard model for text-only seed 43 and for multimodal seed 42 with visual refresh. With matched visual input, the looped model scores higher for all three seeds.
Figure 4: (a) BLiMP accuracy on the fast-evaluation subset at 28 checkpoints, using one submitted seed per condition. (b) Multimodal minus text-only accuracy within each architecture. Both panels plot cumulative training words on a logarithmic horizontal axis.
Figure 5: Held-out caption loss (nats per token) for the standard and looped text-only models, using 20,000 captions and seed 42. Cumulative training words are plotted on a log scale for all 28 checkpoints in (a), and on a linear scale from 100M onward in (b).
Cell
correct
shuf.
zero
Δshuf
std_mm
1.87
3.33
3.78
1.453±0.005
std_mm_id
1.87
3.32
3.31
1.451±0.006
loop_mm
1.84
3.35
3.74
1.516±0.002
loop_mm_prefix
1.85
3.34
3.78
1.485±0.009
Appendix
Table 7: Caption negative log-likelihood (nats per token) under correct, shuffled, and zeroed image features, averaged equally across captions and then across three training seeds. Δshuf reports the shuffled-minus-correct difference as the mean and standard deviation across training seeds.
Cell
correct
shuf.
zero
Δcor−shuf
std_mm
47.4±1.1
42.9±0.6
44.2±2.3
+4.6±1.0
std_mm_id
45.1±3.6
41.8±3.1
42.6±3.6
+3.3±1.2
loop_mm
45.1±4.8
41.0±5.3
35.1±14.1
+4.1±1.0
loop_mm_prefix
41.2±4.8
37.3±4.8
40.2±3.5
+3.8±0.6
std2x_mm
38.6±13.3
34.9±12.4
33.0±12.9
+3.7±1.0
std4x_mm
39.1±6.7
35.7±5.4
30.6±6.5
+3.4±1.4
Appendix
Table 8: VQA accuracy (%) with correct, shuffled, and zeroed image features. Δcor−shuf is the correct-minus-shuffled difference in percentage points. Values are mean ± standard deviation across three training seeds, except for the single-seed exploratory models ( ‡ ) trained on a reduced image subset.
Cell
(adds)
BLiMP
loop_mm_subset
anchor
70.60
loop_mm_rich
+ richness
70.47
loop_mm_hive
+ staging
70.87
loop_mm_hive_k8
+ recipe
55.36
Appendix
Table 9: BLiMP accuracy (%) on the full evaluation set for the four exploratory conditions, each trained with seed 42. All conditions use the same reduced image subset and a peak learning rate of 6×10−4 . VQA results for these models are reported in Appendix H , Table 8 .
1B-word runs
K=8 probes
K
peak LR
seeds
BLiMP
LR
loss
1
1.5×10−3
3
75.21±0.68
3×10−4
3.03
2
1.5×10−3
1
75.28
6×10−4
2.96
4
1.5×10−3
3
76.16±0.45
1×10−3
2.93
8
1.5×10−3
1
76.35
1.5×10−3
2.86
8
1×10−3
1
75.42
Appendix
Table 10: Left: BLiMP accuracy on the full evaluation set after 1B cumulative training words. The K=1 and K=4 rows report means and standard deviations across three seeds; the other rows report individual runs with seed 42. Right: final training loss from the four K=8 learning-rate probes, each trained for 100M cumulative words.
Cell
ptext
BLiMP
VQA
Δcor−shuf
flat
0.5
72.31±0.63
41.0±4.9
+3.7±0.6
v2l
0→1
72.70±0.47
46.4±5.4
+6.0±0.8
l2v
1→0
66.62±1.13
35.5±5.6
+2.8±1.9
v2l_short
0→1 , len.
70.66
23.7\lx@sectionsign
+3.2\lx@sectionsign
Appendix
Table 11: BLiMP accuracy on the full evaluation set and VQA accuracy with correct image features (%). Scores with ± are means and standard deviations across three seeds; scores without ± use seed 42. ptext is the probability of selecting a text-only batch, and “len.” denotes the caption-length schedule. The VQA columns for the three-seed conditions come from a re-evaluation of the final checkpoints in fp32; Table 8 reports the original bf16 evaluation of the flat model. § Single-seed value from the original bf16 evaluation. Δcor−shuf is the correct-minus-shuffled VQA difference in percentage points.
Language-data and batching change (text-only − matched control)
looped
6.50
4.33
6.29
5.71±1.20
[2.73,8.68]
Appendix
Table 12: Loop-side batch-matched no-image control, BLiMP accuracy (%) and paired contrasts in percentage points. The comparisons use loop_mm_prefix , loop_mm_no_image_matched , and loop_text , with three seeds per condition.