Authors: Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer, Mengye Ren, Jakob Verbeek, Saining Xie, Hu Xu
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.
Figures & tables
Figure 1: A controlled study of multimodal vision encoding. (Top) Every recipe shares one backbone architecture (“B”) and trains all modules from scratch on common data at matched lifetime compute (§ 2 ). (Bottom) Each panel unbundles one vision encoder design choice, marks the favored recipe ( ), and states its takeaway; orange marks a pretrained prior and blue dedicated vision capacity. § 6 goes beyond lifetime compute by treating encoder pretraining as free, adding vision to a pretrained language model, and measuring training and inference speed.
Table 1: Unbundling the encoder’s design choices. A pretrained encoder bundles all three; in-fusion keeps only dedicated vision capacity.
Figure 2: Encoder pretraining is not beneficial at matched lifetime compute. (a) At 1.7B and 1.04×1022 -FLOP lifetime budget, spending a larger share of compute on encoder pretraining is comparable to the from-scratch encoder (horizontal dashed blue ) up to 15% and lowers performance beyond it. (b) MetaCLIP pretraining uses 1.54×1021 FLOPs at every backbone size ( orange floor ); the from-scratch encoder matches or exceeds MetaCLIP at every budget. The vertical dashed gray line at 1.7B marks the budget of (a) and the top axis gives the from-scratch encoder’s TPP.
Figure 3: Encoder-free trails the encoder on Avg. VQA, while Avg. Text stays comparable. Each checkpoint is decayed, and arrows mark the compute factor encoder-free needs to match the encoder. Section D.1 gives per-category results.
Table 2: Matching total parameters does not close the encoder-free gap. All rows use a 1.7B backbone and 2.6×1021 lifetime FLOPs. Indented rows ( + ) add the encoder’s 0.4B parameters to the backbone. Bold: within one point of the best.
Figure 4: In-fusion vision capacity adds to the shared backbone instead of replacing it. (a) All text and vision tokens are processed by shared modules ( gray ); vision tokens also pass through vision-only modules ( blue with pink outline) that add to the shared outputs; in-fusion (FFN-only) omits the vision attention projections ( Δ QKV, Δ O). (b) MoT uses entirely separate modules for text and vision, so the modality streams interact only through attention.
Table 3: Added in-fusion capacity is most effective when placed early and reserved for vision tokens. Every variant adds 0.4B parameters to the 1.7B backbone. Schematics show backbone layers from early to late; gray serves all tokens, blue only vision tokens, and wider layers are taller. Highlighted : our default placement. Bold: within one point of the best.
Figure 5: In-fusion is comparable to FFN-only at 1.7B and slightly ahead at every 4B budget.
Figure 6: In-fusion is comparable to the pre-fusion encoder. Encoder-free is shown for reference. Table 26 gives per-category results at the longest budget.
Figure 7: Encoder-free trails the encoder at every size, while in-fusion tracks it. Points and curves shown are 64 TPP. Each recipe’s log-linear VQA fit (a) and power-law loss fit (b) use all of its runs, across sizes and TPPs ( appendix G ).
Figure 11
Figure 10: In-fusion serves images faster than the encoder at every vision parameter budget. Designs are parameter-matched on the 1.7B backbone.
Appendix figures & tables54 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Vision encoder
1.7B
4B
0.4B
Configuration
Qwen3-1.7B
Qwen3-4B
MoonViT-V2
Layers
28
36
27
Width
2,048
2,560
1,024
Query heads
16
32
12
FFN width
6,144
9,728
4,096
Appendix
Table 4: Every multimodal recipe shares one backbone configuration per scale, and the encoder adds a 0.4B ViT. Parameter counts exclude the linear projection from the encoder into the backbone.
Table 5: The default 27-layer, 0.4B encoder gives the strongest VQA and comparable performance on text. Larger encoders do not automatically translate to stronger visual understanding. All rows use a 1.7B backbone at 2.6×1021 FLOPs of lifetime compute. Bold: within one point of the best.
Table 6: On the 4B backbone, the Large encoder attains the strongest VQA. All variants are trained to 5.8×1021 FLOPs of lifetime compute. Bold: within one point of the best.
Setup
VQA (%) ↑
Text (%) ↑
TPP
FLOPs
0.4B encoder
1.7B encoder
Encoder- free
0.4B encoder
1.7B encoder
Encoder- free
16
3.90×1021
45.9
46.0
40.4
52.3
53.0
52.6
32
7.81×1021
49.7
50.2
44.3
56.4
56.4
56.4
64
1.56×1022
53.1
52.8
47.5
59.9
59.1
58.9
Appendix
Table 7: At 6.4B, keeping the encoder at 0.4B is comparable to scaling it to 1.7B. All runs follow the size ladder’s recipe ( appendix G ). Bold: within one point of the best.
Multimodal training
Instruction tuning
Optimizer
Muon, AdamW
AdamW
Peak learning rate
3×10−4 (1.7B), 2×10−4 (4B)
1×10−5
Warmup updates
4,000
1,000
Schedule
warmup–stable–decay
cosine
Final learning rate
5% of peak
5% of peak
Weight decay
0.1
0.1
Appendix
Table 8: The 1.7B and 4B backbones share one optimization recipe, with the peak learning rate scaled to backbone size. Muon and AdamW use the same peak learning rate.
Table 9: Pretraining the encoder on regenerated captions slightly outperforms alt-text. All variants are trained to either 1.8×1020 or 3.7×1020 FLOPs of multimodal training (32 and 64 TPP, respectively).
Data mixture
VQA (%) ↑
Text (%) ↑
Text/image–text
Avg
Gen
Knw
OCR
Vis
Avg
Com
Sci
Knw
Rdg
100:0
28.0 ∗
33.4 ∗
31.3 ∗
3.7 ∗
43.5 ∗
50.1
64.3
44.8
36.2
55.3
90:10
41.2
53.1
34.5
31.0
46.3
49.3
62.8
44.3
36.3
53.9
70:30
43.4
55.2
36.0
37.0
45.6
48.0
62.6
44.5
31.7
53.2
50:50
43.0
54.3
35.2
35.9
46.4
47.7
63.7
44.2
31.4
51.5
Appendix
Table 10: The 70:30 mixture favors visual understanding, whereas 90:10 preserves more language ability. All mixtures use a 1.7B backbone at 2.34×1021 FLOPs of lifetime compute at a constant learning rate. For 100:0, images are replaced with zeros during instruction tuning and evaluation, so its starred VQA entries are blank-image controls ( section B.1 ).
Figure 11: Later starts concentrate the same image–text data into fewer updates. Every schedule sees the same number of image–text examples, so the blue areas are about equal. Labels give each schedule’s final image–text fraction.
Image–text start
VQA (%) ↑
Text (%) ↑
Avg
Gen
Knw
OCR
Vis
Avg
Com
Sci
Knw
Rdg
From first update
42.5
55.1
36.0
32.5
46.2
50.8
64.6
46.2
38.0
54.4
25% start
35.0
43.8
31.9
19.4
44.8
52.0
64.6
46.5
41.9
55.0
50% start
35.1
43.8
34.0
17.0
45.5
51.0
64.2
46.3
40.2
53.1
75% start
34.4
42.8
33.7
17.2
44.0
50.8
64.6
46.0
39.2
53.4
Appendix
Table 11: Starting image–text training from the first update gives the strongest visual understanding. All rows use a 1.7B backbone at 2.6×1021 FLOPs of lifetime compute and a 90:10 mixture overall, scheduled as in fig. 11 . Bold: within one point of the best.
1.7B
4B
Budget
FLOPs
TPP
FLOPs
TPP
55.6B
6.5×1020
39
1.45×1021
15
111.1B
1.30×1021
79
2.90×1021
31
222.2B
2.60×1021
158
5.79×1021
61
444.4B
5.21×1021
315
1.16×1022
122
666.7B
7.81×1021
473
1.74×1022
183
Appendix
Table 12: Compute budgets can be expressed as token-equivalents, FLOPs, or TPP. TPP divides by each backbone’s non-embedding parameter count.
Benchmark
Reference
Split
Metric
Examples
General (Gen)
MME-P
Fu et al. (2025)
test
MME score /20
2,114
MMBench-EN
Liu et al. (2024a)
dev
Exact match
4,329
SEED-Bench
Li et al. (2024)
test
Exact match
5,000
GQA
Hudson and Manning (2019)
test-dev
Exact match
5,000
Knowledge (Knw)
Appendix
Table 13: Avg. VQA weights four categories of visual understanding benchmarks, Each category contains four tasks. MME-P is normalized to [0,100] before averaging. Examples counts the questions evaluated.
Task
Reference
Shots
Scoring
Commonsense reasoning (Com)
COPA
Roemmele et al. (2011)
0
Summed log-likelihood
HellaSwag
Zellers et al. (2019)
0
Mean token log-likelihood
PIQA
Bisk et al. (2020)
0
Mean token log-likelihood
Social IQa
Sap et al. (2019)
0
Summed log-likelihood
WinoGrande
Sakaguchi et al. (2020)
0
Summed log-likelihood
Appendix
Table 14: Avg. Text averages four categories of text benchmarks.
Figure 12: At the same text-token budget, the multimodal recipes score about as well on Avg. Text as the text-only reference. Each point is one decayed endpoint, placed by the text-only tokens it trained on (70% of multimodal training).
MetaCLIP
WebSSL
Cap and CapPa
Global batch (examples)
32,768
3,072
8,192
Optimizer
Muon, AdamW
Muon, AdamW
Muon, AdamW
Peak learning rate
4×10−4
3.5×10−4
2.5×10−4
Warmup updates
2,000
100,000
10,000
Schedule
cosine to 0
cosine to 10−6
cosine to 0
Weight decay
0.2
0.04 to 0.2 (cosine)
10−4
Appendix
Table 15: Every objective pretrains the same encoder on the same data and compute. The training schedules span the entire horizon of each encoder pretraining run.
Table 16: The from-scratch encoder matches the best pretrained encoders at 1.7B and leads every objective at 4B. The models are trained to 1.04×1022 FLOPs at 1.7B and 5.8×1021 FLOPs at 4B, including encoder pretraining. ImageNet accuracy is measured before multimodal training. Bold: within one point of the best at each backbone size.
Figure 13: Joint training with the scratch encoder is comparable to a pretrained encoder. All models use a 1.7B backbone, and the curves start after the encoder’s pretraining compute (dashed line).
Replace
Swap
Add
Replace and swap
Model
Object
Attribute
Relation
Object
Attribute
Object
Attribute
Image
Shuffled
1.7B, 1.04×1022 FLOPs
Scratch encoder
79.2
82.0
77.0
71.4
84.8
95.5
96.5
78.9
64.7
Encoder-free
79.7
82.2
77.3
72.2
86.2
95.9
97.4
79.5
66.8
MetaCLIP
78.2
80.6
77.4
73.5
84.2
96.1
96.7
78.8
64.7
WebSSL
78.8
81.2
79.2
73.9
86.6
96.0
96.4
79.9
65.1
Appendix
Table 17: All recipes are comparable on compositional caption matching. Accuracy (%) is measured before instruction tuning. The last two columns average the replace and swap subsets, leaving out the add subsets because the text alone decides them. Bold: within one point of the best in each block, except the shuffled control.
Table 18: At fixed multimodal compute, longer encoder pretraining improves ImageNet accuracy but not visual understanding. All rows use a 1.7B backbone and 2.7×1021 FLOPs of multimodal training; lifetime FLOPs add the encoder pretraining. ImageNet LP is a linear probe of the pretrained encoder before multimodal training. Bold: within one point of the best in each VQA and Text column.
Figure 14: The encoder’s advantage over encoder-free is primarily in OCR and chart questions. Each point compares encoder-free with the encoder at one budget of table 27 .
Table 19: At 4B, a wider encoder-free backbone still trails the encoder on VQA. All variants are trained to 122 TPP, i.e., 1.16×1022 FLOPs. Bold: within one point of the best.
Table 20: Shared capacity needs about twice the encoder’s parameters to match it on VQA. All variants use a 1.7B backbone, train on the encoder’s token count, and use a constant learning rate.
Table 21: Richer patch embeddings narrow the gap to the encoder only when processing higher-resolution images. All variants train a 1.7B backbone at 315 TPP to 5.2×1021 FLOPs. VQA uses 224×224 instruction tuning. Bold: within one point of the best. Each architecture variant is described in the table below.
Figure 15: Comparing on averaged loss misses the encoder’s lead on caption loss and Avg. VQA. All runs use the 1.7B model from the scaling ladder. The averaged loss is the mean of the alt-text caption, interleaved, and text losses.
Setup
Held-out caption loss ↓
Avg. VQA (%) ↑
Recipe
Encoder learning rate
64 TPP
128 TPP
64 TPP
128 TPP
Regenerated captions
Scratch encoder
1× (ours)
1.229
1.096
32.9
36.3
Scratch encoder
0.1×
1.347
1.211
30.6
32.8
Scratch encoder
0.01× ( Shukor et al., 2025 )
1.372
1.230
28.8
31.2
Encoder-free
–
1.388
1.243
29.1
31.0
Appendix
Table 22: Reducing encoder learning rate removes most or all of the encoder’s lead. All variants use the 1B backbone from the scaling ladder at 64 or 128 TPP, i.e., 3.7×1020 or 7.3×1020 FLOPs of lifetime compute. The encoder learning rate scales only the encoder’s peak learning rate. Alt-text rows use the original alt-text instead of regenerated captions, in training and in the held-out loss. Bold: best in each column within each caption source.
Table 23: The in-fusion variant is matched with in-fusion (FFN-only). Both have a fixed added-parameter budget. Note that the proportion spent on vision-only attention projections grows with backbone width. All rows are checkpoints before the learning rate decay, at 2.34×1021 FLOPs (1.7B) and 1.04×1022 FLOPs (4B) of lifetime compute. Bold: within one point of the better allocation at each backbone scale.
Table 24: Reserving the FFN branch for vision tokens keeps its lead with twice the training. All rows use the 1.7B backbone, decayed at the lifetime compute of their block. Both FFN branches add 0.4B parameters to layers 1–8 of the encoder-free backbone. Bold: within one point of the best in each block.
Contrast
VQA (%) ↑
Other metrics
Δ Avg [95% CI]
Δ OOD caption loss
Δ ImageNet top-1
QKVO-only − encoder-free
+1.8 [1.0, 2.5]
− 0.059
+3.73
In-fusion − same-width FFN branch
+1.1 [0.4, 1.9]
− 0.013
+0.75
Appendix
Table 25: Vision-only attention projections improve visual understanding both alone and alongside an FFN branch. Each row is the difference between two 4B models trained on the same number of tokens, before the final decay. Caption-loss changes are in nats, where negative is better.
Table 26: In-fusion is comparable to the scratch encoder at longer training budgets. Both model scales train to 1.11T token-equivalents. Bold: within one point of the best at each scale.
Figure 16: In-fusion capacity confines vision processing to the earliest blocks, while encoder-free spreads it across shared layers. Vision-token change is defined as 1−cos(hL,hL−1) between consecutive blocks. The shaded band marks blocks 1–8, where the in-fusion modules sit. All models use the 1.7B backbone at 1.30×1022 FLOPs of lifetime compute.
Budget
VQA (%) ↑
Text (%) ↑
FLOPs
TPP
Enc
EF
FFN
QKVO
Text-only
Enc
EF
FFN
QKVO
1.7B backbone
6.5×1020
39
37.3
32.0
36.3
37.4
–
45.3
46.4
45.9
45.7
1.30×1021
79
41.2
34.8
41.4
40.4
48.9
47.3
48.3
48.4
48.1
2.60×1021
158
44.4
39.0
45.0
42.6
52.4
49.7
50.7
50.4
50.7
5.21×1021
315
46.5
41.9
46.7
46.5
54.4
52.5
52.6
52.7
53.4
Appendix
Table 27: Encoder-free trails the encoder at every compute budget, while in-fusion is comparable to it. Enc is the from-scratch encoder, EF encoder-free, and FFN and QKVO mark in-fusion (FFN-only) and in-fusion. All models in a row share the same lifetime compute. Dashes mark models not trained to that budget. Bold: within one point of the best in each row, excluding the text-only reference.
Figure 17: The encoder maintains its lead over encoder-free, and in-fusion is comparable to the encoder, at every peak learning rate. All models are 1.7B and trained to 39 and 79 TPP, i.e., 6.5×1020 and 1.3×1021 FLOPs of lifetime compute.
Figure 18: Held-out caption loss tracks Avg. VQA along one trend shared by all four recipes. Held-out caption loss uses images never seen in training. The line is a least-squares fit over all endpoints. All points are evaluated after learning rate decay.
Backbone
Encoder
In-fusion
Optimization
Non-emb. (M)
Width / layers
Width / layers
Vision-only (M)
Ratio
Layers
Width
Peak LR
Batch
Warmup
0.6B
393
1,280 / 20
640 / 19
115
0.29
1–6
3,584
5.1×10−4
448
429
1B
793
1,536 / 28
768 / 27
227
0.29
1–8
4,544
3.9×10−4
640
605
1.7B
1,409
2,048 / 28
1,024 / 27
401
0.28
1–8
6,144
3.0×10−4
1,024
672
3.2B
2,831
2,560 / 36
1,280 / 35
809
0.29
1–10
7,936
2.2×10−4
1,536
900
6.4B
5,848
3,328 / 44
1,664 / 43
1,693
0.29
1–13
9,664
1.6×10−4
2,304
1,239
Appendix
Table 28: Architectures and optimization settings of the scaling ladder. The vision module grows with the backbone size. Ratio is the vision-only parameter count over the backbone’s non-embedding count, which stays at around 0.29. The in-fusion columns describe the FFN+QKVO variant ( section 5.1 ). Batch counts sequences of 4,096 tokens, and warmup counts updates. Backbones are named by total parameters, including the tied embedding.
Setup
VQA (%) ↑
Text (%) ↑
Backbone
TPP
FLOPs
Scratch encoder
Encoder- free
In-fusion
Scratch encoder
Encoder- free
In-fusion
0.6B
128
2.09×1020
31.6
26.1
30.7
43.3
42.4
42.7
256
4.18×1020
33.4
27.9
32.6
44.1
43.3
44.4
1B
64
3.66×1020
32.9
29.1
33.6
43.3
43.5
44.0
128
7.32×1020
36.3
31.0
36.0
46.1
45.7
45.7
256
1.46×1021
40.1
33.4
38.0
47.6
47.9
47.3
Appendix
Table 29: In-fusion is comparable to the encoder at every size, whereas encoder-free trails both on VQA but stays comparable on text. This shows metrics for every scaling ladder run ( fig. 19 ). Bold: within one point of the best in each row.
Metric
Recipe
Law
Factor
Avg. VQA
Scratch encoder
13.7+4.40lnN+5.25lnD
–
Encoder-free
10.8+4.40lnN+4.75lnD
2.6–3.1
In-fusion
19.3+5.29lnN+3.94lnD
1.0–1.1
Caption loss
Scratch encoder
E=0.59 , A=0.28 , α=0.27 , B=7.54 , β=0.77
–
Encoder-free
E=0.70 , A=0.18 , α=0.59 , B=4.18 , β=0.54
2.4–3.8
In-fusion
E=0.63 , A=0.25 , α=0.40 , B=3.39 , β=0.61
0.9–1.0
Appendix
Table 30: Encoder-free needs 2.6 to 3.1 times the encoder’s compute on Avg. VQA, while in-fusion needs about as much as the encoder. N and D are in billions. Each factor is the compute at which a recipe’s law reaches a metric value, divided by the compute at which the encoder’s law reaches the same value. The factors are read at the compute of each run at 64 TPP from 1.7B to 6.4B.
Fit
Size out
6.4B out
Metric
RMS
Max
Mean
Max
Mean
Max
Avg. VQA (pts)
0.93
2.27
1.03
3.40
0.95
1.78
Caption loss (%)
1.20
4.81
2.03
7.42
1.91
5.00
Text loss (%)
0.36
0.83
0.41
1.52
0.45
0.95
Appendix
Table 31: One law per recipe fits every run and predicts each left-out backbone size. Fit gives the RMS and largest residual over all runs. Size out gives the mean and largest absolute error over all runs, each predicted by the laws refitted without its backbone size. The 6.4B out columns give the same over the 6.4B runs, predicted by the laws fitted on 0.6B to 3.2B.
Token-equivalents
Backbone
Recipe
111B
222B
444B
667B
889B
1.11T
1.7B
Scratch encoder
− 1.6
− 0.4
+ 0.6
+ 0.9
+ 1.2
+ 1.0
Encoder-free
− 2.7
− 1.3
+ 0.9
+ 1.4
+ 1.9
+ 2.1
In-fusion
− 0.8
− 0.7
− 0.5
–
–
− 0.6
4B
Scratch encoder
− 4.3
− 1.5
+ 0.9
+ 3.0
–
+ 3.3
Encoder-free
− 5.0
− 4.0
− 1.9
− 1.7
–
− 1.0
Appendix
Table 32: The ladder’s caption-loss laws predict the separately trained 1.7B and 4B runs within a few percent. Each entry is the measured held-out caption loss divided by the loss that the recipe’s law of table 30 predicts, minus one, in percent.
Figure 19: A single caption-loss law per recipe fits the runs at every backbone size. Points are the held-out caption loss of every ladder run. Each line is the recipe’s law evaluated at one backbone size.
Table 33: In-fusion vision capacity and the from-scratch encoder maintain their VQA lead in midtraining. From start shows VQA performance when training the same recipe with vision from the first step. Each recipe extends the 1.7B text-only model from 1T token-equivalents. Bold: within one point of the best.
Table 34: Tuning the encoder learning rate does not impact midtraining findings. LR multiple scales the learning rate of each recipe’s respective vision pathway, relative to the shared peak learning rate ( 3×10−4 ; table 8 ). Bold: within one point of the best on VQA. Avg. Text varies by only 0.6 points across all rows, so Text metrics are not bolded.
Architecture
Training time
Variant
Δ vs. encoder-free (%)
95% CI
Encoder
+2.9
[+1.9,+4.2]
In-fusion (FFN-only)
−0.3
[−0.8,+0.4]
In-fusion
+0.3
[−0.5,+1.1]
Appendix
Table 35: In-fusion vision capacity trains as fast as encoder-free, while the encoder-based model takes longer. Training time is for a 4B model projected to 1.04×1022 FLOPs of lifetime compute (400B token-equivalents). Intervals are paired 95% bootstrap intervals over 17 common timing windows.
Figure 20: Encoder-free and in-fusion (FFN-only) reduce training step time relative to encoder. All runs use the 1.7B backbone. Intervals are paired 95% bootstrap intervals.
Figure 21: Both encoder-free and in-fusion (FFN-only) reduce TTFT at higher resolution. TTFT is measured relative to the encoder at 1.7B with 16 concurrent requests, where intervals are full ranges over four runs.
Table 36: In-fusion capacity serves image requests faster than the encoder at every vision budget. Encoder-free takes 146 ms at 1.7B and 340 ms at 4B, 31% and 16% less than the Large encoder. Each request has one 448-pixel image (1,024 vision tokens) and 256 text tokens, served on one H200. Budgets describe the encoder’s vision-only parameters, which the in-fusion modules match. TTFT change relative to the scratch encoder is shown in parentheses.
Table 37: Per-benchmark results for fig. 2 . All runs use a 1.7B backbone and a 1.04×1022 -FLOP lifetime budget, of which Share is spent on encoder pretraining.
Table 38: Per-benchmark results for fig. 2 . MetaCLIP’s lifetime FLOPs include its 1.54×1021 -FLOP encoder pretraining.
Table 39: Per-benchmark results for fig. 3 .
Table 40: Per-benchmark results for table 2 . All rows use a 1.7B backbone and 2.6×1021 lifetime FLOPs.
Table 41: Per-benchmark results for table 3 . All rows use a 1.7B backbone and 2.6×1021 lifetime FLOPs.
Table 42: Per-benchmark results for fig. 5 .
Table 43: Per-benchmark results for fig. 6 .
Table 44: Per-benchmark results for fig. 7 . All runs train to 64 TPP.
Table 45: Per-benchmark results for fig. 9 . All runs use a 1.7B backbone and the same 2.7×1021 -FLOP multimodal stage; Pretraining is the encoder’s pretraining compute in FLOPs.
Table 46: Per-benchmark results for fig. 9 . Each run continues a 1.7B text-only model from 1T to 1.33T token-equivalents.
Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss--compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around 1022 FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.
Lin Chen, Bolin Ni, Qi Yang +6
CASIA · UCAS · Foundation Model Department, Tencent
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.
Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features can discard text, local attributes, and spatial relationships, while high-resolution inputs substantially increase context length and inference latency. We introduce \method, a Multi-scale Adaptive Vision Encoder. \method uses position-dependent gates to fuse shallow, intermediate, and deep features from a vision Transformer, preserving global semantics while enhancing edges, text, and local structure. It then performs question-conditioned token routing according to question relevance, local information content, global semantics, and spatial coverage, with a token budget that adapts to image complexity. To mitigate compression loss, we further introduce full-to-compressed representation distillation and a spatial diversity regularizer. In an illustrative simulation under a unified 7B language-model framework, \method reduces the average number of SigLIP-SO400M visual tokens from 729 to 146 (approximately 80.0%) and improves the mean score on VQAv2, GQA, TextVQA, ScienceQA-IMG, and MMBench by 2.2 percentage points, while reducing single-image time to first token from 228,ms to 129,ms. We provide the full model design and evaluation protocol. All reported numbers currently serve only as placeholders for paper organization and experimental design; formal claims require real training runs, independent replications, and official benchmark evaluation.