Emergent Vision-Shaped Thinking in Text-Only Reasoning through Multimodal Training
Organizations: Nanyang Technological University · Singapore Management University
Abstract
Does multimodal training shape how a language model thinks, even when the input is text only? We test this with targeted causal ablations across 10 multimodal configurations from 6 model families, with matched text-only backbones and cross-family transfer controls. We contrast spatial-imagery tasks that invite mental scene construction with symbol-manipulation tasks solvable by symbolic rules alone. We identify a vision-language axis in hidden-state space and ablate its projection at inference. In Chameleon-7B, this ablation drops accuracy on BIG-Bench Hard (BBH) navigate by 27.6 pp while leaving boolean expressions intact (+5.6 pp), a 33.2 pp double dissociation. The effect is direction-specific and layer-localized, extends to mental-simulation tasks such as mental rotation, paper folding, and perspective taking, and is not explained by task difficulty, reasoning depth, or sequential update load. Together, these results reveal emergent vision-shaped thinking (EVT): text-only reasoning that causally recruits representations introduced or strengthened by vision-language training, without assuming an image-like internal format. The measured effect is spatially concentrated at 7-8B and broadens to non-spatial visual attribute binding in the tested 32B configuration. In dual-path unified models (BAGEL, Janus-Pro), the causal handle localizes to the generation pathway as a compact one-dimensional carrier. The understanding pathway also encodes spatial information in a higher-rank, probe-decodable form, but is not the causal handle. To separate decodability from causal use, we analyze concentration, alignment, and semantic location. These diagnostics explain why behavior, probing, and intervention can diverge. Multimodal training thus reshapes the geometry of text-only reasoning, not just how visual input is processed.
Figures & tables
| Model | Fam. | Axis | Hub | DD | Specificity | ||
|---|---|---|---|---|---|---|---|
| Cham-7B | VQ | Emb. | L5 | 1.0 | pp | rand.; CI pp | |
| Emu3 | VQ | Diff. | L3+F | .50/.50 | pp | – | early/final rand.; CI pp |
| Liquid | FT | Diff. | L27 | 1.0 | pp | – | rand.; CI pp |
| BAGEL | Dual | Gen-Diff. | L21–27 | .5 | pp | rand.; CI pp | |
| Janus | Dual | Gen-Diff. (VQ) | L22–28 | .7 | pp | rand.; CI pp | |
| Qwen3-2B | ViT | Diff. | L3 | 1.0 | – | rand.; CI n.e. |
| Path | DiffMean (1D, ) | Top-1 EV | Top-16 EV | PCA rank- ablation |
|---|---|---|---|---|
| DD (pp) | (selectivity) | |||
| Generation | ( ) | 19% | 69.5% | non-selective at all ; controls drop 56–74 pp |
| Understanding | (n.s., ) | 53% | 86.2% | non-selective at all ; : controls vs. spatial pp |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Experiment | Model | tasks | seeds/dirs | GPU-h | Hardware | |
|---|---|---|---|---|---|---|
| Cham-7B core (Exp1) | Cham-7B | 250 | 2 | 3 GPU | ||
| Cham-7B 60-dir random control | Cham-7B | 250 | 2 | 60 | 8 GPU | |
| Cham-7B axis-config sweep | Cham-7B | 250 | 2 | 3 | GPU | |
| Cham-30B Exp1 + Exp2A | Cham-30B | 250 | 2 | 4 GPU | ||
| Cham-30B Exp2B (LOO axes) | Cham-30B | 250 | 2 | 12 pairs | 2 GPU | |
| Janus random control | Janus | 100 | 5 | 22 | 3 GPU |
| Model | Method | Sample source | Seed scheme | |
|---|---|---|---|---|
| Chameleon-7B/30B (main) | Embedding centroid | Embedding rows only (no runtime samples) | — | — |
| Cham-7B (task-contrastive axis) | Per-layer DiffMean | Synthetic random-pixel images | 50 | 4096+i for dir |
| Emu3-Chat | Per-layer DiffMean | Synthetic visual-token IDs | 50 | not seeded (fixed prompt loop) |
| BAGEL gen (main) | Per-layer DiffMean | Real COCO val2017 images | 50 | 4000+i |
| BAGEL und (main) | Per-layer DiffMean | Synthetic random images | 50 | 1000+i |
| BAGEL gen v2 (cross-check) | Per-layer DiffMean | Synthetic random images | 50 | 2000+i |
| Model (path) | Hub layers | at hub | Median (all layers) | Min (all layers) |
|---|---|---|---|---|
| Qwen3-VL-2B | L3 | (L0) | ||
| Qwen3-VL-4B | L6–8 | (L0) | ||
| Qwen3-VL-8B | L8 | (L0) | ||
| Qwen3-VL-32B | L8 | (L0) | ||
| Qwen3-VL-30B-A3B | L6–18 | – | (L0) | |
| Emu3-Chat | L0–7 | – | (L31) |
| Model (hub, ) | Axis built from | Spatial acc. | Control acc. | DD [95% CI] | |
|---|---|---|---|---|---|
| Qwen3-VL-8B | random pixels | ||||
| (L0–8, ) | real COCO | [ ] | |||
| Qwen3-VL-32B | random pixels | — a | CI | ||
| (L0–15, ) | real COCO | [ ] | |||
| Emu3-Chat | random token IDs | ||||
| (L0–7, ) | real COCO | – | [ ] |
| Model (block, ) | Image source | Spatial acc. | Control acc. | DD [95% CI] | Random null |
|---|---|---|---|---|---|
| Cham-7B (L0–7, ) | random pixels | [ ] | |||
| Cham-7B (L0–7, ) | real COCO | [ ] | ; | ||
| Cham-7B (L5 only, ) | real COCO | [ ] | |||
| Cham-30B (L12–23, ) | random pixels | [ ] | |||
| Cham-30B (L12–23, ) | real COCO | [ ] | ; | ||
| Cham-30B (L36–47, ) | random pixels | [ ] | ; all rand. |
| Model | Axis / path | Selection note | |
|---|---|---|---|
| Chameleon-7B | Embedding VL axis | Full removal; strongest layer-localized DD at L5. | |
| Emu3 | Early / final VL axes | / | Same dosage for both carriers (L0–7, L24–31); also degrades controls. |
| Liquid | DiffMean axis (L27) | Single layer. The L21–27 block has a sharp cliff: DD pp at , pp at , non-selective collapse at . | |
| BAGEL | Generation-path DiffMean | over-suppresses control tasks. | |
| Janus-Pro | Generation-path visual-token | Most significant in the sweep ( , L22–28); significant for , marginal at ( ). | |
| Qwen3-VL (all four) | Per-layer DiffMean axis | Full removal at the hub; 30B-A3B is a preliminary specificity check. |
| 0.0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| navigate (%) | 41.2 | 34.4 | 25.6 | 22.4 | 18.0 | 17.2 | 16.4 | 14.8 | 13.2 | 11.2 | 11.6 |
| boolean (%) | 38.8 | 42.0 | 48.4 | 46.4 | 47.2 | 45.6 | 48.0 | 48.4 | 48.4 | 48.8 | 50.0 |
| Experiment | Base seed | Use |
|---|---|---|
| Cham-7B/30B random-direction control | 2026 | seed = 2026 + i , |
| Cham-30B axis build | 2026 | single seed, default |
| Cham-7B task-contrastive random control | 4096 | seed = 4096 + i , |
| EXP4 phase 0 split 1 | 1000 | default_rng(1000+idx) per random image |
| EXP4 phase 0 split 2 (stability) | 2000 | 2000 + idx per random image |
| BAGEL gen (main) axis | 4000 | 4000 + idx per sample |
| Task | GPT | Gemini | Claude | DeepSeek | LLM Mean | Range | Human-A | Human-B |
|---|---|---|---|---|---|---|---|---|
| navigate | 5 | 5 | 5 | 5 | 5.00 | 0 | 5 | 5 |
| relative_position | 3 | 4 | 4 | 3 | 3.50 | 1 | 4 | 3 |
| colored_objects | 3 | 3 | 3 | 3 | 3.00 | 0 | 3 | 3 |
| temporal_sequences | 2 | 2 | 2 | 2 | 2.00 | 0 | 2 | 2 |
| causal_judgement | 1 | 1 | 1 | 1 | 1.00 | 0 | 1 | 1 |
| boolean_expressions | 0 | 0 | 0 | 0 | 0.00 | 0 | 0 | 0 |
| Direction | DD (pp) | Bootstrap 95% CI | Random DD | |
|---|---|---|---|---|
| VL axis, L0–7 |
| Layer region | Mean acc | vs. random | Note |
|---|---|---|---|
| L0–L6 (early, includes hub) | up to | L5 (hub) | |
| L7–L15 (mid) | up to | sustained signal | |
| L16–L31 (late) | near or below baseline | axis no longer aligned |
| Layer | 1D probe acc | vs. random |
|---|---|---|
| L25 | ||
| L26 | ||
| L27 | ||
| 1D DiffMean intervention (block L21–27, ) : DD pp ( , null) | ||
| Rank-16 PCA ablation at hub : damaging but control-dominated (spatial vs. control pp) | ||
| Aspect | Chameleon-7B (L5 hub) | BAGEL und (L21–26 hub) |
|---|---|---|
| Full-dim probe | at every layer | at every layer |
| 1D probe at the hub | (informative) | ( , informative) |
| 1D intervention at the hub | DD pp (L5; block ) | DD pp, (null) |
| Rank- intervention | not needed (1D succeeds) | control-dominated ( vs. pp); not selective |
| Diagnostic shape | Probe and causality co-localize | Probe localizes; no selective handle at 1D or rank- |
| Configuration | Endpoint acc. | Endpoint balanced acc. [95% CI] | Displacement acc. [95% CI] |
|---|---|---|---|
| Chameleon-7B (L5 hub) | 0.842 | 0.807 [0.726, 0.881] | 0.664 [0.605, 0.722] |
| BAGEL-und (L27 hub) | 0.825 | 0.795 [0.714, 0.872] | 0.784 [0.733, 0.835] |
| Qwen3-VL-8B | 0.842 | 0.819 [0.741, 0.891] | 0.828 [0.781, 0.873] |
| Qwen3-VL-2B | 0.881 | 0.849 [0.774, 0.917] | 0.764 [0.711, 0.816] |
| Qwen3-VL-4B | 0.872 | 0.855 [0.784, 0.921] | 0.796 [0.745, 0.844] |
| Qwen3-VL-32B | 0.889 | 0.867 [0.796, 0.931] | 0.832 [0.786, 0.877] |
| Direction (at hub L8) | Task-identity acc. | Causal role |
|---|---|---|
| Random 1D directions (mean) | 0.825 | none (0/100 match real axis) |
| Causal DiffMean axis | 0.540 ( vs. random) | under intervention |
| Full-dimensional probe | 1.000 | — |
| Condition | navigate accuracy | vs. baseline |
|---|---|---|
| Baseline (no intervention) | — | |
| L5 ablation ( , hub) | pp | |
| L5 ablation L6 reinjection ( ) | pp (recovered pp) | |
| L5 ablation L7 reinjection ( ) | pp (recovered pp) | |
| L20 ablation ( , non-hub) | pp (null) | |
| L20 ablation L21 reinjection ( ) | pp (no recovery effect) |
| navigate accuracy | Status | |
|---|---|---|
| ( ) | productive (recovers pp) | |
| ( ) | overshoot (token-prediction collapse) |
| Model / block | Condition | DD | |||
|---|---|---|---|---|---|
| Chameleon-7B, early block L0–7 (baseline: nav = 37.3%, bool = 35.3%) | |||||
| Removal ( ) | 1.0 | 22.0 pp | 2.0 pp | 20.0 pp | |
| Multiplicative ( ) | 0.3 | 15.3 pp | 0.7 pp | 16.0 pp | |
| Multiplicative ( ) | 0.5 | 30.7 pp | 14.0 pp | 16.7 pp | |
| Multiplicative ( ) | 1.0 | 37.3 pp | 35.3 pp | 2.0 pp | |
| Additive shift | 0.3 | 4.7 pp | 4.7 pp | 0.0 pp | |
| Condition | Navigate | Geometric | Control avg. | ||
|---|---|---|---|---|---|
| Baseline | |||||
| Gen-path ablation | |||||
| Rescue |
| Dataset | Subtask | ratio | ||
| Single-hop spatial reasoning | ||||
| bAbI | task 17 (positional reasoning) | pp | pp | — (sign-reversed) |
| bAbI | task 19 (path finding) | pp | pp | |
| bAbI | task 20 (agent motivation) | pp | pp | |
| SpartQA | yes/no + multi-choice | pp | pp | |
| StepGame | (single hop) | pp | pp | |
| Task | Baseline | Gen | Und | Gen–Und | Pattern |
|---|---|---|---|---|---|
| boolean_expressions | 98.7 | both | |||
| causal_judgement | 64.0 | und gen | |||
| date_understanding | 40.0 | und gen | |||
| disambiguation_qa | 50.7 | ||||
| dyck_languages | 12.0 | ||||
| formal_fallacies | 24.7 | gen und |
| Model | Task | Base Int. | (pp) | 95% CI | McNemar | |
| Qwen3-VL-32B | Mental rotation (easy) | 100 | ||||
| Qwen3-VL-32B | Mental rotation (hard) | 100 | ||||
| Qwen3-VL-32B | Paper folding (easy) | 100 | ||||
| Qwen3-VL-32B | Paper folding (hard) | 100 | (n.s.) | |||
| Qwen3-VL-32B | Perspective taking (easy) | 100 | ||||
| Qwen3-VL-32B | Perspective taking (hard) | 100 |
| Model | Base Int. | (pp) | 95% CI | McNemar | |
|---|---|---|---|---|---|
| Chameleon-7B | 250 | (n.s.) | |||
| Qwen3-VL-8B | 150 | ||||
| Qwen3-VL-32B | 100 |
| Task | Update load | Base Int. | (pp) | Completion |
|---|---|---|---|---|
| Perspective taking (spatial; easy/hard) | 0 | / | / | |
| Color tracking (non-spatial) | 3 | |||
| Color tracking (non-spatial) | 5 | |||
| Color tracking (non-spatial) | 8 | |||
| Color tracking (non-spatial) | 12 |
| Configuration | Baseline-controlled slope | Note |
|---|---|---|
| Qwen3-VL-32B | ||
| Chameleon-7B | Spearman , | |
| Emu3 | ||
| Janus-Pro | ||
| Liquid | ||
| BAGEL |
| Task | Base | Int. | Task | Base | Int. | ||
|---|---|---|---|---|---|---|---|
| navigate | 43.3 | 22.7 | date_understanding | 22.0 | 26.7 | ||
| tracking_shuffled_3_obj | 29.3 | 18.7 | boolean_expressions | 50.0 | 54.7 | ||
| movie_recommendation | 42.7 | 36.7 | ruin_names | 24.0 | 28.0 | ||
| snarks | 47.3 | 41.3 | logical_deduction_7_obj | 11.3 | 15.3 | ||
| tracking_shuffled_5_obj | 22.7 | 18.0 | formal_fallacies | 84.0 | 86.7 | ||
| salient_translation_error | 16.0 | 12.0 | reasoning_colored_objects | 15.3 | 18.0 |
| # | Method | Effect | Apparent cause |
|---|---|---|---|
| 1 | Embedding axis, , 7 layer configs | DD | Axis exists geometrically (separation 7B’s ) but is causally inert |
| 2 | Alpha sweep | DD | Stronger removal still inert |
| 3 | Alpha sweep | non-selective collapse | exceeds residual-norm budget |
| 4 | PCA subspace , two depth ranges | DD | Position-PCA is not task-causal-PCA; 2.4% variance subspace contains no causal direction |
| 5 | Single-task probe direction | non-selective | Probe fits prompt-template distribution, not spatial content |
| 6 | Multi-task LOTO probe (12 layers tested) | 0% across all layers | No linearly shared spatial-reasoning feature across tasks |
| Task | Group | Baseline | Intervened | |
|---|---|---|---|---|
| relative_position | spatial | pp | ||
| reasoning_about_colored_objects | spatial | pp | ||
| temporal_sequences | spatial | pp | ||
| web_of_lies | control | pp | ||
| causal_judgement | control | pp |
| Model | Layers | Mean | Max |
|---|---|---|---|
| Chameleon-7B | at L9 | ||
| Chameleon-30B | at L47 |
| navigate | boolean | nav (pp) | DD (pp) | status | |
|---|---|---|---|---|---|
| 0.40 | 51.2 | 51.6 | weak | ||
| 0.50 | 44.8 | 50.4 | block-level sweet spot | ||
| 0.55 | 8.8 | 48.8 | cliff: navigate collapses | ||
| 0.60 | 0.0 | 49.2 | navigate at zero | ||
| 0.65 | 0.0 | 48.4 | plateau (nav dead, bool stable) | ||
| 0.70 | 0.0 | 34.4 | boolean begins to collapse |
| Transfer | Effect | Verdict | Takeaway |
|---|---|---|---|
| Janus DeepSeek-LLM-7B | DD – pp; | null | No task-causal coupling in the unrelated text backbone. |
| BAGEL Qwen2.5-7B base | navigate pp; ; DD | non-selective | Active direction, but no spatial selectivity. |
| Liquid text-mode self-transfer | navigate pp; | sign-reversed | Text-mode axis amplifies rather than suppresses spatial reasoning. |
| Qwen2.5-VL Qwen2.5-7B base | DD pp ; | positive | Matched-family transfer supports amplification of a pre-existing Qwen spatial scaffold. |
| BAGEL und | BAGEL gen | Qwen3-VL-8B | |
| (L21–26, hub) | (L21–27, hub) | (L0–8, hub) | |
| Regime | concentric | projected-diffuse | mixed |
| Mean in hub | 0.94 | 0.42 | 0.44 (layer-bimodal) |
| Cumulative EV at | 52.9% | 18.9% | 47.1% |
| Cumulative EV at | 86.2% | 69.5% | 77.4% |
| DD @ (pp) | [ , ] | ctrl crash | ctrl crash |
| Model / axis | Layer | Random | ||||
|---|---|---|---|---|---|---|
| Chameleon-7B / VL axis | L3 | |||||
| Chameleon-7B / VL axis | L4 | |||||
| Chameleon-7B / VL axis | L5 | |||||
| Chameleon-7B / VL axis | L6 | |||||
| Chameleon-7B / VL axis | L7 | |||||
| Chameleon-7B / VL axis | L20 ctrl |