On the Out-of-Distribution Generalization of Reasoning in Multimodal LLMs for Simple Visual Planning Tasks

Yannic Neuhaus; Nicolas Flammarion; Matthias Hein; Francesco Croce

cs.LGFeb 17, 2026

On the Out-of-Distribution Generalization of Reasoning in Multimodal LLMs for Simple Visual Planning Tasks

Authors: Yannic Neuhaus, Nicolas Flammarion, Matthias Hein, Francesco Croce

Organizations: Tübingen AI Center – University of Tübingen · EPFL · ELLIS Institute Finland – Aalto University

Abstract

Integrating reasoning in large language models and large vision-language models has recently led to significant improvement of their capabilities. However, the generalization of reasoning models is still vaguely defined and poorly understood. In this work, we present an evaluation framework to rigorously examine how well chain-of-thought (CoT) approaches generalize on a simple planning task. Specifically, we consider a grid-based navigation task in which a model is provided with a map and must output a sequence of moves that guides a player from a start position to a goal while avoiding obstacles. The versatility of the task and its data allows us to fine-tune model variants using different input representations (visual and textual) and CoT reasoning strategies, and systematically evaluate them under both in-distribution (ID) and out-of-distribution (OOD) test conditions. Our experiments show that, while CoT reasoning improves in-distribution generalization across all representations, out-of-distribution generalization (e.g., to larger maps) remains very limited in most cases when controlling for trivial matches with the ID data. Surprisingly, we find that reasoning traces which combine multiple text formats yield the best (and non-trivial) OOD generalization. Finally, purely text-based models consistently outperform those utilizing image-based inputs, including a recently proposed approach relying on latent space reasoning.

Figures & tables

Figure 1: Chain-of-thought (CoT) format impacts out-of-distribution (OOD) generalization. We fine-tune Qwen2.5-VL-7B-Instruct on the FrozenLake datasets with maps of size up to 6×6, in grid format. The models trained without CoT or with CoT in description format lead to no OOD (larger maps) generalization, while combining grid and description reasoning steps yield non-trivial results until 10×10 maps. This shows that generalization of reasoning is influenced by the format of the CoT traces.

Figure 2: Maze representations. In Wu et al. (2024) , the planning task is introduced with several representations of the maze: as an image , a text description and a table . We also introduce an ASCII-based grid representation requiring less tokens to encode the map.

Figure 3: Reasoning traces in different formats. We illustrate an example reasoning trace, from first step to the solution, in the various formats. While Xu et al. (2025) use the sequence of maze representations as images , we generate the corresponding steps as text-only descriptions , tables and grids . In the description format at each step we formulate a discussion of which the next step should be, while for the other formats we have the visual representation of the map after the next move.

Input format	CoT format	ID test maps ( $d_{\infty}\leq 5$ )					OOD maps (random $d_{\infty}$ )					OOD maps ( $d_{\infty}\geq 6$ )
Input format	CoT format	3x3	4x4	5x5	6x6	Avg	7x7	8x8	9x9	10x10	Avg	7x7	8x8	9x9	10x10	Avg
	zero-shot	0.14	0.05	0.03	0.02	0.06	0.02	0.02	0.01	0.01	0.01	0.01	0.00	0.00	0.00	0.00
Image	no CoT	0.89	0.84	0.72	0.52	0.74	0.51	0.37	0.23	0.12	0.31	0.02	0.00	0.00	0.00	0.01
	Descr.	0.95	0.87	0.76	0.62	0.80	0.47	0.35	0.21	0.14	0.29	0.07	0.03	0.00	0.01	0.03
Descr.	zero-shot	0.41	0.25	0.23	0.13	0.26	0.12	0.14	0.10	0.10	0.12	0.01	0.06	0.04	0.03	0.03
	no CoT	0.92	0.91	0.86	0.64	0.83	0.65	0.55	0.49	0.41	0.52	0.19	0.08	0.04	0.02	0.08

Table 1: Influence of data format on ID and OOD generalization. We report accuracy on both ID and OOD map sizes of zero-shot and fine-tuned LLMs using different combination of input and CoT traces (if any) formats. For OOD maps, we distinguish test sets where the distance between start and goal position is

d_{\infty}\geq 6

, i.e. not seen during training. CoTs with mixed format ( grid/table + description ) lead to the best in- and out-of-distribution performance, with non-trivial accuracy until 10×10 maps while being trained on up to 6×6 maps.

	ID test maps ( $d_{\infty}\leq 5$ )				OOD maps (random $d_{\infty}$ )				OOD maps ( $d_{\infty}\geq 6$ )
	3x3	4x4	5x5	6x6	7x7	8x8	9x9	10x10	7x7	8x8	9x9	10x10
Mean $d_{\infty}$	1.50	1.79	2.16	2.75	3.27	3.66	4.01	4.47	6.00	6.35	6.72	7.00

Table 2: Average start-goal distance. We report the average distance

d_{\infty}

between the start and goal positions for each map size. For the OOD maps with

d_{\infty}\geq 6

we enforce that both map size and start-goal distance are out-of-distribution compared to the training data.

Input format	CoT format	ID test maps ( $d_{\infty}\leq 5$ )					OOD maps ( $d_{\infty}\geq 6$ )
Input format	CoT format	2	3	4	5	Avg	6	7	8	9	Avg
Image	no CoT	0.70	0.46	0.21	0.03	0.35	0.03	0.01	0.00	0.00	0.01
Image	Descr.	0.69	0.41	0.16	0.06	0.33	0.01	0.00	0.00	0.00	0.00
Descr.	no CoT	0.82	0.75	0.57	0.34	0.62	0.01	0.01	0.00	0.00	0.01
	Descr.	0.89	0.80	0.68	0.41	0.69	0.00	0.00	0.00	0.00	0.00
	Grid	0.82	0.76	0.67	0.49	0.69	0.34	0.20	0.10	0.04	0.17

Table 3: Generalization w.r.t. start-end distance (

d_{\infty}

) with fixed 10×10 map size. We compare different input and CoT format when the training maps have fixed size 10x10 and

d_{\infty}<6

, test maps size 10×10 and

d_{\infty}\in\{2,\ldots,5\}

(ID) or

d_{\infty}\in\{6,\ldots,9\}

(OOD). Using grid input with grid or grid + description CoT format yields the best results, while the accuracy of models with no CoT or CoTs in other formats drop close to

0\%

already at

d_{\infty}=6

Panel of Figure 4 — Figure 4: OOD generalization w.r.t. optimal solution length. Top: we show the distribution of the length of the optimal (shortest) solution paths for both training and test maps (aggregated across map sizes). Bottom: we show success rate of models using grid as input representation and various CoT formats. Even when using only training maps with optimal solution length $\leq 10$ (dashed line), the grid + description CoT yields non-trivial success rate on maps with solutions of length $11$ and $12$ .

Model	ID test maps ( $d_{\infty}\leq 5$ )
Model	3x3	4x4	5x5	6x6	Avg
Direct SFT ( Yang et al., 2025 )	0.88	0.81	0.73	0.47	0.72
CoT SFT ( Yang et al., 2025 )	0.68	0.53	0.35	0.31	0.47
Mirage Direct ( Yang et al., 2025 )	0.93	0.83	0.76	0.51	0.76
Mirage Direct (retrained)	0.91	0.82	0.79	0.51	0.76
Mirage Direct shuffled	0.92	0.85	0.82	0.5	0.77

Table 4: Comparison to Mirage. We compare the ID performance of models from Yang et al. (2025) to ours (20 epochs training, * trained for 10 epochs only). The continuous reasoning of Mirage does not provide benefit over our model trained without CoT data.

Panel of Figure 5 — Figure 5: Example of the CoT reasoning of our models on OOD maps. We show an example of the reasoning trace produced by the model train on grid input and grid + description CoT. Even on an OOD map, the model first reasons in natural language on the next move, then produces the map after such move.

Input format	CoT format	ID maps	OOD maps (random $d_{\infty}$ )	OOD maps ( $d_{\infty}\geq 6$ )
Image	zero-shot	328	562	564
	no CoT	10	11	19
	Descr.	65	69	192
Descr.	zero-shot	383	732	870
	no CoT	10	13	20
	Descr.	69	89	141

Table 5: Output length. We report the average length (in tokens) of the correct solutions, including reasoning, across models.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	ID test maps ( $d_{\infty}\leq 5$ )					OOD maps (random $d_{\infty}$ )					OOD maps ( $d_{\infty}\geq 6$ )
Model	3x3	4x4	5x5	6x6	Avg	7x7	8x8	9x9	10x10	Avg	7x7	8x8	9x9	10x10	Avg
Direct SFT ( Yang et al., 2025 )	0.88	0.81	0.73	0.47	0.72	-	-	-	-	-	-	-	-	-	-
CoT SFT ( Yang et al., 2025 )	0.68	0.53	0.35	0.31	0.47	-	-	-	-	-	-	-	-	-	-
Mirage Direct ( Yang et al., 2025 )	0.93	0.83	0.76	0.51	0.76	-	-	-	-	-	-	-	-	-	-
Mirage Direct (retrained)	0.91	0.82	0.79	0.51	0.76	0.45	0.35	0.22	0.15	0.29	0.01	0.01	0.01	0.00	0.00
Mirage Direct shuffled	0.92	0.85	0.82	0.5	0.77	0.46	0.41	0.26	0.14	0.31	0.05	0.06	0.02	0.00	0.03

Appendix

Table 6: Comparison to Mirage. We additionally report the OOD performance of the models of Table 4 . The latent reasoning approach of Mirage does not provide better OOD generalization than simple supervised fine-tuning without CoT.

	3x3	4x4	5x5	6x6	Total
Number of samples	100	200	300	400	1000
Mean $d_{\infty}$	1.40	1.80	2.23	2.72	2.26
Mean solution length	2.03	2.65	3.58	4.18	3.48

Appendix

Table 7: Statistics of training data. We report the mean

d_{\infty}

as well as the mean length of the ground truth solution sequence on the training set.

Hyperparameter	Value
epochs	10
per_device_train_batch_size	1
gradient_accumulation_steps	1
warmup_steps	10
learning rate	1e-5
weight_decay	0.01

Appendix

Table 8: SFT hyperparameters. These values were used for all text-based models. The image models use the same hyperparameters as Yang et al. (2025)

Model	ID test maps ( $d_{\infty}\leq 5$ )
Model	3x3	4x4	5x5	6x6	Avg
Gemini-1.0 ( Xu et al., 2025 )	0.31	0.26	0.15	0.06	0.20
Claude-3 ( Xu et al., 2025 )	0.52	0.33	0.16	0.15	0.29
GPT-4o ( Xu et al., 2025 )	0.68	0.58	0.35	0.24	0.46
Anole Direct ( Li et al., 2025 )	0.83	0.80	0.75	0.75	0.78
Anole CoT ( Li et al., 2025 )	0.94	0.72	0.50	0.39	0.64

Appendix

Table 9: Other baselines. We report the performance of other baselines models and visual reasoning methods on the ID tasks. The results are taken from the previous works reporting them.

Input format	CoT format	ID test maps ( $d_{\infty}\leq 5$ )					OOD maps (random $d_{\infty}$ )					OOD maps ( $d_{\infty}\geq 6$ )
Input format	CoT format	3x3	4x4	5x5	6x6	Avg	7x7	8x8	9x9	10x10	Avg	7x7	8x8	9x9	10x10	Avg
	zero-shot	210	429	309	363	328	450	590	662	547	562	524	457	774	502	564
Image	no CoT	9	10	10	11	10	12	11	10	9	11	19	-	-	-	19
	Descr.	52	59	64	86	65	81	74	65	54	69	196	217	-	162	192
Descr.	zero-shot	222	264	505	541	383	588	836	832	675	732	848	791	1010	833	870
	no CoT	9	10	11	12	10	13	13	13	13	13	21	19	18	22	20

Appendix

Table 10: Output length. We report the average length (in tokens) of the correct solutions, including reasoning, our the various fine-tuned models for each map size.

Panel of Figure 6 — Figure 6: Number of epochs. We extended the fine-tuning to 20 and 30 epochs for some of the models and evaluated them on the ID and the OOD (random $d_{\infty}$ ) test maps. However, the overall performance and model ranking remain largely unchanged.

Panel of Figure 7 — Figure 7: OOD generalization w.r.t. optimal solution length. We show success rate of models using different input and CoT representation over solution length, complementing the results of Fig. 4 for the grid format.

OOD Generalization

Out-of-distribution generalization studies how machine-learning models perform when test data, domains, tasks, or environments differ from training.

See all 384 papers

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Jun 1, 2026Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2Fine-Tuning Multimodal Large Language Models
University of Virginia
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
Jun 21, 2026Zhuoran Jin, Kejian Zhu, Hongbang Yuan +5Multimodal Reasoning VLM Reasoning
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Tool Use Reduces Depth-Induced Collapse in OOD Reasoning
Feb 24, 2026David Koplow, Tomer Galanti, Tomaso PoggioOOD Generalization LLM Tool Use
Massachusetts Institute of Technology · Texas A&M University

Jun 1, 2026J/K move · Enter open · S save

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

Sanchit Sinha, Guangzhi Xiong, Bohan Liu, Zhenghao He+1

University of Virginia

Fine-Tuning Multimodal Large Language Models VLM Reasoning Multimodal CoT Reasoning Visually Grounded Reasoning Multimodal Understanding

Open paper AskPDF arXiv

The effectiveness of Chain-of-Thought (CoT) prompting in Multimodal Large Language Models (MLLMs) remains uncertain: across several visual reasoning benchmarks, CoT prompting often degrades performance compared to direct prompting. In this paper, we provide a systematic analysis of CoT behavior in three modern MLLM families across model scales on datasets requiring step-wise visual evidence. Our analysis identifies two recurring failure modes: premature answer commitment and limited direct visual-token access during rationale generation. We further find that standard CoT-style Supervised Fine-Tuning (CoT-SFT) can mitigate these issues only partially, while often increasing reliance on textual priors and reducing counterfactual visual dependence. Motivated by these findings, we propose Attentive-CoT (Att-CoT), an attention-guided fine-tuning objective that encourages CoT trajectories to delay answer commitment while maintaining sustained visual-token access. Att-CoT can be plugged into any CoT-SFT training run without architectural changes. Experiments on three visual reasoning benchmarks across six MLLMs show that Att-CoT enhances CoT performance over standard fine-tuning.

Abstract

OOD Generalization

Explore similar work

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do

Tool Use Reduces Depth-Induced Collapse in OOD Reasoning