FloodVLM-Pix: Language-Guided Pixel-Level Flood Understanding

Chengyang Zhang; Zhihao Wang; Zhiyu Long; Xueqian Wang; Quanming Yao; Wei Wu; Gang Li

cs.CVOct 16, 2026

FloodVLM-Pix: Language-Guided Pixel-Level Flood Understanding

Authors: Chengyang Zhang, Zhihao Wang, Zhiyu Long, Xueqian Wang, Quanming Yao, Wei Wu, Gang Li

Organizations: Tsinghua University · National Disaster Reduction Center

Abstract

As climate change intensifies, floods are becoming more frequent and destructive, making accurate flood understanding an urgent priority. However, existing vision-language models perform poorly on flood-related tasks, suffering from attentional bias and inaccurate descriptions of flood magnitude. To bridge this gap, we construct the first large-scale remote sensing flood multi-modal benchmark that supports language-level flood understanding and pixel-level flood segmentation. The benchmark contains 131,200 flood image patches and corresponding 443,100 textual descriptions for flood severity and evolution. Based on this benchmark, we propose FloodVLM-Pix, a language-guided pixel-level flood understanding model that includes a flood prior module and a multi-temporal flood pixel decoder to achieve accurate flood understanding. The flood prior module automatically distills sub-region flood spatial priors and inputs priors into FloodVLM-Pix, solving the attentional bias problem in flood language understanding. The flood pixel decoder introduces a novel general-specific feature fusion mechanism to address performance degradation caused by interference (e.g., clouds or mountain shadows) during flood segmentation. Experiments on FloodVL-Pix demonstrate that FloodVLM-Pix improves both flood description quality and pixel-level flood segmentation performance compared with state-of-the-art VLMs and flood segmentation models. Our code and data will be released at https://github.com/Haomiaoyanmeng/FloodPix.

Figures & tables

Figure 1: A SAR image patch with flood areas in the upper and left parts. Inference results of Qwen3.6-Plus [ 3 ] : (a) without prior, (b) with prior. In the absence of a prior (a), the Qwen3.6-Plus misinterprets large-scale flooding as background and thus ignores it. After incorporating the prior in (b), the Qwen3.6-Plus yields the correct result.

Figure 2: (a) Clouds and cloud shadows, (b) agriculture, (c) mountain shadows, and (d) wet surface areas in optical or SAR images, degrading the performance of flood segmentation.

Models (Benchmarks)		Flood-Oriented	Language	Pixel Results	Multi-Temporal ( $\geq$ 3)
Flood Models	S1GFloods [ 39 ]	✓	✗	✓	✗
	SEN12-FLOOD [ 36 ]	✓	✗	✓	✓
	CAU-Flood [ 13 ]	✓	✗	✓	✗
	FloodNet [ 35 ]	✓	✗	✓	✗
	Kuro Siwo [ 6 ]	✓	✗	✓	✓
Vision-Language Remote Sensing Models	RingMo-GPT [ 47 ]	✗	✓	✗	✗

Table 1: Comparison of different models/benchmarks in terms of flood-oriented design, language support, pixel-level labels, and multi-temporal analysis capability.

Figure 3: The statistics of the FloodVL-Pix dataset. (a) Flood coverage ratio distribution. (b) Continent distribution. (c) Modality distribution. (d) Temporal image patch number distribution per region. Numbers inside/above bars denote counts/percentage of flood image patches or regions.

Figure 4: Structure of our FloodVLM-Pix model. Our model consists of a vision encoder, a projector, an LLM, a prior module, and a pixel decoder. The vision encoder and projector convert image patches into tokens for LLM understanding. The prior module provides spatial priors for flood-related cues. The pixel decoder decodes image patch features to pixel-level flood segmentation results. The yellow and blue small rectangles below the LLM represent text tokens and visual tokens, respectively. “Image 1/2/3” denotes the order of input image patches.

Figure 5: Structures of our pixel decoder. (a) Overview of our pixel decoder. (b) Optical/SAR image patch encoder. (c) General-specific fusion module. (d) Flood mask decoder.

Models	ROUGE-L	CIDER	CLAIR
Kimi-2.6 [ 43 ]	0.2526	0.1056	0.5599
Doubao-Seed-2.0-Lite [ 40 ]	0.2449	0.0706	0.4097
Doubao-Seed-2.0-Pro [ 40 ]	0.2653	0.0924	0.4498
Doubao-Seed-2.1-Turbo [ 40 ]	0.2696	0.0929	0.3945
GLM-5.3-Flash [ 53 ]	0.2590	0.0888	0.5209
DeepSeek-V4-Flash [ 52 ]	0.2721	0.0967	0.4230

Table 2: Language-level flood understanding results of SOTA methods and FloodVLM-Pix. † denotes models fine-tuned on our dataset and models without † are evaluated by direct inference. The best result in each column is shown in blue , and the second-best is shown in red .

Figure 6: Results of the SOTA flood segmentation models and our proposed models. The white, red, blue, and black pixels are the regions of true positives, false alarms, missed targets, and true negatives, respectively. Our method yields fewer missed detections, demonstrating the effectiveness of the proposed pixel decoder.

Model	Precision	Recall	OA	F1	IoU
FTransUNet [ 24 ]	0.6524	0.9583	0.8921	0.7763	0.6344
LSNet [ 46 ]	0.9156	0.8191	0.9557	0.8647	0.7616
MFGF-UNet [ 48 ]	0.8134	0.9294	0.9510	0.8675	0.7661
MFNet [ 23 ]	0.7514	0.9600	0.9301	0.8430	0.7286
SMAGNet [ 17 ]	0.7679	0.9622	0.9358	0.8541	0.7454
FloodVLM-Pix-2B	0.8407	0.9497	0.9606	0.8919	0.8049

Table 3: Flood segmentation results of existing SOTA methods and FloodVLM-Pix. The best and second best results are in blue and red , respectively. Our FloodVLM-Pix-2B/4B achieve higher overall segmentation performance, i.e., OA, F1, IoU.

Figure 7: Flood description CLAIR of direct inference (baseline), fine-tuning with/without the prior module on Qwen3-VL-2B. Fine-tuning refers to LoRA-based fine-tuning [ 14 ] of the LLM and full fine-tuning of the projection layer, while the prior module is kept frozen. “w”/“w/o” prior module denotes fine-tuning with and without the flood prior information output by the prior module, respectively. Flood ratio is the proportion of the flood area in the image patch, and the last column (0%-100%) summarizes performance across all flood ratios, equating to the average CLAIR. Fine-tuning substantially improves performance, and incorporating the prior module further yields an average gain of 6.9% in CLAIR.

Model	OA	F1	IoU
w/o LN + Add	0.9447	0.8525	0.7429
LN + Add	0.9539	0.8764	0.7800
LN + Concat	0.9606	0.8919	0.8049
w/o General-Specific Fusion Module	0.9564	0.8823	0.7894
w/o Features from Qwen	0.9556	0.8799	0.7856

Table 4: Ablation experiments of different pixel decoders. The ”w/o LN”/”LN” means the Qwen feature extraction architecture in Fig. 5 (b) without/with layer normalization layers. “Add”/“Concat” denotes that either element-wise addition or channel-wise concatenation is employed to fuse features from the flood image patch and those from Qwen in Fig. 5 (b). The best and second best results are in blue and red , respectively.

Remote Sensing VLMs

See all 41 papers

Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation
Aug 2, 2026Nafis Fuad, Xiaodong Qian, Dongxiao ZhuVLM Adaptation Depth Estimation
Department of Civil & Environmental Engineering Wayne State University Detroit, MI, USA · Department of Civil & Environmental Engineering Institute for AI and Data Science (AIDaS) Wayne State University Detroit, MI, USA · Department of Computer Science, Institute for AI and Data Science (AIDaS) Wayne State University Detroit, MI, USA
FloodReasonBench: Benchmarking VLM Reasoning Segmentation for Embodied Flood Response at the Edge
Aug 15, 2026Rajat Bhattacharjya, Yoomee Jung, Minwoo Kim +4Reasoning Segmentation VLM Evaluation
University of California, Irvine · Kookmin University, South Korea
Flood Mapping from RGB imagery using a Vision Foundation Model
Jun 23, 2026Vladyslav Polushko, Tilman Bucher, Ronald Rösch +3Remote Sensing Image Segmentation Geospatial Foundation Models
Image Processing Department, Fraunhofer ITWM, Kaiserslautern, Germany · Institut für Optische Sensorsysteme, DLR, Berlin, Germany · ACIDA Lab, Hochschule Darmstadt, Darmstadt, Germany +1

Aug 2, 2026J/K move · Enter open · S save

Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation

Nafis Fuad, Xiaodong Qian, Dongxiao Zhu

Department of Civil & Environmental Engineering Wayne State University Detroit, MI, USA · Department of Civil & Environmental Engineering Institute for AI and Data Science (AIDaS) Wayne State University Detroit, MI, USA · Department of Computer Science, Institute for AI and Data Science (AIDaS) Wayne State University Detroit, MI, USA

VLM Adaptation Depth Estimation VLM Interpretability Mechanistic Interpretability Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Urban flooding poses an escalating threat to transportation infrastructure, yet no operational system provides real-time, street-level flood-depth estimates at centimeter resolution. This paper presents three vision-language models fine-tuned for continuous flood-depth estimation from street-level imagery: FloodLlama-Dense, a fully fine-tuned QLoRA baseline, and FloodLlama-MI5 and FloodLlama-MI6, interpretability-guided sparse variants that fine-tune only the top five and six causally relevant cross-attention layers identified through mechanistic interpretability analysis, respectively. Training uses an approximately 610,000-image subset of a 2.81-million-image synthetic corpus generated in Unreal Engine 5. The dataset combines single-vehicle subsets with 5 cm depth increments and mixed-vehicle subsets with 1 cm depth increments, spanning seven vehicle types, four weather conditions, and flood depths from 0 to 40 cm. FloodLlama-Dense achieves an MAE of 0.40 cm, an RMSE of 1.97 cm, and an Acc@5cm of 97.59%. Mechanistic interpretability analysis combining linear probing, logit lens, centered kernel alignment (CKA), and cross-attention entropy reveals a two-stage adaptation pattern: layers L13-L22 restructure visual representations, while depth first becomes linearly decodable at layer L23. FloodLlama-MI5 and FloodLlama-MI6 leverage this insight by fine-tuning only five or six of the eight cross-attention layers, achieving an 86-88% reduction in trainable parameters (6.55-7.86 million versus 54.4 million) with minimal accuracy loss. On a real-world benchmark, FloodLlama-MI6 achieves 98.62% accuracy, compared with 86.61% for the published STURM-FloodDepth baseline.

Abstract

Remote Sensing VLMs

Explore similar work

Mechanistic Interpretability-Guided Selective Fine-Tuning of Vision-Language Models for Centimeter-Level Flood Depth Estimation

FloodReasonBench: Benchmarking VLM Reasoning Segmentation for Embodied Flood Response at the Edge

Flood Mapping from RGB imagery using a Vision Foundation Model