Anthropogenic Regional Adaptation in Multimodal Vision-Language Model
Authors: Samuel Cahyawijaya, Peerat Limkonchotiwat, Tack Hwa Wong, Hitesh Laxmichand Patel, Amit Agarwal, Manuel Antonio Rufino, Carlos Rafael Catalan, Muhammad Reza Qorib, Vicky Feliren, Holy Lovenia, Aye Hninn Khine, Frederikus Hudi, David Anugraha, Alham Fikri Aji, Romrawin Chumpu, Viet-Thanh Pham, Minghan Wang, Mohamed Fazli Imam, Ruochen Zhang, Joseph Marvin Imperial, Khumaisa Nur'aini, Do Xuan Long, Musa Izzanardi Wijanarko, Joel Ruben Antony Moniz, Patrick Amadeus Irawan, Hanif Muhammad Zhafran, Isaiah Flores, Salsabila Zahirah Pranida, Jun Kevin, Jostin Jerico Rosal, Patricia Nicole Monderin, Kun Kerdthaisong, Ahmad Mustafid, My Chiffon Nguyen, Natchapon Jongwiriyanurak, Siva Worajitwannakul, Haochen Li, Adrian Xuan Wei Lim, Bin Wang, Muhammad Ravi Shulthan Habibi, Lynnette Hui Xian Ng, Mithil Bangera, Yeshil Bangera, Priyaranjan Pattnayak, Dun Li Chan, Sherissa Caren Djuniwar, Cho Chan Myei Oo, Hee Ming Shan
While the field of vision-language (VL) has achieved remarkable success in integrating visual and textual information across multiple languages and domains, there is still no dedicated framework for assessing human-centric alignment in vision-language systems. We offer two contributions to address this gap. First, we introduce Anthropogenic Regional Adaptation: a novel paradigm that aims to optimize model relevance to specific regional contexts while ensuring the retention of global generalization capabilities. Second, we present a simple, but effective adaptation method named Geographical-generalization-made-easy (GG-EZ), which utilizes regional data filtering and model merging. Through comprehensive experiments on 3 VL architectures: large vision-language models, text-to-image diffusion models, and vision-language embedding models, and a case study in Southeast Asia (SEA) regional adaptation, we demonstrate the importance of Anthropogenic Regional Adaptation and the effectiveness of GG-EZ, showing 5-15% gains in cultural relevance metrics across SEA while maintaining over 98% of global performance and even occasionally surpassing it. Our findings establish Anthropogenic Regional Alignment as a foundational paradigm towards applicability of multimodal vision-language models in diverse regions and demonstrate a simple-yet-effective baseline method that optimizes regional value alignment while preserving global generalization.
Figures & tables
Figure 1: Through anthropogenic regional adaptation, we identify two primary model archetypes: (left) Global model with strong overall global performance, but struggle to represent certain regions appropriately, and (right) Regional-specific model that has a strong representation towards certain regions, but fall short on the global context.
Figure 2: Overview of our Geographical-generalization-made-easy (GG-EZ) framework. Our framework consist of 3 constituents: (1) High-quality regional data filtering pipeline; (2) supervised fine-tuning to create a high quality regional-specific model; and (3) model merging to capture the best combination between regional-specific and global represention while also maintaining the generalization capabilities of the model.
Model
GRP
Global
SEA-Specific
Avg.
WC
CVQA
Avg.
SEAVQA
WC
CVQA
Google Gemma-3
59.4
63.5
59.8
67.2
56.3
41.0
60.1
67.8
SEA-Gemma-3 5%
64.0
64.3
60.0
68.7
63.7
61.2
60.3
69.5
SEA-Gemma-3 10%
64.1
64.4
60.0
68.8
63.8
61.7
60.2
69.5
SEA-Gemma-3 50%
57.3
56.7
51.6
61.8
57.8
59.5
51.4
62.6
SEA-Gemma-3 70%
56.1
56.3
51.9
60.6
56.0
54.0
52.6
61.3
Table 1: SEA-VLM evaluation results on CVQA, SEA-VQA, and WorldCuisine (WC). <MODEL> (X%) denotes a merged model where X% is the weight β use in the linear merging process of the region-specific model.
Model
Overall
Language Breakdown
GRP
Global
SEA
fil
ind
tha
vie
zsm
Google Gemma-3
2.29
2.54
2.09
1.69
2.15
2.17
2.37
2.07
SEA-Gemma-3 10%
2.31
2.42
2.22
1.88
2.07
2.29
2.61
2.25
SEA-Gemma-3 (w/o merge)
1.74
1.18
2.23
2.75
2.29
2.33
1.76
2.00
Table 2: SEA-VLM human evaluation on SEA-AYA and SEA-VL VQA datasets. We report the average rank across different models (Higher is better). <MODEL> (X%) denotes a linear merge where X% is the weight β of the region-specific model.
Model
Overall
Aspects
Score
Attribute
Relation
Entity
Other
Global
StabilityAI SDXL
73.75
79.18
86.38
81.07
60.00
83.89
SEA-SDXL 25%
74.75
80.43
86.34
81.86
61.60
85.41
SEA-SDXL 50%
74.61
80.93
87.12
81.15
60.00
86.02
SEA-SDXL 75%
74.39
79.98
85.96
81.36
63.20
82.67
SEA-SDXL (w/o merge)
74.32
80.79
86.92
81.57
65.20
81.16
Table 3: SEA-SDXL Performance on Image Generation Benchmark DPGBench (Higher is better). <MODEL> (X%) denotes a linear merge where X% is the weight β of the region-specific model.
Model
Correctness
Naturalness
Overall
T
L
C
Overall
T
L
C
StabilityAI SDXL
1.491
1.470
1.636
1.387
1.675
1.436
2.023
1.613
SEA-SDXL 25%
1.569
1.587
1.729
1.413
1.767
1.473
2.124
1.753
SEA-SDXL (w/o merge)
1.431
1.473
1.527
1.307
1.557
1.340
1.806
1.560
Table 4: SEA-SDXL human evaluation results on 3 distinct cultural aspects. “T” denotes Tradition, “L” denotes Landmark, and “C” denotes Cuisine
Model
GRP
CVQA
SEAVQA
Global
SEA
Others
SEA
Khm
Ind
Vie
Google SigLIP2
25.17
25.51
24.02
25.84
25.81
28.95
25.00
25.56
SEA-SigLIP2 25%
24.96
24.38
24.44
24.36
26.36
27.63
25.40
23.96
SEA-SigLIP2 50%
27.10
27.52
25.50
27.97
28.06
31.25
26.99
27.16
SEA-SigLIP2 75%
27.96
27.12
27.51
27.03
29.66
28.62
30.05
28.75
SEA-SigLIP2 (w/o merge)
26.96
26.75
25.29
27.07
28.96
28.29
28.59
29.07
Table 5: SEA-VL Embedding Zero-Shot Performance on Cultural VQA Benchmarks. <MODEL> (X%) denotes a linear merge where X% is the weight β of the region-specific model. For SEA-VQA, we provide additional breakdown of 3 of the biggest country data in the dataset with “Khm”=Cambodia, “Ind”=Indonesia, “Vie”=Vietnam.
Figure 3: Impact of regional-specific data curation strategy on SEA-Gemma-3.
Figure 4: (left) Impact of globalization factor alpha to GRP across different models. Optimizing on a misaligned α can lead to suboptimal performance. (right) We derive α from the KOF globalization index [ 26 , 25 ] to better reflect the degree of globalization across regions. The globalization index is distinct across regions and evolves over time.
Language
Model
Grammar
Naturalness
Burmese (mya)
NLLB
4.6 ± 0.8
4.1 ± 1.1
Gemma-3-27b
4.1 ± 1.2
3.6 ± 1.2
Gemini-2.5-flash
4.8 ± 0.6
4.6 ± 0.8
Gemini-2.5-pro
4.9 ± 0.4
4.9 ± 0.5
Thai (tha)
NLLB
3.3 ± 1.1
3.0 ± 1.2
Gemma-3-27b
4.2 ± 0.9
4.1 ± 0.9
Table 6: Human evaluation of English→5 SEA languages translation quality using a five-point scale (mean ± standard deviation)
Reward Model
Rate where srma>srmb
ImageReward
0.394
HPSv2
0.384
VisionReward-Image
0.466
UnifiedReward
0.442
Table 7: Rate at which srma>srmb for each reward model under consideration.
Llama-3.2
Pixtral
Gemma-3
SEA-Gemma-3
SEA-Gemma-3
Aya-Vision
Qwen-3 VL
Qwen-2.5-VL
Molmo
Language
Vision (11B)
(12B)
(27B)
(27B)
10% (27B)
(32B)
(35B)
(72B)
(72B)
Indonesian
56.31
62.86
66.50
21.60
66.26
67.88
66.26
66.50
63.83
Filipino
51.72
64.53
66.01
24.14
70.94
66.34
64.04
65.02
64.53
Malay
56.19
61.90
67.62
29.84
71.43
72.38
70.16
62.50
69.84
Chinese
62.80
68.93
72.66
40.73
75.14
77.48
78.28
80.98
79.29
Tamil
58.41
51.87
73.36
40.65
74.30
61.68
61.68
58.88
58.41
Table 8: Comparison of our best model (SEA-Gemma-3 10%) againsts other competitor models and the regionally-adapted Gemma-3 (SEA-Gemma-3) model on CVQA.
Gemma-3
SEA-Gemma-3
SEA-Gemma-3
Aya-Vision
Qwen-3 VL
Region
(27B)
(27B)
10% (27B)
(32B)
(35B)
Cambodia
50.7
40.5
52.3
56.25
50.99
Indonesia
43.5
49.3
69.0
66.09
66.09
Laos
0.0
50.0
68.1
45.83
63.89
Malaysia
37.5
44.4
62.4
69.31
60.32
Philippines
63.4
46.6
58.8
63.40
56.21
Table 9: Comparison of our best model (SEA-Gemma-3 10%) againsts other competitor models and the regionally-adapted Gemma-3 (SEA-Gemma-3) model on SEA-VQA.
Avg.
Country
Model
Score
Camb.
Indo.
Laos
Mala.
Phil.
Sing.
Thai.
Viet.
Google Gemma-3
41.0
50.7
43.5
0.0
37.5
63.4
56.3
51.6
18.9
SEA-Gemma-3 5%
61.2
50.3
68.8
66.7
62.4
59.5
65.6
51.1
57.8
SEA-Gemma-3 10%
61.7
52.3
69.0
68.1
62.4
58.8
68.8
52.2
57.5
SEA-Gemma-3 50%
59.5
52.0
67.3
62.5
57.1
56.9
53.1
52.7
55.0
SEA-Gemma-3 70%
54.0
51.3
67.3
63.9
56.6
58.2
53.1
51.6
20.5
Table 10: SEA-VLM Zero-shot performance on SEA-VQA. The result is broken down per country. “Camb.” = Cambodia, “Indo.” = Indonesia, “Mala.” = Malaysia, ‘Phil.’ = Phillipines, “Sing.”=Singapore, “Thai.” = Thailand, “Viet.” = Vietname
Overall
SEA
By Prompt Type
Model
Score
Avg.
Task 1 (NC)
Task 2
Task 1 (C)
Task 1 (Adv.)
Google Gemma-3
59.8
60.1
64.8
48.3
78.5
47.5
SEA-Gemma-3 5%
60.0
60.3
64.5
48.1
79.2
48.1
SEA-Gemma-3 10%
60.0
60.2
64.5
48.0
79.1
48.1
SEA-Gemma-3 50%
51.6
51.4
53.7
42.0
67.0
43.6
SEA-Gemma-3 70%
51.9
52.6
53.5
43.5
67.1
43.4
Table 11: SEA-VLM Zero-shot performance on World Cuisine. (NC) = No Context, (C) = Context, (Adv.) = Adversarial
Model
Overall SEA
Prompt Type
Score
Avg.
English
Local Lang.
Gemma-3
67.2
67.83
67.96
66.44
SEA-Gemma-3 5%
68.72
69.45
69.39
68.05
SEA-Gemma-3 10%
68.76
69.45
69.42
68.09
SEA-Gemma-3 50%
61.84
62.62
62.9
60.79
SEA-Gemma-3 70%
60.58
61.27
61.51
59.66
Table 12: SEA-VLM Zero-shot performance on CVQA.
Attribute
Entity
Other
Relation
Model
color
other
shape
size
texture
part
state
whole
count
text
non-spatial
spatial
StabilityAI SDXL
80.88
79.49
69.87
67.36
79.98
78.52
74.23
82.71
56.00
76.00
81.13
86.73
SEA-SDXL 25%
82.34
81.49
75.11
63.22
80.82
82.81
73.39
83.45
59.50
70.00
77.36
86.93
SEA-SDXL 50%
82.39
80.49
73.80
66.53
82.52
80.08
74.13
82.67
58.50
66.00
76.73
87.80
SEA-SDXL 75%
81.59
79.49
72.49
67.36
81.19
79.49
73.71
83.09
61.50
70.00
81.76
86.23
SEA-SDXL 100%
82.14
81.26
77.73
64.46
81.73
79.69
74.45
83.20
62.00
78.00
84.91
87.06
Table 13: Detailed per aspect breakdown (Attribute, Entity, Other, and Relation) of the DPGBench evaluation.
Model
SEAVQA Zero-Shot Performance
Overall
Cambodia
Indonesia
Laos
Malaysia
Phillipines
Singapore
Thailand
Vietnam
Google SigLIP2
25.81
28.9
25.0
22.2
24.3
28.8
21.9
25.5
25.6
SEA-SigLIP2 25%
26.36
27.6
25.4
29.2
29.1
28.8
28.1
26.1
24.0
SEA-SigLIP2 50%
28.06
31.3
27.0
22.2
29.6
27.5
21.9
31.0
27.2
SEA-SigLIP2 75%
29.66
28.6
30.1
19.4
32.3
34.0
25.0
29.9
28.8
SEA-SigLIP2 100%
28.96
28.3
28.6
29.2
26.5
32.7
31.3
30.4
29.1
Table 14: Detailed per country zero-shot performance of the SEAVQA evaluation.
Avg.
Country
Model
Score
Khm
Ind
Lao
Mys
Phl
Sgp
Tha
Vie
SEA-Mammoth 50k
13.0
9.9
18.8
11.1
11.1
11.8
18.8
14.67
2.88
SEA-Mammoth 250k
39.6
13.8
52.9
52.8
42.9
47.1
46.9
44.0
20.8
SEA-Mammoth 250k + CG OE
41.9
40.5
49.3
50.0
44.4
46.6
46.9
39.1
18.5
SEA-Mammoth 250k + CG MC
21.6
26.6
20.9
26.4
16.4
24.8
28.1
28.3
14.1
SEA-Mammoth 250k + WC
30.4
10.2
39.9
34.7
37.6
34.0
50.0
32.6
16.9
Table 15: SEA-VLM Zero-shot performance on SEA-VQA. The result is broken down per country. “Khm” = Cambodia, “Lao” = Laos, “Ind” = Indonesia, “Mys” = Malaysia, ‘Phl’ = Phillipines, “Sgp”=Singapore, “Th.” = Thailand, “Vie” = Vietnam. “CG” denotes CulturalGround, “WC” denotes WorldCuisine, while “OE” and “MC” denote open-ended and multiple-choice, respectively.
Figure 5: Generated responses using different model architypes. From left-to-right: global model (Gemma-3), our regional model (SEA-Gemma-3), our merged model (SEA-Gemma-3 10%) along with the prompts. Our model produces the most correct image among others, while retaining the image naturalness and overall quality of the original Gemma-3.
Figure 6: Generated image using different model architypes. From left-to-right: global model (SDXL), our regional model (SEA-SDXL), our merged model (SEA-SDXL 25%), and reference of natural images.
Year
EAP
ECA
LAC
MENA
NA
SA
SEA
SSA
World
1993
49.32
58.82
51.20
57.00
68.59
28.05
36.04
34.37
48.86
1994
51.07
59.15
51.00
56.85
68.58
28.25
36.40
34.26
49.15
1995
51.40
59.72
51.67
56.62
68.66
28.39
37.34
34.13
49.42
1996
52.02
60.23
51.67
56.74
69.43
28.37
37.89
33.99
49.64
1997
51.68
60.43
51.75
56.79
69.48
28.35
37.76
33.43
49.53
1998
51.63
60.71
51.86
56.89
69.83
28.33
37.70
33.55
49.66
Table 16: Globalization index by region by yea from 1993 to 2023. “EAP” = East Asia and Pacific , ‘ECA” = Europe and Central Asia,“LAC” = Latin America and Caribbean, “MENA” Middle East and North Africa, “NA” = North America, “SA” = South Asia, “SEA” = Southeast Asian, “SSA” = Sub-Saharan Africa.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
Existing geospatial vision-language models (Geo-VLMs) typically optimize diverse geospatial tasks through a unified multi-task adaptation paradigm without explicitly accounting for the heterogeneous optimization characteristics. Our empirical observations reveal heterogeneous gradient characteristics across tasks, including vision-language differences, intra-branch gradient relationships, and task interference, which hinder effective multi-task optimization. Motivated by these observations, we propose Gradient-Guided Decoupled Adaptation (G2DA), a gradient-aware optimization framework for multi-task Geo-VLM learning. G2DA first partitions tasks into vision- and language-centric groups through gradient-guided cross-modal decoupling. It then constructs modality-specific curricula based on task gradient similarity and employs bidirectional rehearsal to mitigate the recency effects introduced by sequential optimization. We evaluate G2DA on three Geo-VLM benchmarks using six InternVL3 and Qwen3.5-VL variants, along with GeoChat and GeoLLaVA. Across all 24 benchmark-model combinations, G2DA consistently outperforms representative baselines, improving over the strongest competitor by 3.08, 4.30, and 2.81 percentage points on UrBench-MCQ, XLRS-Bench-Lite, and VRS-Bench-VQA, respectively. These results demonstrate the effectiveness of gradient-guided task organization for Geo-VLM adaptation.
Dongdong Wang, Deepak Balakrishnan, Ravi Srinivasan +1
Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent reasoning and perceptual capabilities. CoA introduces a structured reasoning format that enhances domain alignment without sacrificing general multimodal competence by reinforcement learning. Experiments on standard surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-language abilities, providing a reliable pathway for domain specialization in VLMs.
Vision-Language Models (VLMs) achieve strong cross-modal performance, yet recent evidence suggests they over-rely on textual descriptions while under-utilizing visual evidence -- a phenomenon termed ``text shortcut learning.'' We propose an adversarial evaluation framework that quantifies this cross-modal dependency by measuring accuracy degradation (Drop) when semantically conflicting text is paired with unchanged images. Four adversarial strategies -- shape_swap, color_swap, position_swap, and random_text -- are applied to a controlled geometric-shapes dataset (n=1,000). We compare three configurations: Baseline CLIP (ViT-B/32), LoRA fine-tuning, and LoRA Optimized (integrating Hard Negative Mining, Label Smoothing, layer-wise learning rates, Cosine Restarts, curriculum learning, and data augmentation). The optimized model reduces average Drop from 27.5% to 9.8% (64.4% relative improvement, p<0.001) while maintaining 97% normal accuracy. Attention visualization and embedding-space analysis confirm that the optimized model attends more to visual features and achieves tighter cross-modal alignment.
Lijie Zhou
School of Computer Science University of Nottingham Ningbo China Ningbo, China