Multimodal Foundation Models Adaptation based on Domain-Aware Relaxed Orthogonal Subspace for Remote Sensing

Han Luo; Ruoyu Yang; Yinhe Liu; Yanfei Zhong

cs.CVSep 12, 2026

Multimodal Foundation Models Adaptation based on Domain-Aware Relaxed Orthogonal Subspace for Remote Sensing

Authors: Han Luo, Ruoyu Yang, Yinhe Liu, Yanfei Zhong

Organizations: State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430072, China

Abstract

Pretrained foundation models (FMs) have achieved remarkable success in computer vision, yet their high fine-tuning cost limits practical deployment. Parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) improve efficiency by constraining updates to a predefined low-rank subspace. However, when applied to remote sensing tasks with substantial domain shifts, the fixed subspace is constructed without observing the downstream activation distribution and can therefore provide a poor coordinate system for adaptation, a phenomenon herein termed subspace mismatch. To address this issue, a unified framework is introduced, termed Domain-aware Relaxed Orthogonal Subspace adaptation (DROS), which reformulates low-rank adaptation as data-conditioned subspace learning and flexible subspace adaptation. Specifically, the weight decomposition is conditioned on second-order activation statistics estimated from the downstream training distribution, so that the initialization reflects the feature geometry actually induced by the remote-sensing data, followed by flexible geometric transformations enabled by a relaxed orthogonal parameterization. Furthermore, the framework is extended to multimodal settings (MM-DROS) by sharing transformation structures across modality-specific subspaces, facilitating efficient cross-modal interaction. Extensive experiments on multiple remote sensing benchmarks demonstrate that DROS achieves state-of-the-art performance, even surpassing full fine-tuning, without additional inference overhead.

Figures & tables

Fig. 1: Conceptual illustration of subspace mismatch in parameter-efficient fine-tuning (PEFT) under domain shifts. (a) LoRA [ 17 ] constructs a low-rank update within a randomly initialized subspace without using downstream activation statistics. (b) PiSSA [ 18 ] derives the adaptation subspace from pretrained weights, improving optimization efficiency but remaining independent of the downstream activation distribution. (c) DROS conditions the subspace initialization on second-order activation statistics from representative training data and further enables structured transformations within this data-conditioned subspace.

Fig. 2: Overview of DROS. Panel a presents the overall framework, where a trainable DROS side branch is attached to each of the $n$ adapted linear layers in a frozen foundation model. Panel b details the construction for layer $k$ , where $k\in\{1,\ldots,n\}$ . The frozen backbone collects $H_{k,i}$ from each of $N$ representative training samples, and averaging the products $H_{k,i}H_{k,i}^{\top}$ gives $C_{k}$ . The index $k$ is omitted after a single layer is fixed. Covariance-conditioned SVD of $W^{\mathrm{rw}}=W^{\mathrm{pre}}C$ selects the bottom- $r$ components to initialize $B_{0}$ and $A_{0}$ . The pretrained weight is split as $W^{\mathrm{pre}}=W^{\mathrm{res}}+B_{0}A_{0}$ , and the trainable side block $W^{\mathrm{side}}=B\mathcal{T}A$ yields $W^{\mathrm{tuned}}=W^{\mathrm{res}}+W^{\mathrm{side}}$ .

Fig. 3: Overview of MM-DROS at one adapted linear layer. The layer index $k$ is suppressed. The optical and SAR branches retain modality-specific residual weights and bases $B^{\mathrm{opt}},A^{\mathrm{opt}}$ and $B^{\mathrm{sar}},A^{\mathrm{sar}}$ . The relaxed operator $\mathcal{T}_{\mathrm{shared}}=\operatorname{diag}(\alpha)O\operatorname{diag}(\beta)$ is shared across the two branches. Each transformed low-rank branch is added to its corresponding residual weight.

Task Type	Dataset	Source (Region, Reported Dataset Year)	Resolution	Classes	Train Size	Test Size	Total Images
Classification	EuroSAT	Sentinel-2, Europe (2017)	10m	10	16,200	10,800	27,000
	RESISC45	Google Earth, Global (2015)	0.3m	45	18,900	6,300	25,200
	UCM	Aerial, California, US (2010)	0.3m	21	1,260	420	1,680
Semantic Segmentation	Cashew	Sentinel-2, Benin (GEO-Bench)	10m	7	1,350	400	1,750
	Chesap.	RGBN, Chesapeake Bay, US (GEO-Bench)	1.0m	7	3,000	1,000	4,000
	Cattle	RGB, New Zealand (GEO-Bench)	0.1m	2	524	66	590

TABLE I: Detailed statistics of the experimental datasets. For classification and GEO-Bench tasks, the official data splits provided by TorchGeo and GEO-Bench are adopted. For DGSS tasks, the number of cropped

512\times 512

patches used for training and evaluation is reported. The data source/sensor, covered region, and reported dataset year of each dataset are also given.

Method	#Params	MARS				#Params	DINOv2-L
Method	#Params	EuroSAT	UCM	RESISC45	Avg	#Params	EuroSAT	UCM	RESISC45	Avg
FT	89.00M	96.47 $\pm$ 0.31	98.57 $\pm$ 0.00	96.85 $\pm$ 0.17	97.30	304.20M	96.63 $\pm$ 0.12	98.41 $\pm$ 0.69	95.25 $\pm$ 0.18	96.76
FZ	0.00M	86.83 $\pm$ 0.17	93.17 $\pm$ 0.11	87.84 $\pm$ 0.04	89.28	0.00M	86.13 $\pm$ 0.49	90.00 $\pm$ 1.04	84.23 $\pm$ 0.12	86.79
LoRA	0.28M	95.77 $\pm$ 0.29	98.73 $\pm$ 0.36	96.49 $\pm$ 0.10	97.00	0.59M	96.83 $\pm$ 0.31	98.10 $\pm$ 0.41	95.33 $\pm$ 0.20	96.75
Earth-Adapter	1.55M	96.50 $\pm$ 0.10	98.69 $\pm$ 0.11	96.75 $\pm$ 0.05	97.31	9.59M	96.37 $\pm$ 0.12	98.41 $\pm$ 1.10	95.62 $\pm$ 0.19	96.80
VPT	0.24M	86.97 $\pm$ 0.06	92.54 $\pm$ 0.14	87.80 $\pm$ 0.06	89.10	0.49M	96.43 $\pm$ 0.25	97.86 $\pm$ 0.24	95.12 $\pm$ 0.06	96.47

TABLE II: Comparative results on classification tasks using MARS and DINOv2-L backbones. Red and blue indicate the best and second-best results, respectively.

Fig. 4: Qualitative comparison of Top-5 classification predictions on RESISC45 using DINOv2-L. Red bars identify the ground-truth category, and Ours denotes DROS.

Method	#Params	MARS				#Params	DINOv2-L
Method	#Params	Cashew	Chesap.	Cattle	Avg	#Params	Cashew	Chesap.	Cattle	Avg
FT	89.00M	78.30 $\pm$ 0.20	75.53 $\pm$ 0.75	83.69 $\pm$ 0.19	79.17	304.20M	62.65 $\pm$ 1.69	74.59 $\pm$ 0.18	83.18 $\pm$ 0.54	73.47
FZ	0.00M	69.05 $\pm$ 1.06	74.15 $\pm$ 0.37	84.02 $\pm$ 0.48	75.74	0.00M	55.68 $\pm$ 4.26	73.35 $\pm$ 0.16	82.55 $\pm$ 0.14	70.53
LoRA	3.03M	74.82 $\pm$ 1.13	75.25 $\pm$ 0.17	83.82 $\pm$ 0.11	77.96	6.21M	63.53 $\pm$ 1.62	75.13 $\pm$ 0.18	83.10 $\pm$ 0.31	73.92
Earth-Adapter	1.55M	70.29 $\pm$ 0.80	74.56 $\pm$ 0.41	83.57 $\pm$ 0.10	76.14	9.59M	58.98 $\pm$ 1.80	74.04 $\pm$ 0.16	82.89 $\pm$ 0.31	71.97
VPT	3.08M	69.83 $\pm$ 0.30	74.59 $\pm$ 0.24	84.03 $\pm$ 0.56	76.15	6.29M	50.51 $\pm$ 1.46	74.22 $\pm$ 0.26	82.94 $\pm$ 0.55	69.22

TABLE III: Comparative results on segmentation tasks using MARS and DINOv2-L backbones. For clarity, dataset names are abbreviated: Cashew : m-cashew-plant; Chesap. : m-chesapeake; Cattle : m-nz-cattle. Red and blue indicate the best and second-best performance, respectively.

Fig. 5: Qualitative comparison on Chesapeake semantic segmentation using DINOv2-L.

Method	#Params	MARS					#Params	DINOv2-L
Method	#Params	U2R	R2U	P2V	V2P	Avg	#Params	U2R	R2U	P2V	V2P	Avg
FT	89.00M	38.82 $\pm$ 0.76	52.90 $\pm$ 0.54	41.22 $\pm$ 0.56	40.78 $\pm$ 0.94	43.43	304.20M	45.55 $\pm$ 0.82	58.38 $\pm$ 0.39	60.72 $\pm$ 1.31	55.57 $\pm$ 0.88	55.05
FZ	0.00M	36.69 $\pm$ 0.62	50.87 $\pm$ 1.16	37.32 $\pm$ 0.79	32.79 $\pm$ 1.15	39.42	0.00M	45.23 $\pm$ 1.40	57.37 $\pm$ 0.04	55.74 $\pm$ 1.14	49.93 $\pm$ 1.28	52.07
LoRA	3.03M	39.67 $\pm$ 0.60	53.84 $\pm$ 0.08	41.32 $\pm$ 1.63	39.39 $\pm$ 0.70	43.56	6.21M	45.27 $\pm$ 1.11	58.02 $\pm$ 0.30	60.76 $\pm$ 0.65	54.14 $\pm$ 0.86	54.55
Earth-Adapter	1.55M	40.33 $\pm$ 1.25	52.34 $\pm$ 0.70	39.31 $\pm$ 0.93	40.59 $\pm$ 0.53	43.14	9.59M	45.99 $\pm$ 0.94	57.72 $\pm$ 0.35	59.49 $\pm$ 1.00	55.66 $\pm$ 1.51	54.72
VPT	0.77M	36.78 $\pm$ 1.69	51.47 $\pm$ 1.17	37.51 $\pm$ 0.28	35.51 $\pm$ 0.19	40.32	1.57M	43.51 $\pm$ 1.99	57.46 $\pm$ 0.10	58.27 $\pm$ 0.49	55.75 $\pm$ 0.93	53.75

TABLE IV: Domain generalization results (mIoU %) on MARS and DINOv2-L backbones. Tasks include cross-city ( P2V , V2P ) and cross-scene ( U2R , R2U ) transfers. Red and blue indicate the best and second-best results, respectively.

Fig. 6: Qualitative comparison on the LoveDA U2R setting using DINOv2-L.

Method	#Params	MARS			#Params	DINOv2-L
Method	#Params	BRIGHT	WHU	Avg	#Params	BRIGHT	WHU	Avg
FT	178.00M	63.78 $\pm$ 0.36	58.59 $\pm$ 0.59	61.19	608.40M	65.28 $\pm$ 0.62	60.36 $\pm$ 0.05	62.82
FZ	0.00M	55.22 $\pm$ 1.35	59.22 $\pm$ 0.08	57.22	0.00M	62.08 $\pm$ 1.69	59.80 $\pm$ 0.07	60.94
LoRA	6.06M	63.29 $\pm$ 0.61	59.89 $\pm$ 0.27	61.59	12.42M	64.88 $\pm$ 1.48	60.32 $\pm$ 0.33	62.60
Earth-Adapter	3.11M	62.62 $\pm$ 1.61	59.79 $\pm$ 0.06	61.20	19.18M	65.08 $\pm$ 0.87	60.32 $\pm$ 0.15	62.70
VPT	6.16M	54.65 $\pm$ 1.06	59.14 $\pm$ 0.13	56.89	12.58M	64.11 $\pm$ 1.20	60.25 $\pm$ 0.19	62.18

TABLE V: Multimodal segmentation results (mIoU %) on MARS and DINOv2-L backbones. For clarity, dataset names are abbreviated: WHU : WHU-OPT-SAR. Red and blue indicate the best and second-best results, respectively.

Fig. 7: Qualitative comparison on BRIGHT multimodal segmentation using DINOv2-L.

Component	RESISC45	Chesap.	U2R	Avg
Dominant (Top- $r$ )	25.21 $\pm$ 0.32	70.32 $\pm$ 0.59	40.97 $\pm$ 1.19	45.50
Minor (Bot- $r$ )	95.37 $\pm$ 0.10	75.58 $\pm$ 0.13	45.77 $\pm$ 0.61	72.24

TABLE VI: Mechanism verification of spectral component selection on DINOv2-L. Adapting the top-

r

principal components collapses classification accuracy and degrades dense tasks, whereas adapting the bottom-

r

components remains stable.

Subset Ratio	RESISC45	Chesap.	U2R	Avg
1%	94.80 $\pm$ 0.77	74.89 $\pm$ 0.45	45.48 $\pm$ 0.80	71.72
10%	95.37 $\pm$ 0.10	75.58 $\pm$ 0.13	45.77 $\pm$ 0.61	72.24
50%	94.74 $\pm$ 0.27	74.91 $\pm$ 0.20	45.59 $\pm$ 0.53	71.75
100%	94.63 $\pm$ 0.59	74.96 $\pm$ 0.08	45.08 $\pm$ 0.89	71.56

TABLE VII: Ablation of training subset ratio for domain-aware covariance estimation on DINOv2-L.

Fig. 8: Comparison of different subspace tuning principles: LoRA performs fine-tuning by randomly assigning subspaces, PiSSA restricts subspaces to the principal components of weights. DROS achieves optimal matching.

Method	RESISC45	Chesap.	U2R	Avg
FT	88.32 $\pm$ 0.71	69.72 $\pm$ 0.26	42.62 $\pm$ 0.59	66.89
FZ	70.00 $\pm$ 0.06	68.37 $\pm$ 0.32	42.23 $\pm$ 0.57	60.20
LoRA	87.19 $\pm$ 0.65	69.69 $\pm$ 0.79	42.60 $\pm$ 0.20	66.49
Earth-Adapter	88.68 $\pm$ 0.27	69.44 $\pm$ 0.63	42.20 $\pm$ 0.69	66.77
VPT	82.79 $\pm$ 2.08	68.45 $\pm$ 0.73	41.88 $\pm$ 0.14	64.37
PiSSA	86.21 $\pm$ 1.62	68.91 $\pm$ 1.13	42.49 $\pm$ 0.85	65.87

TABLE VIII: Comparative results under 10% training data on DINOv2-L.

Operator	RESISC45	Chesap.	U2R	Avg
None (w/o operator)	95.18 $\pm$ 0.10	74.80 $\pm$ 0.19	44.52 $\pm$ 0.23	71.50
$O$	95.28 $\pm$ 0.10	75.07 $\pm$ 0.10	44.78 $\pm$ 0.61	71.71
$O$ + $\alpha$	95.30 $\pm$ 0.05	74.91 $\pm$ 0.19	44.12 $\pm$ 0.98	71.44
$O$ + $\beta$	95.33 $\pm$ 0.23	75.17 $\pm$ 0.11	45.61 $\pm$ 0.35	72.04
$O$ + $\alpha$ + $\beta$	95.37 $\pm$ 0.10	75.58 $\pm$ 0.13	45.77 $\pm$ 0.61	72.24

TABLE IX: Ablation study of the relaxed orthogonal operator components on DINOv2-L.

Shared Operator	BRIGHT
None (non-shared)	65.41 $\pm$ 0.84
$O$	65.46 $\pm$ 0.87
$O$ + $\alpha$	65.47 $\pm$ 0.47
$O$ + $\beta$	65.50 $\pm$ 0.71
$O$ + $\alpha$ + $\beta$	65.60 $\pm$ 0.84

TABLE X: Ablation study of the shared subspace in MM-DROS on BRIGHT.

Fig. 9: Qualitative case study of MM-DROS on the unseen Los Angeles wildfire scenario.

Fig. 10: Comparison of loss curves and accuracy curves for different PEFT methods.

Method	RESISC45		Chesap.		U2R
	#Params	Mem. (GB)	#Params	Mem. (GB)	#Params	Mem. (GB)
FT	304.20M	5.21	304.20M	6.41	304.20M	13.71
FZ	0.00M	1.24	0.00M	2.42	0.00M	4.65
LoRA	0.59M	2.75	6.21M	4.02	6.21M	10.77
Earth-Adapter	9.59M	3.18	9.59M	4.35	9.59M	11.24
VPT	0.49M	2.68	6.29M	3.95	1.57M	10.58

TABLE XI: Efficiency comparison on DINOv2-L across different methods. The number of learnable parameters (#Params) and the peak GPU memory in GB are reported for each task.

Vision Foundation Model Adaptation

Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.

See all 319 papers

GeoRoPE: Ground-Aware Rotary Adaptation for Remote Sensing Foundation Models
Jun 8, 2026Yu Luo, Kun Hu, Mengwei He +7Rotary Positional Embeddings Geospatial Foundation Models
The University of Sydney · Edith Cowan University · Adelaide University +3
Scalable and Trustworthy Earth Observation Foundation Models
Jul 8, 2026Syed Usama Imtiaz, Mitra Nasr Azadani, Nasrin AlamdariGeospatial Foundation Models Vision Foundation Model Adaptation
Department of Civil and Environmental Engineering, Florida State University
A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting
Apr 7, 2026Yongchuan Cui, Peng LiuAll-in-One Image Restoration Image Restoration
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China · School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing, China

Jun 8, 2026J/K move · Enter open · S save

GeoRoPE: Ground-Aware Rotary Adaptation for Remote Sensing Foundation Models

Yu Luo, Kun Hu, Mengwei He, Xiaogang Zhu+6

The University of Sydney · Edith Cowan University · Adelaide University+3

Rotary Positional Embeddings Geospatial Foundation Models Vision Foundation Model Adaptation Positional Encoding

Open paper AskPDF arXiv

Remote-sensing foundation models (RSFMs) benefit from pretraining on imagery from multiple sensors and ground sampling distances (GSDs), but such exposure alone does not resolve scale mismatch during downstream adaptation. A fixed token-grid offset can correspond to different ground distances across sensors, making grid-based positional priors physically inconsistent. Meanwhile, heterogeneous spatial granularity means that compact urban regions and homogeneous landscapes may require different positional sensitivities even under the same GSD. Therefore, we propose {GeoRoPE}, a ground-aware, RoPE-compatible, and parameter-efficient spatial adaptation method for RSFMs. GeoRoPE recalibrates token-level positional interactions from two complementary aspects. First, Geo-Coordinate Calibration (GCC) rescales raw token-grid offsets according to the ground distance represented by one token-grid step, producing geo-calibrated relative coordinates across GSDs. Second, Geo-Frequency Calibration (GFC) adjusts the native RoPE frequency with a relation-specific factor, enabling position sensitive adaptation to scene-dependent spatial granularity. GeoRoPE is injected into pretrained RSFMs through a lightweight adapter, preserving the frozen spatial prior while adding geo-aware positional corrections. Experiments across multiple RSFMs, sensors, resolutions, and downstream tasks demonstrate that GeoRoPE improves cross-resolution robustness and scale-sensitive representation learning.

Abstract

Vision Foundation Model Adaptation

Explore similar work

GeoRoPE: Ground-Aware Rotary Adaptation for Remote Sensing Foundation Models

Scalable and Trustworthy Earth Observation Foundation Models

A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting