MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search

Minkyoung Cho; Insu Jang; Shuowei Jin; Zesen Zhao; Adityan Jothi; Ethem F. Can; Min-Hung Chen; Z. Morley Mao

cs.LGFeb 28, 2026

MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search

Authors: Minkyoung Cho, Insu Jang, Shuowei Jin, Zesen Zhao, Adityan Jothi, Ethem F. Can, Min-Hung Chen, Z. Morley Mao

Organizations: University of Michigan · NVIDIA

Abstract

Fine-tuning Multimodal Large Language Models (MLLMs) with parameter-efficient methods like Low-Rank Adaptation (LoRA) is crucial for task adaptation. However, imbalanced training dynamics across modalities often lead to suboptimal accuracy due to negative interference, a challenge typically addressed with inefficient heuristic methods such as manually tuning separate learning rates. To overcome this, we introduce MARS (Multimodal Adaptive Rank Search), an approach to discover optimal rank pairs that balance training dynamics while maximizing performance. Our key innovation, a proposed framework of dual scaling laws, enables this search: one law models module-specific convergence time to prune the search space to candidates with aligned dynamics, while the other predicts final task performance to select the optimal pair from the pruned set. By re-purposing the LoRA rank as a controller for modality-specific convergence speed, MARS outperforms baseline methods and provides a robust, automated strategy for optimizing MLLM fine-tuning.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation: Imbalanced training dynamics lead to suboptimal performance. Conceptual cases of module imbalance (left) and their corresponding empirical results (right).

Panel of Figure 2 — Figure 2: The proposed dual scaling laws from LLaVA-OV-0.5B* (VE and LLM have a similar parameter count without task-specific knowledge). This pattern holds consistently across different fixed-rank settings (e.g., $r_{\mathrm{ve}}=8$ and $r_{\mathrm{ve}}=32$ ).

Table 3

Table 4

Source of the coefficients	$R^{2}$	MAPE
From-scratch LLaVA-OV	$-5.309$	57.23%
Per-model calibration	$0.994$	0.96%

Table 6: Prediction reliability of Scaling Law-P on pre-trained LLaVA-OV.

R^{2}

: fit quality; MAPE: mean abs. percentage error.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Optimal VE rank size in the small-dataset regime. Lower values (lighter colors) indicate better performance (i.e., lower perplexity).

Figure 8: Validating Scaling Law-P on Pre-trained Models

Panel of Figure 9 — Figure 9: Scaling Law-C Validation: LLaVA-OV-0.5B Models for the ScienceQA Dataset

Figure 10: Scaling Law-C Validation: Qwen2.5-VL Models for the ScienceQA Dataset

Table 12

Benchmark	Focus	Metric in Figure 3
MME [ 23 ]	General perception and cognition	Cognition + Perception
POPE [ 25 ]	Object hallucination	F1 Score
MMStar [ 24 ]	Visual perception and reasoning	Avg. Accuracy
GQA [ 28 ]	Compositional visual reasoning	Accuracy
TextCaps [ 26 ]	OCR-based image captioning	ROUGE_L
AI2D [ 27 ]	Scientific diagram understanding	Accuracy

Appendix

Table 9: Summary of Benchmarks and Evaluation Metrics

Method	MME (sum)	POPE	MMStar (avg)	TextCaps (Val)	AI2D	GQA
AdaLoRA [ 29 ]	1524.0	79.7	28.6	38.0	45.5	52.7
LoRA ( $\bigstar$ , 64)	1662.4	86.8	36.1	42.6	48.0	55.1
MARS	1695.6	86.3	39.4	42.9	49.9	55.7

Appendix

Table 10: Performance comparison on diverse multimodal benchmarks demonstrating the broad generalization of generalist models when MARS is applied to the connected VE and LLM.

Method	Coarse Per.	Fine-Grained Per.	Instance Reason.	Logical Reason.	Sci & Tech	Math
AdaLoRA	34.7	23.2	27.8	29.9	24.4	31.5
LoRA ( $\bigstar$ , 64)	55.5	27.8	49.8	32.1	22.2	29.0
MARS	58.3	28.4	48.4	34.9	33.0	33.4

Appendix

Table 11: Fine-grained capability breakdown on MMStar [ 24 ]

Figure 11: Simultaneous Multi-Rank Tuning Mechanism

Model	Benchmark	Item	$(r_{\text{ve}},r_{\text{llm}})$	Performance
LLaVA-OV-7B	LLaVABench	Baseline	(16, 16)	2.2323 $\pm$ 0.0038
LLaVA-OV-7B	LLaVABench	MARS	(10, 16)	2.1875 $\pm$ 0.0025
Qwen2.5-VL-3B	LLaVABench	Baseline	(16, 8)	3.5914 $\pm$ 0.0021
Qwen2.5-VL-3B	LLaVABench	MARS	(10, 8)	3.5825 $\pm$ 0.0017
LLaVA-OV-7B	ScienceQA	Baseline	(16, 32)	73.64 $\pm$ 0.29%
LLaVA-OV-7B	ScienceQA	MARS	(22, 32)	74.30 $\pm$ 0.25%

Appendix

Table 12: Stability of MARS across three random seeds. Performance is reported as mean

\pm

std.

Coefficient	Within-Family CV	Cross-Family CV	Transfers?
$\beta$ (dataset-size)	2–3%	3.7%	Yes
$E$ (irreducible loss)	$\sim$ 1%	1.1%	Yes
$\alpha_{v}$ (VE rank)	6–7%	31.1%	No (need calibration)
$\alpha_{l}$ (LLM rank)	4–6%	35.0%	No (need calibration)

Appendix

Table 13: Coefficient of Variation (CV) analysis for Scaling Law-P coefficients.

Grouping	Mean $\delta$	CV
Within LLaVA-OV (pre-trained)	0.697	3.0%
Within Qwen2.5-VL (pre-trained)	0.570	2.5%
Cross-family (all pre-trained)	0.646	11.0%
From-scratch vs. Pre-trained	0.633	14.1%

Appendix

Table 14: Coefficient of Variation (CV) analysis for Scaling Law-C convergence exponent

\delta

Benchmark	Fine-tuning Data	Pearson $r$	$p$ -value
ScienceQA	ScienceQA	$-0.97$	$<0.001$
MMStar	LLaVA-665K	$-0.89$	$<0.001$
AI2D	LLaVA-665K	$-0.79$	$<0.01$
MME	LLaVA-665K	$-0.78$	$<0.001$
GQA	LLaVA-665K	$-0.74$	$<0.01$
POPE	LLaVA-665K	$-0.72$	$<0.01$

Appendix

Table 15: Correlation between validation perplexity and downstream task performance across diverse benchmarks.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,085 papers

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning
May 8, 2026Ionut-Vlad Modoranu, Mher Safaryan, Dan AlistarhAdapter Tuning LLM Fine-Tuning
Institute of Science and Technology Austria (ISTA). · Lancaster University, UK · ISTA
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
May 29, 2026Valérie Castin, Kimia Nadjahi, Pierre Ablin +1LLM Fine-Tuning Low-Rank Adaptation
1 ´Ecole Normale Sup´erieure PSL, Paris, France · 2CNRS · 3Apple, Paris, France.
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization
Sep 24, 2026Zebang Xie, Chuanyang Zheng, Yik-Chung Wu +1Low-Rank Adaptation Adaptive Rank Allocation
National University of Singapore, Singapore · Huawei, Hong Kong · The University of Hong Kong, Hong Kong +1

May 8, 2026J/K move · Enter open · S save

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning

Ionut-Vlad Modoranu, Mher Safaryan, Dan Alistarh

Institute of Science and Technology Austria (ISTA). · Lancaster University, UK · ISTA

Adapter Tuning LLM Fine-Tuning Hierarchical Representation Learning Nonlinear LoRA Low-Rank Adaptation Parameter-Efficient Fine-Tuning+1

Open paper AskPDF arXiv

With the rise in scale for deep learning models to billions of parameters, the computational cost of fine-tuning remains a significant barrier to deployment. While Low-Rank Adaptation (LoRA) has become the standard for parameter-efficient fine-tuning, the need to set a predefined, static rank

r

requires exhaustive grid searches to balance efficiency and performance. Existing rank-adaptive solutions such as DyLoRA mitigate this by sampling ranks during the training from a predefined distribution. However, they often yield sub-optimal results at higher ranks due to lack of consistent gradient signals across the full hierarchy of ranks, thus making these methods data-inefficient. In this paper, we propose MatryoshkaLoRA, a general, Matryoshka-inspired training framework for LoRA that learns accurate hierarchical low-rank representations by inserting a fixed, carefully crafted diagonal matrix

P

between the existing LoRA adapters to scale their sub-ranks accordingly. By introducing this simple modification, our general framework recovers LoRA and DyLoRA only by changing

P

and ensures all sub-ranks embed the available gradient information efficiently. Our MatryoshkaLoRA supports dynamic rank selection with minimal degradation in accuracy. We further propose Area Under the Rank Accuracy Curve (AURAC), a metric that consistently evaluates the performance of hierarchical low-rank adapters. Our results demonstrate that MatryoshkaLoRA learns more accurate hierarchical low-rank representations than prior rank-adaptive approaches and achieves superior accuracy-performance trade-offs across ranks on the evaluated datasets. Our code is available at https://github.com/IST-DASLab/MatryoshkaLoRA.

Abstract

Multimodal Large Language Models

Explore similar work

MatryoshkaLoRA: Learning Accurate Hierarchical Low-Rank Representations for LLM Fine-Tuning

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization