Enhancing Lightweight Vision Language Models through Group Competitive Learning for Socially Compliant Navigation

Xinyu Zhang; Atsushi Konno; Toshihiko Yamasaki; Ling Xiao

cs.ROMar 12, 2026

Enhancing Lightweight Vision Language Models through Group Competitive Learning for Socially Compliant Navigation

Authors: Xinyu Zhang, Atsushi Konno, Toshihiko Yamasaki, Ling Xiao

Organizations: Hokkaido University, Sapporo, Japan · The University of Tokyo, Tokyo, Japan

Abstract

Social robot navigation requires a sophisticated integration of scene semantics and human social norms. Scaling up Vision Language Models (VLMs) generally improves reasoning and decision-making capabilities for socially compliant navigation. However, increased model size incurs substantial computational overhead, limiting suitability for real-time robotic deployment. Conversely, lightweight VLMs enable efficient inference but often exhibit weaker reasoning and decision-making performance in socially complex environments. Achieving both strong reasoning ability and efficiency remains an open challenge. To bridge this gap, we propose Group Competitive Learning (GCL), a strategy designed to amplify the capabilities of lightweight VLMs. Our strategy introduces the Group Competitive Objective (GCO) to harmonize global semantics with distributional regularization, alongside Asymmetric Group Optimization (AGO) to explore the upper limits of model performance. Empirical evaluations on social navigation benchmarks demonstrate that GCL significantly elevates VLM performance. Specifically, GCL enables the Qwen2.5-VL-3B learner model and guide Qwen3-VL-4B to achieve an F1 score of 0.968 and 0.914, representing 40% and 12% improvement over vanilla supervised fine-tuning (SFT). Notably, under vanilla SFT, the 3B model initially trails the 8B model (F1: 0.692 vs. 0.755). However, through the GCL, the 3B model outperforms (28%) the 8B baseline model. These results suggest that GCL provides an effective solution for achieving both high accuracy and computational efficiency in real-world deployment.

Figures & tables

Figure 1: The Overview of GCL. The left of the figure illustrates the GCL strategy’s inputs examples: texts and images across social navigation dataset. The middle shows the GCL strategy’s components: Group Competitive Objective, to align VLMs at both semantic and distributional token levels, and Asymmetric Group Optimization, to explore the optimization dynamics and performance boundaries of GCL. The right shows the socially compliant instruction of GCL outputs.

Regime A: Reshaping Mode Capacity L < Capacity G , Small $\rightarrow$ Large, $\bm{\eta_{learner}=1\times 10^{-5}}$ $\bm{\eta_{guide}=5\times 10^{-6}}$
Model Group (Learner / Guide)	$\mathbf{\tau_{L}}$ / $\mathbf{\tau_{G}}$	Baseline ( $F1_{L}$ / $F1_{G}$ )	GCL ( $F1_{L}$ / $F1_{G}$ )
Qwen3-VL-2B / Qwen3-VL-4B	3.0 / 2.0	0.797 / 0.816	0.954 (+0.157) / 0.910 (+0.094)
Qwen2.5-VL-3B / Qwen3-VL-4B	3.0 / 2.0	0.692 / 0.816	0.968 (+0.276) / 0.914 (+0.098)
Qwen2.5-VL-3B / Qwen2.5-VL-7B	3.0 / 2.0	0.692 / 0.777	0.869 (+0.177) / 0.915 (+0.138)
Qwen2.5-VL-3B / Qwen2.5-VL-8B	3.0 / 2.0	0.692 / 0.755	0.964 (+0.272) / 0.811 (+0.056)
Qwen3-VL-2B / Qwen3-VL-2B	3.0 / 2.0	0.797 / 0.797	0.841 (+0.044) / 0.884 (+0.087)

Table 1: Performance of GCL across 12 competitive group.

\eta_{guide}

and

\eta_{learner}

are allocated to SFT baseline superiority. The performance hierarchy from lowest to highest is: Qwen2.5-VL-3B (0.692)

<

Qwen3-VL-8B (0.755)

<

Qwen2.5-VL-7B (0.777)

<

Qwen3-VL-2B (0.797)

<

Qwen3-VL-4B (0.816). Meanwhile, temperature parameters (

\tau_{L}

\tau_{G}

) are assigned solely based on parameter size (larger capacity = 2.0, smaller capacity = 3.0).

F1

denotes Action-F1. Bold text indicates optimal performance of the model.( For MUSON results, details provided in the Supplementary Material )

	Method		Model	SNEI			MUSON
	Method		Model	F1 $\uparrow$	Per-cos $\uparrow$	Rea-cos $\uparrow$	F1 $\uparrow$	Per-cos $\uparrow$	Rea-cos $\uparrow$
SFT (baseline)	$\mathcal{L}_{sup}$		Qwen2.5-VL-3B	0.692	0.834	0.767	0.811	0.810	0.803
SFT (baseline)	$\mathcal{L}_{sup}$		Qwen3-VL-4B	0.816	0.773	0.775	0.863	0.855	0.849
GCL	$\mathcal{L}_{GCO}$	$\mathcal{L}_{sup}+\mathcal{L}_{GSL}$	Qwen2.5-VL-3B	0.818	0.880	0.846	0.877	0.872	0.864
		$\mathcal{L}_{sup}+\mathcal{L}_{GSL}$	Qwen3-VL-4B	0.821	0.806	0.838	0.905	0.879	0.874
		$\mathcal{L}_{sup}+\mathcal{L}_{DRL}$	Qwen2.5-VL-3B	0.832	0.903	0.851	0.851	0.840	0.837

Table 2: Performance comparison of GCL components and vanilla SFT. We report Action-F1 (F1), Perception-cos (Per-cos) and Reasoning-cos (Rea-cos) on two datasets SNEI and MUSON. These results demonstrate that GCL, enables compact models to internalize intricate structural knowledge, ultimately surpassing the generalization capabilities of larger models trained in isolation.

Panel of Figure 2 — Figure 2: Ablation Study of Loss Components in GCO. Red and blue bars represent the learner (Qwen2.5-VL-3B) and guide (Qwen3-VL-4B) models, respectively, with bold values indicating peak performance. The left illustrates the ablation of the GSL weight, where both models achieve optimal results at 0.5. The right plot presents the ablation of the DRL weight, demonstrating that a value of 0.4 yields the highest performance for both models.

Panel of Figure 3 — Figure 3: Impact of Capability Gaps on Asymmetric Learning Rate Dynamics. Action-F1 is evaluated against the learning rate ratio ( $r=\eta_{learner}/\eta_{guide}$ ), with $\eta_{guide}$ fixed at $\eta=5\times 10^{-6}$ . Shaded regions indicate the Optimization Best Spot. Fig. 3(a) shows that large capability gap group tolerate aggressive learner updates ( $1.0\leq r\leq 3.0$ ), enabling effective feature space alignment. In contrast, Fig. 3(b) reveals that minimal initial gaps confine the optimal window to ( $1.0\leq r\leq 2.0$ ). These confirms that GCL necessitates performance based asymmetric learning rate scheduling.

Figure 4: Optimal Asymmetric Temperature Gradients in VLMs Group. Heatmaps of Action-F1 scores for Qwen2.5-VL-3B learner (left) and Qwen3-VL-4B guide (right) across temperature grids. Symmetric settings $\tau_{learner}=\tau_{guide}$ ) result in suboptimal consensus or collapse (e.g., $0.720$ at $\tau=3.0$ ). The global peak (learner: 0.968) occurs at $(\tau_{learner}=3.0,\tau_{guide}=2.0)$ , highlighting the necessity of asymmetric thermal gradients to enhance learner plasticity through guide discriminative signaling. ( For small gap group, details provided in the Supplementary Material )

Panel of Figure 5 — Figure 5: Visualization Comparison. SFT models (yellow/green) miss critical constraints and produce incorrect actions, while GCL models (red/blue) generate socially compliant instructions aligned with GT.

Regime A: Reshaping Mode Capacity L < Capacity G , Small $\rightarrow$ Large, $\bm{\eta_{learner}=1\times 10^{-5}}$ $\bm{\eta_{guide}=5\times 10^{-6}}$
Model Group (Learner / Guide)	$\mathbf{\tau_{L}}$ / $\mathbf{\tau_{G}}$	Baseline ( $F1_{L}$ / $F1_{G}$ )	GCL ( $F1_{L}$ / $F1_{G}$ )
Qwen3-VL-2B / Qwen2.5-VL-3B	3.0 / 2.0	0.805 / 0.811	0.989 (+0.184) / 0.974 (+0.163)
Qwen3-VL-2B / Qwen3-VL-4B	3.0 / 2.0	0.805 / 0.863	0.963 (+0.158) / 0.969 (+0.106)
Qwen2.5-VL-3B / Qwen3-VL-4B	3.0 / 2.0	0.811 / 0.863	0.975 (+0.164) / 0.961 (+0.098)
Qwen3-VL-2B / Qwen2.5-VL-7B	3.0 / 2.0	0.805 / 0.829	0.907 (+0.102) / 0.943 (+0.114)
Qwen2.5-VL-3B / Qwen2.5-VL-7B	3.0 / 2.0	0.811 / 0.829	0.966 (+0.155) / 0.930 (+0.101)

Table 3: Performance of GCL across 12 competitive group on MUSON.

\eta_{guide}

and

\eta_{learner}

are allocated to SFT baseline superiority. The performance hierarchy from lowest to highest is: Qwen3-VL-2B (0.805)

<

Qwen2.5-VL-3B (0.811)

<

Qwen2.5-VL-7B (0.829)

<

Qwen3-VL-4B (0.863)

<

Qwen3-VL-8B (0.873). Meanwhile, temperature parameters (

\tau_{L}

\tau_{G}

) are assigned solely based on parameter size (larger capacity = 2.0, smaller capacity = 3.0).

F1

denotes Action-F1. Bold text indicates optimal performance of the model.

Figure 7: Asymmetric Temperature in Small Gap Group. Heatmaps of Action-F1 scores for Qwen3-VL-2B learner (left) and Qwen3-VL-4B guide (right) across temperature grids. The global peak (learner: 0.954) occurs at $(\tau_{learner}=3.0,\tau_{guide}=2.0)$ . Symmetric configuration ( $\tau_{learner}=\tau_{guide}$ ) leads to suboptimal performance and may even induce severe performance degradation (guide: 0.732).

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 657 papers

SOPD-SocialNav: Selective On-Policy Distillation for Vision-Language Social Navigation
Jul 22, 2026Xinyu Zhang, Zishuo Wang, Ling XiaoEfficient VLM Inference VLM Distillation
Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan
Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies
Jul 13, 2026Ali Ahmadi, Hamed Rahimi, Adrien Jacquet Cretides +3RL for Robotics Social Robot Navigation
Institut des Syst`emes Intelligents et de Robotique (ISIR) Sorbonne Universit´e France
STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction
Sep 30, 2026Nathan Tsoi, Michael J. Munje, Tejas Oberoi +5VLM Evaluation Social Robot Navigation
Department of Computer Science, The University of Texas at Austin

Jul 22, 2026J/K move · Enter open · S save

SOPD-SocialNav: Selective On-Policy Distillation for Vision-Language Social Navigation

Xinyu Zhang, Zishuo Wang, Ling Xiao

Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan

Efficient VLM Inference VLM Distillation Social Robot Navigation On-Policy Distillation VLMs for Robotics

Open paper AskPDF arXiv

Vision-language models have shown strong potential for social robot navigation by leveraging rich semantic understanding of complex environments and human behaviors. However, large scale VLMs are difficult to deploy on resource-constrained robotic platforms, while lightweight VLMs often lack sufficient social reasoning capability. To address this problem, we propose SOPD-SocialNav, a selective on-policy distillation (SOPD) method that transfers social navigation knowledge from a large teacher VLM to a lightweight student VLM. SOPD introduces an entropy-based token selection mechanism that uses teacher uncertainty to identify socially informative decision tokens, while suppressing gradients from low-entropy tokens corresponding to trivial navigation states. A temperature-controlled Jensen-Shannon divergence objective is then used to align the student and teacher distributions on the selected tokens. Experiments on the SNEI and MUSON benchmarks demonstrate that SOPD consistently outperforms supervised fine-tuning, off-policy distillation, and standard on-policy distillation baselines in action prediction, perception consistency, and reasoning consistency. Real-world deployment on a Scout Mini robot further shows that the distilled model can generate more socially appropriate navigation behaviors in conversational and queuing scenarios. These results suggest that SOPD is an effective strategy for building lightweight yet socially aware VLM-based navigation systems.

Abstract

VLMs for Robotics

Explore similar work

SOPD-SocialNav: Selective On-Policy Distillation for Vision-Language Social Navigation

Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies

STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction