Towards Efficient Multimodal and Multilingual Opinion Extraction for STI: A QLoRA-Based Fine-Tuning Approach

Sheng Hong; Xuanqi Wang; Jiacheng Wang; Yuwei Wang

cs.AIAug 14, 2026

Towards Efficient Multimodal and Multilingual Opinion Extraction for STI: A QLoRA-Based Fine-Tuning Approach

Authors: Sheng Hong, Xuanqi Wang, Jiacheng Wang, Yuwei Wang

Organizations: School of Cyber Science and Technology, Beihang University, Beijing, 100191, China · School of Information and Engineering, Nanchang University, Nanchang, 330031, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China

Abstract

Recent advances in large language models (LLMs) have reshaped semantic analysis. Opinion Extraction (OE) for Science and Technology Intelligence (STI) requires concise core opinions from large information streams. Off-the-shelf models struggle to filter noise from these streams and show limited structured-output reliability in zero-shot multilingual and multi-modal settings. To address information overload and extraction defocus, this study proposes a multimodal core-opinion extraction framework in which visual evidence serves as a contextual anchor for textual judgment. Using VideoLLaMA2 (VL2) and VideoLLaMA2.1 (VL2.1) as the base models, we apply Quantized Low-Rank Adaptation (QLoRA) fine-tuning on a curated dataset of 2,194 multilingual and multimodal samples. Under the selected Image-Augmented setting, fine-tuned VL2.1 generates structured JSON core-opinion outputs, achieving 64.98% Precision, 42.15% Recall, 51.14% F1-score, and 74.00% sample-level accuracy. Relative to the zero-shot VL2.1 setting, it raises the F1-scores of Spanish and Russian from 4.83% and 0.45% to 46.05% and 51.93%, respectively. The framework further incorporates a Fuzzy Cumulative Prospect Theory-based post-extraction triage module for case-level value assessment, providing a case-level value signal for downstream STI screening.

Figures & tables

Fig. 1: Overall framework of the proposed method.

Fig. 2: Construction process of the proposed dataset.

Modality	Samples	Language	Samples
Image	858	English	608
Video	138	Chinese	610
Text	1198	Russian	482
		Spanish	494
Total	2194	Total	2194

TABLE I: DISTRIBUTION BY MODALITY AND LANGUAGE

Model	Context Length	Visual Encoder	Language Decoder
Video-LLaVA	2K	CLIP Vision Encoder	Vicuna-7B
Video-LLaMA	4K	ViT-G/14 + BLIP-2 Q-Former	LLaMA2-7B
VideoLLaMA2	32K	CLIP-ViT-Large- Patch14-336	Mistral-7B- Instruct-v0.2
VideoLLaMA2.1	131K	SigLIP-So400m- Patch14-384	Qwen2-7B- Instruct

TABLE II: COMPARISON OF MULTIMODAL LARGE LANGUAGE MODELS

Score Range	Stars	Typical Guidance
$\mathcal{V}\geq 0.74$	$\bigstar\bigstar\bigstar\bigstar\bigstar$	Priority: focused attention.
$0.61\leq\mathcal{V}<0.74$	$\bigstar\bigstar\bigstar\bigstar$	High value: continued monitoring.
$0.46\leq\mathcal{V}<0.61$	$\bigstar\bigstar\bigstar$	Reference: supporting context.
$\mathcal{V}<0.46$	$\bigstar\bigstar$	Limited value: background information.

TABLE III: Fuzzy-CPT Value Categories for Case-Level Assessment

Model	Language	Precision (%)	Recall (%)	F1-score (%)	Sample-level Acc (%)
VL2 (Mistral-based)	English	28.70	51.20	36.78	66.00
	Chinese	12.11	32.29	17.61	36.00
	Spanish	10.58	24.18	14.72	32.00
	Russian	4.60	9.57	6.21	10.00
	Overall	13.82	29.98	18.92	36.00
VL2.1 (Qwen-based)	English	24.15	68.00	35.64	84.00

TABLE IV: ZERO-SHOT BASELINE PERFORMANCE OF VL2 AND VL2.1 ACROSS LANGUAGES

Fig. 3: Zero-shot F1-score across four languages.

Model	Prompt Strategy	Language	Precision (%)	Recall (%)	F1-score (%)	Sample-level Acc (%)
VL2	Zero-Shot	English	28.70	51.20	36.78	66.00
		Chinese	12.11	32.29	17.61	36.00
		Spanish	10.58	24.18	14.72	32.00
		Russian	4.60	9.57	6.21	10.00
		Overall	13.82	29.98	18.92	36.00
	4-Shot	English	21.53	58.40	31.47	76.00

TABLE V: PERFORMANCE COMPARISON OF PROMPT ENGINEERING TECHNIQUES ACROSS LANGUAGES

Fig. 4: Overall sample-level accuracy comparison across prompt-based baseline strategies.

Hyperparameters	Language	Precision (%)	Recall (%)	F1-score (%)	Sample-level Acc (%)
$r=64$ $\alpha=128$	English	40.74	26.40	32.04	56.00
	Chinese	68.63	36.46	47.62	68.00
	Spanish	44.83	28.57	34.90	52.00
	Russian	56.90	28.70	38.15	64.00
	Overall	51.21	29.74	37.63	60.00
$r=128$ $\alpha=256$	English	52.22	37.60	43.72	76.00

TABLE VI: PERFORMANCE COMPARISON OF QLORA HYPERPARAMETER SETTINGS ON VL2 MODEL

Model	Language	Precision (%)	Recall (%)	F1-score (%)	Sample-level Acc (%)
Fine-tuned VL2	English	56.52	41.60	47.93	78.00
	Chinese	58.33	43.75	50.00	74.00
	Spanish	53.52	41.76	46.91	70.00
	Russian	54.00	46.96	50.23	78.00
	Overall	55.52	43.56	48.82	75.00
Fine-tuned VL2.1	English	60.22	44.80	51.38	88.00

TABLE VII: PERFORMANCE OF THE SELECTED IMAGE-AUGMENTED QLORA FINE-TUNING ACROSS LANGUAGES

Fig. 5: Overall precision comparison among the zero-shot, 4-Shot, and QLoRA fine-tuning settings.

Model	Modality Setting	Prec. (%)	Rec. (%)	F1 (%)
VL2	Text-Centered	53.59	41.92	47.04
	Image-Augmented	55.52	43.56	48.82
	Full Modality	55.36	43.56	48.75
VL2.1	Text-Centered	63.32	42.86	51.12
	Image-Augmented	64.98	42.15	51.14
	Full Modality	64.75	42.15	51.06

TABLE VIII: ABLATION STUDY ON VISUAL GROUNDING SETTINGS ACROSS DIFFERENT MODELS

Fig. 6: Ablation study comparing Text-Centered, Image-Augmented, and Full Modality visual grounding settings.

Model	Precision (%)	Recall (%)	F1-score (%)	Sample-level Acc (%)
DeepSeek-V3.2	19.37	80.80	31.25	85.50
Qwen3-Omni-Flash	29.11	45.20	35.41	12.50
Claude-Sonnet-4-6	28.46	73.53	41.04	15.50
Ours (Fine-tuned VL2)	55.52	43.56	48.82	75.00
Ours (Fine-tuned VL2.1)	64.98	42.15	51.14	74.00

TABLE IX: PERFORMANCE COMPARISON WITH EXTERNAL MULTIMODAL LARGE LANGUAGE MODELS

Fig. 7: Performance comparison with external multimodal large language models.

Fig. 8: Distribution of value categories among predicted opinion-level case records.

Topic	4/5-Star Records	Source Types
Frontier AI	17	5
Electric and Autonomous Vehicles	9	4
Clean Energy Transition	7	3
Digital Transformation	6	4

TABLE X: Topic-Level Summary of High-Value Cases

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,092 papers

SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systems
Mar 25, 2026Chung-En Johnny Yu, Brian Jalaian, Nathaniel D. BastianHallucination Detection in VLMs
University of West Florida Pensacola, Florida, USA · United States Military Academy West Point, New York, USA
Omni-Modal LLMs Know More Than They Say
Oct 12, 2026Na Min An, Taekyung Kim, Jaemyung Yu +2Multimodal Large Language Models Cross-Modal Alignment
KAIST AI · NAVER AI Lab
Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation
Jun 29, 2026Xin Zou, Haolin Deng, Yibo Yan +5Multimodal Large Language Models Multimodal Hallucination
HKUST (GZ) · HKUST · OPPO AI Center

Mar 25, 2026J/K move · Enter open · S save

SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systems

Chung-En Johnny Yu, Brian Jalaian, Nathaniel D. Bastian

University of West Florida Pensacola, Florida, USA · United States Military Academy West Point, New York, USA

Hallucination Detection in VLMs

Open paper AskPDF arXiv

Combining multiple Vision-Language Models (VLMs) can enhance multimodal reasoning and robustness, but aggregating heterogeneous models' outputs amplifies uncertainty and increases the risk of hallucinations. We propose SCoOP (Semantic-Consistent Opinion Pooling), a training-free uncertainty quantification (UQ) framework for multi-VLM systems through uncertainty-weighted linear opinion pooling. The core idea is to treat each VLM as a probabilistic "expert," sample multiple outputs, map them to a unified space, aggregate their opinions, and produce a system-level uncertainty score. Unlike prior UQ methods designed for single models, SCoOP explicitly measures collective, system-level uncertainty across multiple VLMs, enabling effective hallucination detection and abstention for highly uncertain samples. On ScienceQA, SCoOP achieves an AUROC of 0.866 for hallucination detection, outperforming baselines (0.732-0.757) by approximately 10-13%. For abstention, it attains an AURAC of 0.907, exceeding baselines (0.818-0.840) by 7-9%. Despite these gains, SCoOP introduces only microsecond-level aggregation overhead relative to the baselines, which is trivial compared to typical VLM inference time (on the order of seconds). These results demonstrate that SCoOP provides an efficient and principled mechanism for uncertainty-aware aggregation, advancing the reliability of multimodal AI systems. Our code is publicly available at https://github.com/chungenyu6/SCoOP.

Abstract

Multimodal Large Language Models

Explore similar work

SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systems

Omni-Modal LLMs Know More Than They Say

Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation