FineGen: A VLM-based Multi-Agent Framework for Fine-Grained Image-Text Dataset Construction

Chang Kong; Yuebing Li; Peng Mo; Haigang Zhang; Qiuming Luo

cs.CVJun 2, 2026

FineGen: A VLM-based Multi-Agent Framework for Fine-Grained Image-Text Dataset Construction

Authors: Chang Kong, Yuebing Li, Peng Mo, Haigang Zhang, Qiuming Luo

Organizations: Shenzhen Polytechnic University, Shenzhen, China · Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong Macao2026 Greater Bay Area, Shenzhen, China · Shenzhen University, China

Abstract

The scarcity of hard negative samples in current vision-language datasets significantly hinders fine-grained perception. To address this, we propose FineGen, a VLM-based Multi-Agent framework for automated dataset construction. By employing a collaborative Generation-Verification-Correction pipeline with a closed-loop feedback mechanism, FineGen ensures synthesized hard negatives are semantically valid yet strictly contradictory to visual content. Applying this to ImageNet, we construct FineGen-100K, a hierarchical dataset containing over 147,000 attribute-specific hard negatives with a rigorous 1:10 positive-to-negative ratio. Extensive evaluations confirm a 96.7% attribute validity rate. Crucially, downstream validation on the FG-OVD benchmark shows that fine-tuning on FineGen-100K yields a substantial +14.4% accuracy improvement on hard samples, significantly outperforming state-of-the-art methods.

Explore similar work

PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models
Jun 23, 2026Simone Gallivanone, Hossein Khodadadi, Mauro Dore +2Adversarial Examples Adversarial Evaluation
DataComp-VLM: Improved Open Datasets for Vision-Language Models
Jun 26, 2026Matteo Farina, Vishaal Udandarao, Thao Nguyen +33Large Vision Language Models Vision-Language
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
Jun 17, 2026Hong-Tao Yu, Chen-Wei Xie, Yuxin Peng +2Fine-Grained Perception Large Vision Language Models

Jun 23, 2026 · cs.AIJ/K move · Enter open · S save

PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models

Simone Gallivanone, Hossein Khodadadi, Mauro Dore, Mauro Medda+1

The Italian Institute of Artificial Intelligence (AI4I), Turin, Italy · HikmaAI S.r.l., Pula, Italy

We introduce a large-scale, open-source dataset of pre-generated adversarial attacks for vision-language models (VLMs). The dataset is designed to be diverse, representative, and practical, extending existing benchmarks by covering 10 high-level categories and 55 subcategories of harmful intents. Our primary goal is to make adversarial data accessible to the research community, given the computational cost and complexity of generating large numbers of attacks. The dataset comprises 47 524 adversarial samples, generated using state-of-the-art attack strategies from recent literature. Our work complements existing efforts by consolidating and extending prior benchmarks from multiple established sources, resulting in 7 826 intents, and introduce an additional category to broaden coverage. This provides realistic evaluation resources for studying model robustness and alignment. Our dataset intends to enable researchers and practitioners to systematically evaluate the robustness and safety of VLMs, fine-tune attack-generation models, and develop or stress-test defensive guardrails under diverse adversarial conditions. By releasing this resource, we aim to lower the barrier to adversarial research and foster more reproducible, comprehensive, and comparable evaluations of VLM safety.

Adversarial Examples Adversarial Evaluation Multimodal Learning

Open paper Ask about it PDF

Abstract

Explore similar work

PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis