Vision-Language Models

Also known as VLM

Vision-language models study how systems can connect images or videos with language to recognize, describe, retrieve, and reason about visual content. Common approaches include contrastive image-text encoders such as CLIP, generative models that connect vision encoders to language models, and instruction tuning, sometimes combined with retrieval or tools. Researchers evaluate them on image-text retrieval, visual question answering, OCR and document understanding, visual grounding, and video benchmarks. A central challenge is that fluent answers do not guarantee visual grounding: hallucinations, reliance on textual shortcuts, weak compositional and temporal reasoning, and the costs of efficient deployment remain important concerns.

Latest papers

738 papers77 papers in the last four weeks, up 20% on the four weeks before. 0.4% of all new papers.

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models
Mar 25, 2026Qijia He, Xunmei Liu, Hammaad Memon +6Vector Graphics Generation Vision-Language Models
University of Washington · Allen Institute for Artificial Intelligence · UNC-Chapel Hill
Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification
Mar 25, 2026Han Sun, Qin Li, Peixin Wang +1Object Hallucination in VLMs VLM Hallucination Mitigation
Shanghai Key Laboratory of Trustworthy Computing, East China Normal University
VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions
Mar 24, 2026Adrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas +2Efficient VLM Inference Vision-Language Models
Samsung AI Cambridge · Technical University of Iasi · Queen Mary University of London
ARGENT: Adaptive Hierarchical Image-Text Representations
Mar 24, 2026Chuong Huynh, Hossein Souri, Abhinav Kumar +3Vision-Language Models Hierarchical Representation Learning
University of Maryland, College Park · Samsung Research America, AI Center – Mountain View
Gaze-Regularized VLMs for Ego-Centric Behavior Understanding
Mar 24, 2026Anupam Pani, Yanchao YangVision-Language Models Egocentric Video Understanding
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong
A Vision Language Model for Generating Procedural Plant Architecture Representations from Simulated Images
Mar 23, 2026Heesup Yun, Isaac Kazuo Uyehara, Ioannis Droutsas +4Procedural Content Generation Plant Phenotyping
Biological and Agricultural Engineering, One Shields Avenue, Davis, 95616, CA, USA · Department of Plant Sciences, One Shields Avenue, Davis, 95616, CA, USA · Wageningen University & Research, P.O. Box 9101, 6700 HB Wageningen, The Netherlands +1
Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models
Mar 23, 2026Hayeon Kim, Ji Ha Jang, Junghun James Kim +1Vision-Language Models Hierarchical Representation Learning
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · INMC & IPAI, Seoul National University, Republic of Korea
A Large-Scale Remote Sensing Dataset and VLM-based Algorithm for Fine-Grained Road Hierarchy Classification
Mar 22, 2026Ting Han, Xiangyi Xie, Yiping Chen +5Remote Sensing VLMs Classification
School of Geospatial Engineering and Science, Sun Yat-Sen University, Zhuhai, 519082, China · Transport Bureau of Nanhai District, Foshan, 528200, China · Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China, Chengdu, 611731, China +2
Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
Mar 22, 2026Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Shubhashis Roy Dipta +6VLM Evaluation Multilingual VLM Evaluation
ME-IQA: Memory-Enhanced Image Quality Assessment via Re-Ranking
Mar 21, 2026Kanglong Fan, Tianhe Wu, Wen Wen +6Memory-Augmented VLMs Vision-Language Models
City University of Hong Kong, Hong Kong SAR, China · ByteDance Inc., China
Satellite-to-Street: Synthesizing Post-Disaster Views from Satellite Imagery via Generative Vision Models
Mar 21, 2026Yifan Yang, Lei Zou, Wendy JepsonImage Generation Disaster Damage Assessment
Department of Geography, Texas A&M University, College Station, TX, USA
PEARL: Personalized Streaming Video Understanding Model
Mar 20, 2026Yuanhong Zheng, Ruichuan An, Xiaopeng Lin +10Streaming Video Understanding Vision-Language Models
Peking University · CASIA · Stepfun +3
Contextual inference from single objects in Vision-Language models
Mar 20, 2026Martina G. Vilas, Timothy Schaumlöffel, Gemma RoigVision-Language Models VLM Reasoning
Goethe University Frankfurt · The Hessian Center for AI
Geometric Risk Control for Vision-Language Model OCR
Mar 20, 2026Weile Gong, Zijian Lu, Mingcai Chen +3Selective Prediction for VLMs VLM Hallucination Mitigation
Nanjing University of Posts and Telecommunications, Nanjing, China
dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
Mar 19, 2026Saikat Dutta, Biplab Banerjee, Hamid RezatofighiOpen-Vocabulary Semantic Segmentation Vision-Language Models
IITB-Monash Research Academy · IIT Bombay · Monash University
Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders
Mar 19, 2026Shang-Jui Ray Kuo, Paola Cascante-BonillaVision-Language Models State Space Models
Stony Brook University
Tinted Frames: Question Framing Blinds Vision-Language Models
Mar 19, 2026Wan-Cyuan Fan, Jiayun Luo, Declan Kutscher +2Vision-Language Models VLM Robustness
University of British Columbia · Vector Institute for AI · University of California, Berkeley
SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues
Mar 19, 2026Carlos Hinojosa, Clemens Grange, Bernard GhanemCultural Safety Evaluation of VLMs Vision-Language Models
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Technical University of Munich (TUM), Germany
SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
Mar 19, 2026Quentin Guimard, Federico Bartsch, Simone Caldarella +3Cultural Bias in VLMs Vision-Language Models
University of Trento · Toyota Motor Europe · Fondazione Bruno Kessler
Balanced Thinking: Improving Chain of Thought Training in Vision Language Models
Mar 19, 2026Shaked Perek, Ben Wiesel, Avihu Dekel +2PEFT of VLMs VLM Reasoning
IBM Research
Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness
Mar 19, 2026Lu Yu, Haiyang Zhang, Changsheng XuAttention-Based Adversarial Attacks Contrastive Learning for VLMs
School of Computer Science and Engineering, Tianjin University of Technology, 300384, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China · School of Artificial Intelligence, University of the Chinese Academy of Sciences, Beijing, 100049, China
To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
Mar 19, 2026Rui Hong, Shuxue QuanSelective Prediction for VLMs Sycophancy in VLMs
George Mason University, Fairfax, VA, USA · Independent Researcher
CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning
Mar 18, 2026Marios Krestenitis, Christos Tzelepis, Konstantinos Ioannidis +5Cycle Consistency Image Captioning
Queen Mary, University of London · Centre for Research and Technology Hellas · City St George’s, University of London
VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events
Mar 18, 2026Mohammad Qazim Bhat, Yufan Huang, Niket Agarwal +7VLM Post-Training VLMs for Autonomous Driving
NVIDIA, California, USA
Learning Transferable Temporal Primitives for Video Reasoning via Synthetic Videos
Mar 18, 2026Songtao Jiang, Sibo Song, Chenyi Zhou +12VLM Post-Training Temporal Video Understanding
Zhejiang University · Qwen Team, Alibaba Group · Shanghai Jiao Tong University +1
VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation
Mar 18, 2026Junyoung Kim, Woojoo Kim, Wonbin Kweon +3Multimodal Recommendation Multimodal Embedding
Pohang University of Science and Technology · University of Illinois Urbana-Champaign
FMMC: Harnessing the Power of Foundation Models for Accurate Material Classification
Mar 18, 2026Qingran Lin, Fengwei Yang, Chaolun ZhuVision-Language Models Synthetic Data Generation
Georgia Institute of Technology · Duke University · Waseda University
Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles
Mar 17, 2026Zacharie BugaudVision-Language Models Visual Question Answering
Astera Institute
Empirical Recipes for Efficient and Compact Vision-Language Models
Mar 17, 2026Jiabo Huang, Zhizhong Li, Sina Sajadmanesh +2Vision-Language Models Efficient VLM Inference
Sony AI.
How to Utilize Complementary Vision-Text Information for 2D Structure Understanding
Mar 17, 2026Jiancheng Dong, Pengyue Jia, Derong Xu +9Document Understanding Vision-Language Models
City University of Hong Kong · Baidu Inc.

Mar 25, 2026J/K move · Enter open · S save

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

Qijia He, Xunmei Liu, Hammaad Memon, Ziang Li+5

University of Washington · Allen Institute for Artificial Intelligence · UNC-Chapel Hill

Vector Graphics Generation Vision-Language Models Image-to-Image Translation VLM Evaluation

Open paper AskPDF arXiv

Scalable Vector Graphics (SVG) are essential for technical illustration and digital design, offering resolution independence and semantic editability. In practice, original vector files are frequently lost, leaving only rasterized versions (e.g., PNG, JPEG) that resist modification, while manual reconstruction is prohibitively expensive. Progress on automating raster-to-SVG conversion has been bottlenecked by two gaps: existing SVG datasets are dominated by icons and decorative graphics that lack the complexity of professional diagrams, and existing benchmarks rely on pixel- or embedding-level similarity that fails to capture structural correctness (e.g., broken connectivity, misplaced arrows). We close both gaps with paired contributions targeting diagram-centric figures (e.g., model architectures, flowcharts, schematics). For training, we introduce VFIG-Data, the largest figure-to-SVG dataset of its kind at 66K pairs, combining real paper figures converted via a describe-and-generate pipeline with programmatic diagrams that supply noise-free supervision over arrow styles, fonts, and geometry. For evaluation, we introduce VFIG-Bench, a structure-aware evaluation suite, paired with VFIG-Bench-OOD, an out-of-distribution set of figures manually curated from highly cited arXiv papers. Beyond pixel and embedding similarity, our protocol reports rubric-based VLM-Judge scores and Elo ratings from pairwise human preference evaluation. Built on these contributions, VFIG is a VLM family trained with a simple-to-complex SFT curriculum followed by RL with rendering-aware rewards. VFIG achieves state-of-the-art open-source performance, outperforming the best open-source VLM baseline by over 30%, and matches Claude Sonnet 4.6 on VFIG-BENCH: Gemini-Judge 78.2% vs. 76.7% and GPT-Judge 87.5% vs. 87.4%. It remains slightly behind the strongest proprietary models GPT-5.2 and Gemini-3.