VLM Adaptation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

34 papers in the last four weeks, up 209% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 395

All topics

From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction
Oct 8, 2026Uddipan Basu Bir, Vincent Christlein, Andreas Maier +1Document Information Extraction VLM Adaptation
Friedrich-Alexander-Universität Erlangen–Nürnberg, Germany
Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder
Oct 8, 2026Zidan Wang, Yaqian Li, Xiaokai Zhang +3VLM Adaptation Vision-Language Models
Huazhong University of Science and Technology · Li Auto Inc.
DiscoVL: Unveiling Disentangled C ross-Modal Representation Learning via Orthogonal Adversarial Regularization for V ision-Language Models
Oct 8, 2026Mengping Dong, Jinbao Li, Fei LiVision-Language Models VLM Adaptation
Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences), Jinan, China · University of Florida, Florida, USA
Unsupervised Long-Tailed Adaptation of Vision-Language Models
Oct 6, 2026Keliang Chen, Yaxin Hou, Hui Liu +1Long-Tail Learning VLM Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · School of Computing and Information Sciences, Saint Francis University, Hong Kong, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China
CueRator: Agentic Search for Symbolic Rules to Adapt Frozen Multimodal Encoders
Oct 6, 2026Sunchan Park, Beomkwon Cho, Kyeongbo KongAudio-Visual Understanding Agentic Search
Pusan National University
Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling
Oct 6, 2026Juntong Li, Lingwei Dang, Haomin Wu +3VLM Adaptation Dense Prediction
School of Software Engineering, South China University of Technology
Two Vectors Replace In-Context Demos: Structured Task Adaptation via Embeddings
Oct 6, 2026Xi Ding, Naichen Shi, Jiawei ZhangMultimodal ICL Task Vectors
University of Wisconsin–Madison · Northwestern University
Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuning
Oct 5, 2026Suguru Onda, Matthew Bailey, Ryan FarrellVision-Language Models VLM Adaptation
Brigham Young University · Carnegie Mellon University
Sim-to-Real Transfer of Vision-Language Navigation in Continuous Environments Using an Ackermann-Steered Mobile Robot
Oct 5, 2026Chalindu Abeywansa, Sahan Gunasekara, Devindi De Silva +3VLM Adaptation Sim-to-Real Transfer
Department of Electronic and Telecommunication Engineering, University of Moratuwa, Colombo, Sri Lanka.
ReMem: Streaming Video Understanding With Long Context Retention
Oct 5, 2026Li Yiheng, He Xu, Wang Shaobo +2Memory-Augmented VLMs Streaming Video Understanding
Nanyang Technological University · Shanghai Jiaotong University · University of the Chinese Academy of Sciences
Fitting Vision Adapters at Frontier Scales
Oct 5, 2026Jaehoon Lee, Harry Partridge, Mudith Jayasekara +3Vision-Language Models VLM Adaptation
Base Labs
LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder
Oct 2, 2026Anh-Khoa Dinh-Duc, Duc-Tai Dinh, Tam V. Nguyen +1VLM Adaptation Zero-Shot Image Classification
University of Science, Ho Chi Minh City, Vietnam · Viet Nam National University, Ho Chi Minh City, Vietnam · University of Dayton, USA
Beyond Domain-Level Adaptation: Margin-Oriented Semantic-Appearance Interaction Correction for Personalized Federated Vision-Language Models
Oct 1, 2026Wentao Yue, Qingyu Mao, Tianyou Lai +2VLM Adaptation Federated Fine-Tuning
Concept Driven Domain Adaptation: Finding an Abstract Needle in a Haystack
Oct 1, 2026Haiming Zhao, Tai Wang, Kun Zhang +2VLM Adaptation Vision-Language Alignment
Laboratory for Artificial Intelligence and New Forms of Education Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan, China · Wuhan United Imaging Healthcare Co., Ltd., Wuhan, China
Platonic Task Arithmetic
Oct 1, 2026Junghwan Park, Woojin ChoVLM Adaptation Task Arithmetic
TelePIX
Comparative study of adapting pre-trained models for driving behavior video captioning
Sep 30, 2026Sayak Mallick, Philipp Geiger, Augustin KelavaVLM Adaptation Autonomous Driving
University of Tübingen · Bosch Center for Artificial Intelligence
Inductive Visual Logic for Few-Shot Out-Of-Distribution Adaptation in VLMs
Sep 29, 2026Hung-Jen Chen, Yu-Heng Ho, Ting-Yao Huang +4Vision-Language Models Few-Shot Domain Adaptation
National Tsing Hua University, Hsinchu, Taiwan · National Taiwan University, Taipei, Taiwan
Scaling Full Conformal Image Classifiers
Sep 29, 2026Julio Silva-Rodríguez, Ender KonukogluVLM Adaptation Conformal Prediction
Computer Vision Lab, ETH Zürich
EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception
Sep 29, 2026Yaoxin Niu, Zhangquan Chen, Yang Zhang +5VLM Adaptation Efficient ViTs
Tsinghua University · Peng Cheng Laboratory · The Hong Kong University of Science and Technology +3
Reprogramming Vision-Language Models via Structured Prompt Reparameterization
Sep 29, 2026Zizhao Li, Chengyi Cai, Mohammed Yaqoob Ansari +3VLM Adaptation Few-Shot Image Classification
The University of Melbourne, Melbourne, Australia
You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming
Sep 29, 2026Zizhao Li, Mohammed Yaqoob Ansari, Xinyu Su +3VLM Adaptation Vision Foundation Model Adaptation
The University of Melbourne · Fudan University
Benchmarking Vision-Language Models on Synapse Detection and Proofreading in Connectomics
Sep 29, 2026Yicong Li, Junjie Wang, Leander Lauenburg +5VLM Evaluation VLM Adaptation
Harvard University, Cambridge, MA, USA · Boston College, Chestnut Hill, MA, USA · Nanyang Technological University, Singapore
ConvCue: Complementary Visual Inductive Biases for Vision-Language Models
Sep 28, 2026Zixuan Lan, Shichu SunVisual Question Answering VLM Adaptation
The University of Chicago · University of Chinese Academy of Sciences
BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation
Sep 27, 2026Anglin Liu, Yanlin Wu, Ruichao Chen +6VLM Adaptation VLM Reasoning
HKUST(GZ) · HKUST
Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models
Sep 27, 2026Jingxuan Kang, Qianying Yue, Che Liu +1VLM Adaptation Few-Shot Learning
Imperial College London · The Chinese University of Hong Kong
When Does Geometric View Synthesis Help Wine Label Retrieval? A Public One-Shot Benchmark Across Self-Supervised and Vision-Language Backbones
Sep 27, 2026Yueh-Cheng HuangVLM Adaptation Novel View Synthesis
Department of Computer Science and Information Engineering, National Dong Hwa University, Hualien 974301, Taiwan
From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Model
Sep 24, 2026Xunlan Zhou, Xianliang Yang, Li ZhaoVisual Question Answering VLM Adaptation
School of Intelligence Science and Technology, Nanjing University, Suzhou, China · Microsoft Research Asia
Small yet Assistive: Spatially-Aware Post-Training for Low Vision
Sep 23, 2026Rishabh Choudhary, Shreyansh Raj, Umesh Goyal +6Vision-Language Models VLM Adaptation
Indian Institute of Technology Mandi · Mohamed bin Zayed University of Artificial Intelligence
PrismGPT: Proxy-Guided Learning for Region-Aware Photo Editing with Self-Synthesized Reasoning
Sep 21, 2026Ke Zhao, Hue Nguyen, Abhijith Punnappurath +3VLM Adaptation Image Editing
AI Center-Toronto, Samsung Electronics Toronto, Ontario, Canada
HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space
Sep 21, 2026Zelin Peng, Zhengqin Xu, Changsong Wen +4VLM Adaptation Open-Vocabulary Semantic Segmentation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · State Key Laboratory of Infrared Physics, Shanghai Institute of Technical Physics, Chinese Academy of Science, Shanghai, 200083, China · Meituan, Shanghai, 200051, China

Oct 8, 2026, cs.CVJ/K move · Enter open · S save

From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction

Uddipan Basu Bir, Vincent Christlein, Andreas Maier, Mathias Zinnen

Friedrich-Alexander-Universität Erlangen–Nürnberg, Germany

Document Information Extraction VLM Adaptation Optical Character Recognition VLM Evaluation Cultural Heritage

Open paper AskPDF arXiv

While massive, closed-source Vision-Language Models (VLMs) set strong benchmarks for document understanding, their dependence on commercial APIs limits adoption in institutional archives due to data autonomy concerns, recurring costs, and the environmental footprint of hyperscale computing. This is especially acute in heritage digitization, where documents include historical handwriting, domain-specific terminology (e.g., jewelry, prehistory, architecture), and non-standard layouts requiring high-dimensional structured extraction. We present a comparative study of eight open-source lightweight VLMs (up to 7B parameters) for Optical Character Recognition (OCR)-to-structure across three university heritage collections. Given a document image, models must extract text and generate schema-compliant JSON, enabling automatic validation and downstream use. We evaluate models under a constraint-aware protocol across zero-shot, few-shot, and fine-tuning settings, measuring extraction fidelity and structured-output quality using Character Error Rate (CER), Approximate Normalized Levenshtein Similarity (ANLS*), and mean Average Precision F1 (mAP-F1). Against a fine-tuning baseline, we further test the independent impact of (i) hyperparameter optimization, (ii) classical image preprocessing (illumination flattening, denoising, and CLAHE), and (iii) multi-stage training. Finally, we analyze the trade-off between dataset-specific fine-tuning and a single multi-dataset checkpoint, where joint training enables one model to operate across collections but can shift performance between datasets. Overall, we show that carefully adapted VLMs with up to 7B parameters can provide a sustainable, private, high-performing alternative to manual transcription or commercial black-box systems, and we offer actionable guidance for heritage institutions seeking institution-controlled OCR-to-JSON extraction.

Momentum

Related topics

Latest papers 395