Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models

Hiroshi Sasaki

cs.CVFeb 27, 2026

Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models

Authors: Hiroshi Sasaki

Organizations: The Japan Research Institute, Limited

Abstract

Recent multimodal models such as Contrastive Language-Image Pre-training (CLIP) have shown remarkable ability to align visual and linguistic representations. However, domains where small visual differences carry large semantic significance, such as diagram understanding, remain challenging due to the models' limited sensitivity to fine-grained structural variations. We propose a new training paradigm designed to enhance diagram comprehension in vision-language models. Our approach introduces pseudo contrastive samples generated by a diagram renderer that creates synthetic diagrams using randomly picked text elements. These samples highlight structural differences in diagrammatic imagery without requiring any modification or editing of the original data. By incorporating these pseudo contrastive samples into the training objective, the model learns to capture more precise and semantically consistent diagram structures. Empirical evaluations on a benchmark dataset of flowcharts demonstrate substantial improvements over standard CLIP and hard-negative CLIP training in both image-text matching and visual question answering tasks. The results underscore the value of domain-specific training strategies and contribute to advancing diagrammatic understanding within the broader context of vision-language learning.

Figures & tables

Figure 1: Conceptual overview of our proposed method. Text is extracted from a raster image via OCR to generate pseudo diagrams with random connections. These are rendered into an editable format, from which hard positive and negative pairs are created through rule-based edits. The resulting samples are used to train a VLM with structure-aware contrastive learning, enhancing its ability to distinguish fine-grained structural differences.

Figure 2: The editable pseudo diagram synthesis flow. OCR-extracted text is grouped by its proximity into combinations to form nodes. After adding random edge connections, this structure is finalised as definition code. This code is then used by a renderer to create both an editable image and a corresponding rule-based caption.

Figure 3: The synthesis of hard positive and negative samples. Hard positive images are generated via minor visual edits, such as altering node positions or flow orientation. Hard negatives are created by permuting nodes and edges. The original diagram code serves as the hard positive caption, while hard negative captions are generated by swapping labels within the code to create a structural mismatch.

	Query: images, Retrieve: captions				Query: captions, Retrieve: images
	R@1	R@5	R@10	MRR	R@1	R@5	R@10	MRR
w/o FT [ 14 ]	0.65726	0.88790	0.92231	0.75842	0.66586	0.89812	0.93575	0.76584
CROP - CLIP [ 16 ]	0.48118	0.70161	0.77419	0.57887	0.48118	0.65081	0.70726	0.55689
GRAN [ 17 ] - CLIP [ 16 ]	0.76694	0.92850	0.94758	0.84036	0.77930	0.93925	0.94893	0.85005
GRAN [ 17 ] - SaCLIP [ 17 ]	0.76882	0.92796	0.94785	0.84148	0.77903	0.93253	0.94570	0.84869
PSEUDO (ours) - CLIP [ 16 ]	0.76667	0.93952	0.95645	0.84315	0.78253	0.93790	0.95591	0.85319

Table 1: Evaluation results for the image-text matching task, reporting Recall@1 (R@1), Recall@5 (R@5), Recall@10 (R@10), and Mean Reciprocal Rank (MRR) across different data preparation and finetuning strategies.

	Query: images, Retrieve: captions			Query: captions, Retrieve: images
	R@1	R@5	MRR	R@1	R@5	MRR
w/o FT [ 14 ]	0.16613	0.16613	0.16613	0.60538	0.82419	0.71632
CROP - CLIP [ 16 ]	0.20887	0.20887	0.20887	0.49140	0.75403	0.62629
GRAN [ 17 ] - CLIP [ 16 ]	0.46667	0.46667	0.46667	0.61640	0.82634	0.72529
GRAN [ 17 ] - SaCLIP [ 17 ]	0.47527	0.47527	0.47527	0.89973	0.98602	0.94133
PSEUDO (ours) - CLIP [ 16 ]	0.35269	0.35269	0.35269	0.50511	0.76022	0.63644

Table 2: Evaluation results for the image-text matching task in the presence of hard negative samples, reported using Recall@1 (R@1), Recall@3 (R@3), and Mean Reciprocal Rank (MRR) across different data preparation and finetuning strategies.

Vision Encoder	Precision	Recall	F1
w/o FT	0.483461	0.425924	0.494744
CROP - CLIP [ 16 ]	0.713844	0.543008	0.618101
GRAN [ 17 ] - CLIP [ 16 ]	0.532666	0.467946	0.507943
GRAN [ 17 ] - SaCLIP [ 17 ]	0.620144	0.601086	0.634233
PSEUDO - CLIP (ours)	0.558813	0.488994	0.536966
PSEUDO - SaCLIP (ours)	0.660028	0.612537	0.647963

Table 3: A comparison of the BERTScore [ 26 ] performance across different LLaVA [ 10 ] models with their vision encoders swapped.

Contrastive Learning

Contrastive learning trains representations by bringing related examples together and separating unrelated ones for recognition, retrieval, and transfer.

See all 322 papers

DiagramNet: An End-to-End Recognition Framework and Dataset for Non-Standard System-Level Diagrams
May 2, 2026Jincheng Lou, Ruohan Xu, Jiapeng Li +6Multi-Agent LLM Systems Electronic Design Automation
School of IC, Peking University, Beijing, China · College of Artificial Intelligence, Xi’an Jiaotong University, Xi’an, China · Department of Precision Instruments, Tsinghua University, Beijing, China +4
Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designs
Mar 13, 2026Zixin Wen, Yifu Cai, Kyle Lee +5Synthetic Data Generation Tool-Using Vision-Language Agents
Carnegie Mellon University Pittsburgh, PA 15213, USA
DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
Apr 28, 2026Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande +4VLM Evaluation Visual Reasoning
Arizona State University · Adobe Research · University of Maryland

May 2, 2026J/K move · Enter open · S save

DiagramNet: An End-to-End Recognition Framework and Dataset for Non-Standard System-Level Diagrams

Jincheng Lou, Ruohan Xu, Jiapeng Li, Junyin Pi+5

School of IC, Peking University, Beijing, China · College of Artificial Intelligence, Xi’an Jiaotong University, Xi’an, China · Department of Precision Instruments, Tsinghua University, Beijing, China+4

Multi-Agent LLM Systems Electronic Design Automation Multimodal Large Language Models Multimodal QA Multimodal Reasoning

Open paper AskPDF arXiv

System-level diagrams encode the architectural blueprint of chip design, specifying module functions, dataflows, and interface protocols. However, non-standardized symbols and the scarcity of structured training data hinder existing multimodal large language models (MLLMs) from recognizing these diagrams. To address this gap, we introduce DiagramNet, the first multimodal dataset for system-level diagrams, comprising 10,977 connection annotations and 15,515 chain-of-thought QA pairs across four tasks: Listing, Localization, Connection, and Circuit QA. Building on this dataset, we propose a progressive training pipeline together with a decoupled multi-agent workflow that decomposes complex visual reasoning into Perception, Reasoning, and Knowledge stages. On the DiagramNet benchmark, integrating our 3B-parameter model with the proposed workflow surpasses the 2025 EDA Elite Challenge winner and outperforms GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro by over 2x in end-to-end evaluation. Notably, the workflow generalizes beyond our model, boosting Task 1 performance by 128.7x for Gemini-2.5-Pro and 12.4x for GPT-5. Furthermore, with only 60 images for detector adaptation, the method transfers effectively to AMSBench, achieving zero-shot connectivity reasoning on par with GPT-5 and Claude-Sonnet-4 while surpassing the AMS state-of-the-art method Netlistify.

Abstract

Contrastive Learning

Explore similar work

DiagramNet: An End-to-End Recognition Framework and Dataset for Non-Standard System-Level Diagrams

Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designs

DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams