XRF-to-Optical Field-of-View Localization with Vision Language Models

Xiangyu Yin; Tatjana Paunesku; Letonia Copeland-Hardin; Martina Ralle; Zichao Wendy Di; Si Chen; Gayle E. Woloschak; Barry Lai

cs.CVAug 18, 2026

XRF-to-Optical Field-of-View Localization with Vision Language Models

Authors: Xiangyu Yin, Tatjana Paunesku, Letonia Copeland-Hardin, Martina Ralle, Zichao Wendy Di, Si Chen, Gayle E. Woloschak, Barry Lai,

Organizations: Advanced Photon Source, Argonne National Laboratory, Lemont, Illinois, USA · Northwestern University, Chicago, Illinois, USA · University of Chicago, Chicago, Illinois, USA · Oregon Health and Science University, Portland, Oregon, USA

Abstract

Registering images acquired with different microscopy modalities is essential for relating complementary measurements of the same specimen. In correlative X-ray fluorescence (XRF) and optical microscopy, the XRF map often covers only a small region of an optical image acquired from the same or an adjacent tissue section. Field-of-view (FOV) localization is necessary but can be difficult when appearance and structure differ across modalities. Here we evaluate training-free vision language model (VLM) localization on two datasets representing same-section high-correspondence and adjacent-section low-correspondence imaging. We test unconstrained and metadata-constrained search and compare VLMs with geometric controls, classical template matching, and two alternative training-free approaches (DINOv2 and multiGradICON). Direct VLM prompting produced content-dependent spatial signals but was not reliable alone. Classical matching was most accurate when cross-modal structure was preserved but failed in the low-correspondence collection. A proposal-and-verify workflow used repeated VLM predictions as candidates and image-based similarity to select the final location. This workflow recovered useful localization in the low-correspondence regime.

Figures & tables

Figure 1: Task definition for XRF-to-optical FOV localization. Given an optical reference $O_{i}$ and an XRF map $X_{i}$ , the goal is to estimate the XRF FOV $b_{i}^{\ast}$ in optical-image coordinates. In unconstrained search, the method estimates the full box $\hat{b}_{i}$ . In metadata-constrained search, the supplied box size $(w_{i}^{m},h_{i}^{m})$ fixes scale and the method estimates only the translation $t_{i}$ . All inputs are pre-rotated.

Figure 2: VLM prompting design. Four output formats were tested for direct localization: top-left pixels, center pixels, full bounding boxes, and normalized top-left fractions over the valid placement range. Few-shot prompting supplies completed examples in the prompt before the query image pair, adapting the requested response format without task-specific weight updates.

Figure 3: Representative outcomes from the four dataset–setting cells, including improvements and one small regression under challenging unconstrained search. Top: optical reference with ground truth (ochre), direct VLM prediction (dashed blue), and proposal-and-verify prediction (navy). Bottom: XRF template in false color. IoU values list direct prompting followed by proposal-and-verify. The challenging collection contains adjacent-section pairs with ground-truth FOV area below 20% of the optical reference. All inputs are pre-rotated, so the metadata-constrained task estimates translation rather than rotation. Residual appearance differences can arise from the adjacent physical sections.

Geometric controls
	Controlled ( $n{=}5$ )		Challenging ( $n{=}9$ )
Method	Unconstr.	Meta-constr.	Unconstr.	Meta-constr.
Image-center prior	0.00 [0.00, 0.00]	0.00 [0.00, 0.00]	0.25 [0.06, 0.48]	0.25 [0.06, 0.48]
Random-box prior	0.01 [0.01, 0.02]	0.01 [0.01, 0.02]	0.06 [0.01, 0.13]	0.06 [0.01, 0.13]
Constant prior	0.11 [0.00, 0.33]	0.11 [0.00, 0.33]	0.20 [0.03, 0.42]	0.20 [0.03, 0.42]
Baselines and pretrained controls

Table 1: Localization performance (mean IoU with bootstrap 95% confidence intervals over image pairs). Each method is reported on both datasets and in both settings. Best mean per column is bold. Geometric controls and the two pretrained baselines use the same box size in both settings because the constraint does not change their computation, so their values are repeated across the two setting columns within each dataset. Setting and policy details by cell are in Appendix C and Appendix D .

Figure 4: Pair-level comparison between direct VLM prompting and proposal-and-verify for all four dataset–setting cells in Table 1 . Each row links the direct-prompting IoU to the proposal-and-verify IoU for one pair. The dotted line at IoU = 0.30 marks the threshold above which predictions count as approximate localizations. Rows are sorted by proposal-and-verify improvement within each panel.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Pair	Optical size (px)	XRF FOV size in optical pixels	FOV area (%)
Challenging	2xfm_0009	$5120\times 4112$	$2013\times 2019$	19.3
Challenging	2xfm_0010	$5120\times 4112$	$1249\times 1538$	9.1
Challenging	2xfm_0012	$5120\times 4112$	$335\times 324$	0.5
Challenging	2xfm_0013	$5120\times 4112$	$261\times 232$	0.3
Challenging	2xfm_0016	$3840\times 2528$	$467\times 515$	2.5
Challenging	2xfm_0017	$3840\times 2528$	$423\times 500$	2.2

Appendix

Table A1: Geometry of evaluated image pairs. FOV area is the ground-truth XRF field-of-view area as a percentage of the optical reference image area. For adjacent-section pairs, the dimensions describe the evaluation coordinate frames and do not imply identical physical tissue outlines.

Ablation	Condition	Examples	GPT-5	Claude Opus 4.8	Gemma 4 31B Q8_0
Output format	Pixel top-left	3	0.07	0.06	0.13
Output format	Pixel center	3	0.06	0.08	0.08
Output format	Normalized top-left	3	0.19	0.17	0.15
Few-shot count	Normalized top-left	0	0.18	0.19	0.12
Few-shot count	Normalized top-left	1	0.12	0.12	0.10
Few-shot count	Normalized top-left	2	0.17	0.15	0.11

Appendix

Table B1: Direct VLM ablations on the challenging metadata-constrained collection (

n{=}9

). Each entry is the mean IoU over nine pair means, with three calls per pair (27 calls per entry).

Method	Implementation	Role in interpretation
Image-center prior	Places the evaluated box at the optical-image center. The box size is metadata-derived on the challenging dataset, and uses the ground-truth size on the controlled dataset where no probe metadata are recorded. The same box-size convention applies to the random-box and constant priors.	Measures how much IoU can arise from center bias alone.
Random-box prior	Samples valid top-left locations uniformly and reports expected IoU from 5000 samples per pair.	Measures chance overlap under the same box-size convention.
Constant prior	Places each box at the empirical mean ground-truth center across the evaluated setting.	Measures dataset-level location bias.
Classical template matching	Uses gradient-magnitude features and OpenCV normalized cross-correlation over valid translations. Additional development variants included raw NCC, percentile-normalized NCC, and gradient mutual information.	Strong training-free reference when cross-modal structure is shared.
DINOv2 dense features	Uses facebook/dinov2-base . Grayscale inputs are replicated to RGB, dense patch features are normalized, and candidate optical windows are scored by cosine similarity to the template feature.	Tests whether generic self-supervised vision features align optical and XRF appearance without task-specific training.
multiGradICON pretrained	Places the resized XRF map into an optical-sized canvas and runs unigradicon-register with the multiGradICON model for 30 iterations. Both inputs are passed through the same medical-image registration interface.	Tests no-adaptation transfer of a pretrained multimodal registration model.

Appendix

Table C1: Implementation details for baselines and controls.

Dataset (setting)	Proposal pool	Verification and guards	Selected policy summary
Controlled (unconstr.)	Three cached global full-box direct proposals plus three zoom proposals per pair.	Six candidates are scored after resizing crops to the XRF map. The local stage refines proposals inside cropped search regions.	Bbox-seeded zoom policy with six scored candidates per pair.
Controlled (meta-constr.)	Three primary few-shot proposals per pair, scored with translation-only candidates.	Candidates are scored at the metadata-derived box size with the same scoring family used on challenging, selected on controlled.	NMI re-rank with small offset jitter.
Challenging (unconstr.)	Full-box global proposals seed one zoom crop per pair, followed by three local VLM proposals.	Direct and local candidates are rescored with image evidence, and the highest-scoring candidate is retained.	Bbox-seeded zoom top-1 policy filtered to the nine small-target challenging pairs.
Challenging (meta-constr.)	Three primary few-shot proposals plus three supplemental zero-shot proposals per pair.	The primary pool is scored first. A supplemental proposal can override only if it improves the score and overlaps the primary top candidate by at least 0.5 IoU. Templates with area below 60000 pixels use the NMI fallback.	Edge-guard policy with seven primary-only selections and two supplemental overrides.
Leave-one-pair-out	Candidate generation is fixed; the scoring policy is selected using all pairs except the held-out pair and applied once to the held-out pair.	Held-out estimates equal the full estimates in the two primary policy-selection checks.	Challenging metadata-constrained mean 0.35; controlled metadata-constrained mean 0.20.

Appendix

Table D1: Proposal-and-verify policy details by dataset and setting, corresponding to the cells in Table 1 . Policy variant names refer to the internal experiment registry and are preserved for reproducibility.

Method	Unconstrained mean (median; 95% CI)	Metadata-constrained mean (median; 95% CI)
Image-center prior	0.00 (0.00; 0.00–0.00)	0.00 (0.00; 0.00–0.00)
Random-box prior	0.01 (0.01; 0.01–0.02)	0.01 (0.01; 0.01–0.02)
Constant prior	0.11 (0.00; 0.00–0.33)	0.11 (0.00; 0.00–0.33)
DINOv2 dense features	0.06 (0.00; 0.00–0.14)	0.06 (0.00; 0.00–0.14)
multiGradICON pretrained	0.00 (0.00; 0.00–0.00)	0.00 (0.00; 0.00–0.00)
Classical template matching	0.12 (0.00; 0.00–0.36)	0.92 (0.90; 0.89–0.95)

Appendix

Table E1: Summary statistics for localization IoU on the controlled dataset (

n{=}5

), both settings. Confidence intervals are bootstrap 95% intervals over image pairs. The unconstrained classical entry uses multi-scale gradient+NCC search, and the metadata-constrained entry uses a fixed template at the ground-truth box size. The unconstrained direct VLM entry asks for a full four-parameter box, and the metadata-constrained entry asks for a normalized top-left position over the valid placement range. The unconstrained proposal-and-verify entry uses a coarse-to-fine zoom workflow, and the metadata-constrained entry uses the best image-based scoring policy fitted on the controlled dataset itself. As a transfer diagnostic, the metadata-constrained policy selected on challenging yields 0.11 [0.03, 0.20] when applied to controlled.

Method	Unconstrained mean (median; 95% CI)	Metadata-constrained mean (median; 95% CI)
Image-center prior	0.25 (0.07; 0.06–0.48)	0.25 (0.07; 0.06–0.48)
Random-box prior	0.06 (0.02; 0.01–0.13)	0.06 (0.02; 0.01–0.13)
Constant prior	0.20 (0.09; 0.03–0.42)	0.20 (0.09; 0.03–0.42)
DINOv2 dense features	0.20 (0.00; 0.00–0.40)	0.20 (0.00; 0.00–0.40)
multiGradICON pretrained	0.19 (0.00; 0.02–0.37)	0.19 (0.00; 0.02–0.37)
Classical template matching	0.00 (0.00; 0.00–0.00)	0.01 (0.00; 0.00–0.02)

Appendix

Table E2: Summary statistics for localization IoU on the challenging dataset (

n{=}9

small-target pairs), both settings. Confidence intervals are bootstrap 95% intervals over image pairs. The unconstrained classical entry uses multi-scale gradient+NCC search, and the metadata-constrained entry uses a fixed template at the probe-metadata-derived box size. The unconstrained direct VLM entry asks for a full four-parameter box, and the metadata-constrained entry asks for a normalized top-left position over the valid placement range. The unconstrained proposal-and-verify entry uses a coarse-to-fine zoom workflow, and the metadata-constrained entry uses the edge-chamfer scoring policy with NMI fallback selected via leave-one-pair-out on the challenging dataset.

Dataset	Pair	Area	Center	Random	Const.	DINOv2	mGICON	Unconstrained			Metadata-constrained
		(%)						Class.	Direct	P&V	Class.	Direct	P&V
Controlled	60fb54	1.8	0.00	0.02	0.56	0.21	0.00	0.60	0.00	0.00	0.97	0.00	0.00
	6dfe75	1.1	0.00	0.01	0.00	0.00	0.00	0.00	0.11	0.00	0.89	0.13	0.47
	85e863	1.6	0.00	0.01	0.00	0.10	0.00	0.00	0.00	0.19	0.90	0.07	0.21
	97acdf	1.2	0.00	0.01	0.00	0.00	0.00	0.00	0.01	0.00	0.97	0.08	0.02
	e1c7e5	1.5	0.00	0.01	0.00	0.00	0.00	0.00	0.11	0.25	0.89	0.21	0.30

Appendix

Table E3: Pair-level IoU values for evaluated methods, with one row per pair from each dataset. Geometric controls and pretrained baselines are setting-invariant under the evaluated box-size convention. Classical template matching, direct VLM, and proposal-and-verify are reported separately for unconstrained and metadata-constrained search. Setting and policy choices per cell are documented in Tables E1 – E2 and Table D1 .

Figure E1: Qualitative atlas for the evaluated pairs and both search settings, visualizing the direct VLM and proposal-and-verify columns of Table E3 . Each optical reference shows ground truth, direct VLM prediction, and proposal-and-verify prediction using the same color convention as Fig. 3 . Outcome codes distinguish rescue (RS), refinement (RF), regression (RG), unchanged (EQ), and shared failure (FL) cases, matching the legend rendered on the figure.

Dataset	Setting	Direct	P&V	$p$ (P&V $>$ direct)	$p$ (P&V $>$ center)
Controlled	unconstrained	0.05	0.09	$0.250$	$0.250$
Controlled	metadata-constrained	0.10	0.20	$0.125$	$0.062$
Challenging	unconstrained	0.11	0.19	$0.125$	$0.648$
Challenging	metadata-constrained	0.19	0.35	$0.031$	$0.266$

Appendix

Table E4: Statistical comparisons for the proposal-and-verify workflow in all four dataset–setting cells. Paired tests are exact one-sided sign-flip tests over image pairs.

Cross-Modal Image Registration

Cross-modal image registration aligns images from different sensors or modalities into a shared spatial frame.

See all 70 papers

Acquisition Geometry-Assisted Whole-Group Localization of X-ray Fluorescence Maps in Optical Microscopy Images
Aug 18, 2026Xiangyu Yin, Tatjana Paunesku, Letonia Copeland-Hardin +7Microscopy Image Analysis Cross-Modal Image Registration
Advanced Photon Source, Argonne National Laboratory, Lemont, Illinois, USA · Northwestern University, Chicago, Illinois, USA · University of Chicago, Chicago, Illinois, USA +1
Localize Any Object in X-Ray Security Scans without Human Annotation
Oct 5, 2026Yaqi Cai, Mingxuan Liu, Lorenzo Vaquero +5Self-Supervised Learning Synthetic Data Generation
University of Trento · Fondazione Bruno Kessler · Dalian University of Technology +1
Vision-Language Grounding as Bidirectional Concept Correspondence
Aug 8, 2026Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer +1Vision-Language Grounding Vision-Language Alignment
University of Washington · Allen Institute for AI · FAIR at Meta

Aug 18, 2026J/K move · Enter open · S save

Acquisition Geometry-Assisted Whole-Group Localization of X-ray Fluorescence Maps in Optical Microscopy Images

Xiangyu Yin, Tatjana Paunesku, Letonia Copeland-Hardin, Martina Ralle+6

Advanced Photon Source, Argonne National Laboratory, Lemont, Illinois, USA · Northwestern University, Chicago, Illinois, USA · University of Chicago, Chicago, Illinois, USA+1

Microscopy Image Analysis Cross-Modal Image Registration

Open paper AskPDF arXiv

X-ray fluorescence (XRF) microscopy maps elemental distributions, while optical microscopy can provide complementary morphological context. Localizing XRF fields of view (FOVs) in optical images is difficult because the two modalities differ in contrast mechanism and resolution. Most current workflows place each XRF tile independently, even when acquisition metadata already record the tiles' relative scan positions. This study formalizes XRF tile-group localization, in which one optical-frame placement is estimated for the whole group, constrained by acquisition geometry and quantified using group intersection-over-union (GroupIoU). In a controlled case study, independent localization failed with GroupIoU 0.000, whereas group localization achieved 0.931. Replacing the normalized cross-correlation (NCC) metric with mutual information (MI) gave nearly identical results, showing that the outcome is not specific to one local similarity metric. In another multiscale case study, using a coarse XRF survey scan to connect the fine-scale tile group to the optical image increased mean GroupIoU from 0.694 to 0.856. These case studies support using acquisition geometry as an explicit constraint when localizing related XRF tiles.

Abstract

Cross-Modal Image Registration

Explore similar work

Acquisition Geometry-Assisted Whole-Group Localization of X-ray Fluorescence Maps in Optical Microscopy Images

Localize Any Object in X-Ray Security Scans without Human Annotation

Vision-Language Grounding as Bidirectional Concept Correspondence