Grounding a text query in an image needs a matrix scoring every term against every patch. Vision-language models compute it with an aligned text encoder, which many image encoders, such as histopathology-related models trained on slides alone, lack. The Caption-Anchored Patch-Term Occurrence Representation (CAPTOR) estimates the matrix without a text encoder. It clusters a frozen image encoder's patch tokens into visual words and describes each captioned image by its share of patches in each word. A term signature is the standardized difference of those shares between images whose captions mention the term and the rest. The matrix factorizes into a term-by-word table and a query-independent image factor, so an image is encoded once and a new term adds one table row. Against every training-free baseline, CAPTOR leads on Common Objects in Context (COCO), Pascal Visual Object Classes (VOC) and Cityscapes. On COCO, scored on the cells one object owns, it reaches 0.565 mean intersection over union (mIoU) against 0.502. On all three, it is also level with a text alignment trained on the same backbone, which needs gradients, a text encoder and several times the captions. On held-out breast-cancer slides, it exceeds the strongest baseline scoring patches with a pathology text encoder by 0.160 mIoU. Read through CAPTOR, UNI, an encoder trained on slides alone, exceeds such baselines by 0.132 on a common grid. Counting can therefore give a text interface to any image encoder trained without language, wherever captioned images exist.
Figures & tables
Figure 1: The method, drawn from its own artifacts rather than as a schematic. Offline preparation. Patch features of 15,000 captioned COCO images are clustered into K=640 visual words, and each term signature is counted from them. The table is the factor S of Equation ( 1 ), one row per term and one column per word. Red means a word covers more of an image when captions mention the term and blue less, in multiples of what shuffled captions produce. Per-image inference. One forward pass assigns each patch of a new image to its nearest word on the encoder’s 14×14 grid, by the rule of Equation ( 2 ) that described the corpus. That word map is Φ , and scoring a term at a patch reads the row of the term in S at the word of the patch, the product SΦ entry by entry. The table holds five terms, the row shows the three the image contains, and the three maps share one color scale.
Figure 2: The permutation null and the ceiling on COCO, drawn with one vocabulary and the hard assignment. (a) The permutation null of Equation ( 4 ) for a common term and a rare one, where π is the fraction of captions that name a term. “Person”, with π=0.465 , reaches 0.631 against a null of 0.060 . “Handbag”, named in 34 of 15,000 captions, with π=0.002 , reaches 3.57 against 1.85 . On the raw scale the rare term is more than five times larger. Measured in multiples of its own null, however, it is the weaker of the two, at 1.9 times its null against 10.5 for “person”. (b) The largest raw signature of every term, its null and the ceiling of Equation ( 5 ). Both rise as a term becomes rare, and no term exceeds 0.84 of its ceiling.
Figure 3: Cityscapes at its full resolution of 2048×1024 , on four typical images, one per row. Among the reporting-half images whose annotation contains at least two of the seven object classes, we take the four whose CAPTOR score lies closest to the median, 0.59 . They show what a reader should expect rather than our best cases. Every method is drawn by the code that produced its row of Table 3 , at the configuration that row reports. The number in each panel is the mIoU of that image.
Figure 4: One row of the matrix, the score of a single term at every cell, on ten high-resolution photographs, each above its map. The term of each photograph is the one it was chosen for. Red is high and blue is low, and each map is standardized over its own cells, so color shows where the term applies rather than how strongly. Photographs from Wikimedia Commons, chosen by the rule the text gives and credited as follows. “Cat”, “dog” and “bicycle” are by Basile Morin, “train” by Kabelleger and David Gubler, “pizza” and “car” by Rhododendrites, “horse” by Alvesgaspar, “boat” by Norbert Nagel, “airplane” by Carlos Delgado, all CC BY-SA 3.0 or 4.0 , and “laptop” by PantheraLeo1359531, CC BY 4.0 .
Figure 5: Named segmentation of breast fields with K=80 , four vocabularies and the propagation. The columns are the field, its annotation, the term CAPTOR scores highest, and SCLIP reading the CONCH text encoder with the same propagation and standardization, the strongest baseline of Table 4 that does not borrow the words of CAPTOR. Each row shows the field whose intersection over union (IoU) for its tissue is the median among the fields that tissue dominates, so the panel shows the typical case. Maps are drawn on the 28×28 scoring grid, and each number is that field’s mIoU. CAPTOR is ahead on four of the five fields.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Two further steps of the method on COCO, from one vocabulary of the reported table. (a) The contrast Equation ( 3 ) counts, for “cat” and the word its signature is largest on: the share of each image in that word, for images whose captions name “cat” and for the rest, with each group’s mean marked. Images whose captions name “cat” give that word a mean share of 0.071 , against 0.00006 for the rest. (b) Why the score of Equation ( 15 ) blends two estimates. A joint word combines one word from each of the four vocabularies. The bars count joint words by the number of captioned images that contain them, and the red curve is the trust weight γℓ at λ=3 . Of the 178,828 joint words, 83% appear in three images or fewer, where γℓ is at most one half. Most patches therefore rely mainly on the better-supported average of the four vocabularies, and only frequent joint words are trusted on their own.
Figure S2: COCO as a named segmentation, at the reported configuration. Each column is one validation image, chosen from the annotation alone and never from any method’s output. It contains exactly two of the terms shown, each covering at least 35% of the center crop and together at least 70% , and no pair of terms appears twice. Row two is the term CAPTOR scores highest, and row three the same patches’ visual words with no names attached. Rows four to six are Talk2DINO, SCLIP and ProxyCLIP from their authors’ released code, and row seven is the annotation.
Figure S3: Which images each method ranks highest for a term. Each row is one term. The left four images are CAPTOR’s top four, and the right four are the top four of the released CLIP model. A green border means the image contains the term, and a red one means it does not. Terms are spread evenly from rare to common.
Figure S4: Score maps for one term at a time. Each row shows an image containing the term, CAPTOR’s map, the same term read by Talk2DINO, SCLIP and ProxyCLIP, and the annotation. Each image is the first validation image in which the term covers between a fifth and two thirds of the center crop, a rule fixed before drawing. Every map is rescaled over its own patches, so the colors compare shapes rather than strengths.
Figure S5: Pascal VOC read with the table counted on COCO captions, with nothing re-estimated for VOC and the joint estimate the development half picks. The three images are those on which the methods disagree most, among validation images with two or more terms each covering over a quarter of the center crop. This rule never looks at which method is right, so it can show rows that favor any method. Columns are the image, the term CAPTOR scores highest, the visual words themselves without names, Talk2DINO, SCLIP and ProxyCLIP from their released code, and the annotation. Only the cells the annotation labels, which are the ones scored, are colored. In the first row SCLIP gives almost the whole image to “person” and misses the chair that CAPTOR and ProxyCLIP both find.
Figure S6: COCO-Object images from the reporting half, chosen as typical, with every pixel scored and background as an eighty-first class. The four images were picked before any prediction was drawn, as those with two or more object classes whose CAPTOR mIoU is closest to the median of that pool, 0.344 . Every method runs with the code and setting of its row in Table S14 , and each number is that image’s mIoU. CAPTOR is ahead on the second image and behind at least one baseline on the other three. Over the four images it gives the right name to 70% of object pixels, against 26% to 53% for the baselines, but calls 20% of the background an object, against 3% to 8% . In the last row “train” spreads over the platform, the error Section S6 measures.
Tumor
Stroma
Lymphocyte
Necrosis
Adipose
CAPTOR, one vocabulary
0.670
0.497
0.434
0.266
0.125
Region-pooled, standardized
0.420
0.395
0.377
0.283
0.056
Dense (MaskCLIP)
0.267
0.456
0.331
0.146
0.122
CAPTOR, reported configuration
0.738
0.470
0.502
0.577
0.181
Supervised probe
0.810
0.683
0.584
0.679
0.304
Appendix
Table S16: Table S15 broken down by tissue, in IoU on the same fields. The first three rows are CAPTOR with one vocabulary at K=40 and two baselines from that table, and bold marks the best of the three. CAPTOR is best on four of the five tissues, and region-pooled scoring leads on necrosis, 0.283 against 0.266 . The fourth row is the configuration reported in Table 4 , K=80 with four vocabularies and the propagation, at 0.494 mIoU and 0.735 accuracy on all 200 fields, with the cell labels of Table S15 . Its largest gain over one vocabulary is on necrosis. The supervised probe, trained on the labels, is an upper reference.
Figure S7: Lung tiles from the LUAD-HistoSeg reporting patients, chosen as typical, with the settings of Table S18 . For each tissue, the row shows the tile, among those mostly of that tissue, on which CAPTOR on CONCH scores the median. Maps are drawn on the 14×14 grid that is scored, and each number is that tile’s mIoU. White cells are less than half annotated and are not scored. The baseline reads CONCH’s text encoder with CAPTOR’s propagation and standardization. CAPTOR on CONCH leads on the lymphocyte tile, is level on necrosis and falls behind on tumor and stroma. On the stroma tile both CAPTOR rows name most cells “lymphocyte” or “necrosis”, while the baseline names about half of them stroma.
Tissue
CAPTOR
CAPTOR
Region-
Self-
SCLIP
Dense
Re-
Hard
Soft
Pooled
Affinity
Encoding
Adipose
0.887
0.958
0.985
0.980
0.980
0.972
0.733
Necrosis
0.802
0.875
0.912
0.912
0.940
0.900
0.794
Lymphocyte
0.830
0.909
0.895
0.891
0.920
0.868
0.630
Stroma
0.676
0.753
0.833
0.836
0.788
0.819
0.603
Tumor
0.761
0.838
0.742
0.724
0.684
0.699
0.677
Appendix
Table S20: How well each method finds one named tissue at a time. This is easier than segmentation, because the scores of different terms never need to be compared. Each tissue is scored patch by patch against the BCSS annotation on the 200 fields of Table S15 , in AUC, where 0.5 is chance. Hard gives each patch one visual word and soft lets it belong to several (Equations ( 10 ) and ( 12 )), and the other columns are the text-encoder methods of Table S15 . Every method except re-encoding needs one pass of the encoder, and none uses labels. Intervals over patients run to about ±0.02 , and bold marks the best method in each row. The soft assignment raises CAPTOR from 0.791 to 0.867 , level with the best method, region-pooled scoring, at 0.873 . This suggests that CAPTOR’s lead in Table S15 comes from putting all terms on one scale rather than from finding each tissue better.
Figure S8: Score maps for one tissue term at a time on BCSS fields. Each field is the first in which the queried tissue covers more than a third of the annotation, a rule fixed before drawing. Row two is CAPTOR’s map, which looks up the term’s signature at each patch’s visual word. Row three is the same term read by the adapted CONCH encoder, and row four is the annotation for that term. All maps are rescaled to a common range.
Encoder
Dataset
CAPTOR
CONCH text
Adapted CLIP
CAPTOR − adapted
CONCH
Colorectal
0.835
0.684
0.773
+0.062
Lung
0.794
0.784
0.790
+0.004
19 organs
0.711
0.666
0.720
−0.009
PLIP
Colorectal
0.714
0.798
0.663
+0.051
Lung
0.767
0.777
0.756
+0.011
19 organs
0.685
0.730
0.657
+0.028
Appendix
Table S23: Whether counting can name the visual words of other pathology encoders as well as a text encoder can. Each of six encoders has its own K=40 visual words, and each method names every word with one of twelve tissue terms. The names are tested on three labelled datasets no method saw, colorectal (NCT-CRC), lung (LUAD-HistoSeg) and 19 organs (PanNuke), in mean AUC. CAPTOR names a word from the captions alone. CONCH text and adapted CLIP instead look at the PathCap figures where the word is most common and pick the term whose text embedding is closest. They use CONCH’s released text encoder or a CLIP text encoder trained on the same captions. The last column is CAPTOR minus adapted CLIP. CAPTOR is ahead in 10 of 12 cells on the four encoders trained with captions and in 2 of 6 on UNI and Phikon, which were trained without text. Overall it is level with the adapted encoder. So counting names any encoder’s visual words about as well as a text encoder trained on the same captions, without gradient training. Of the three routes, it is the only one that needs nothing beyond the encoder and the captions.
CAPTOR
CLIP text encoder (SCLIP, ProxyCLIP)
Talk2DINO
Learned values
16.1 M
63.4 M
72.0 M
Preparation
8.6 s, counting and k -means
Pretrained by OpenAI
CLIP pretrained, projection trained on COCO captions
Inference time
3.2 ms
37.3 ms
6.3 ms
Inference compute
15.4 GFLOPs
476.8 GFLOPs
41.7 GFLOPs
Appendix
Table S24: The term side of each method on COCO, measured on the same GPU, since every method encodes the image with a comparable backbone. Learned values are the numbers each method estimates from data, by gradient descent for the text routes and by clustering and counting for CAPTOR. Preparation is what must happen before the first query. CAPTOR builds its table from the 15,000 captions, after the image encoder pass all methods share. The text routes use CLIP’s text encoder as pretrained on about 400 million image and text pairs, and Talk2DINO adds a projection its authors trained on COCO captions. Inference makes one new term queryable, a text encoder pass over 80 prompts for CLIP and 7 for Talk2DINO, or counting the captions for CAPTOR. That count covers caption matching from a prebuilt word index, the signatures and their 200 permutation draws for four vocabularies, and the joint words, but not the caption-conditioned reweighting. So CAPTOR learns its values from captions in seconds, without gradients or a pretrained text model, and adds a term with about thirty times less computation than CLIP. Once a term is prepared, scoring an image against it takes every method a fraction of a millisecond. For scale, encoding an image takes 2.29 ms for CAPTOR and 2.17 ms for Talk2DINO at a batch of 32 . Labelling one image with all 78 terms takes 8.2 ms for CAPTOR against 387 ms for Talk2DINO’s released refinement.
Caption pairs
CAPTOR counting
Adapter training
Adapter steps
Adapter parameters
500
0.0014 s
16.6 s
600
1.25 M
1,000
0.0018 s
16.2 s
620
1.25 M
2,000
0.0033 s
16.2 s
620
1.25 M
4,000
0.0063 s
19.4 s
750
1.25 M
8,000
0.0123 s
41.5 s
1,500
1.25 M
32,463
0.0484 s
160.5 s
6,084
1.25 M
Appendix
Table S25: The time it takes to turn a set of captions into a naming table, the table that gives each visual word a name, for CAPTOR and for a CLIP text encoder adapted with low-rank adapters. Both first need the image embeddings of the captioned figures, and that shared step is left out. Both are timed on the same GPU. The table shows the part that grows with the number of caption pairs. For CAPTOR this is counting, and for the adapter it is training, with its steps and trained parameters. The adapter trains rank- 16 LoRA on the query and value projections of CLIP’s text encoder and a linear head, with a cosine regression loss, for six epochs or more where needed to reach at least 600 steps. CAPTOR also runs two steps only once, building its visual words with k -means ( 4.2 s) and the permutation nulls ( 0.12 s). Including them it needs 4.3 s in total, against 160.5 s of training and 1.3 GB of GPU memory for the adapter. On the CPU alone CAPTOR needs 57.2 s. Training uses batch size 32 , and counting times are the median of three repeats. Counting takes milliseconds, and even with its one-off steps CAPTOR builds the table far faster than the adapter trains, and still faster on the CPU alone.
An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still judge every image-caption pair in isolation. We ask what these neighboring captions contribute when a frozen encoder consults them at inference time. Fusing image similarity with similarity to an image's own captions improves text-to-image R@1 by 3.4 to 16.8 points across nine frozen encoders when the captions come from the same annotators as the queries. The gain falls to 1.9 to 7.2 points when an off-the-shelf captioner writes them, and to at most one point when they come from a different annotation source; a paraphrase-trained text encoder strengthens the text side further. A natural refinement, scoring the part of the image that the captions leave unexplained, appears to add much more under a leave-one-out evaluation. We show that this advantage arises because the correct candidate's supports are built differently from those of the negative candidates, with set size and query removal each contributing about half, and that under a fixed support index the residual reduces to a reweighting of what the captions already provide. On compositional benchmarks, the captions help by matching candidates to human descriptions of the image, without improving image-text grounding. The code is available at: https://github.com/nhatkhangcs/clip_consensus
Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging. Existing methods that combine dense captioning with pixel-level grounding often produce either incomplete descriptions or inaccurate segmentation masks. We study this problem through panoptic grounded captioning, a task that requires a VLM to describe both foreground objects and background regions while grounding each referring phrase with pixel-level masks. We make three contributions. First, we introduce PanoCaps, a human-annotated benchmark constructed from panoptic segmentation datasets. It provides dense captions with near-complete pixel coverage and image-text alignments at the entity level, supporting both training and evaluation. We further propose a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly evaluates textual and mask agreement. Second, we formulate phrase grounding as selection from a phrase-conditioned pool of mask proposals and introduce PANORAMA, a VLM that conditions a pretrained segmenter on contextualized phrase representations to obtain candidate masks and learns to select those corresponding to each phrase. Training this interface jointly with caption generation enables PANORAMA to produce high-quality masks while allowing each phrase to refer to a single region or multiple instances. Third, PANORAMA achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks. Experiments show that our method produces precise entity-level segmentations while maintaining detailed, mask-consistent captions. Code, data and models are available at https://www.di.ens.fr/willow/research/panorama/.
Sara Pieri, Evangelos Kazakos, Shizhe Chen +2
Inria, École normale supérieure, CNRS, PSL Research University · Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague
Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image (T2I) generators. As T2I models improve, benchmarking has become demanding, requiring metrics capable of finding a series of issues like missing objects, swapped attributes, miscounts, and ignored negations. Recent work addresses this by fine-tuning evaluators on preference data or by prompting a vision-language model, either holistically with the caption or with decomposed verification questions. However, existing approaches fall short: fine-tuned metrics remain bound to one backbone and training distribution; holistic metrics miss fine-grained details; and decomposed metrics rely on a fixed-YES assumption that penalizes faithful images whenever that assumption fails. In contrast, we propose DEPICT, a training-free metric that replaces fixed reference answers with expected agreement between image-based and caption-only answers, weighting questions by how decisively the caption determines them. By replacing fixed references, our agreement rule increases negation accuracy from 19% to 88%. To recover the context lost during decomposition, DEPICT merges this agreement score with a holistic score. We evaluate DEPICT on five benchmarks and eleven backbones from three model families and find that it surpasses all training-free metrics and exceeds fine-tuned evaluators on two out of three human-correlation benchmarks.