cs.CVSep 26, 2026

Attribution Gaps in Zero-Training LLM+OVOD Pipelines: A Fine-Grained Analysis of the CAAP--SNAP Discrepancy

Authors: Yu-Feng Yen

Organizations: Independent Researcher Taiwan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LAOD and similar zero-training LLM+open-vocabulary-detector (OVOD) pipelines score two things separately: class-agnostic localization accuracy (CAAP) and semantic naming accuracy (SNAP). The two consistently diverge, and nobody has asked why. This paper asks why, on the full 5,000-image COCO-Val split (27,273 detections) rather than the small subset the original work evaluated on. Object visual complexity turns out not to be the driver -- small and occluded objects are, if anything, localized better than large ones. Vocabulary novelty is: once the LLM's wording falls outside the detector's native category set, localization accuracy falls from 80.9% to 31.6%. That drop is not spread evenly across unfamiliar phrasing, though. Almost all of it comes from cases where the novel wording actually names a different object than the one COCO annotated (true synonyms still score 89.3%; semantically unrelated "noise" labels score 12.0%). A closer look at a further failure subset tells a similar story: 78-88% of what looks like complete localization failure is really the model correctly finding a real object that COCO's non-exhaustive 80-category scheme simply never labeled, not hallucination. Swap the detector backbone (YOLO-World for Grounding DINO) or the LLM (Gemma-3 for Qwen2.5-VL) and both the effect and its rough size hold up, so this looks like a general property of the pipeline family rather than a quirk of one model pairing. The upshot is that a large share of the apparent CAAP--SNAP gap traces back to closed-category annotation limits rather than a real grounding failure, which matters for how we detect hallucination, analyze failure modes, and design evaluation for grounded multimodal systems meant to work in the open world.

Figures & tables

Figure 1: Localization accuracy by vocabulary commonness. Out-of-vocabulary labels see a >2.6×>2.6\times drop relative to native COCO-80 vocabulary.
Figure 2: Failure-mode decomposition of the 11,981 detections where SNAP judged the label correct but CAAP judged the box incorrect. The dominant mode is “right object, loose box.”
Categorynn (matched detections)Weighted CAAP-TP rateRepresentative labels
synonym1,87689.3%man, sofa, phone, television, horses
finer_grained1,58265.8%table, woman, flowers, boy, faucet
noise5,93911.3%plate, cabinet, helmet, hat, shoes
Table 1: Vocabulary deepdive: CAAP-TP rate by semantic relation to ground truth (150 most-frequent out-of-vocabulary labels).
Figure 3: Out-of-vocabulary labels by semantic relation to ground truth (full coverage, 99.3%). Dashed line marks the native COCO-80 baseline (80.9%).
Categoryn=60n=60n=200n=200 (merged)
gt_gap (model correct; COCO did not annotate)46 (76.7%)167 (83.5%)
real_failure (model genuinely mislocalizes)8 (13.3%)15 (7.5%)
ambiguous6 (10.0%)18 (9.0%)
Table 2: Manual classification of the “no overlap” subset.
Figure 4: Manual inspection of the “no overlap” subset ( n=200n=200 ). Error bar shows the Wilson 95% CI for gt_gap.
MetricYOLO-World + Gemma-3Grounding DINOQwen2.5-VL 7B
native CAAP-TP rate80.9%81.1%80.7%
out-of-vocabulary CAAP-TP rate31.6%26.0%29.4%
native–OOV gap∼\sim 49.3 pp∼\sim 55.0 pp∼\sim 51.3 pp
Table 3: Native vs. out-of-vocabulary CAAP-TP rate across detector and LLM swaps (uniform denominator).
More in this topic

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Explore similar work

CardsList
Jul 14, 2025

MLLM-Guided Autonomous Object Detection for Open-World Understanding

Object detection typically relies on predefined category vocabularies, while open-world and open-vocabulary approaches remain limited by generic unknown labels or externally specified detection concepts. We propose LAOD, an MLLM-Guided Autonomous Object Detection framework in which a Multimodal Large Language Model (MLLM) generates an image-specific detection vocabulary that is grounded by an open-vocabulary detector. This decomposition separates semantic concept discovery from spatial localization and removes the need to manually enumerate target categories at inference time. We evaluate LAOD using Class-Agnostic Average Precision (CAAP), Semantic Naming Average Precision (SNAP), and Semantic Localization Average Precision (SemLoc-AP), which measure localization, semantic naming, and their correct association, respectively. Experiments on COCO-Val, LVIS-minival, and COCO-OOD across multiple MLLMs and detectors demonstrate strong performance across common, long-tailed, and out-of-distribution objects, while showing that grounding quality remains a major determinant of end-to-end performance. LAOD also substantially outperforms both direct MLLM-based object detection and dedicated models trained specifically for open-ended detection. Overall, the results support a modular formulation of autonomous object detection in which MLLMs determine what to detect and specialized open-vocabulary detectors determine where those concepts occur.

Furkan Mumcu, Michael J. Jones, Anoop Cherian +1

University of South Florida · Mitsubishi Electric Research Laboratories (MERL)

Open-Vocabulary Object DetectionVision-Language GroundingLarge Vision-Language Models
AskPDF
Jul 7, 2026

Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention

Multimodal large language models can emit localized predictions, bounding boxes for objects and temporal windows for video and audio events, but they hallucinate these regions prolifically. The model's own token log-probabilities are nearly uninformative: they conflate grounding quality with input ambiguity, and coordinate tokens become near-deterministic once the model commits. We propose Multi-Token Localized Attention (MTLA): a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim. Prior attention-based detectors, which sum attention over the entire input modality and read a single response token, are weaker special cases; we show that summing only within the claimed region and aggregating across all prediction tokens recovers a stronger grounding signal. The same recipe applies almost trivially to other modalities and tasks: object detection in images and temporal localization in video and audio. Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline. Used as a confidence score for re-ranking, it nearly doubles the zero-shot COCO detection AP of an open-source 8B generalist (from 20.4 to 37.0), narrowing the gap to supervised detectors without any task-specific training.

Daniel Shalam, Emanuel Ben Baruch, Avi Ben Cohen +1

Amazon

Multimodal GroundingLLM Uncertainty EstimationMultimodal Model EvaluationVLM Hallucination Mitigation
AskPDF
Jul 31, 2026

Is It Time for the Renaissance of Salient Object Detection in the Era of MLLMs?

The zero-shot capabilities of multimodal large language models (MLLMs) are pushing salient object detection (SOD) beyond task-specific supervision. To disentangle MLLMs beyond conventional mask-based evaluation, we decompose SOD into localization and segmentation, and re-engineer datasets with phrases, boxes, and attributes, establishing a diagnostic benchmark for MLLM saliency perception (SaliLLM). SaliLLM uncovers a striking capability mismatch: MLLMs outperform state-of-the-art (SOTA) methods in localization, yet remain substantially weaker in segmentation. Further analyses attribute this gap primarily to mismatches between MLLMs and annotations over foreground cardinality, granularity, and extent. Motivated by this diagnosis, we recast zero-shot SOD as protocol-aligned Foreground Organization and introduce the first training-free framework that leverages Gestalt-inspired Collaborative attention for Unified SOD (FOCUS). FOCUS couples top-down Bayesian-surprise calibration of protocol-conditioned foreground granularity with bottom-up propagation of MLLMs evidence over entity-centric perceptual manifolds induced by self-supervised features, yielding coherent object extents as prompts for a general segmenter. Across 13 RGB, RGB-D, and RGB-T SOD benchmarks, FOCUS generally surpasses SOTA methods without training, reducing mean absolute error by 11%, 34%, and 48% compared with fully, weakly, and self-supervised methods, respectively. Our findings signal the renaissance of SOD: from task-specific supervision to zero-shot foreground organization. Code is available in the supplementary material.

Wenzhuo Zhao, Xiuzhi Li, Zhongkuan Mao +6

College of Computer Science, Sichuan University, China · National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, China · School of Information and Communication Engineering, University of ElectronicScience and Technology of China, China

VLM EvaluationSalient Object DetectionMultimodal Large Language ModelsVisual Saliency
AskPDF