From Routing Signals to Selective Review: Visual regrounding in MoE VLMs

Hongzhu Guo; Mohsen Fayyaz; Nanyun Peng

cs.CLSep 29, 2026

From Routing Signals to Selective Review: Visual regrounding in MoE VLMs

Authors: Hongzhu Guo, Mohsen Fayyaz, Nanyun Peng

Organizations: University of California, Los Angeles · Peking University

Abstract

Vision-language models (VLMs) may accept false visual premises, answering questions about a target object's color, count, location, or state even when it is absent. We call this reliability-critical behavior a target-absence grounding failure. Existing visual-grounding detectors primarily rely on generated responses, hidden states, or uncertainty measures. We present the first framework to leverage internal routing decisions in Mixture-of-Experts (MoE) VLMs to detect target absence before generation and guide selective correction. We extract target-token routing probabilities from Qwen3-VL-30B-A3B-Instruct and Gemma-4-26B-A4B-it, train a separate L2-regularized linear detector for each model, and use its predictions to selectively invoke a target-aware review prompt. Using routing alone, the Qwen and Gemma detectors achieve ROC-AUCs of 0.9988 and 0.9956 on GQA-Inpaint and retain 0.8095 and 0.7781 on the external OBER dataset, respectively. The resulting routing-gated policy improves end-to-end accuracy on GQA-Inpaint and OBER by +22.25% and +12.17% for Qwen, and by +13.42% and +1.39% for Gemma, without modifying model weights. Further analysis shows that the signal is localized to the target-object token, emerges in early MoE layers, and is distributed across partially substitutable experts. Although cross-dataset threshold shifts require recalibration, false-positive review causes limited harm overall, suggesting that intervention risk can be controlled through joint selection of the detector threshold and review prompt. Overall, we show that routing probabilities alone preserve actionable information about visual perception, allowing computation already produced by an MoE VLM to support low-cost detection and selective visual regrounding.

Figures & tables

Figure 1: Overview of routing-gated visual regrounding. A linear detector is trained offline on target-token routing probabilities collected across MoE layers and experts. At inference time, the detector scores routing from the standard-prompt prefill before answer generation. Predicted-present inputs follow standard decoding, while predicted-absent inputs trigger a target-aware prompt that asks the unchanged VLM to verify the queried object’s presence and explicitly report its absence when unsupported.

Figure 2: Routing-detector ROC-AUC and accuracy across models and datasets. Each model uses its GQA validation threshold on both datasets. The manually audited GQA set includes validation images and is used to test the review policy. Detection remains informative across datasets, with thresholds requiring recalibration for transfer.

Representation	ROC-AUC	Accuracy
Input embedding	0.5000	50.00%
Mean of hidden layers	0.9979	98.06%
Last hidden layer (L47)	0.9980	98.16%
All-layer routing	0.9988	98.54%

Table 1: Qwen detection on the GQA-Inpaint held-out test split. All probes use validation-selected regularization and thresholds.

Figure 3: Test ROC-AUC on GQA-Inpaint using the top

K

routing features. Performance nearly plateaus at

K=500

Figure 4: End-to-end accuracy of standard answering and routing-gated review. Accuracy is computed jointly over target-present and target-absent inputs. Target-aware review yields positive gains for both models and both datasets, while generic review is consistently weaker.

Dataset	Target	Standard	Always	Selective
GQA	Absent	37.17%	83.17%	82.50%
GQA	Present	96.33%	89.33%	95.50%
GQA	Total	66.75%	86.25%	89.00%
OBER	Absent	66.78%	95.83%	95.83%
OBER	Present	100.00%	94.78%	95.30%
OBER	Total	83.39%	95.30%	95.57%

Table 2: Qwen target-aware accuracy by true target presence. Always review replaces every answer; selective review retains the standard answer when the gate does not trigger.

Figure 5: Correct-to-wrong harm after false review triggers. Bars show the percentage of initially correct answers on target-present images that become wrong after review; labels give the numerator and denominator. Qwen has 2 false-triggered images in GQA and 79 in OBER; Gemma has 2 and 36, respectively. Each image has five sampled answers.

Rank	Expert	Mean weight
1	L04.E067	$-0.2675$
2	L02.E013	$+0.2274$
3	L45.E063	$-0.2225$
4	L02.E055	$+0.1949$
5	L13.E018	$+0.1821$
6	L02.E089	$-0.1817$

Table 3: Highest-ranked experts by absolute mean standardized weight. Signs indicate the direction of the absence score.

Figure 6: Routing analysis in Qwen. (a) Standardized coefficients of the full 6,144-feature probe, averaged across three training seeds. Red and blue indicate positive and negative weights; the color scale is clipped at the 99th percentile of absolute weights. (b) Test ROC-AUC of 48 single-layer probes. The dashed line shows the full 48-layer probe and the marker highlights Layer 2.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Cohort	Pairs	Images	Role
GQA training, automatic QC	20,000	40,000	Probe fitting
GQA validation, automatic QC	5,446	10,892	Model and threshold selection
GQA held-out test, automatic QC	5,776	11,552	Frozen detector test
GQA manually audited review cohort	120	240	Same-domain policy evaluation
OBER manually audited primary cohort	115	230	External detector and policy evaluation

Appendix

Table 4: Cohorts and their roles. Image counts include both present and absent conditions. The 120-pair GQA review cohort includes validation examples; held-out detector performance is measured on the separate 5,776-pair GQA test split.

Setting	Qwen	Gemma
Routing features	6,144	3,840
Feature statistics	Per-feature GQA-train mean/std	Per-feature GQA-train mean/std
Loss and optimizer	BCE; AdamW	BCE; AdamW
Learning rate; batch size	$10^{-3}$ ; 512	$10^{-3}$ ; 512
Maximum epochs; patience	30; 5	30; 5
Epoch selection	Best validation ROC-AUC	Best validation ROC-AUC

Appendix

Table 5: Probe training and selection settings. Each model uses the same GQA-Inpaint pair splits (20,000 train, 5,446 validation, and 5,776 test). Standardization and model selection use only training and validation data; thresholds remain frozen on the held-out GQA test and OBER.

Condition	Added instruction before the question
Standard	None
Generic	“Review the image carefully before answering.”
Target-aware	“Before answering, carefully verify whether the image actually contains any {target} . If it does not, explicitly state that the target is absent.”

Appendix

Table 6: Prompt instructions prepended to the unchanged question and short-answer request. Standard generation has no added instruction. The generic and target-aware rows reproduce the instruction text used for review; Qwen and Gemma differ only in the separator between the target-aware sentences.

Model	Cohort	ROC-AUC	Accuracy	Absence recall	Specificity	Trigger rate
Qwen	GQA held-out test	0.9988	98.54	98.96	98.11	50.42
Qwen	GQA audited review	0.9995	98.75	99.17	98.33	50.42
Qwen	OBER external	0.8095	65.22	99.13	31.30	83.91
Gemma	GQA held-out test	0.9956	96.68	97.07	96.30	50.39
Gemma	GQA audited review	0.9998	99.17	100.00	98.33	50.83
Gemma	OBER external	0.7781	70.87	73.04	68.70	52.17

Appendix

Table 7: Detector performance with the GQA-validation-selected threshold frozen for both datasets. Accuracy, recall, specificity, and trigger rate are percentages. GQA audited is a policy cohort that contains validation images.

Layer	ROC-AUC	Accuracy	Layer	ROC-AUC	Accuracy
0	0.9948	97.22	24	0.9956	97.00
1	0.9980	98.24	25	0.9962	97.48
2	0.9986	98.54	26	0.9961	97.27
3	0.9987	98.69	27	0.9955	97.05
4	0.9987	98.61	28	0.9953	96.95
5	0.9987	98.61	29	0.9948	96.73

Appendix

Table 8: Qwen target-token hidden-state probes on the GQA-Inpaint held-out test split. Each probe uses one decoder layer and the same data splits as the routing probe. Regularization and thresholds are selected on validation data. Layer indices start at zero; accuracy is a percentage.

Model	Threshold	Balanced acc.	Recall	Specificity	Trigger rate
Qwen	Frozen GQA	65.22	99.13	31.30	83.91
Qwen	OBER split-calibrated	77.75	94.86	60.64	67.11
Gemma	Frozen GQA	70.87	73.04	68.70	52.17
Gemma	OBER split-calibrated	73.19	68.61	77.77	45.42

Appendix

Table 9: OBER threshold sensitivity. Calibration uses 200 pair-grouped 50/50 calibration–evaluation splits; the split-calibrated values are held-out means. All rate columns are percentages.

Figure 7: Gemma validation ROC-AUC after 20 independently shuffled-label runs. The dashed line marks chance; the red line marks the original single run.

Model	Dataset	Standard	Generic review	Target-aware review	Target-aware gain
Qwen	GQA	66.75	67.67	89.00	+22.25
Qwen	OBER	83.39	89.57	95.57	+12.17
Gemma	GQA	80.00	80.75	93.42	+13.42
Gemma	OBER	93.91	94.09	95.30	+1.39

Appendix

Table 10: End-to-end answer accuracy for standard generation and two routing-gated review prompts. All values except gain are percentages.

Figure 8: Qwen target-aware accuracy gains across five sampled generations per image. Points show individual repetitions; diamonds show dataset means.

Dataset	Prompt	Always acc.	Selective acc.	Review rate
GQA	Generic	67.25%	67.67%	50.42%
GQA	Target-aware	86.25%	89.00%	50.42%
OBER	Generic	89.39%	89.57%	83.91%
OBER	Target-aware	95.30%	95.57%	83.91%

Appendix

Table 11: Qwen selective versus always review with manually checked answers. Accuracy combines target-present and target-absent inputs. Always review has a 100% review rate; the final column gives the selective rate.

Dataset	Target	Review prompt	Standard	Always	Selective
GQA	Absent	Generic	37.17	39.00	39.00
GQA	Present	Generic	96.33	95.50	96.33
GQA	Absent	Target-aware	37.17	83.17	82.50
GQA	Present	Target-aware	96.33	89.33	95.50
OBER	Absent	Generic	66.78	79.83	79.83
OBER	Present	Generic	100.00	98.96	99.30

Appendix

Table 12: Qwen accuracy (%) by true target presence. Each condition has 600 sampled answers on GQA or 575 on OBER. Selective review keeps the standard answer when the gate does not trigger; always review replaces every answer.

Figure 9: Answer accuracy on target-absent images that trigger review, shown separately for GQA and OBER. Within each model and dataset, all three prompts use the same triggered subset with five answers per image. Gains are target-aware accuracy minus standard accuracy, in percentage points.

Model	Dataset	Generic review	Target-aware review
Qwen	GQA	0/10 (0.00%)	5/10 (50.00%)
Qwen	OBER	4/395 (1.01%)	27/395 (6.84%)
Qwen	Combined	4/405 (0.99%)	32/405 (7.90%)
Gemma	GQA	0/10 (0.00%)	0/10 (0.00%)
Gemma	OBER	10/178 (5.62%)	15/178 (8.43%)
Gemma	Combined	10/188 (5.32%)	15/188 (7.98%)

Appendix

Table 13: Correct-to-wrong changes among initially correct answers on false-triggered target-present images. Each cell shows the number of harmed answers over the eligible answer count. Five answers from one image are correlated.

Layer	ROC-AUC	Accuracy (%)	Layer	ROC-AUC	Accuracy (%)
0	0.9006	83.19	24	0.9377	86.50
1	0.9360	87.50	25	0.9599	89.34
2	0.9836	94.33	26	0.9489	87.69
3	0.9454	89.08	27	0.9359	86.28
4	0.9618	90.74	28	0.9407	86.92
5	0.9559	89.58	29	0.9385	86.42

Appendix

Table 14: Test results for all 48 Qwen layer-wise classifiers. Each classifier uses 128 routing probabilities from one layer. Model selection and thresholds use validation data only. Accuracy is reported as a percentage.

Explore similar work

Jul 30, 2026cs.CV

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

Large vision-language models (LVLMs) often hallucinate objects that are absent from an image. Despite recent progress, existing mitigation methods still lack reliable object-level grounding diagnostics and therefore tend to apply coarse-grained interventions, which can impair visual understanding, shorten responses, and reduce coverage of genuinely grounded objects. The key challenge is thus to detect, during generation, whether each emerging object mention is supported by reliable visual evidence, so that hallucination can be mitigated selectively. Yet output confidence reflects next-token plausibility rather than visual support, allowing language priors to make absent objects appear certain. We show that the missing diagnostic evidence is encoded in an Intrinsic Grounding Signature (IGS), a distributed signed attention pattern that remains informative for such confident hallucinations. Based on IGS, we propose Verifier-Guided Decoding (VGD), a decoding framework in which a lightweight verifier examines each emerging object mention, rolls back the KV cache when the mention is identified as high risk, suppresses the object and its synonyms, and regenerates the affected continuation. Because VGD intervenes only on object mentions identified as high risk, it reduces object hallucination while preserving the model's original visual understanding and grounded object coverage. Experiments on CHAIR and AMBER-G show that VGD achieves state-of-the-art object hallucination reduction: at @rec90, it cuts AMBER-G CHAIR by 43.6% while retaining 99.6% of grounded-object coverage, and reduces CHAIR-MSCOCO CHAIR

_i

/CHAIR

_s

by 37.0%/30.4% without shortening captions.

Lei Yang, Xinze Liu, Dayan Wu +7

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA

Object Hallucination Visual Evidence Objects Decoding Signatures

Ask PDF

Aug 31, 2026cs.CV

VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs

Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding. Recent training-free detectors use internal signals such as token likelihood, attention, visual confidence, or image-text similarity to identify hallucinated objects. These signals are useful, but they are often source-confounded. They measure how strongly an object is supported inside the model without distinguishing whether that support comes from object-specific visual evidence or the generated text prefix. In difficult cases, a hallucinated object can still receive high internal support because it fits the scene, is associated with nearby visual cues, or follows naturally from the generated text prefix. We propose VisER, a training-free two-sided metric for object-level hallucination detection. VisER evaluates each generated object mention from two complementary views. Visual Evidence measures whether object-context compatibility is backed by object-specific evidence from image tokens. Visual Reliance measures whether the object is supported more by the image than by the generated prefix. Combining these views gives a more source-aware grounding score, while avoiding additional object-level verification generations. Across multiple LVLMs and benchmarks, VisER improves AUROC and AUPR over a range of baselines.

Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie +1

School of Computing and Information Systems The University of Melbourne Melbourne, Australia

Citation Hallucination Detection Visual Evidence Vista Objects

Ask PDF

Jun 11, 2026cs.CV

Mirage Probes: How Vision Models Fake Visual Understanding

Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.

Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5

Columbia University · Intuit · Technion +2

Weak Visual Grounding Vision Foundation Models Representational Capacity Prior Knowledge Integration

Ask PDF