Sckwoky at SHROOM-Visions: Character-Level Hallucination-Span Detection in Image-Conditioned VLMs via Dual-Prompted LoRA Decoders
Organizations: Novosibirsk State University
Abstract
We present our system for the SHROOM-visions 2026 shared task on character-level hallucination span detection in image-conditioned vision-language model (VLM) responses, evaluated across English, French, Italian, and Chinese. The detector is built on a frozen Gemma-4-12B-it backbone with trainable LoRA adapters (approximately 10.7M parameters). Because the backbone is a causal decoder, hidden states read from a single copy of an answer cannot attend to subsequent tokens; we therefore repeat the candidate answer within one prompt and extract features from its second occurrence, giving every position approximate bidirectional context in a single non-autoregressive pass. A character-level locator combines intermediate hidden states to predict hallucination probabilities, BIO boundaries, and a response-level abstention score, while separate language-specific classifiers assign one of five hallucination types to each predicted span. On the closed test set, our system reaches a macro-averaged character IoU of 0.4404, more than doubling the official baseline (0.1854), and ranks 5th-6th on IoU across the four languages. On the English tuning split, controlled ablations attribute a gain to the repeated answer that is comparable to that of LoRA adaptation (its held-out gain is smaller and concentrated on hallucinated answers), and they show that the detector relies on the image-answer pairing: shuffling images across examples drives IoU down to the level of the predict-nothing floor or below it in every language. In deployments where the same Gemma-4-12B-it model also generates answers, its base weights can be reused, making the approach practical for local question-answering systems that cannot run a separate verifier.
Figures & tables
| Track | System | Cor+Lbl | Cor | IoU |
|---|---|---|---|---|
| EN | Baseline | 0.1549 | 0.1549 | 0.1549 |
| Our system | 0.3129 | 0.4491 | 0.4189 | |
| FR | Baseline | 0.1679 | 0.1679 | 0.1679 |
| Our system | 0.3181 | 0.4378 | 0.4468 | |
| IT | Baseline | 0.1635 | 0.1635 | 0.1635 |
| Our system | 0.3373 | 0.4926 | 0.4293 |
| Configuration change (dev set) | IoU |
|---|---|
| Add LoRA adapters (vs. frozen backbone) | |
| Add visual-token cross-attention | |
| Joint multilingual training (ZH dev) | |
| Single answer copy (vs. echo), tune-202 | |
| held-out-157 | |
| hallucinated answers only, tune-202 |
| Split | Floor | Correct | Shuffled | No image |
|---|---|---|---|---|
| EN tune | 0.213 | 0.422/0.453 | 0.183/0.130 | 0.297/0.332 |
| EN held-out | 0.210 | 0.381/0.423 | 0.211/0.169 | 0.271/0.311 |
| FR dev | 0.297 | 0.450/0.437 | 0.224/0.143 | 0.347/0.339 |
| IT dev | 0.289 | 0.416/0.450 | 0.299/0.318 | 0.366/0.403 |
| ZH dev | 0.355 | 0.500/0.523 | 0.225/0.223 | 0.398/0.414 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Split | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| EN tune-202 | 0.416 | 0.421 | 0.422 | 0.405 | 0.397 |
| EN held-out-157 | 0.386 | 0.388 | 0.381 | 0.373 | 0.364 |
| FR dev | 0.448 | 0.449 | 0.450 | 0.451 | 0.440 |
| IT dev | 0.418 | 0.419 | 0.416 | 0.415 | 0.404 |
| ZH dev | 0.515 | 0.513 | 0.500 | 0.477 | 0.458 |
Hallucination Detection in VLMs
Hallucination detection in vision-language models identifies claims that are unsupported by images, video, or other visual evidence.
See all 77 papers