Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models

Raúl Vázquez; Aman Sinha; Chuyuan Li; Artem Shelmanov; Artem Vazhentsev; Claudio Savelli; Eduardo Calò; Emilio Raimond

cs.CLAug 26, 2026

Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models

Authors: Raúl Vázquez, Aman Sinha, Chuyuan Li, Artem Shelmanov, Artem Vazhentsev, Claudio Savelli, Eduardo Calò, Emilio Raimond,

Organizations: University of Helsinki · Politecnico di Torino · Universiteit Utrecht · Université Bretagne Sud ·

Abstract

In 2026, we held the fourth iteration of the SHROOM Shared Task series: SHROOM-Visions (Shared-task on Hallucinations and Related Observable Overgeneration Mistakes in Vision language models), which is hosted at the UncertaiNLP Workshop co-located with EMNLP 2026. Following the success of the 2024 and 2025 tasks, this time we aim to tackle hallucinations through a model-agnostic detection task focused on large vision-language models. Building on the recently introduced SHEEP dataset, designed for long-term evaluation across model generations, the task invites participants to detect and classify fine-grained hallucination spans in image-conditioned text generation (VQA, image captioning, etc.). The evaluation uses a five-class taxonomy of hallucinations spanning four languages: Chinese, English, French, and Italian. The shared task generated strong interest in the NLP community worldwide, with 27 teams contributing 600+ system submissions. The best systems achieve average scores of 0.58 in character-level correlation, 0.46 in label-conditioned correlation, and 0.51 in intersection-over-union (IoU) across four languages, outperforming the baselines by 30-40 points.

Figures & tables

Figure 1: Shroom-Visions 2026 Shared Task logo.

Panel of Figure 2 — Figure 2: Sample data point. The original data contain additional meta data including raw annotation and sampling strategy. Red highlight in the “response” corresponds to the marked hallucination span, darker shade denotes overlapped spans.

	EN	FR	IT	ZH
Train – Random	1,918	1,925	1,936	1,948
Train – Silver (MAP)	1,881	1,842	1,810	1,842
Test – Random	382	375	364	352
Test – Silver (MAP)	419	458	490	458
Test – Human	400	400	400	400

Table 1: Statistics per language, split, and sampling strategy from SHEEP dataset Mickus et al. (2026) .

Team	Lang	Description
Bit-by-bit Branescu and Dascalu (2026)	All	QLoRA-finetuned Qwen2.5-VL-7B with teacher-forced token scoring and hallucination probability heads.
Bubus Bubus (2026)	EN	Fusion of teacher-forced Qwen3-VL-32B and InternVL3-38B with hidden-state logistic regression and claim verification.
CPS Lab Srivastava et al. (2026)	All	Linear probe on frozen Qwen3-VL-8B-Instruct with soft-label multi-class hallucination span detection.
DSR	All	Character-level XLM-RoBERTa–SigLIP ensemble for hallucination span and category prediction.
HalluVision Sohail Pasha Qureshi et al. (2026)	All	LoRA-finetuned Qwen2-VL-2B trained on 15K image-text samples for hallucination prediction.
HalluciNauts Vo (2026)	EN	Output-only soft token tagger trained on English annotations for hallucination span detection.

Table 2: Participating teams and the language tracks in which they submitted systems in alphabetical order.

Figure 3: Overall Performance of all participating teams across 4 languages.

Panel of Figure 4 — Figure 4: Bootstrap rank distributions (25,000 samples) with means and confidence intervals at 95% for the top-5 ranking submissions per language under $\text{Corr}_{\text{lbl}}$ metric.

Figure 5: Rank flows across data partitions for the top-10 French systems on $\text{Corr}_{\text{lbl}}$ . Each node represents a system’s rank within a partition (General vs. Human/Silver/Random subsets). The variation in ranks illustrate that individual system positions can vary substantially with data sampling strategy.

Lang.	Metric	R–H	R–S	H–S	Fried. $p$
	Corr	0.802	0.933	0.905	0.0015
EN	Corr ${}_{\text{lbl}}$	0.684	0.864	0.861	<.0001
	IoU	0.828	0.941	0.920	0.0344
	Corr	0.805	0.784	0.951	<.0001
ZH	Corr ${}_{\text{lbl}}$	0.575	0.667	0.893	<.0001
	IoU	0.734	0.746	0.970	<.0001

Table 3: Rank correlation (Spearman

\rho

) between team rankings computed on Random (R), Human-written (H), and Silver-label (S) data partitions, per language and metric. Friedman

p

-values test for systematic raw scores differences across the three strategies computed via bootstrapping. Lower R–H and R–S values indicate that randomly sampling outputs without any prior signal can induce greater reordering of systems.

Figure 6: Scatter plot of IoU versus Corr scores for all participating teams, ranked by average IoU scores.

Figure 7: Scatter plot of IoU versus

\text{Corr}_{\text{lbl}}

scores for all participating teams, ranked by average IoU scores.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Overview of the performance by the best systems from each team in each language. From top to bottom: EN, FR, IT, ZH.

#	Team	Score [95%CI]	MR	RI	Score [95%CI]	MR	RI	Score [95%CI]	MR	RI	$P(>\!\text{next})$
		Corr ${}_{\text{lbl}}$			Corr			IoU
1	TÜRKSAT	0.433 [.29,.57]	2.9	[1, 10]	0.554 [.41,.69]	2.8	[1, 10]	0.486 [.39,.58]	2.0	[1, 7]	0.45
2	vroom-vroom	0.425 [.29,.56]	3.1	[1, 9]	0.533 [.39,.68]	3.5	[1, 10]	0.448 [.35,.55]	4.1	[1, 11]	0.45
3	medusa	0.417 [.29,.55]	3.7	[1, 12]	0.526 [.39,.66]	4.1	[1, 13]	0.455 [.36,.55]	3.7	[1, 10]	0.32
4	dynamos	0.388 [.26,.52]	5.2	[1, 12]	0.478 [.34,.61]	7.3	[2, 15]	0.393 [.30,.48]	9.4	[3, 15]	0.38
5	champ	0.372 [.24,.51]	6.5	[1, 15]	0.490 [.35,.63]	6.4	[1, 15]	0.432 [.34,.53]	5.5	[1, 12]	0.40

Appendix

Table 4: Leaderboard for English sorted by

\text{Corr}_{\text{lbl}}

score. MR = mean rank, RI = 95% rank interval.

P(>\!\text{next})

: bootstrap probability that the next-lower-ranked team outperforms this team on

\text{Corr}_{\text{lbl}}

; bold

\geq 0.95

= robust separation, italic

\leq 0.20

= possible inversion.

#	Team	Score [95%CI]	MR	RI	Score [95%CI]	MR	RI	Score [95%CI]	MR	RI	$P(>\!\text{next})$
		Corr ${}_{\text{lbl}}$			Corr			IoU
1	vroom-vroom	0.504 [.40,.61]	2.1	[1, 6]	0.608 [.51,.71]	1.7	[1, 5]	0.534 [.43,.64]	2.4	[1, 7]	0.35
2	TÜRKSAT	0.485 [.39,.59]	3.4	[1, 9]	0.581 [.48,.68]	3.2	[1, 9]	0.524 [.42,.62]	3.2	[1, 10]	0.38
3	dynamos	0.469 [.36,.57]	4.0	[1, 8]	0.530 [.42,.64]	6.0	[2, 10]	0.458 [.35,.56]	7.5	[2, 12]	0.48
4	champ	0.467 [.36,.57]	4.2	[1, 8]	0.550 [.45,.65]	4.5	[1, 9]	0.489 [.39,.59]	5.0	[1, 9]	0.41
5	SmurfCat	0.457 [.35,.56]	5.0	[1, 10]	0.544 [.44,.65]	5.1	[1, 10]	0.501 [.40,.61]	4.4	[1, 10]	0.48

Appendix

Table 5: Leaderboard for Chinese sorted by

\text{Corr}_{\text{lbl}}

score. MR = mean rank, RI = 95% rank interval.

P(>\!\text{next})

: bootstrap probability that the next-lower-ranked team outperforms this team on

\text{Corr}_{\text{lbl}}

; bold

\geq 0.95

= robust separation, italic

\leq 0.20

= possible inversion.

#	Team	Score [95%CI]	MR	RI	Score [95%CI]	MR	RI	Score [95%CI]	MR	RI	$P(>\!\text{next})$
		Corr ${}_{\text{lbl}}$			Corr			IoU
1	vroom-vroom	0.477 [.34,.61]	2.0	[1, 7]	0.586 [.45,.72]	1.9	[1, 7]	0.514 [.42,.62]	2.0	[1, 6]	0.35
2	TÜRKSAT	0.452 [.31,.60]	3.3	[1, 11]	0.545 [.40,.68]	3.7	[1, 11]	0.499 [.40,.60]	2.8	[1, 9]	0.27
3	champ	0.408 [.28,.54]	5.3	[1, 12]	0.516 [.39,.65]	4.9	[1, 11]	0.477 [.39,.57]	3.7	[1, 8]	0.47
4	dynamos	0.404 [.28,.53]	5.5	[1, 13]	0.502 [.37,.63]	5.8	[1, 12]	0.440 [.35,.53]	6.6	[2, 11]	0.45
5	SmurfCat	0.397 [.25,.55]	6.2	[1, 14]	0.483 [.34,.63]	7.2	[1, 14]	0.436 [.34,.54]	7.1	[2, 12]	0.44

Appendix

Table 6: Leaderboard for French sorted by

\text{Corr}_{\text{lbl}}

score. MR = mean rank, RI = 95% rank interval.

P(>\!\text{next})

: bootstrap probability that the next-lower-ranked team outperforms this team on

\text{Corr}_{\text{lbl}}

; bold

\geq 0.95

= robust separation, italic

\leq 0.20

= possible inversion.

#	Team	Score [95%CI]	MR	RI	Score [95%CI]	MR	RI	Score [95%CI]	MR	RI	$P(>\!\text{next})$
		Corr ${}_{\text{lbl}}$			Corr			IoU
1	vroom-vroom	0.450 [.32,.58]	2.9	[1, 9]	0.563 [.42,.70]	3.2	[1, 9]	0.480 [.38,.57]	3.0	[1, 8]	0.47
2	TÜRKSAT	0.445 [.31,.58]	3.5	[1, 11]	0.552 [.41,.69]	4.1	[1, 12]	0.490 [.39,.59]	2.7	[1, 8]	0.41
3	Risotto AI Funghi	0.429 [.30,.56]	4.3	[1, 11]	0.548 [.41,.69]	4.2	[1, 11]	0.471 [.37,.56]	3.7	[1, 9]	0.43
4	champ	0.421 [.29,.55]	4.6	[1, 11]	0.540 [.41,.67]	4.4	[1, 10]	0.462 [.37,.55]	4.0	[1, 9]	0.34
5	dynamos	0.400 [.28,.53]	6.0	[1, 14]	0.513 [.38,.64]	6.2	[1, 13]	0.425 [.34,.51]	7.0	[2, 12]	0.45

Appendix

Table 7: Leaderboard for Italian sorted by

\text{Corr}_{\text{lbl}}

score. MR = mean rank, RI = 95% rank interval.

P(>\!\text{next})

: bootstrap probability that the next-lower-ranked team outperforms this team on

\text{Corr}_{\text{lbl}}

; bold

\geq 0.95

= robust separation, italic

\leq 0.20

= possible inversion.

#	Team	Global	Human	Rndm	Silver
1	TÜRKSAT	0.553	0.646	0.492	0.522
2	vroom-vroom	0.533	0.684	0.440	0.474
3	medusa	0.527	0.626	0.462	0.491
4	champ	0.490	0.562	0.428	0.478
5	dynamos	0.478	0.583	0.420	0.429
6	eye-be-am	0.469	0.562	0.419	0.424

Appendix

Table 8: Strategy-specific Corr scores for the English leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Random	Silver
1	TÜRKSAT	0.434	0.488	0.407	0.406
2	vroom-vroom	0.425	0.536	0.374	0.366
3	medusa	0.416	0.477	0.388	0.383
4	dynamos	0.388	0.453	0.368	0.344
5	champ	0.372	0.398	0.361	0.358
6	eye-be-am	0.357	0.409	0.353	0.311

Appendix

Table 9: Strategy-specific

\text{Corr}_{\text{lbl}}

scores for the English leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	TÜRKSAT	0.487	0.551	0.444	0.464
2	medusa	0.455	0.516	0.424	0.425
3	vroom-vroom	0.449	0.565	0.388	0.393
4	champ	0.432	0.482	0.390	0.422
5	Sckwoky	0.419	0.453	0.408	0.396
6	Risotto_AI_F.	0.418	0.483	0.383	0.389

Appendix

Table 10: Strategy-specific IoU scores for the English leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	vroom-vroom	0.608	0.691	0.599	0.543
2	TÜRKSAT	0.581	0.667	0.540	0.539
3	Risotto_AI_F.	0.558	0.623	0.573	0.489
4	champ	0.551	0.563	0.610	0.495
5	SmurfCat	0.544	0.589	0.588	0.472
6	dynamos	0.530	0.551	0.581	0.472

Appendix

Table 11: Strategy-specific Corr scores for the Chinese leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Random	Silver
1	vroom-vroom	0.504	0.555	0.528	0.441
2	TÜRKSAT	0.484	0.531	0.487	0.442
3	dynamos	0.469	0.462	0.538	0.422
4	champ	0.467	0.442	0.560	0.417
5	SmurfCat	0.456	0.445	0.544	0.399
6	Risotto_AI_Funghi	0.454	0.487	0.511	0.381

Appendix

Table 12: Strategy-specific

\text{Corr}_{\text{lbl}}

scores for the Chinese leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	vroom-vroom	0.534	0.574	0.566	0.476
2	TÜRKSAT	0.524	0.575	0.515	0.487
3	Risotto_AI_F.	0.516	0.557	0.553	0.452
4	SmurfCat	0.501	0.507	0.570	0.442
5	champ	0.489	0.458	0.579	0.448
6	Sckwoky	0.466	0.468	0.551	0.401

Appendix

Table 13: Strategy-specific IoU scores for the Chinese leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	vroom-vroom	0.563	0.624	0.535	0.534
2	TÜRKSAT	0.553	0.665	0.483	0.514
3	Risotto_AI_F.	0.548	0.610	0.520	0.519
4	champ	0.540	0.600	0.497	0.523
5	dynamos	0.513	0.572	0.463	0.501
6	Bit-by-bit	0.505	0.538	0.464	0.507

Appendix

Table 14: Strategy-specific Corr scores for the Italian leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	vroom-vroom	0.451	0.508	0.456	0.401
2	TÜRKSAT	0.444	0.530	0.419	0.391
3	Risotto_AI_F.	0.430	0.482	0.428	0.388
4	champ	0.421	0.460	0.418	0.393
5	dynamos	0.399	0.437	0.382	0.382
6	SmurfCat	0.393	0.404	0.421	0.362

Appendix

Table 15: Strategy-specific

\text{Corr}_{\text{lbl}}

scores for the Italian leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	TÜRKSAT	0.490	0.571	0.445	0.458
2	vroom-vroom	0.479	0.508	0.476	0.458
3	Risotto_AI_F.	0.470	0.501	0.463	0.450
4	champ	0.461	0.476	0.445	0.461
5	SmurfCat	0.443	0.464	0.428	0.438
6	Sckwoky	0.430	0.424	0.457	0.415

Appendix

Table 16: Strategy-specific IoU scores for the Italian leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	vroom-vroom	0.587	0.678	0.537	0.550
2	TÜRKSAT	0.547	0.621	0.484	0.536
3	champ	0.517	0.554	0.496	0.501
4	Risotto_AI_F.	0.506	0.548	0.504	0.472
5	dynamos	0.504	0.583	0.462	0.468
6	eye-be-am	0.484	0.560	0.487	0.416

Appendix

Table 17: Strategy-specific Corr scores for the French leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	vroom-vroom	0.474	0.540	0.451	0.435
2	TÜRKSAT	0.451	0.493	0.421	0.440
3	champ	0.410	0.403	0.426	0.402
4	dynamos	0.407	0.442	0.391	0.389
5	SmurfCat	0.397	0.437	0.421	0.342
6	eye-be-am	0.388	0.407	0.426	0.341

Appendix

Table 18: Strategy-specific

\text{Corr}_{\text{lbl}}

scores for the French leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

#	Team	Global	Human	Rndm	Silver
1	vroom-vroom	0.515	0.572	0.487	0.488
2	TÜRKSAT	0.500	0.541	0.460	0.497
3	champ	0.478	0.485	0.466	0.481
4	Risotto_AI_F.	0.457	0.463	0.477	0.434
5	Sckwoky	0.447	0.417	0.474	0.451
6	dynamos	0.440	0.466	0.414	0.439

Appendix

Table 19: Strategy-specific IoU scores for the French leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.

Lang.	Annot. abs. %	Total	System abs.%
EN	15.48	1201	31.78
FR	16.78	1233	30.65
IT	16.34	1254	32.91
ZH	25.53	1210	45.98
Total	18.51	4898	34.87

Appendix

Table 20: Comparison of empty (abstinence) annotation versus participation system submissions on test set.

Panel of Figure 13 — Figure 13: Bootstrap rank distributions with means and confidence intervals at 95% for ZH, IT, FR submissions under $\text{Corr}_{\text{lbl}}$ metric. Distributions show rank frequencies across 25,000 bootstrap samples.

Figure 14: SHROOM-Visions platform participant submission interface.

Class	Lang	Precision	Recall	F1-Score	Span-Acc
OCR	EN	0.274 ±0.197 /0.672	0.233 ±0.151 /0.533	0.209 ±0.136 /0.516	0.123 ±0.087 /0.347
	FR	0.225 ±0.177 /0.583	0.260 ±0.172 /0.601	0.195 ±0.130 /0.512	0.114 ±0.083 /0.344
	IT	0.241 ±0.193 /0.685	0.274 ±0.175 /0.611	0.195 ±0.138 /0.536	0.115 ±0.089 /0.366
	ZH	0.251 ±0.181 /0.652	0.240 ±0.164 /0.663	0.188 ±0.115 /0.510	0.108 ±0.072 /0.342
invention	EN	0.136 ±0.104 /0.453	0.148 ±0.103 /0.466	0.110 ±0.077 /0.319	0.060 ±0.044 /0.190
	FR	0.113 ±0.103 /0.448	0.172 ±0.094 /0.431	0.111 ±0.083 /0.288	0.061 ±0.047 /0.168

Appendix

Table 21: Hallucination class-level summary of the participants system across all the languages. Each entry denotes mean

{}_{\pm\text{std}}

/max score format (min score = 0.0 for all, so redacted for better reability).

Hallucination Detection in VLMs

Hallucination detection in vision-language models identifies claims that are unsupported by images, video, or other visual evidence.

See all 72 papers

Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputs
Sep 15, 2026Toqeer Ehsan, Nico Penttilä, Richard Schmidt +2VLM Evaluation Hallucination Detection in VLMs
Reliable Intelligence Team, Physical AI, VTT Technical Research Centre of Finland
Two-Token Features and Small-Large Ensembles for VLM Hallucination Detection
Sep 9, 2026Eli SchwartzHallucination Detection in VLMs Vision-Language Models
IBM Research
SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Models
Sep 21, 2026Ali Athar, Imran Ahsan, Joon-Yong JungHallucination Detection in VLMs Vision-Language Models
Department of Radiology, Seoul St. Mary’s Hospital, The Catholic University of Korea · Department of Smart City, Chung-Ang University

Sep 15, 2026J/K move · Enter open · S save

Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputs

Toqeer Ehsan, Nico Penttilä, Richard Schmidt, Arash Hajikhani+1

Reliable Intelligence Team, Physical AI, VTT Technical Research Centre of Finland

VLM Evaluation Hallucination Detection in VLMs Hallucination Detection Multilingual VLM Evaluation

Open paper AskPDF arXiv

This paper describes our submission to the SHROOM-Visions shared task on detecting and classifying hallucinated character spans in vision-language model outputs across four languages. We employ several fine-tuned vision-language models as independent annotators and combine their span predictions through character-level majority voting, and additionally explore activation probes. The approach ranks first in three of four languages and places on the podium in every language and metric. Our analysis indicates that disagreement among diverse models tracks disagreement among human annotators.

Abstract

Hallucination Detection in VLMs

Explore similar work

Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputs

Two-Token Features and Small-Large Ensembles for VLM Hallucination Detection

SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Models