Organizations: University of Helsinki · Politecnico di Torino · Universiteit Utrecht · Université Bretagne Sud · University of Copenhagen · University Grenoble Alpes · University of Lorraine · MBZUAI
In 2026, we held the fourth iteration of the SHROOM Shared Task series: SHROOM-Visions (Shared-task on Hallucinations and Related Observable Overgeneration Mistakes in Vision language models), which is hosted at the UncertaiNLP Workshop co-located with EMNLP 2026. Following the success of the 2024 and 2025 tasks, this time we aim to tackle hallucinations through a model-agnostic detection task focused on large vision-language models. Building on the recently introduced SHEEP dataset, designed for long-term evaluation across model generations, the task invites participants to detect and classify fine-grained hallucination spans in image-conditioned text generation (VQA, image captioning, etc.). The evaluation uses a five-class taxonomy of hallucinations spanning four languages: Chinese, English, French, and Italian. The shared task generated strong interest in the NLP community worldwide, with 27 teams contributing 600+ system submissions. The best systems achieve average scores of 0.58 in character-level correlation, 0.46 in label-conditioned correlation, and 0.51 in intersection-over-union (IoU) across four languages, outperforming the baselines by 30-40 points.
Figures & tables
Figure 1: Shroom-Visions 2026 Shared Task logo.
Figure 2: Sample data point. The original data contain additional meta data including raw annotation and sampling strategy. Red highlight in the “response” corresponds to the marked hallucination span, darker shade denotes overlapped spans.
EN
FR
IT
ZH
Train – Random
1,918
1,925
1,936
1,948
Train – Silver (MAP)
1,881
1,842
1,810
1,842
Test – Random
382
375
364
352
Test – Silver (MAP)
419
458
490
458
Test – Human
400
400
400
400
Table 1: Statistics per language, split, and sampling strategy from SHEEP dataset Mickus et al. (2026) .
Team
Lang
Description
Bit-by-bit Branescu and Dascalu (2026)
All
QLoRA-finetuned Qwen2.5-VL-7B with teacher-forced token scoring and hallucination probability heads.
Bubus Bubus (2026)
EN
Fusion of teacher-forced Qwen3-VL-32B and InternVL3-38B with hidden-state logistic regression and claim verification.
CPS Lab Srivastava et al. (2026)
All
Linear probe on frozen Qwen3-VL-8B-Instruct with soft-label multi-class hallucination span detection.
DSR
All
Character-level XLM-RoBERTa–SigLIP ensemble for hallucination span and category prediction.
HalluVision Sohail Pasha Qureshi et al. (2026)
All
LoRA-finetuned Qwen2-VL-2B trained on 15K image-text samples for hallucination prediction.
HalluciNauts Vo (2026)
EN
Output-only soft token tagger trained on English annotations for hallucination span detection.
Table 2: Participating teams and the language tracks in which they submitted systems in alphabetical order.
Figure 3: Overall Performance of all participating teams across 4 languages.
Figure 4: Bootstrap rank distributions (25,000 samples) with means and confidence intervals at 95% for the top-5 ranking submissions per language under Corrlbl metric.
Figure 5: Rank flows across data partitions for the top-10 French systems on Corrlbl . Each node represents a system’s rank within a partition (General vs. Human/Silver/Random subsets). The variation in ranks illustrate that individual system positions can vary substantially with data sampling strategy.
Lang.
Metric
R–H
R–S
H–S
Fried. p
Corr
0.802
0.933
0.905
0.0015
EN
Corr lbl
0.684
0.864
0.861
<.0001
IoU
0.828
0.941
0.920
0.0344
Corr
0.805
0.784
0.951
<.0001
ZH
Corr lbl
0.575
0.667
0.893
<.0001
IoU
0.734
0.746
0.970
<.0001
Table 3: Rank correlation (Spearman ρ ) between team rankings computed on Random (R), Human-written (H), and Silver-label (S) data partitions, per language and metric. Friedman p -values test for systematic raw scores differences across the three strategies computed via bootstrapping. Lower R–H and R–S values indicate that randomly sampling outputs without any prior signal can induce greater reordering of systems.
Figure 6: Scatter plot of IoU versus Corr scores for all participating teams, ranked by average IoU scores.
Figure 7: Scatter plot of IoU versus Corrlbl scores for all participating teams, ranked by average IoU scores.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Overview of the performance by the best systems from each team in each language. From top to bottom: EN, FR, IT, ZH.
Corr lbl
Corr
IoU
#
Team
Score [95%CI]
MR
RI
Score [95%CI]
MR
RI
Score [95%CI]
MR
RI
P(>next)
1
TÜRKSAT
0.433 [.29,.57]
2.9
[1, 10]
0.554 [.41,.69]
2.8
[1, 10]
0.486 [.39,.58]
2.0
[1, 7]
0.45
2
vroom-vroom
0.425 [.29,.56]
3.1
[1, 9]
0.533 [.39,.68]
3.5
[1, 10]
0.448 [.35,.55]
4.1
[1, 11]
0.45
3
medusa
0.417 [.29,.55]
3.7
[1, 12]
0.526 [.39,.66]
4.1
[1, 13]
0.455 [.36,.55]
3.7
[1, 10]
0.32
4
dynamos
0.388 [.26,.52]
5.2
[1, 12]
0.478 [.34,.61]
7.3
[2, 15]
0.393 [.30,.48]
9.4
[3, 15]
0.38
5
champ
0.372 [.24,.51]
6.5
[1, 15]
0.490 [.35,.63]
6.4
[1, 15]
0.432 [.34,.53]
5.5
[1, 12]
0.40
Appendix
Table 4: Leaderboard for English sorted by Corrlbl score. MR = mean rank, RI = 95% rank interval. P(>next) : bootstrap probability that the next-lower-ranked team outperforms this team on Corrlbl ; bold ≥0.95 = robust separation, italic ≤0.20 = possible inversion.
Corr lbl
Corr
IoU
#
Team
Score [95%CI]
MR
RI
Score [95%CI]
MR
RI
Score [95%CI]
MR
RI
P(>next)
1
vroom-vroom
0.504 [.40,.61]
2.1
[1, 6]
0.608 [.51,.71]
1.7
[1, 5]
0.534 [.43,.64]
2.4
[1, 7]
0.35
2
TÜRKSAT
0.485 [.39,.59]
3.4
[1, 9]
0.581 [.48,.68]
3.2
[1, 9]
0.524 [.42,.62]
3.2
[1, 10]
0.38
3
dynamos
0.469 [.36,.57]
4.0
[1, 8]
0.530 [.42,.64]
6.0
[2, 10]
0.458 [.35,.56]
7.5
[2, 12]
0.48
4
champ
0.467 [.36,.57]
4.2
[1, 8]
0.550 [.45,.65]
4.5
[1, 9]
0.489 [.39,.59]
5.0
[1, 9]
0.41
5
SmurfCat
0.457 [.35,.56]
5.0
[1, 10]
0.544 [.44,.65]
5.1
[1, 10]
0.501 [.40,.61]
4.4
[1, 10]
0.48
Appendix
Table 5: Leaderboard for Chinese sorted by Corrlbl score. MR = mean rank, RI = 95% rank interval. P(>next) : bootstrap probability that the next-lower-ranked team outperforms this team on Corrlbl ; bold ≥0.95 = robust separation, italic ≤0.20 = possible inversion.
Corr lbl
Corr
IoU
#
Team
Score [95%CI]
MR
RI
Score [95%CI]
MR
RI
Score [95%CI]
MR
RI
P(>next)
1
vroom-vroom
0.477 [.34,.61]
2.0
[1, 7]
0.586 [.45,.72]
1.9
[1, 7]
0.514 [.42,.62]
2.0
[1, 6]
0.35
2
TÜRKSAT
0.452 [.31,.60]
3.3
[1, 11]
0.545 [.40,.68]
3.7
[1, 11]
0.499 [.40,.60]
2.8
[1, 9]
0.27
3
champ
0.408 [.28,.54]
5.3
[1, 12]
0.516 [.39,.65]
4.9
[1, 11]
0.477 [.39,.57]
3.7
[1, 8]
0.47
4
dynamos
0.404 [.28,.53]
5.5
[1, 13]
0.502 [.37,.63]
5.8
[1, 12]
0.440 [.35,.53]
6.6
[2, 11]
0.45
5
SmurfCat
0.397 [.25,.55]
6.2
[1, 14]
0.483 [.34,.63]
7.2
[1, 14]
0.436 [.34,.54]
7.1
[2, 12]
0.44
Appendix
Table 6: Leaderboard for French sorted by Corrlbl score. MR = mean rank, RI = 95% rank interval. P(>next) : bootstrap probability that the next-lower-ranked team outperforms this team on Corrlbl ; bold ≥0.95 = robust separation, italic ≤0.20 = possible inversion.
Corr lbl
Corr
IoU
#
Team
Score [95%CI]
MR
RI
Score [95%CI]
MR
RI
Score [95%CI]
MR
RI
P(>next)
1
vroom-vroom
0.450 [.32,.58]
2.9
[1, 9]
0.563 [.42,.70]
3.2
[1, 9]
0.480 [.38,.57]
3.0
[1, 8]
0.47
2
TÜRKSAT
0.445 [.31,.58]
3.5
[1, 11]
0.552 [.41,.69]
4.1
[1, 12]
0.490 [.39,.59]
2.7
[1, 8]
0.41
3
Risotto AI Funghi
0.429 [.30,.56]
4.3
[1, 11]
0.548 [.41,.69]
4.2
[1, 11]
0.471 [.37,.56]
3.7
[1, 9]
0.43
4
champ
0.421 [.29,.55]
4.6
[1, 11]
0.540 [.41,.67]
4.4
[1, 10]
0.462 [.37,.55]
4.0
[1, 9]
0.34
5
dynamos
0.400 [.28,.53]
6.0
[1, 14]
0.513 [.38,.64]
6.2
[1, 13]
0.425 [.34,.51]
7.0
[2, 12]
0.45
Appendix
Table 7: Leaderboard for Italian sorted by Corrlbl score. MR = mean rank, RI = 95% rank interval. P(>next) : bootstrap probability that the next-lower-ranked team outperforms this team on Corrlbl ; bold ≥0.95 = robust separation, italic ≤0.20 = possible inversion.
#
Team
Global
Human
Rndm
Silver
1
TÜRKSAT
0.553
0.646
0.492
0.522
2
vroom-vroom
0.533
0.684
0.440
0.474
3
medusa
0.527
0.626
0.462
0.491
4
champ
0.490
0.562
0.428
0.478
5
dynamos
0.478
0.583
0.420
0.429
6
eye-be-am
0.469
0.562
0.419
0.424
Appendix
Table 8: Strategy-specific Corr scores for the English leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
Figure 17
#
Team
Global
Human
Random
Silver
1
TÜRKSAT
0.434
0.488
0.407
0.406
2
vroom-vroom
0.425
0.536
0.374
0.366
3
medusa
0.416
0.477
0.388
0.383
4
dynamos
0.388
0.453
0.368
0.344
5
champ
0.372
0.398
0.361
0.358
6
eye-be-am
0.357
0.409
0.353
0.311
Appendix
Table 9: Strategy-specific Corrlbl scores for the English leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
TÜRKSAT
0.487
0.551
0.444
0.464
2
medusa
0.455
0.516
0.424
0.425
3
vroom-vroom
0.449
0.565
0.388
0.393
4
champ
0.432
0.482
0.390
0.422
5
Sckwoky
0.419
0.453
0.408
0.396
6
Risotto_AI_F.
0.418
0.483
0.383
0.389
Appendix
Table 10: Strategy-specific IoU scores for the English leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
vroom-vroom
0.608
0.691
0.599
0.543
2
TÜRKSAT
0.581
0.667
0.540
0.539
3
Risotto_AI_F.
0.558
0.623
0.573
0.489
4
champ
0.551
0.563
0.610
0.495
5
SmurfCat
0.544
0.589
0.588
0.472
6
dynamos
0.530
0.551
0.581
0.472
Appendix
Table 11: Strategy-specific Corr scores for the Chinese leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Random
Silver
1
vroom-vroom
0.504
0.555
0.528
0.441
2
TÜRKSAT
0.484
0.531
0.487
0.442
3
dynamos
0.469
0.462
0.538
0.422
4
champ
0.467
0.442
0.560
0.417
5
SmurfCat
0.456
0.445
0.544
0.399
6
Risotto_AI_Funghi
0.454
0.487
0.511
0.381
Appendix
Table 12: Strategy-specific Corrlbl scores for the Chinese leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
vroom-vroom
0.534
0.574
0.566
0.476
2
TÜRKSAT
0.524
0.575
0.515
0.487
3
Risotto_AI_F.
0.516
0.557
0.553
0.452
4
SmurfCat
0.501
0.507
0.570
0.442
5
champ
0.489
0.458
0.579
0.448
6
Sckwoky
0.466
0.468
0.551
0.401
Appendix
Table 13: Strategy-specific IoU scores for the Chinese leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
vroom-vroom
0.563
0.624
0.535
0.534
2
TÜRKSAT
0.553
0.665
0.483
0.514
3
Risotto_AI_F.
0.548
0.610
0.520
0.519
4
champ
0.540
0.600
0.497
0.523
5
dynamos
0.513
0.572
0.463
0.501
6
Bit-by-bit
0.505
0.538
0.464
0.507
Appendix
Table 14: Strategy-specific Corr scores for the Italian leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
vroom-vroom
0.451
0.508
0.456
0.401
2
TÜRKSAT
0.444
0.530
0.419
0.391
3
Risotto_AI_F.
0.430
0.482
0.428
0.388
4
champ
0.421
0.460
0.418
0.393
5
dynamos
0.399
0.437
0.382
0.382
6
SmurfCat
0.393
0.404
0.421
0.362
Appendix
Table 15: Strategy-specific Corrlbl scores for the Italian leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
TÜRKSAT
0.490
0.571
0.445
0.458
2
vroom-vroom
0.479
0.508
0.476
0.458
3
Risotto_AI_F.
0.470
0.501
0.463
0.450
4
champ
0.461
0.476
0.445
0.461
5
SmurfCat
0.443
0.464
0.428
0.438
6
Sckwoky
0.430
0.424
0.457
0.415
Appendix
Table 16: Strategy-specific IoU scores for the Italian leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
vroom-vroom
0.587
0.678
0.537
0.550
2
TÜRKSAT
0.547
0.621
0.484
0.536
3
champ
0.517
0.554
0.496
0.501
4
Risotto_AI_F.
0.506
0.548
0.504
0.472
5
dynamos
0.504
0.583
0.462
0.468
6
eye-be-am
0.484
0.560
0.487
0.416
Appendix
Table 17: Strategy-specific Corr scores for the French leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
vroom-vroom
0.474
0.540
0.451
0.435
2
TÜRKSAT
0.451
0.493
0.421
0.440
3
champ
0.410
0.403
0.426
0.402
4
dynamos
0.407
0.442
0.391
0.389
5
SmurfCat
0.397
0.437
0.421
0.342
6
eye-be-am
0.388
0.407
0.426
0.341
Appendix
Table 18: Strategy-specific Corrlbl scores for the French leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
#
Team
Global
Human
Rndm
Silver
1
vroom-vroom
0.515
0.572
0.487
0.488
2
TÜRKSAT
0.500
0.541
0.460
0.497
3
champ
0.478
0.485
0.466
0.481
4
Risotto_AI_F.
0.457
0.463
0.477
0.434
5
Sckwoky
0.447
0.417
0.474
0.451
6
dynamos
0.440
0.466
0.414
0.439
Appendix
Table 19: Strategy-specific IoU scores for the French leaderboard. Teams are ordered by their score on the full dataset (Global). Human-written, Random, and Silver-label report scores computed on the corresponding data partitions. Boldface indicates the highest score within each column.
Lang.
Annot. abs. %
Total
System abs.%
EN
15.48
1201
31.78
FR
16.78
1233
30.65
IT
16.34
1254
32.91
ZH
25.53
1210
45.98
Total
18.51
4898
34.87
Appendix
Table 20: Comparison of empty (abstinence) annotation versus participation system submissions on test set.
Figure 13: Bootstrap rank distributions with means and confidence intervals at 95% for ZH, IT, FR submissions under Corrlbl metric. Distributions show rank frequencies across 25,000 bootstrap samples.
Table 21: Hallucination class-level summary of the participants system across all the languages. Each entry denotes mean ±std /max score format (min score = 0.0 for all, so redacted for better reability).
This paper describes our submission to the SHROOM-Visions shared task on detecting and classifying hallucinated character spans in vision-language model outputs across four languages. We employ several fine-tuned vision-language models as independent annotators and combine their span predictions through character-level majority voting, and additionally explore activation probes. The approach ranks first in three of four languages and places on the podium in every language and metric. Our analysis indicates that disagreement among diverse models tracks disagreement among human annotators.
Toqeer Ehsan, Nico Penttilä, Richard Schmidt +2
Reliable Intelligence Team, Physical AI, VTT Technical Research Centre of Finland
We present our system for the SHROOM-Visions 2026 shared task on character-level VLM hallucination detection. A small (4B-parameter) VLM is fine-tuned as a per-token classifier reading a two-token feature from its own hidden states, and is ensembled with a ∼400B zero-shot VLM judge at prediction time. Both components see off-the-shelf OCR of any visible in-image text. We use synthetic hallucination data generated by the large model as a source of ensemble diversity, and use validation to select feature layer, training data and OCR grounding. Our official entry reaches mean Cor 0.487 / Cor-lbl 0.387 on the hidden test set, placing 6th/28 (EN), 6th/21 (FR), 8th/21 (IT) and 7th/22 (ZH) on the task's primary Cor-lbl metric.
This paper describes the SKstars submission to SHROOM-Visions 2026, a shared task on fine-grained hallucination detection in large vision-language model outputs. The task requires systems to identify hallucinated character spans, assign hallucination categories, and provide confidence estimates for their predictions. Our approach combines zero-shot predictions from Qwen2.5-VL-72B-Instruct with those of a LoRA-adapted Qwen2.5-VL-7B-Instruct model. The outputs of the two models are integrated through a lightweight ensemble procedure, followed by span refinement and confidence adjustment. We evaluate the main system components on a small internal development subset and report the performance of the submitted system on the official English test set. SKstars achieved a Cor+Lbl score of 0.2902, ranking 15th among 29 teams, and obtained Cor and IoU scores of 0.3642 and 0.3151, respectively, ranking 18th on both metrics. The results show that combining a large zero-shot model with a smaller adapted model provides a practical framework for multilingual and fine-grained hallucination localization, while also highlighting the difficulty of transferring development-set improvements to hidden test data. Code and predictions: https://github.com/aliathar1401/SK-Stars-shroom-visions-2026
Ali Athar, Imran Ahsan, Joon-Yong Jung
Department of Radiology, Seoul St. Mary’s Hospital, The Catholic University of Korea · Department of Smart City, Chung-Ang University