Correct Case Description Problem Type Tasks No obvious inconsistencies The data license is clear, accessible, consistently applied across all sources, appropriate for the data type, and is consistent with stated reuse conditions. No ambiguities are present, including with respect to license scope, consistency, and stated reuse conditions. None 20% Uncertain / borderline Cases in which data licensing or access conditions are formally permissible but lack clarity, suitability, or transparency. These practices do not necessarily violate legal terms but complicate reuse and reproducibility. Case Description Problem Type Tasks Platform registration under restrictive license A dataset under a restrictive license requires registration before access. While this may be justified for ethical, legal, or governance reasons, it adds additional barriers and reduces transparency when no justification for the restriction is provided. It can be legally sound, but introduces a hurdle to usability and transparency. Data access 33% Additional data usage agreement mentioned A dataset is accompanied by an additional data usage agreement that exists alongside the stated license. The presence of two parallel sets of terms introduces uncertainty about their relationship and whether the agreement merely clarifies or partially modifies the license. Such unclear interaction can, in some cases, result in overlapping or even contradictory obligations, creating potential legal risks for both data providers and users. Data access & License 23% Approval-based access under restrictive license Approval is required for data under restrictive licensing conditions. Although this may align with the terms of the license, the absence of clear criteria creates uncertainty about permissible reuse. In addition, the absence of the researchers who managed the data (e.g., after defending theses) may make the dataset inaccessible. Data access 20% NoDerivatives clause A dataset is released under a license containing a NoDerivatives (ND) clause. In the context of AI and machine learning, it remains unclear whether model training (e.g., data augmentation) constitutes a derivative work, creating uncertainty about lawful use. License 18% Context-limited access under restrictive license Geographic, temporal, or event-related limitations are imposed under a restrictive license. These may be legitimate within a restricted-use framework, but they should be clearly specified to avoid confusion. The restriction is valid, but lack of clarity hinders reuse and may complicate interpretation of permissible use conditions. Data access 10% Software-style license applied to dataset A software license such as MIT or GNU General Public License (GPL) is applied to data rather than code. Because these licenses were not designed for datasets or databases, they fail to address relevant rights and create uncertainty about allowable reuse, which may be legally and practically critical. This choice of those licenses may not cover relevant rights (e.g., database rights), leading to an incomplete grant. License 1% Inconsistent / misleading (pseudo-open) Cases in which data sharing was presented as open but access or licensing conditions contradicted the openness claim. Such practices create an illusion of openness, undermining transparency and user trust. Case Description Problem Type Tasks Platform registration despite open license Access to a dataset that claims to be openly licensed still requires prior registration on a hosting platform. This contradicts the principle of unrestricted accessibility that defines open data and limits practical reuse. Data access 13% License provided but data unavailable A license is publicly declared, but the corresponding dataset is no longer accessible, for example because a challenge has ended or a platform was shut down. Declaring a license without maintaining access misleads users and undermines reproducibility, as openness requires both permission and availability. Data access & License 8% Approval-based access despite open license Data usage requires explicit organizer approval even though the dataset is offered under an open license. Potential absence of the researchers who managed the data (e.g., after defending theses), may make the dataset even inaccessible. This turns open access into a permission-based process and potentially deters legitimate reuse. It is a misleading practice that undermines the core principle of an open license. Data access 5% Context-limited access despite open license Access to an openly licensed dataset is limited to certain countries, time periods, or participation in a specific event. Such conditions contradict the universal reuse rights guaranteed by open licenses. Data access 3% Potentially non-compliant Cases involving licensing practices that raise concerns about legal or procedural compliance. They may conflict with commonly used licensing frameworks or stated license conditions and may introduce legal and ethical risks to data reuse. Case Description Problem Type Tasks Conflicting license statements across sources Different sources, such as the challenge or data website, challenge design document, and accompanying publication, provide conflicting licenses for the same dataset. This inconsistency leaves users uncertain which license terms apply and creates a risk of unintentional non-compliance due to reliance on an incorrect or outdated license. License 20% License missing or not findable The dataset is made publicly available, but no explicit license or terms are provided on any source. In the absence of an explicit license, the default copyright status is ”All Rights Reserved.” Reuse beyond legally mandated exceptions may not be legally permitted. License 16% Conflicting or contradictory license terms The same license contains mutually inconsistent conditions, such as a Creative Commons statement combined with additional restrictions. Such internal contradictions can make the licensing legally problematic and prevent users from determining which terms actually apply. License 3% Violation of general license terms A data provider, organizer, or redistributor re-uses or distributes licensed data but fails to meet a core mandatory license requirement. Examples include not providing required attribution (BY) or re-licensing derivative works without the required Share-Alike (SA) clause. Such practice constitutes a breach of mandatory license terms and may result in copyright infringement. License 2% Incompatible license combination within a single source A dataset is distributed within a single source under multiple license statements that impose mutually exclusive obligations, for instance both requiring attribution and prohibiting redistribution. These conditions may not be fulfilled simultaneously and can make lawful reuse difficult or impossible. License 2%
3.4 Widespread reporting gaps hinder reusability and reproducibility
Even where data access is granted, many datasets are not reusable in practice due to missing documentation (Fig. ). Across multiple dimensions of reporting, we observed substantial gaps that undermine transparency and long-term usability.
Technical and organizational information was frequently incomplete. For 3% of tasks, the challenge website was no longer functional. The type of award was unspecified in 43% of tasks. Even when monetary awards were given, the exact prize amounts were missing in 56% of cases. Information on the acquisition devices (e.g., scanner type) was absent in 43% of tasks.
Reporting on basic dataset organization and ethics approval was limited. Only 19% of tasks provided a sufficient description of the data splitting strategy, and metadata was not described at all in 25% of tasks. De-identification procedures were not reported in 40% of tasks, and 35% of applicable tasks lacked information on ethics committee approval had been obtained.
Information on cohort composition and study design was often missing. The study population was not described in 41% of tasks, and clinical information such as disease status, diagnosis, or treatment history per subjects was absent in 40% of tasks. Case selection criteria were not reported in 49% of tasks, and eligibility criteria were missing in 50% of tasks. Although the overall country of data origin was often reported, the geographic composition of the training and test sets remained unspecified in 15% and 13% of tasks, respectively. Sex distribution was reported in only 30% of tasks, and sufficiently detailed age distribution in only 5%, with 26% of tasks reporting partial information such as mean age.
Reporting related to data annotation was similarly sparse. Annotation protocols were fully documented in only 40% of tasks, annotation tools were not described in 42%, and inter- or intra-rater variability was not reported in 74%.
4 Discussion
Biomedical image analysis challenges are growing in number and visibility, shaping methodological development and performance standards. They aim to enable fair comparison and foster clinically translatable AI models. However, as their influence expands, so does their responsibility: benchmarks that fail to represent clinical diversity or that cannot be legally reused risk undermining fairness they aim to ensure.
Our analysis revealed that specific regions, modalities, and anatomical sites dominate challenges. This limited representation may constrain model generalizability and perpetuate gaps in geographic and clinical coverage. Notably, some patterns diverge markedly from clinical practice. For example, although MRI was the most common imaging modality in challenges (Fig. 2 c), it accounts for a comparatively small share of clinical imaging. MRI imaging rates for older adults were only 139 per 1,000 person-years, compared to 428 for CT and 495 for ultrasound in the US in 2016 [ 26 ] . Restricting the comparison to these three modalities, MRI was 4.4-fold overrepresented in challenges relative to the reported clinical utilization distribution, whereas ultrasound was substantially underrepresented (ratio: 0.2). Beyond questions of clinical utility, this mismatch raises fairness concerns: prioritizing MRI-centric benchmarks implicitly centers resource-rich healthcare systems that can support high-cost modalities, thereby disadvantaging regions where ultrasound or X-ray are the primary diagnostic tools. This mismatch may reinforce the geographic underrepresentation of certain countries.
Similarly, segmentation was by far the most common task category across challenges, yet its direct clinical relevance is often constrained in real-world settings. Manual segmentation is rarely used in routine care, and segmented outputs primarily serve as intermediate steps such as for volume estimation or planning [ 9 ] . Our additional analyses suggest that these task distributions are not solely driven by clinical priorities. Dataset sizes differed across problem categories, indicating that practical considerations related to data availability and annotation effort may influence which tasks are preferentially addressed in biomedical image analysis challenges. While our study design does not allow us to determine what drives these associations, several practical factors may plausibly contribute. Annotation requirements differ substantially between problem categories [ 34 ] . For example, some classification tasks require only image-level labels, whereas instance segmentation requires identifying and delineating individual objects. Other tasks may depend on data that are inherently more difficult to collect at scale, such as outcome prediction requiring longitudinal follow-up or linked clinical information. Dataset scale may also depend on examination frequency and the resources required for data acquisition and curation. For example, high-volume procedures such as cholecystectomy can generate large amounts of laparoscopic video data [ 28 ] , whereas assembling similarly sized datasets for rare diseases may require substantially longer acquisition periods. This interpretation is supported by the relationship between imaging modality and problem category. While MRI dominated most problem categories, detection tasks were considerably more common in laparoscopy and endoscopy than in other imaging modalities (Fig. 3b), suggesting that different imaging domains tend to emphasize different methodological questions and practical application scenarios. Regulatory, institutional, and political frameworks may further determine which data can be collected, accessed, and shared, shaping dataset availability for challenges. A similar interpretation may apply to anatomical regions. The predominance of brain imaging tasks likely reflects the historical availability of large, open-access neuroimaging datasets (e.g., [ 30 ] ) rather than population-level disease burden or unmet clinical needs.
Overall, these patterns suggest that benchmarks may address problems that are readily measurable and feasible to study rather than those of greatest clinical relevance This challenges the implicit assumption that leaderboard success equates to clinical readiness [ 10 ] . More generally, challenge leaderboards should always be interpreted with care. As discussed in prior work (e.g., [ 17 , 16 , 32 ] ), small differences in benchmark performance are often difficult to interpret and do not necessarily translate into clinically meaningful improvements. An excessive focus on leaderboard position may encourage optimization for potentially unsuitable benchmark-specific metrics rather than properties required for successful clinical deployment, such as robustness or workflow integration [ 31 ] . More broadly, benchmark performance can serve as a valuable proxy for clinical translation when evaluated on representative and clinically meaningful benchmarks, as illustrated by the Digital Mammography DREAM Challenge. Clinical deployment, however, additionally depends on rigorous external validation, regulatory approval, and integration into clinical workflows.
We found that even when challenge data are nominally public, access is frequently restricted through organizer approval requirements, registration barriers, or ambiguous licensing. Registration or approval procedures may be justified for ethical, legal, or governance reasons, particularly for sensitive medical data. However, in combination with open licenses, additional access requirements should be avoided wherever possible, as they contradict the universal reuse rights such licenses guarantee. In extreme cases, datasets may become unreachable, for example after organizers leave their institutions, rendering datasets practically unavailable [ 12 ] . Prior studies in clinical epidemiology have shown that in, ”data available upon request” statements have extremely low compliance, similar to having no such statement at all [ 7 ] . These issues are not administrative details; they actively prevent legitimate reuse and slow down scientific progress. Notably, [ 4 ] showed that articles that make their data available receive more citations in other fields such as economics and political science, underscoring the broader value of open data.
While the majority of tasks claim to provide public data, we found that more than 80% include substantial reuse restrictions, especially in a sense of ambiguous licensing terms. Although additional data access restrictions can be legally valid, several challenges used impermissible licenses or terms. A common example includes conflicting license statements across different sources, such as stating a different license for the same dataset on the (potentially outdated) challenge website and the challenge publication. This creates uncertainty about which license governs reuse and may expose users to legal risks, including takedown requests, disputes, or claims arising from unintended violations. Other clear violations include conflicting or contradictory license terms, often induced by an additional data usage agreement, or the violation of general license terms, such as re-licensing works without a required Share-Alike (SA) clause. Beyond these clear violations, large grey zones persist. A common example is the usage of NoDerivatives (ND) clauses (e.g., CC-BY ND), where the notion of “derivative work” remains poorly defined in the context of AI [ 21 ] . For example, it can be unclear whether trained models, learned representations, data augmentation, or weights constitute derivative works under current copyright frameworks [ 2 ] . While studies in both AI law and EU copyright interpretation suggest that model training typically lacks the originality required for a new protected work unless the modification reflects creative input [ 2 , 6 ] , ND clauses may still formally prohibit reuse while offering no practical guidance for compliance.
Our results mirror a broader “crisis in misattribution” discussed in the general AI community [ 15 ] , where incomplete licensing information and inconsistent data provenance lead to widespread misattribution [ 27 ] and uncertainty about permissible use. Limited data licensing expertise in the biomedical AI community may contribute to practices whose legal and practical implications are insufficiently considered [ 25 ] . For example, challenge organizers may introduce additional data usage agreements without recognizing potential conflicts with existing licenses or barriers for participants and future users. This is rarely due to negligence but rather limited access to specialized legal expertise or inconsistent institutional guidance, combined with ambiguities in interpreting license terms (e.g., commercial use or publication-related exemptions). These structural uncertainties become even more visible in practical conflicts: additional agreements may contradict open licenses, while attribution and Share-Alike requirements become nearly impossible to satisfy when models are trained on several datasets with incompatible terms [ 29 ] . For challenge organizers, these problems are amplified by restrictive Institutional Review Board (IRB) interpretations, cautious or inconsistent legal risk assessments that may diverge from evidence-based reuse risk evaluations, and insufficient data stewardship infrastructure, often leading to more restrictive or ad-hoc licensing than intended. At the same time, unclear or restrictive terms can substantially undermine reproducibility and long-term utility of benchmark datasets. That said, restrictive licenses or usage conditions can, in some situations, be necessary to preserve scientific integrity, and ensure ethical use. These access and licensing issues also fall short of FAIR expectations for clear terms, stable availability, and unambiguous conditions for responsible and practically feasible data reuse.
A related limitation is incomplete reporting in challenge publications, particularly of socio-demographic and clinical details. Basic descriptors such as age and sex were frequently missing, and clinical characterization was often incomplete. From a medical perspective, such information is essential for understanding the cohort and for assessing whether model performance is likely to generalize to real clinical populations. Similarly, insufficient reporting on study design, data splitting, acquisition devices, or annotation procedures restricts reproducibility and prevents meaningful comparison across challenges. Poor documentation therefore constitutes a fundamental obstacle to reusability, independent of access or licensing conditions.
The consequences for AI development are substantial: because biomedical AI depends heavily on benchmarks and challenge datasets that are frequently reused for subsequent method development, limited accessibility or unclear licensing conditions slow scientific progress and may distort how model performance is interpreted within and beyond leaderboard results. When benchmarks are unrepresentative or legally unusable, they lose their function as proxies for real-world performance and may inadvertently lead to misguided decisions, especially in safety-critical clinical settings. Ultimately, these problems raise the broader question of whether current benchmarking practices are aligned with real-world application needs or whether they increasingly serve academic signaling rather than supporting clinically meaningful AI development. Recent bans on studies using public health datasets such as NHANES [ 19 ] illustrate that openness alone is not sufficient; openly available data can still be misused, which may appropriately trigger policy interventions, while highlighting how misuse and unclear governance can erode trust in open science ecosystems.
Our work comes with several limitations. The study was descriptive rather than prescriptive; due to substantial licensing gray zones and institution-specific rules, we did not aim to propose a universal guideline for challenge data governance. Our analysis was restricted to challenges conducted between 2018 and 2023, excluding recent editions that typically lack stable documentation or peer-reviewed publications. All information was extracted from publicly available sources, which were sometimes incomplete or outdated; missing information may therefore reflect either documentation gaps or limitations of what could be retrieved. Thus, we could not independently verify institutional or legal procedures (e.g., IRB approvals, internal data-sharing agreements), which limited our ability to assess compliance beyond what was reported. Although data extraction followed a structured double-blind process with 30 observers and conflict resolution, some assessments necessarily involved subjective interpretation. Our analysis focused on challenge datasets and documentation, without assessing downstream reuse, later publications, or the broader benchmark research lifecycle. Consequently, we cannot determine how observed dataset characteristics may influence subsequent methodological development or the trajectory of biomedical image analysis. Future work should investigate how benchmark datasets shape dataset reuse, research directions, and algorithm development. Meaningful external reference distributions were unavailable for several assessed dimensions. We therefore limited comparisons to geography and imaging modality, for which suitable reference data were available. However, establishing a reliable reference distribution for imaging modality remained challenging. Available studies on imaging utilization are often limited to specific countries or institutions and do not cover all modalities assessed in our study. Consequently, our modality comparison should be interpreted as an illustrative external proxy rather than a reference for global clinical practice. Finally, we did not assess the legal enforceability of identified licensing issues, which can vary across jurisdictions and may depend on context-specific factors beyond the scope of this study. Accordingly, our categorization reflects observed licensing practices and reported terms, rather than conclusions based on jurisdiction-specific legal standards or case-by-case legal interpretation.
Given the discussed problems, several emerging approaches aim to improve transparency, provenance, and legal certainty in dataset design. Complementary infrastructure-oriented work has proposed machine-readable, policy-as-code governance layers to decouple policy definition from enforcement, aiming to increase transparency, auditability, and legal clarity in decentralized AI data ecosystems [ 13 ] . Related work from other areas highlighted that legal risks related to dataset reuse and redistribution cannot be assessed from license terms alone, but tracing dataset redistribution and lifecycle is essential [ 14 ] . New technical efforts address this, including automated compliance agents and provenance-tracking systems such as the Data Provenance Explorer [ 15 ] . The Re(usable) Data Project offers a structured ruleset for assessing reusability, which we also applied in our analysis [ 3 ] . Licensing frameworks like the Montreal Data License provide AI-specific license language [ 2 ] . Complementary documentation frameworks such as ”Datasheets for Datasets” [ 8 ] outline standardized descriptions of dataset motivation, composition, and collection. Major data-hosting platforms have already begun to establish clearer licensing best practices that could serve as a model for biomedical challenges. For example, Hugging Face 4 4 4 https://huggingface.co/docs/hub/en/repositories-licenses or Kaggle 5 5 5 https://www.kaggle.com/discussions/general/116302 provide overviews of acceptable license types, reuse conditions, and contributor responsibilities, although these aspects are not reviewed and users often leave respective documentation fields empty [ 12 ] . Open collaborative efforts such as the GA4GH Data Use Ontology (DUO) 6 6 6 https://www.ga4gh.org/product/data-use-ontology-duo/ and Croissant metadata standard 7 7 7 https://research.google/blog/croissant-a-metadata-format-for-ml-ready-datasets/ further support this direction by providing controlled vocabularies and structured schemas for describing permissible data uses. Notably, schema.org 8 8 8 https://research.google/blog/croissant-a-metadata-format-for-ml-ready-datasets/ already defines a standard license field, enabling consistent, machine-readable expressions of licensing terms. These initiatives address three identified issues: (1) difficulty identifying license types, (2) confusion between access restrictions and use permissions, and (3) inconsistent interpretation of use cases, including geographic limitations. Integrating these standards into biomedical image analysis challenges could substantially reduce ambiguity and improve data governance transparency.
One step toward improving the usability and fairness of challenge datasets would be the introduction of baseline requirements for metadata and licensing. Leading conferences such as MICCAI and ISBI could require challenge organizers to provide machine-readable metadata and clear, standardized licensing information, for example through the RDP or Croissant schemas. Establishing such requirements during submission and peer review would reduce ambiguity, support lawful reuse, and strengthen the long-term value of challenge datasets.
Beyond technical standards for metadata and licensing, improving future challenges also requires more comprehensive and targeted reporting. Motivated by the findings of this work, we are currently in the process of updating the BIAS (Transparent Reporting of Biomedical Image Analysis Challenges) reporting guideline [ 18 ] through an international Delphi consensus process. The revision extends the original checklist with additional items identified in this study, including data accessibility, justification for licensing, dataset representativeness, demographic and clinical characteristics, potential confounders, and fairness-related considerations. We hope that integrating these aspects into an established reporting standard will facilitate their broad adoption by future challenge organizers.
Overall, our findings indicate that current challenges fall far short of these emerging standards and therefore do not yet provide legally or clinically meaningful benchmarks for real-world translation. Our intention is not to criticize individual challenge organizers or prescribe how biomedical image analysis challenges should be organized. Many of the limitations identified in this work reflect broader structural constraints, including limited resources, restricted access to diverse clinical data, and legal or institutional barriers to data sharing that are largely beyond the control of individual organizers. Instead, we aim to raise awareness of these limitations, provide an empirical basis for discussion, and encourage feasible improvements in reporting quality, metadata documentation, and licensing practices. We hope these findings contribute to more transparent, reusable, and clinically meaningful benchmarking practices.
5 Methods
In this study, we use the terms “challenge” and “task” as defined by [ 16 ] . A ”challenge” denotes an open competition on a dedicated scientific problem in biomedical image analysis, typically organized by a consortium issuing a call for participation and potentially comprising multiple tasks. A “task” refers to a subproblem within a challenge for which a separate assessment or leaderboard is provided, potentially with its own performance metrics.