cs.SEFeb 17, 2026
SaveFrom Reflection to Repair: A Scoping Review of Dataset Documentation Tools
Organizations: Robotics and AI Institute · Robotics and AI Institute Cambridge, MA 02142 · Escuela Superior Politécnica del Litoral · Escuela Superior Politécnica del Litoral Guayaquil - Ecuador ·
Abstract
Dataset documentation is widely recognized as essential for the responsible development of automated systems. Despite growing efforts to support documentation through different kinds of artifacts, little is known about the motivations shaping documentation tool design or the factors hindering their adoption. We present a systematic review supported by mixed-methods analysis of 59 dataset documentation publications to examine the motivations behind building documentation tools, how authors conceptualize documentation practices, and how these tools connect to existing systems, regulations, and cultural norms. Our analysis shows four persistent patterns in dataset documentation conceptualization that potentially impede adoption and standardization: unclear operationalizations of documentation's value, decontextualized designs, unaddressed labor demands, and a tendency to treat integration as future work. Building on these findings, we propose a shift in Responsible AI tool design toward institutional rather than individual solutions, and outline actions the HCI community can take to enable sustainable documentation practices.
Figures & tables
| Tools Definitions | ||
|---|---|---|
| Type of tool | Definition | Items |
| Application | Programs, web interfaces or digital tools provided to users as support to create dataset documentation | [ 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 ] |
| Datasheet | Items named "datasheets", as well as proposals that provided structured batteries of questions that once answered became the documentation artifact | [ 83 , 56 , 84 , 85 , 86 , 87 , 1 , 88 , 57 , 89 , 90 , 91 , 92 , 93 ] |
| Framework | Structured, reflexive guidelines (sometimes in the form of questions) aimed at helping users define key information to include in a documentation artifact | [ 94 , 95 , 46 , 96 , 59 , 97 , 98 , 99 , 100 , 101 , 102 , 38 , 58 , 2 , 6 ] |
| Markup format | Machine-readable format, usually implemented as an application in the form of a plugin or package | [ 103 , 104 , 105 , 106 , 107 ] |
| Schema | Proposals that combined metadata information along with structured questions meant to become documentation, similar to datasheets | [ 108 , 109 ] |
Table 1: Definitions and items for each type of dataset documentation tool identified in the systematic review. The table categorizes seven tool types: Application (13 items), Datasheet (14 items), Framework (15 items), Markup format (5 items), Schema (2 items), Study (8 items), and Toolkit (2 items). Classification was based on the language used by authors in each paper when introducing their proposals.
Figure 1: Distribution of different types of tools over time based on the sample in our corpus. It is worth noting that 2025 is an outlier year given our sample was finalized in March of that year. 3 3 3 Distribution of different types of dataset documentation tools over time. Data shows a noticeable increase in the production of these tools between 2021 and 2024
Figure 2: Distribution over time of different approaches to the production of dataset documentation across the six categories of tools in our sample
Figure 3: Comparative analysis of stakeholder engagement and integration pipelines including the percentage of proposals that specifically mentioned stakeholders as part of the design/use stages compared to the number of proposals from within that group that included any stakeholder in the process. It also showcases the percentage of proposals that included stakeholders during the design/testing stages compared to the number of proposals that included concrete features or guidance towards integration to stakeholders.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Distribution of approaches to the construction of dataset documentation over the years. Data shows a prevalence of manual approaches with a sustained increase in automated and semi-automated approaches between 2022 and 2025.
Figure 5: Comparative analysis of different types of tools that included tool evaluation or features towards integration. Left: Count of proposals that included an evaluation study at any stage of the design or integration of the tool. Right: Percentage of proposals that included stakeholders during the design/testing stages compared to the number of proposals that included concrete features or guidance towards integration to stakeholders.
Figure 6: Overlap of different types of tools across different audiences as stated by authors. Data suggests ‘dataset creators’ as a main category of interest for authors, and ‘applications, datasheets and frameworks, as the most preferred categories of tools.
Figure 7: Adapted flowchart from PRISMA outline, describing demonstrating how we searched, screened, and selected items for inclusion in our corpus
| Keywords Queries | ||||
| Resource | Query | Date | Field | Notes |
| NeurIPS | "dataset documentation" OR "dataset transparency" OR "dataset accountability" OR "dataset provenance" OR "dataset datasheet" OR “dataset management” | Feb 1st | All fields search and manual search over Dataset and Benchmarks repositories | This track has contributions from 2021-2024 |
| ACM | Title:(“dataset” AND (“documentation” OR “transparency” OR “accountability” OR “provenance” OR “datasheet” OR “management”)) OR Abstract:(“dataset” AND (“documentation” OR “transparency” OR “accountability” OR “provenance” OR “datasheet” OR “management”)) OR Keyword:(“dataset” AND (“documentation” OR “transparency” OR “accountability” OR “provenance” OR “datasheet” OR “management”)) | Feb 7th | title-abstract-keywords | |
| IEEE | (“Document Title”:“dataset” OR “Abstract”:“dataset” OR “Author Keywords”:“dataset”) AND (“Document Title”:“documentation” OR “Document Title”:“transparency” OR “Document Title”:“accountability” OR “Document Title”:“provenance” OR “Document Title”:“datasheet” OR “Document Title”:“management” OR “Abstract”:“documentation” OR “Abstract”:“transparency” OR “Abstract”:“accountability” OR “Abstract”:“provenance” OR “Abstract”:“datasheet” OR “Abstract”:“management” OR “Author Keywords”:“documentation” OR “Author Keywords”:“transparency” OR “Author Keywords”:“accountability” OR “Author Keywords”:“provenance” OR “Author Keywords”:“datasheet” OR “Author Keywords”:“management”) | Feb 7th | title-abstract-keywords | |
| ScienceDirect | (“dataset documentation”) OR (“dataset transparency”) OR (“dataset accountability”) OR (“dataset provenance”) OR (“dataset datasheet”) OR (“dataset management”) | Feb 7th | all fields search | |
| ArXiv | order: -announced_date_first; size: 200; classification: Computer Science (cs), Economics (econ), Electrical Engineering and Systems Science (eess), Statistics (stat); include_cross_list: True; terms: AND all=“dataset documentation”; OR all=“dataset transparency”; OR all=“dataset accountability”; OR all=“dataset provenance”; OR all=“dataset datasheet” OR all=“dataset management” | Feb 12th | all fields search | |
Table 2: Detailed queries as used for each different resource. ‘Field’ denotes the search fields that were leveraged within each resource in order to return results.
| Corpus Description | ||||
| Tool/Item | Year | Audiences addressed | Degree of Automation | Type of Tool |
| A domain-specific language for describing machine learning datasets [ 71 ] | 2023 | dataset creators | Manual | Application |
| A Field Study of a Human-Centered Process for Increasing AI Transparency [ 110 ] | 2024 | N/A | N/A | Study |
| A generative benchmark creation framework for detecting common data table versions [ 94 ] | 2024 | dataset creators | Automated | Framework |
| A Methodology for Creating AI FactSheets [ 83 ] | 2020 | dataset creators, data curators | Hybrid | Datasheet |
| A Standardized Machine-readable Dataset Documentation Format for Responsible AI [ 103 ] | 2024 | dataset creators, dataset users, dataset auditors | Manual | Markup format |
Table 3: Items included in the review, along with the level of automation for each tool as described in Section 3.2.3 . Audiences are based on target audiences identified by authors in their contributions.
More in this topic
Responsible AI
Responsible AI studies how to design, evaluate, and govern AI systems to promote fairness, safety, transparency, accountability, privacy, and sustainability.
See all 132 papers