Implementation Guidelines for Data Quality Metrics

Philipp Jung; Katinka Becker; Felix Biessmann; Valerie Restat; Martin Seyferth; Daniel Schwabe; Lisa Ehrlinger

cs.DBOct 7, 2026

Implementation Guidelines for Data Quality Metrics

Authors: Philipp Jung, Katinka Becker, Felix Biessmann, Valerie Restat, Martin Seyferth, Daniel Schwabe, Lisa Ehrlinger

Organizations: Berliner Hochschule für Technik Berlin, Germany · Physikalisch-Technische Bundesanstalt Berlin, Germany · Einstein Center Digital Future Berlin, Germany · FernUniversität in Hagen Hagen, Germany · Johannes Kepler University Linz Linz, Austria · Hasso Plattner Institute Potsdam, Germany

Abstract

Despite decades of data quality (DQ) research, a gap remains between DQ dimensions, such as accuracy or completeness, which the literature defines in textual form, and DQ tools, which typically implement low-level checks that are not aligned with these dimensions. ISO/IEC 25024 and ISO/IEC 5259 attempt to bridge this gap by defining DQ metrics for each dimension. However, these DQ metrics are hardly used, because the standards leave open how to implement them: for example, the metric for syntactic accuracy counts syntactically accurate values, but does not state how to decide that a value is syntactically accurate. This simply moves the problem to another level without solving it. As a result, DQ assessment currently cannot build on the standards. In this paper, we make the ISO DQ metrics executable. We classify all data-level metrics of both standards into (i) generalizable metrics that need no input beyond the data, (ii) parameterized metrics whose parameters can be learned from clean reference data or set by an expert, and (iii) non-generalizable metrics that need qualitative judgment and cannot be automated. For the metrics that can be automated, i.e., categories (i) and (ii), we propose implementation guidelines that resolve what the standards leave open. We realize the guidelines in dqmeasure, an open-source library of 20 metrics that learns these parameters from reference data instead of relying on manually defined rules. Our experiments on real-world and synthetic datasets show that the metric scores decrease monotonically with an increasing number of injected errors, decline together with downstream ML performance, and scale linearly with the number of rows, which enables automated DQ monitoring based on the standards.

Figures & tables

Panel of Figure 1 — Figure 1. Gap between standard-defined DQ dimensions and tool-specific low-level checks: for Acc-I-1, ISO/IEC 25024 defines the metric but leaves open how to decide that a value is “syntactically accurate”. Our contribution (blue) bridges this gap with implementation guidelines that are realized in the open-source library dqmeasure . Figure showing an operationalization gap between standard-defined data quality dimensions and tool-specific low-level checks. Two top boxes labeled DQ dimensions and Low-level checks are separated by a dashed double-headed arrow labeled "gap". Two blue boxes below highlight the contribution: Implementation guidelines and the dqmeasure library, connected by blue arrows.

ISO/IEC 25024	ISO/IEC 5259	In this paper
data value	data value	cell value
data item, field	data item, field	cell
attribute	feature	column
data record, tuple	data record	row
data file, data set	data frame, dataset	table
data type	data type	column dtype

Table 1. Mapping of ISO concepts onto dqmeasure .

Dimension	ID (5259)	ID (25024)	Name	Category	Reference	Expert	Method
Accessibility	Acs-ML-1	Acs-I-1	User accessibility	N		x
Accessibility	Acs-ML-3		Data accessibility	G
Accuracy	Acc-ML-1	Acc-I-1	Syntactic data accuracy	K	x	x
Accuracy	Acc-ML-2	Acc-I-2	Semantic data accuracy	K	x
Accuracy	Acc-ML-4	Acc-I-4	Risk of inaccuracy	K	x		x
Accuracy	Acc-ML-6	Acc-I-7	Data accuracy range	K	x	x

Table 2. List of data-level DQMs from ISO/IEC 25024 ( International Organization for Standardization, 2015 ) and ISO/IEC 5259 ( International Organization for Standardization, 2024 ) with their generalizability category and required inputs. Dark grey cells mark direct references from ISO/IEC 5259 to ISO/IEC 25024. The remaining rows with two IDs compute the same value under the assumptions in Table 1 . Light blue rows mark the DQMs implemented in dqmeasure , which are all DQMs for accuracy, completeness, consistency, currentness and timeliness with category G or K.

Panel of Figure 2 — Figure 2. Overview of the five implementation guidelines. IG1–IG4 turn a textual DQM definition into an executable DQM; DQMs of category G need no parameters and skip IG3. The result is validated in IG5. Four boxes in a row, connected by arrows: Specify (subject, units, condition, reference), Categorize (category G, K or N), Parameterize (from reference data, by an expert, or by default) and Harmonize (orientation, undefined scores, missing values, results per unit). The arrow from Categorize to Parameterize is labelled K; a second arrow labelled G leads from Categorize over Parameterize directly to Harmonize. A fifth box below, Validate, covers applicability and detection of injected errors.

Figure 3. Experimental procedure: errors are injected into one fold of the dataset ( left ); DQMs learn their parameters on the clean folds and score the corrupted fold ( middle ), and an ML model is trained and evaluated on the same folds ( right ). Three panels. Left: a dataset is split into five folds, errors are inserted into one fold, and that fold becomes the test set while the other four form the training set. Middle: a DQM is fit on the training set and scores the test set. Right: an ML model is fit on the training set and predicts on the test set.

Scenario	Description	Cols.	Dimension a
blank record	Empties entire records; error rate is the share of emptied records	17	Compl.
missing	Removes individual cells	17	Compl.
swap	Replaces a value with another from the same column	14 str b	Acc., Cons.
typo	Keyboard-adjacent misspellings	15 str	Acc., Cons.
outlier	Pushes values past the interquartile fence and adds Gaussian noise	2 num	Acc.
wrong unit	Rescales values by a factor of $1/100$	2 num	Acc.

Table 3. Error scenarios injected into

X_{\text{test}}

, each limited to one error type with columns and DQ dimensions affected.

		Error scenario
DQ metric	clean	missing	blank record	typo	swap	outlier	wrong unit
Data accuracy range	0.999	+0.001 $\pm 0.001$	+0.001 $\pm 0.001$	–	–	-0.348 $\pm 0.013$	+0.000 $\pm 0.000$
Risk of data set inaccuracy	0.903	+0.004 $\pm 0.017$	+0.011 $\pm 0.024$	–	–	-0.404 $\pm 0.042$	-0.204 $\pm 0.009$
Semantic data accuracy	0.983	-0.006 $\pm 0.006$	-0.005 $\pm 0.005$	-0.475 $\pm 0.013$	-0.567 $\pm 0.031$	-0.517 $\pm 0.019$	-0.465 $\pm 0.023$
Syntactic data accuracy	0.986	+0.001 $\pm 0.003$	-0.001 $\pm 0.001$	-0.488 $\pm 0.002$	-0.004 $\pm 0.001$	-0.448 $\pm 0.013$	-0.416 $\pm 0.012$
Empty records	1.000	+0.000 $\pm 0.000$	-0.500 $\pm 0.000$	+0.000 $\pm 0.000$	+0.000 $\pm 0.000$	+0.000 $\pm 0.000$	+0.000 $\pm 0.000$

Table 4. Average change in DQM scores from clean data to a 50% per-column error rate. Since each error scenario corrupts multiple columns, we report the average of the change in DQM score over the affected columns, with the standard deviation over five folds. Changes of at least

0.05

are bold, and a dash marks that a DQM could not be evaluated under this scenario.

Figure 4. The DQ scores drop or stay constant as the error rate increases on the HOSP dataset. Each line represents one DQM’s reaction to the error scenario listed in the legend. Measurements are grouped by data quality dimension. Risk of data inconsistency is excluded from the graph for conciseness. Three line plots, for accuracy, completeness and consistency, show the change in DQ score as the share of erroneous cells rises from 0 to 50 percent. Most lines fall roughly linearly and a few stay at zero. Swapped values impact Semantic data accuracy and Semantic consistency most, down by about 0.55 and 0.9. Record completeness falls to about -0.83 by 25 percent missing values and then levels off.

	Share of erroneous cells
Data quality measure	0%	1%	5%	10%	25%	50%	$\Delta$
ROC-AUC	$0.879$ $\pm 0.003$	$0.878$ $\pm 0.004$	$0.868$ $\pm 0.007$	$0.856$ $\pm 0.010$	$0.826$ $\pm 0.019$	$0.776$ $\pm 0.036$	$\mathbf{-0.103}$
Data accuracy range	$1.000$ $\pm 0.000$	$0.990$ $\pm 0.000$	$0.950$ $\pm 0.000$	$0.900$ $\pm 0.000$	$0.750$ $\pm 0.000$	$0.500$ $\pm 0.000$	$\mathbf{-0.500}$
Risk of data set inaccuracy	$1.000$ $\pm 0.000$	$0.994$ $\pm 0.005$	$0.972$ $\pm 0.026$	$0.948$ $\pm 0.050$	$0.884$ $\pm 0.113$	$0.811$ $\pm 0.139$	$\mathbf{-0.189}$
Syntactic data accuracy	$1.000$ $\pm 0.000$	$1.000$ $\pm 0.000$	$0.997$ $\pm 0.000$	$0.995$ $\pm 0.000$	$0.987$ $\pm 0.001$	$0.975$ $\pm 0.001$	$-0.025$
Empty records	$1.000$ $\pm 0.000$	$1.000$ $\pm 0.000$	$1.000$ $\pm 0.000$	$1.000$ $\pm 0.000$	$1.000$ $\pm 0.000$	$1.000$ $\pm 0.000$	$0.000$

Table 5. Mean DQ score over the columns the error scenario damages on the SynTabFall dataset, averaged over five folds (

\pm

standard deviation). The first row is the classifier’s ROC-AUC.

\Delta

is the change from

0\%

50\%

(bold at

0.05

or more), and a dash marks a DQM that returned no score.

Figure 5. On the fall risk prediction ML task (SynTabFall), the scores of the DQMs that react to the errors fall together with the classifier’s ROC-AUC as the error rate rises. The left panel shows DQMs whose score changes by

0.05

or more, the right panel those that change less. Whiskers show the standard deviation of ROC-AUC across folds. Two stacked line plots show the change in DQ score (left axis) and in ROC-AUC (right axis) as the share of erroneous cells rises from 0 to 50 percent. In the top plot, five DQMs fall by between 0.2 and 0.5. In the bottom plot, three DQMs fall by less than 0.05. In both, ROC-AUC falls in step to about -0.10.

Figure 6. DQMs’ fit and score scale linearly with the number of rows and columns. The four panels on the left show that, below around

10^{5}

rows, a fixed overhead per call dominates the runtime. The rightmost panel fixes

10^{7}

rows and shows only the DQMs that read more than one column and are affected by the number of columns. Lines show the mean over ten repetitions, error bars the standard deviation. Four log-log plots, one per DQ dimension accuracy, completeness, consistency and currentness, show seconds per measure against 1,000 to about 50 million rows. Most runtimes stay near one millisecond up to about 100,000 rows and then rise in linearly with the row count; the slowest DQMs, Semantic consistency and Data format consistency, reach about 100 seconds. A fifth plot fixes 10 million rows and shows runtime rising gradually from 5 to 25 columns for five DQMs.

Figure 7. Left: Semantic data accuracy’s runtime over ten repetitions with the per-repetition linear fit and its 95% confidence band. Right: the fit extrapolated to the full dataset. Left: runtime of Semantic data accuracy at 300 to 900 rows, with ten repetitions per size. A linear fit gives 30.5 ± 4.6 ms per request with an intercept of 1.7 ± 2.3 seconds, R2 = 0.76. Right: the fit extended on log-log axes gives 8.5 ± 1.3 hours at about one million rows and 15 ± 2 days at about 40 million rows.

Explore similar work

XQDT: eXplainable and Quantitative Data-Text Alignment Metric with Feedback Signals
Aug 30, 2026Kun Efimov-Zhang, Yifei Song, Claire GardentAutomated Evaluation Explainability Evaluation
CNRS/LORIA and Université de Lorraine
Data Quality Profiling at Scale with Progressive Sampling: A Benchmark for Data-Centric AI Pipelines
Jul 28, 2026Laure Berti-EquilleImportance Sampling Adaptive Sampling
IRD, ESPACE-DEV, 500, rue Jean-François Breton, 34093 Montpellier, France
How Output Format Confounds Data Quality and Capability in Instruction Tuning
Sep 2, 2026Chengguang Gan, Hanjun Wei, Yunhao Liang +3LLM Evaluation Instruction Tuning
1Independent Researcher · University of Chinese Academy of Sciences · 3Pusan National University +2

Aug 30, 2026, cs.CLJ/K move · Enter open · S save

XQDT: eXplainable and Quantitative Data-Text Alignment Metric with Feedback Signals

Kun Efimov-Zhang, Yifei Song, Claire Gardent

CNRS/LORIA and Université de Lorraine

Automated Evaluation Explainability Evaluation Text Generation

Open paper AskPDF arXiv

Evaluating data-text alignment remains challenging: existing metrics often provide limited explanations for the scores, while prompt-based LLM-as-Judge methods can be expensive and unreliable. We present an end-to-end explainable evaluation metric that fine-tunes a language model to identify omitted, extra, incorrect, and correct data units in a data-text pair. These local judgements are aggregated into precision, recall, and F1 scores, providing both fine-grained diagnostic feedback and an interpretable measure of alignment quality. Across benchmarks, our fine-tuned models outperform LLM-as-Judge methods in error prediction and achieve competitive precision, recall, and F1 scores, while maintaining strong correlation with human judgements. Beyond evaluation, our verifier outputs also provide useful feedback signals for downstream correction and refinement, supporting alignment-oriented improvement of data-to-text and text-to-data. Code and resources are available at https://github.com/guihuzhang/xqdt.

Abstract

Explore similar work

XQDT: eXplainable and Quantitative Data-Text Alignment Metric with Feedback Signals

Data Quality Profiling at Scale with Progressive Sampling: A Benchmark for Data-Centric AI Pipelines

How Output Format Confounds Data Quality and Capability in Instruction Tuning