Automated Keyword Generation Using LLMs in a High-Volume Data Virtualization Service
Authors: Saiful Khan, Yiwen Xing, Joyraj Chakraborty, Afrah Farea, Philip Beaucamp, Samuel Jackson, Min Chen
Organizations: UK Research and Innovation (UKRI), STFC, OX11 0QX, UK. · University of Oxford, Oxford OX1 3QG, UK. · United Kingdom Atomic Energy Authority (UKAEA), OX14 3EB, UK.
In modern data infrastructure, a data virtualization service (DVS) allows users to create many versions of source datasets for different processing tasks. For example, a multivariate or multi-modality dataset can be used for different machine learning tasks (e.g., classification, detection, segmentation, forecasting, and so on). Data selection, participation, integration, and preprocessing in different tasks result in many virtual datasets [Khan et al., SSC 2026]. Meanwhile, different users need to discover existing virtual datasets frequently and easily to avoid duplicate efforts. To enable an effective search engine in a DVS, it is desirable to automatically generate keywords for each virtual dataset. In this work, we present a solution for generating keywords for virtual datasets using large language models (LLMs). We propose five strategies for selecting different collections of text data for processing by the LLMs. We applied the five strategies to virtual datasets created in five application case studies with different source datasets. We conducted experiments with three LLM generators and two embedding techniques. The experiments show that the performances of the five strategies depend on the quality of the text descriptions. The results help us adopt a multi-strategy implementation of a new keyword generation service (KGS) as part of an extended DVS. The work demonstrates that the capability of automated keyword generation is indispensable for a high-volume DVS that is dataset-rich and frequently accessed.
Figures & tables
Fig. 1: An example of creating a virtual dataset: (1) locating a parent dataset and invoking a data transformation; (2) using the corresponding UI to define parameters and create a virtual dataset; and (3) the README file of the virtual dataset captures a variety of information for creating a physical dataset as well as user comments. The information in the README file can be used to generate keywords for the dataset automatically.
Fig. 2: Keyword-related components in a DVS. Bottom: users generate numerous virtual datasets in their workflows for different ML tasks. These virtual datasets, together with their corresponding source datasets, are interrelated, and their relationships are conceptualized as a dataset ontology called DVG. Top-Left: The DVS manages a database about all physical and virtual datasets, including their unique IDs, their metadata, their relationships, as well as the keyword set of every dataset. Top-Right: When a user creates a virtual dataset, the user interface invokes the KGS that generates a keyword set for the newly created virtual dataset and stores it in the database.
Fig. 3: The KGS is a new component of an extended DVS [ 1 ] . Top : It is invoked by one of the UIs for data transformation whenever a new virtual dataset is created. The generated keywords are stored in the DVG (data virtualization graph). Bottom : The stored keywords are used by the DQE (data query engine) in the DVS. The DQE is extended with a new LLM-aided capability to match keywords semantically.
Fig. 4: Five different keyword generation strategies studied in this work.
Type
Model
Parameters
Size
Abbr.
Generator
gemma3:27b
27B
17 GB
G27
Generator
llama3.2:3b
3.2B
2 GB
L3
Generator
Mistral-7B
7.2B
13.4 GB
M7
Embedder
Qwen3-Embedding-8B
8B
4.7 GB
Qwen
Embedder
all-MiniLM-L6-v2
22.7M
91 MB
MiniLM
TABLE I: LLM models used in the implementation of our KGS.
Code
% Meta.
Exp.
Len.
% Cmt.
Remark
M
100%
—
—
0%
metadata-only
EL
0%
E
L
100%
comment-only, expert, long
ES
0%
E
S
100%
comment-only, expert, short
NL
0%
N
L
100%
comment-only, non-expert, long
NS
0%
N
S
100%
comment-only, non-expert, short
MEL
100%
E
L
100%
full README, expert, long
TABLE II: Annotation variant codes. Meta. cov. = fraction of workflow metadata included; Cmt. cov. = fraction of entries with a comment; Exp. = annotator expertise; Len. = comment length.
Fig. 5: For RQ1, RQ7 . F1sem of LLM keywords, extractive baselines, and shuffled ground truth (SGT), evaluated with Qwen and MiniLM. Each dataset’s F1 is averaged over seven input dataset variants and all five methods, at temperature 0. Extractive baselines do not depend on the method, so they are averaged over the variants only. Large markers show means with 95% confidence intervals (CIs); small dots represent individual datasets. The yellow band ranges from SGT-A (chance) to the best extractive baseline (dashed line).
F1sem
F1rescaled
Qwen
MiniLM
Qwen
MiniLM
G27
.749
.532
.30
.29
L3
.728
.474
.27
.25
M7
.721
.441
.23
.21
Best extractive a
.705
.422
.19
.16
SGT-W
.700
.425
.18
.17
TABLE III: For RQ1, RQ7 . Raw and rescaled keyword-quality scores.
Workflow
G27
L3
M7
Best extr.
SGT-W
Helmholtz
0.728
0.706
0.704
0.686
0.694
Image
0.777
0.738
0.741
0.729
0.717
Speech
0.736
0.724
0.713
0.699
0.674
Wave
0.753
0.732
0.722
0.691
0.720
Windfarm
0.754
0.736
0.723
0.710
0.723
TABLE IV: For RQ2 . Workflow-level F1sem scores.
Fig. 6: For RQ3, RQ4, RQ5 . The relative quality statistics of each experimental condition, with the mean value of condition M (metadata only) as the reference point. Keywords were generated at temperature 0 by three generators with all five methods, averaged over all runs. The x -axis encodes the length of input data, and the y -axis encodes the relative changes in percentage against M. Error bars show the 95% CIs of the paired difference to M across the 38 datasets.
Qwen
MiniLM
a. SGT-A
.633
.307
b. F1sem (M)
.719
.458
c. F1sem (MEL)
.736
.491
(b – a)
+.086
+.151
(c – b)
+.017
+.033
TABLE V: For RQ5 . Keyword quality with metadata and expert comment.
Fig. 7: For RQ6 . The keyword generation of the current dataset (child) may benefit from the information stored in the parent datasets. The experiments were conducted under two conditions: (a, M) the child dataset has no user comments and (b, MEL) has long comments from an expert user.
embedder
condition
Mt
Mtf
MtfK
MtfT
MtfKT
Qwen
(a, M)
F1sem
.712
.709
.733
.708
.733
change%
–
− 3.2
27.5
− 4.1
28.1
(b, MEL)
F1sem
.727
.729
.748
.732
.744
change%
20.8
23.2
48.2
26.3
42.8
MiniLM
(a, M)
F1sem
.441
.438
.487
.434
.488
change%
–
− 2.4
35.8
− 5.3
36.3
TABLE VI: For RQ6 . Results for five keyword-generation strategies under the M and MEL conditions.
Fig. 8: For RQ7, RQ1 . Embedder comparison per workflow. (a) Raw F1sem under Qwen and MiniLM. Each bar is split into the chance part (hatched; SGT-A, RQ1) and the part above chance (solid). (b) F1rescaled , which divides the part above chance by 1−SGT-A . (c) Embedder-free Jaccard. Bars show the mean with 95% CI across nodes. Scores are averaged over the seven input variants, all models, all methods, and temperature 0 and repeats.
Fig. 9: For RQ8 . Mean F1sem per workflow vs. sampling temperature (strategy MtfK, five applications, three generators, three runs). σ : standard deviation of F1sem across runs. Self-Jaccard: mean pairwise overlap of the generated keywords across runs. Color bands show 95% CIs.
T:
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Scored with Qwen
Helmholtz
0.715
+.001
−.001
−.002
−.002
−.001
−.004
Wave
0.748
+.001
−.001
−.003
−.003
−.006
−.006
Windfarm
0.760
−.004
−.005
−.006
−.006
−.008
−.010
Speech
0.733
−.004
−.002
−.004
−.006
−.005
−.006
Image
0.754
−.001
−.003
−.004
−.006
−.007
−.009
TABLE VII: For RQ8 . Mean F1sem differences by workflow and temperature.
R@5
R-precision
Qwen
MiniLM
Qwen
MiniLM
a. Random ranking
.132
.132
.032
.032
b. M
.618
.618
.240
.241
c. MEL, MES, MNS, MNL
.669–.689
.660–.690
.272–.290
.270–.292
d. EL, ES, NL, NS
.669–.682
.673–.686
.259–.269
.260–.276
(c – b)
.051–.071
.042–.072
.032–.050
.029–.051
TABLE VIII: For RQ9 . Retrieval performance of LLM-generated keywords.