Automated Keyword Generation Using LLMs in a High-Volume Data Virtualization Service
Authors: Saiful Khan, Yiwen Xing, Joyraj Chakraborty, Afrah Farea, Philip Beaucamp, Samuel Jackson, Min Chen
Organizations: UK Research and Innovation (UKRI), STFC, OX11 0QX, UK. · University of Oxford, Oxford OX1 3QG, UK. · United Kingdom Atomic Energy Authority (UKAEA), OX14 3EB, UK.
In modern data infrastructure, a data virtualization service (DVS) allows users to create many versions of source datasets for different processing tasks. For example, a multivariate or multi-modality dataset can be used for different machine learning tasks (e.g., classification, detection, segmentation, forecasting, and so on). Data selection, participation, integration, and preprocessing in different tasks result in many virtual datasets [Khan et al., SSC 2026]. Meanwhile, different users need to discover existing virtual datasets frequently and easily to avoid duplicate efforts. To enable an effective search engine in a DVS, it is desirable to automatically generate keywords for each virtual dataset. In this work, we present a solution for generating keywords for virtual datasets using large language models (LLMs). We propose five strategies for selecting different collections of text data for processing by the LLMs. We applied the five strategies to virtual datasets created in five application case studies with different source datasets. We conducted experiments with three LLM generators and two embedding techniques. The experiments show that the performances of the five strategies depend on the quality of the text descriptions. The results help us adopt a multi-strategy implementation of a new keyword generation service (KGS) as part of an extended DVS. The work demonstrates that the capability of automated keyword generation is indispensable for a high-volume DVS that is dataset-rich and frequently accessed.
Figures & tables
Fig. 1: An example of creating a virtual dataset: (1) locating a parent dataset and invoking a data transformation; (2) using the corresponding UI to define parameters and create a virtual dataset; and (3) the README file of the virtual dataset captures a variety of information for creating a physical dataset as well as user comments. The information in the README file can be used to generate keywords for the dataset automatically.
Fig. 2: Keyword-related components in a DVS. Bottom: users generate numerous virtual datasets in their workflows for different ML tasks. These virtual datasets, together with their corresponding source datasets, are interrelated, and their relationships are conceptualized as a dataset ontology called DVG. Top-Left: The DVS manages a database about all physical and virtual datasets, including their unique IDs, their metadata, their relationships, as well as the keyword set of every dataset. Top-Right: When a user creates a virtual dataset, the user interface invokes the KGS that generates a keyword set for the newly created virtual dataset and stores it in the database.
Fig. 3: The KGS is a new component of an extended DVS [ 1 ] . Top : It is invoked by one of the UIs for data transformation whenever a new virtual dataset is created. The generated keywords are stored in the DVG (data virtualization graph). Bottom : The stored keywords are used by the DQE (data query engine) in the DVS. The DQE is extended with a new LLM-aided capability to match keywords semantically.
Fig. 4: Five different keyword generation strategies studied in this work.
Type
Model
Parameters
Size
Abbr.
Generator
gemma3:27b
27B
17 GB
G27
Generator
llama3.2:3b
3.2B
2 GB
L3
Generator
Mistral-7B
7.2B
13.4 GB
M7
Embedder
Qwen3-Embedding-8B
8B
4.7 GB
Qwen
Embedder
all-MiniLM-L6-v2
22.7M
91 MB
MiniLM
TABLE I: LLM models used in the implementation of our KGS.
Code
% Meta.
Exp.
Len.
% Cmt.
Remark
M
100%
—
—
0%
metadata-only
EL
0%
E
L
100%
comment-only, expert, long
ES
0%
E
S
100%
comment-only, expert, short
NL
0%
N
L
100%
comment-only, non-expert, long
NS
0%
N
S
100%
comment-only, non-expert, short
MEL
100%
E
L
100%
full README, expert, long
TABLE II: Annotation variant codes. Meta. cov. = fraction of workflow metadata included; Cmt. cov. = fraction of entries with a comment; Exp. = annotator expertise; Len. = comment length.
Fig. 5: For RQ1, RQ7 . F1sem of LLM keywords, extractive baselines, and shuffled ground truth (SGT), evaluated with Qwen and MiniLM. Each dataset’s F1 is averaged over seven input dataset variants and all five methods, at temperature 0. Extractive baselines do not depend on the method, so they are averaged over the variants only. Large markers show means with 95% confidence intervals (CIs); small dots represent individual datasets. The yellow band ranges from SGT-A (chance) to the best extractive baseline (dashed line).
F1sem
F1rescaled
Qwen
MiniLM
Qwen
MiniLM
G27
.749
.532
.30
.29
L3
.728
.474
.27
.25
M7
.721
.441
.23
.21
Best extractive a
.705
.422
.19
.16
SGT-W
.700
.425
.18
.17
TABLE III: For RQ1, RQ7 . Raw and rescaled keyword-quality scores.
Workflow
G27
L3
M7
Best extr.
SGT-W
Helmholtz
0.728
0.706
0.704
0.686
0.694
Image
0.777
0.738
0.741
0.729
0.717
Speech
0.736
0.724
0.713
0.699
0.674
Wave
0.753
0.732
0.722
0.691
0.720
Windfarm
0.754
0.736
0.723
0.710
0.723
TABLE IV: For RQ2 . Workflow-level F1sem scores.
Fig. 6: For RQ3, RQ4, RQ5 . The relative quality statistics of each experimental condition, with the mean value of condition M (metadata only) as the reference point. Keywords were generated at temperature 0 by three generators with all five methods, averaged over all runs. The x -axis encodes the length of input data, and the y -axis encodes the relative changes in percentage against M. Error bars show the 95% CIs of the paired difference to M across the 38 datasets.
Qwen
MiniLM
a. SGT-A
.633
.307
b. F1sem (M)
.719
.458
c. F1sem (MEL)
.736
.491
(b – a)
+.086
+.151
(c – b)
+.017
+.033
TABLE V: For RQ5 . Keyword quality with metadata and expert comment.
Fig. 7: For RQ6 . The keyword generation of the current dataset (child) may benefit from the information stored in the parent datasets. The experiments were conducted under two conditions: (a, M) the child dataset has no user comments and (b, MEL) has long comments from an expert user.
embedder
condition
Mt
Mtf
MtfK
MtfT
MtfKT
Qwen
(a, M)
F1sem
.712
.709
.733
.708
.733
change%
–
− 3.2
27.5
− 4.1
28.1
(b, MEL)
F1sem
.727
.729
.748
.732
.744
change%
20.8
23.2
48.2
26.3
42.8
MiniLM
(a, M)
F1sem
.441
.438
.487
.434
.488
change%
–
− 2.4
35.8
− 5.3
36.3
TABLE VI: For RQ6 . Results for five keyword-generation strategies under the M and MEL conditions.
Fig. 8: For RQ7, RQ1 . Embedder comparison per workflow. (a) Raw F1sem under Qwen and MiniLM. Each bar is split into the chance part (hatched; SGT-A, RQ1) and the part above chance (solid). (b) F1rescaled , which divides the part above chance by 1−SGT-A . (c) Embedder-free Jaccard. Bars show the mean with 95% CI across nodes. Scores are averaged over the seven input variants, all models, all methods, and temperature 0 and repeats.
Fig. 9: For RQ8 . Mean F1sem per workflow vs. sampling temperature (strategy MtfK, five applications, three generators, three runs). σ : standard deviation of F1sem across runs. Self-Jaccard: mean pairwise overlap of the generated keywords across runs. Color bands show 95% CIs.
T:
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Scored with Qwen
Helmholtz
0.715
+.001
−.001
−.002
−.002
−.001
−.004
Wave
0.748
+.001
−.001
−.003
−.003
−.006
−.006
Windfarm
0.760
−.004
−.005
−.006
−.006
−.008
−.010
Speech
0.733
−.004
−.002
−.004
−.006
−.005
−.006
Image
0.754
−.001
−.003
−.004
−.006
−.007
−.009
TABLE VII: For RQ8 . Mean F1sem differences by workflow and temperature.
R@5
R-precision
Qwen
MiniLM
Qwen
MiniLM
a. Random ranking
.132
.132
.032
.032
b. M
.618
.618
.240
.241
c. MEL, MES, MNS, MNL
.669–.689
.660–.690
.272–.290
.270–.292
d. EL, ES, NL, NS
.669–.682
.673–.686
.259–.269
.260–.276
(c – b)
.051–.071
.042–.072
.032–.050
.029–.051
TABLE VIII: For RQ9 . Retrieval performance of LLM-generated keywords.
This paper introduces Bangla Key2Text, a large-scale dataset of 2.6 million Bangla keyword--text pairs designed for keyword-driven text generation in a low-resource language. The dataset is constructed using a BERT-based keyword extraction pipeline applied to millions of Bangla news texts, transforming raw articles into structured keyword--text pairs suitable for supervised learning. To establish baseline performance on this new benchmark, we fine-tune two sequence-to-sequence models, mT5 and BanglaT5, and evaluate them using multiple automatic metrics and human judgments. Experimental results show that task-specific fine-tuning substantially improves keyword-conditioned text generation in Bangla compared to zero-shot large language models. The dataset, trained models, and code are publicly released to support future research in Bangla natural language generation and keyword-to-text generation tasks.
Tonmoy Talukder, G M Shahariar
1Ahsanullah University of Science and Technology · University of California - Riverside
Building datasets remains one of the most manual and brittle parts of AI development. In this technical report, we focus on the most extreme but also most prevalent setting real world practitioners face: a zero data regime. Here, practitioners don't have any data for the capability they want to learn. We introduce Invent-A-Dataset which is a prompt based system to go from dataset description to realistic and large scale post-training datasets. We evaluate Invent-A-Dataset against five frontier model APIs including Anthropic, Google, Open AI, DeepSeek, Zai. Across eight task types and dataset sizes up to 20K samples, Invent-A-Dataset significantly outperforms with both the highest quality (17% relative gains) while simultaneously producing the most diverse samples (19% relative gains). Its diversity advantage widens with scale of training dataset size (from parity at 200 samples to 37% relative gains at 20K samples). This translates into considerable downstream training gains, resulting in far more performant post-trained models. Invent-A-Dataset fine-tune consistently ranks higher compared to other generator fine-tunes across different post-trained model architectures.
We introduce VDCook: a self-evolving video data operating system, a configurable video data construction platform for researchers and vertical domain teams. Users initiate data requests via natural language queries and adjustable parameters (scale, retrieval-synthesis ratio, quality threshold). The system automatically performs query optimization, concurrently running real video retrieval and controlled synthesis modules. It ultimately generates in-domain data packages with complete provenance and metadata, along with reproducible Notebooks. Unlike traditional static, one-time-built datasets, VDCook enables continuous updates and domain expansion through its automated data ingestion mechanism based on MCP (Model Context Protocol)\cite{mcp2024anthropic}, transforming datasets into dynamically evolving open ecosystems. The system also provides multi-dimensional metadata annotation (scene segmentation, motion scoring, OCR ratio, automatic captioning, etc.), laying the foundation for flexible subsequent data `cooking' and indexing\cite{vlogger}. This platform aims to significantly lower the barrier to constructing specialized video training datasets through infrastructure-level solutions, while supporting community contributions and a governance-enabled data expansion paradigm. Project demo: https://screenapp.io/app/v/WP0SvffgsH