Latent entity extraction (LEE) tackles the challenge of identifying implicit, contextually inferred entities within free text-an area where traditional entity extraction methods fall short. In this paper, we introduce LentEx, a novel framework for latent entity extraction that leverages synthetic data generation and instruction fine-tuning to optimize smaller, efficient large language models (LLMs). Latent entities, which are often abstract and thematic, are crucial for applications such as retrieval-augmented generation (RAG), customer persona analysis, and knowledge graph enrichment. LentEx addresses the scarcity of labeled datasets by employing a template-based approach to generate diverse, contextually rich synthetic data, ensuring high variability and alignment with real-world distributions. To our knowledge, LentEx is the first to systematically approach LEE through the lens of LLMs. LentEx demonstrates significant performance improvements across multiple tasks, notably surpassing state-of-the-art models on the MTEB Clustering Benchmark. Furthermore, our methodology enables robust generalization to unseen domains, making LentEx highly applicable in real-world NLP tasks, including RAG and clustering, thereby establishing a new paradigm for latent entity understanding and extraction in natural language processing.
Figures & tables
Fig. 1: Overview of the LentEx workflow in three steps: (1) Template-based domain generation using Claude-3 with few-shot examples, (2) Synthetic input-output instance generation with similarity filtering, and (3) Instruction fine-tuning using Mistral-7B-Instruct-v0.2
Fig. 2: Diversity of synthetic data. Inner circle represents the 100 synthetic domains in synthetic data. Outer circle represents the domains clustered based on their similarity
Fig. 3: Synthetic data distributions: Top figure shows the distribution of length of input tokens across instances, middle figure shows the distribution of range of count of domains and the corresponding number of unique entities, while the bottom figure shows the size of dataset at different values of Rouge-L
Task
Prompt
Domain Generation
Generate a synthetic data dictionary containing 10 example entries. For each entry, the input should describe a broad category of text, and the output should specify a general category of latent entities. Latent entities are those that can be inferred from a text, even though they are not explicitly mentioned in the text. Ensure that the examples span a diverse range of topics, including but not limited to entertainment, education, technology, and health. The output categories should be broad, general, in 1-3 words, like ’movie genre’ or ’product type’, indicative of the kind of inference a reader can make from the given text type. Example: seed_task_dict = { ”sports commentary”: ”sport_name”, ”Google place review”: ”place category”, ”book summary”: ”book genre”, ”legal case_summary”: ”legal case type” }
Input-output Instance Generation
Generate a list of 10 detailed instances, each with their corresponding type or category. For each instance, provide comprehensive content that is representative of its category, ensuring clarity, relevance, and authenticity. The instances should cover a diverse range of subjects, showcasing unique and specific characteristics of each category. Format the output as follows: Each instance should be encapsulated in a dictionary with two keys: ”text_type” and ”latent_entity_type”. The ”text_type” key should include the full content of the instance, detailed and structured appropriately for its type. The ”latent_entity_type” key should specify the category or type that the instance belongs to. Structure each dictionary entry as a separate line in a JSONL format, where each line represents a single, complete dictionary corresponding to one instance and its category. Encapsulate the output in tag ¡rows¿.
Paraphrased Instructions for Training and Inference
1. From the given {text_type}, determine the {latent_entity_type}. 2. Based on the provided {text_type}, identify the {latent_entity_type}. 3. From the specified {text_type}, ascertain the {latent_entity_type}. 4. Using the given {text_type}, pinpoint the {latent_entity_type}. 5. Examine the {text_type} provided and determine the {latent_entity_type}.
TABLE I: Prompts for multiple stages. First row prompt is used in step 1, second row in step 2 of LentEx workflow. Third row shows the paraphrased prompts used in step 3 for instruction fine-tuning the model
Model
Mean Semantic Similarity %
Mistral-7B-Instruct-v0.2
61.17
Claude-3-Haiku
66.48
LentEx (Ours)
77.79
TABLE II: Experiment results from automated evaluation of model predictions on latent entity extraction task
Inference without entity options (Clustering Task)
Model
Average
arxiv-*
biorxiv-*
medrxiv-*
reddit-*
stackexchange-*
twenty-newsgroups
p2p
s2s
p2p
s2s
p2p
s2s
p2p
p2p
gte-Qwen2-7B-instruct
56.92
56.46
51.74
50.09
46.65
46.23
44.13
73.55
74.13
79.86
49.41
53.91
bge-en-icl
57.89
54.44
49.33
53.05
48.38
45.86
44.33
72.33
72.72
81.32
46.05
68.98
NV-Embed-v2
58.46
55.80
51.26
54.09
49.60
46.09
44.86
71.10
74.94
82.10
48.36
64.82
Mistral-7B-Instruct-v0.2
39.74
40.18
37.30
36.94
35.16
40.74
39.53
48.10
36.58
45.81
35.41
41.44
TABLE III: Clustering performance (v-measure scores) across different domains. p2p and s2s denote paragraph-to-paragraph and sentence-to-sentence tasks respectively. Best scores are in bold, second-best are underlined
Model
COLIEE
BioASQ
Precision
Recall
F-1
Precision
Recall
F-1
RAG (Baseline)
0.248
0.167
0.199
0.612
0.441
0.513
RAG + Mistral
0.644
0.536
0.585
0.681
0.496
0.574
RAG + Claude
0.715
0.663
0.688
0.704
0.528
0.603
RAG + Pre-trained NER
0.652
0.513
0.574
0.657
0.513
0.576
RAG + Topic Modeling
0.667
0.591
0.627
0.649
0.502
0.566
TABLE IV: Retrieval Metrics for COLIEE and BioASQ Datasets. CAPTAIN results are only available for COLIEE as it is a legal domain-specific system
Fig. 4: LLM evaluation of model responses over 1000 data points. Claude-3-Sonnet model was prompted to rate model responses as A, B, or C based on the SOP shown. Evaluations for different models were conducted independently, allowing for identical ratings across models.
Fig. 5: Effect of dataset size on model performance. Using synthetic outputs as reference labels, semantic similarity is measured between model predictions and labels. The y-axis represents the mean semantic similarity on test dataset.
Quality Review
Yes %
Is synthetic input text aligned with the text type?
98
Is latent entity label correct?
91.5
TABLE V: Manual data quality evaluation of input texts and labels from synthetic data, on a stratified sample of 2000 examples