Interpretable Text Embeddings Based on Hierarchical Questions
Organizations: Japan Advanced Institute of Science and Technology · Independent Researcher
Abstract
QA-based text embeddings represent texts as vectors consisting of 1/0 answers to yes/no questions. Although these embeddings provide detailed information for interpretation and achieve high task performance, they often require humans to examine a large number of questions. Furthermore, the relationship between the number of questions and the appropriate level of question granularity has not been sufficiently studied. To address these challenges, we propose a method for constructing hierarchical questions for text embeddings. As the hierarchy deepens, questions become more specific and their number increases. During interpretation, we reduce the number of questions that humans need to examine by replacing questions at deeper levels with aligned questions at shallower levels. The question hierarchy is constructed through recursive clustering, contrastive prompting, and question alignment. Experimental results on general-purpose tasks show that our method improves interpretability while maintaining task performance. It is also found that, when only a limited number of questions are available, questions at shallower levels tend to achieve higher task performance than those at deeper levels, even with the same number of questions.
Figures & tables
| Level | 1 | 2 | 3 | 4 | Baseline |
| # Clusters | 40 | 200 | 1,000 | 4,996 | 5,000 |
| # Questions | 213 | 861 | 3,462 | 13,914 | 16,419 |
| Level | ||||||||
| 1 | 12 | 36 | 36 | 3 | 5 | 3 | 2 | 0.8 |
| 2 | 10 | 30 | 30 | 3 | 5 | 3 | 2 | 0.8 |
| 3 | 8 | 24 | 24 | 3 | 5 | 3 | 2 | 0.8 |
| 4 | 6 | 18 | 18 | 3 | 5 | 3 | 2 | 0.8 |
| Baseline | 6 | 18 | 18 | 3 | 5 | 3 | 2 | 0.8 |
| Text 1: what level is blood pressure too low | |||
| Text 2: As long as you are not feeling dizzy or passing out, and have a decent quality of life, it is totally okay to have that level of blood pressure. There are a lot of normal people walking around with blood pressures of 90/60 on no medication and it is just normal for them. The only time doctors treat low blood pressure is when there are any symptoms, such as dizziness, light-headedness, or the experience of passing out. Any of these could result in a potentially disastrous fall, so if you experience one of these symptoms then you do want to tell your doctor right away. | |||
| Source Question | A1 | A2 | Target Question |
| Is the article likely to be used in academic settings or coursework? | Yes | Yes | Is the article concerned with |
| Is there a clear intent to explain a concept or process? | Yes | Yes | enhancing understanding or skills? |
| Is the information presented in a clinical or educational context? | Yes | Yes | Does the article discuss physical or |
| Is the information provided related to health or medical conditions? | Yes | Yes | anatomical concepts? |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Model | Spearman Correlation (STS) | |||||||
| STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg. | ||
| Black-box | BERT | 38.78 | 57.98 | 57.98 | 63.15 | 61.06 | 46.35 | 58.40 | 54.81 |
| GloVe | 54.64 | 69.16 | 60.81 | 72.31 | 65.34 | 61.54 | 55.43 | 62.74 | |
| USE | 64.49 | 67.80 | 64.61 | 76.83 | 73.18 | 74.92 | 76.69 | 71.22 | |
| SimCSE (Unsup.) | 66.05 | 81.49 | 73.61 | 79.72 | 78.12 | 76.52 | 72.24 | 75.39 | |
| SBERT (Ori.) | 74.53 | 77.00 | 73.18 | 81.85 | 76.82 | 79.10 | 74.29 | 76.68 | |
| Type | Model | V-Measure (Clustering) | |||||||
| TNG | SE-P2P | BR-P2 | BR-S2S | MR-P2P | MR-S2S | RD-P2P | Avg. | ||
| Black-box | SimCSE (Unsup.) | 23.21 | 28.50 | 24.90 | 19.55 | 23.60 | 21.97 | 45.14 | 26.70 |
| GloVe | 25.83 | 28.51 | 29.27 | 19.18 | 26.12 | 20.38 | 35.82 | 26.44 | |
| BERT | 23.35 | 26.55 | 30.12 | 24.77 | 26.09 | 23.60 | 43.32 | 28.26 | |
| SimCSE (Sup.) | 34.86 | 29.45 | 30.15 | 24.67 | 26.25 | 24.12 | 47.74 | 31.03 | |
| SBERT (New) | 47.47 | 33.13 | 36.99 | 33.21 | 34.25 | 32.24 | 54.80 | 38.87 | |
| Type | Model | nDCG@10 (Retrieval) | ||||||
| MS MARCO | ArguAna | FQA | NFC | SCIDOCS | SciFact | Avg. | ||
| Black-box | BERT | 16.86 | 28.29 | 2.19 | 4.30 | 2.82 | 13.34 | 11.30 |
| SimCSE (Unsup.) | 44.63 | 38.34 | 9.84 | 9.88 | 5.50 | 25.72 | 22.32 | |
| GloVe | 44.27 | 36.30 | 10.09 | 13.87 | 8.04 | 29.58 | 23.69 | |
| SimCSE (Sup.) | 47.86 | 38.33 | 10.41 | 12.42 | 7.53 | 29.59 | 24.36 | |
| SBERT (New) | 88.74 | 47.13 | 37.27 | 32.25 | 21.82 | 62.64 | 48.31 | |
Text Embeddings
Text embeddings map words, sentences, or documents to vectors for measuring meaning, retrieving information, and organizing text.
See all 59 papers