Dense Embeddings

Dense embeddings represent data points (text, images, etc.) as high-dimensional vectors, aiming to capture semantic relationships and facilitate tasks like retrieval and classification. Current research focuses on improving the efficiency and interpretability of these embeddings, exploring techniques like sparse autoencoders to disentangle semantic concepts and non-parametric methods for efficient fine-tuning. These advancements are crucial for improving the accuracy and scalability of various applications, including information retrieval, medical coding prediction, and object tracking, while simultaneously enhancing the explainability of complex models.

Papers

November 6, 2024

From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models
Charles Zhang, Benji Peng, Xintian Sun, Qian Niu, Junyu Liu, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Ming Liu, Yichao Zhang, Cheng Fei, Caitlyn Heqi Yin, Lawrence KQ Yan, Tianyang Wang
Language Model Word Embeddings Future Direction Barzilai Borwein Technique Word Vector Multimodal Embeddings Document Embeddings Dense Embeddings

September 16, 2024

DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction
John Wu, David Wu, Jimeng Sun
Mechanistic Interpretability Label Prediction Label Attention Dense Embeddings Medical Code Prediction

September 4, 2024

NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval
Sepanta Zeighami, Zac Wellmer, Aditya Parameswaran
Retrieval Augmented Generation Jina Embeddings App to App Retrieval Image Retrieval Digital Algorithmic Nudging K Nearest Neighbor Search Dense Embeddings

August 1, 2024

Disentangling Dense Embeddings with Sparse Autoencoders
Charles O'Neill, Christine Ye, Kartheik Iyer, John F. Wu
Sparse Autoencoders Semantic Fidelity Dense Embeddings Dense Text

June 11, 2024

CUPID: Contextual Understanding of Prompt-conditioned Image Distributions
Yayan Zhao, Mingwei Li, Matthew Berger
Generative Model Prompt Learning Contextual Understanding Image Distribution Dense Embeddings

September 1, 2022

Johnson-Lindenstrauss embeddings for noisy vectors -- taking advantage of the noise
Zhen Shao
Industrial Disturbing Noise Advantage Feedback Sparse Embeddings Euclidean Embeddings Dense Embeddings High Dimensional Noisy Johnson Lindenstrauss

April 15, 2022

Knowledgebra: An Algebraic Learning Framework for Knowledge Graph
Tong Yang, Yifei Wang, Long Sha, Jan Engelbrecht, Pengyu Hong
Knowledge Graph Knowledge Representation Dense Embeddings Algebraic Learning

April 1, 2022

Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings
Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Defu Lian, Yeyun Gong, Qi Chen, Fan Yang, Hao Sun, Yingxia Shao, Denvy Deng, Qi Zhang, Xing Xie
Vector Quantization Retrieval Performance Distilling Knowledge Vector Quantisation Dense Embeddings Dataset Quantization

January 14, 2022

Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval
Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Yingxia Shao, Defu Lian, Chaozhuo Li, Hao Sun, Denvy Deng, Liangjie Zhang, Qi Zhang, Xing Xie
App to App Retrieval Potential Scalability Embedding Based Retrieval Sparse Embeddings Dense Embeddings

December 9, 2021

Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddings
Mel Vecerik, Jackie Kay, Raia Hadsell, Lourdes Agapito, Jon Scholz
Computer Vision Task Sparse Network Keypoint Detection Task Adaptation Shot Adaptation Latent Embeddings Dense Embeddings

Dense Embeddings

Papers

From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models

DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction

NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval

Disentangling Dense Embeddings with Sparse Autoencoders

CUPID: Contextual Understanding of Prompt-conditioned Image Distributions

Johnson-Lindenstrauss embeddings for noisy vectors -- taking advantage of the noise

Knowledgebra: An Algebraic Learning Framework for Knowledge Graph

Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings

Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval

Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddings