Publication Date
1-1-2018
Document Type
Conference Proceeding
Publication Title
Proceedings of the 21stInternational Conference on Extending Database Technology (EDBT), March 26-29,2018
DOI
10.5441/002/edbt.2018.33
First Page
373
Last Page
384
Abstract
The concept of similarity is used as the basis for many data exploration and data mining tasks. Nearest Neighbor (NN) queries identify the most similar items, or in terms of distance the closest points to a query point. Similarity is traditionally characterized using a distance function between multi-dimensional feature vectors. However, when the data is high-dimensional, traditional distance functions fail to significantly distinguish between the closest and furthest points, as few dissimilar dimensions dominate the distance function. Localized similarity functions, i.e. functions that only consider dimensions close to the query, quantize each dimension independently and only compute similarity for the dimensions where the query and the points fall into the same bin. These quantizations are query-agnostic. There is potential to improve accuracy when a query-dependent quantization is used. In this paper we propose a Query dependent Equi-Depth (QED) on-the-fly quantization method to improve high-dimensional similarity searches. The quantization is done for each dimension at query time and localized scores are generated for the closest p fraction of the points while a constant penalty is applied for the rest of the points. QED not only improves the quality of the distance metric, but also improves query time performance by filtering out non relevant data. We propose a distributed indexing and query algorithm to efficiently compute QED. Our experimental results show improvements in classification accuracy as well as query performance up to one order of magnitude faster than Manhattan-based sequential scan NN queries over datasets with hundreds of dimensions.
Funding Number
1557578
Funding Sponsor
National Science Foundation
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
Department
Computer Engineering
Recommended Citation
Gheorghi Guzun and Guadalupe Canahuate. "Distributed query-aware quantization for high-dimensional similarity searches" Proceedings of the 21stInternational Conference on Extending Database Technology (EDBT), March 26-29,2018 (2018): 373-384. https://doi.org/10.5441/002/edbt.2018.33
