Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graph

The approximate nearest neighbor search (ANNS) is a fundamental problem in machine learning and data mining. An ANNS algorithm is required to be efficient on both memory use and search performance. Recently, graph-based methods have achieved revolutionary performance on public datasets. The search algorithm on a graph is a greedy algorithm. It can be applied to various graph structures and shows promising performance. Recent improvements of the graph-based methods mainly focus on two aspects: (1) Some provide better initial search position to prevent the search from being stuck in some local optima, which is far away from the correct answers. (2) Others try to construct better graphs for faster traversing and neighbor locating. In our observation, an ideal graph for search should consider four aspects, (1) lowering the average out-degree of the graph for fast traversing; (2) ensuring the connectivity of the graph; (3) avoiding detours; and (4) avoiding additional index structures to reduce the index size. None of the previous methods take all these four aspects into consideration simultaneously, which prevents them from achieving better performance. In this paper, we present a novel graph structure named Navigating Spreading-out Graph (NSG) to take the four aspects into account simultaneously. Extensive experiments show that our algorithm outperforms all the existing algorithms significantly. What's more, our algorithm outperforms the existing approach of Taobao (Alibaba Group) and has been integrated into their search engine for billion-scale search.