Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graph

The approximate nearest neighbor search (ANNS) is a fundamental problem in data base and data mining. A scalable ANNS scheme should be both memory-saving and search-efficient. Traditional graph-based approaches have shown attractive theoretical guarantees on search time complexity, but they all suffer from the problem of high indexing time complexity. Recently, some practical graph-based methods are proposed to reduce the indexing complexity by approximating the traditional graphs and have achieved revolutionary performance on public datasets. However, these works are mainly based on intuitions and lack of rigorous theoretical support. They fail to maximize the potential of graph-based search. In our observation, an ideal graph for ANNS should consider four aspects, (1) ensuring the connectivity of the graph; (2) lowering the average out-degree of the graph for fast traversing; (3) shortening the search path; and (4) avoiding additional index structures to reduce the index size. In this paper, we introduce a new graph structure called Monotonic Relative Neighborhood Graph (MRNG) which takes the four aspects into account simultaneously and guarantees very low search complexity (close to logarithmic time). To further lower the indexing complexity, we propose a novel graph structure named Navigating Spreading-out Graph (NSG) for practical large-scale ANNS problems by approximating the MRNG. Extensive experiments show that NSG outperforms all the existing algorithms significantly. What's more, NSG shows superior performance in the e-commercial search scenario of Taobao (Alibaba Group) and has been integrated into their search engine for billion-scale search.