To solve the approximate nearest neighbor search problem (NNS) on the sphere, we propose a method using locality-sensitive filters (LSF), with the property that nearby vectors have a higher probability of surviving the same filter than vectors which are far apart. We instantiate the filters using spherical caps of height 1 - A, where a vector survives a filter if it is contained in the corresponding spherical cap, and where ideally each filter has an independent, uniformly random direction. For small A, these filters are very similar to the spherical locality-sensitive hash (LSH) family previously studied by Andoni et al. For larger A bounded away from 0, these filters potentially achieve a superior performance, provided we have access to an efficient oracle for finding relevant filters. Whereas existing LSH schemes are limited by a performance parameter of P \geq 1/(2c^2 - 1) to solve approximate NNS with approximation factor c, with spherical LSF we potentially achieve smaller asymptotic values of P, depending on the density of the data set. For sparse data sets where the dimension is super-logarithmic in the size of the data set, we asymptotically obtain P = 1/(2c^2 - 1), while for a logarithmic dimensionality with density constant K we obtain asymptotics of P \sim 1/(4 K c^2). To instantiate the filters and prove the existence of an efficient decoding oracle, we replace the independent filters by filters taken from certain structured random product codes. We show that the additional structure in these concatenation codes allows us to decode efficiently using techniques similar to lattice enumeration, and we can find the relevant filters with low overhead, while at the same time not significantly changing the collision probabilities of the filters. We finally apply spherical LSF to sieving algorithms for solving the shortest vector problem (SVP) on lattices, and show that this leads to a heuristic time complexity for solving SVP in dimension n of (3/2)^{n/2 + o(n)} ~ 2^{0.292 n + o(n)}. This asymptotically improves upon the previous best algorithms for solving SVP which use spherical LSH and cross-polytope LSH and run in time 2^{0.298 n + o(n)}. Experiments with the GaussSieve validate the claimed speedup and show that this method may be practical as well, as the polynomial overhead is small. Our implementation is available under an open-source license.
Additional Metadata
Keywords Lattices, Cryptanalysis, SIeving algorithm
MSC Cryptography (msc 94A60)
THEME Information (theme 2)
ISBN 978-1-61197-433-1
Journal Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms
Conference ACM-SIAM Symposium on Discrete Algorithms
Project Applications of Arithmetic Secret Sharing Schemes in Two-Party Cryptography , Cryptanalysis of Widely-used Hash Function Standards and Beyond , Cryptanalysis of Widely-used Hash Function Standards and Beyond
Note eprint.iacr.org/2015/
Grant This work was funded by the The Netherlands Organisation for Scientific Research (NWO); grant id nwo/617.001.201 - Cryptanalysis of Widely-used Hash Function Standards and Beyond, This work was funded by the The Netherlands Organisation for Scientific Research (NWO); grant id nwo/617.023.117 - Applications of Arithmetic Secret Sharing Schemes in Two-Party Cryptography
Citation
Beck, S, Ducas, L, Gama, N, & Laarhoven, T.M.M. (2016). New directions in nearest neighbor searching with applications to lattice sieving. In Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms (Vol. 1, pp. 10–25).