Distributed Kernel Principal Component Analysis

Kernel Principal Component Analysis (KPCA) is a key technique in machine learning for extracting the nonlinear structure of data and pre-processing it for downstream learning algorithms. We study the distributed setting in which there are multiple servers, each holding a set of points, who wish to compute the principal components of the union of their pointsets. Our main result is a communication efficient algorithm that takes as input arbitrary data points and computes a set of global principal components, that give relative-error approximation for polynomial kernels, or give relative-error approximation with an arbitrarily small additive error for a wide family of kernels including Gaussian kernels. While recent work shows how to do PCA in a distributed setting, the kernel setting is significantly more challenging. Although the "kernel trick" is useful for efficient computation, it is unclear how to use it to reduce communication. The main problem with previous work is that it achieves communication proportional to the dimension of the data points, which if implemented straightforwardly in the kernel setting would give communication either proportional to the dimension of the feature space, or to the number of examples, both of which could be very large. We instead take a roundabout approach, using a careful combination of oblivious subspace embeddings for kernels, oblivious leverage score approximation, adaptive sampling, and sketching for low rank approximation to achieve our result. We also show that the efficacy of our algorithm on large scale datasets.