A Fundamental Tradeoff between Computation and Communication in Distributed Computing
- OT

How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff relationship between computation and communication in distributed computing, i.e., the two are inverse-linearly proportional to each other. More specifically, a general distributed computing framework, motivated by commonly used structures like MapReduce, is considered, where the goal is to compute arbitrary output functions from input files, by decomposing the overall computation into computing a set of "Map" and "Reduce" functions distributedly across computing nodes. A coded scheme, named "Coded Distributed Computing" (CDC), is proposed to demonstrate that increasing the computation load of the Map phase by a factor of (i.e., evaluating each Map function at carefully chosen nodes) can create novel coding opportunities in the data shuffling phase that reduce the communication load by the same factor. An information-theoretic lower bound on the communication load is also provided, which matches the communication load achieved by the CDC scheme. As a result, the optimal computation-communication tradeoff in distributed computing is exactly characterized.
View on arXiv