Pilot-Data: An Abstraction for Distributed Data

The challenge of "Big Data" extends beyond the simple storage and management of large volumes of data. Science that involves and depends upon large amounts of data, also requires overcoming challenges in multiple other areas, including managing large-scale data distribution as well as co-placement and scheduling with computing resources. Although there exist multiple approaches to addressing each of these challenges, an integrative approach is missing; furthermore extending existing functionality or enabling cross-implementation capability for existing implementations remains difficult at best. To address the fundamental challenges of co-placement and scheduling of data and compute in heterogenous and distributed environments with interoperability and extensibility as first-order concerns, we define the concept of Pilot-Data, in analogy with, and symmetrical to, Pilot-Jobs. In this paper, we design and implement a Pilot-Data prototype, and deploy it on multiple production distributed cyberinfrastructure. We validate the concept of Pilot-Data by establishing that it provides a simple abstraction for managing data placement, whilst supporting interoperability across distributed environments and late-binding. Our experiments characterize the performance of an implementation of the Pilot-Data concept. They show a level of performance that is suitable for most large-scale scientific applications on (production) distributed cyberinfrastructure. We demonstrate how the concept of Pilot-Data also provide the basis upon which to build tools and support capabilities like affinity which in turn can be used for advanced data-compute co-placement and scheduling.