Distributed Computing with the Cell Broadband Engine

Abstract

The rapid improvements in the availability of commodity high-performance components has resulted in a proliferation of networked devices, making scalable computing clusters the standard platform for many high-performance and large-scale applications. However, the process of parallelizing applications for such distributed environments is a challenging task, requiring explicit management of concurrency and data locality. While there exists many frameworks and platforms to assist with this process, like Google’s MapReduce, Microsoft’s Dryad and Azure, Yahoo’s Pig Latin programming language, and the Condor framework, they are usually targeted towards off-line batch processing of large quantities of data, contrary to real-time offloading of compute intensive tasks. Moreover, MapReduce, Dryad, and Pig Latin may not be suitable for all application domains, due to their inability to model branching and iterative algorithms. In this thesis, we present a design for a framework able to accelerate applications by offloading compute intensive tasks to a heterogeneous distributed environment, and provide a prototype implementation for the Cell Broadband Engine. We evaluate the framework performance and scalability, and propose several future enhancements to further increase performance. Our results show that compute intensive applications that allow for high numbers of concurrent jobs fits well to our framework, and shows good scalability.