Dynamic Container-based Resource Management Framework of Spark Ecosystem

Abstract

Apache Spark is known for its robustness in processing large-scale datasets in a distributed computing environment. This form of efficiency is highly observing because of the direct use of Random-Access Memory (RAM) in processing its resilient distributed datasets across the ecosystem. Recently, it is observed that, the memory utilization in computing spark jobs is mainly dependent on job containers, which are closely associated to persistent storage media components. Thus, spark jobs processing relevancy is tightly coupled to the type of storage container and in case of any dynamic resource allocation, the job loses its ratio of resource computation in existing container and increases a functional issue of processing large-scale datasets in spark ecosystem. In this paper, we propose dynamic container-based resource management framework, that shifts coupled associations of job profiles to dynamically available resource containers. Also, it relieves static container allocations and presumes them as a fresh piece of resource allocation for new job profile. The experimental evaluation shows that the proposed dynamic framework reduces wastage of resource allocations and increase ecosystem performance than default job profile in spark ecosystem.