To improve the resource utilization jobs running in the production clusters shares the available resources by assigning with different priorities. Preemption is one of the common technique used by the schedulers in Hadoop systems to prevent the delay in the job completion, also to ensure that the available resource is effectively shared among the running jobs. In addition, this prevents large jobs, obtaining too many resources and making others to starve for resources. Jobs submitted to Hadoop systems may have a mixture of lengths, in such environment the existing preemption policies can delay the job completion resulting in the wasting of resources. The main aim of this paper is to size based scheduling using Hadoop Fair Sojourn Protocol with job preemption, which implements a size-based discipline that satisfies simultaneously system responsiveness and fairness requirements.
Download Full PDF Version (Non-Commercial Use)