University of Illinois Urbana-Champaign

Natjam: Eviction Policies For Supporting Priorities and Deadlines in Mapreduce Clusters

Gupta, Indranil; Cho, Brian; Rahman, Muntasir Raihan; Chajed, Tej; Abad, Cristina L.; Roberts, Nathan; Lin, Philbert

Content Files
newpaper.Ideals.Techrep.Jun7.2013.pdf

Permalink

https://hdl.handle.net/2142/44871

Description

Title
Natjam: Eviction Policies For Supporting Priorities and Deadlines in Mapreduce Clusters
Author(s)
  • Gupta, Indranil
  • Cho, Brian
  • Rahman, Muntasir Raihan
  • Chajed, Tej
  • Abad, Cristina L.
  • Roberts, Nathan
  • Lin, Philbert
Issue Date
2013-06-06
Keyword(s)
  • Cloud Computing
  • Mapreduce
  • Scheduling
Date of Ingest
2013-06-07T16:53:22Z
Abstract
This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! commercial cluster, using both synthetic traces as well as Hadoop cluster traces we obtained from Yahoo!. Our results reveal that Natjam incurs overheads of under 7%. Under real Hadoop workloads, Natjam performs better than existing techniques.
Type of Resource
text
Genre of Resource
Article
Language
en
Permalink
http://hdl.handle.net/2142/44871

Owning Collections

Research and Tech Reports - Computer Science PRIMARY