Reliable MapReduce computing on opportunistic resources

Heshan Lin, Xiaosong Ma, Wu chun Feng

Research output: Contribution to journalArticle

12 Citations (Scopus)


MapReduce offers an ease-of-use programming paradigm for processing large data sets, making it an attractive model for opportunistic compute resources. However, unlike dedicated resources, where MapReduce has mostly been deployed, opportunistic resources have significantly higher rates of node volatility. As a consequence, the data and task replication scheme adopted by existing MapReduce implementations is woefully inadequate on such volatile resources. In this paper, we propose MOON, short for MapReduce On Opportunistic eNvironments, which is designed to offer reliable MapReduce service for opportunistic computing. MOON adopts a hybrid resource architecture by supplementing opportunistic compute resources with a small set of dedicated resources, and it extends Hadoop, an open-source implementation of MapReduce, with adaptive task and data scheduling algorithms to take advantage of the hybrid resource architecture. Our results on an emulated opportunistic computing system running atop a 60-node cluster demonstrate that MOON can deliver significant performance improvements to Hadoop on volatile compute resources and even finish jobs that are not able to complete in Hadoop.

Original languageEnglish
Pages (from-to)145-161
Number of pages17
JournalCluster Computing
Issue number2
Publication statusPublished - 1 Jun 2012
Externally publishedYes



  • Cloud computing
  • MapReduce
  • Volunteer computing

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications

Cite this