MOON: MapReduce on opportunistic eNvironments

Heshan Lin, Xiaosong Ma, Jeremy Archuleta, Wu Chun Feng, Mark Gardner, Zhe Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

108 Citations (Scopus)

Abstract

MapReduce offers an ease-of-use programming paradigm for processing large data sets, making it an attractive model for distributed volunteer computing systems. However, unlike on dedicated resources, where MapReduce has mostly been deployed, such volunteer computing systems have significantly higher rates of node unavailability. Furthermore, nodes are not fully controlled by the MapReduce framework. Consequently, we found the data and task replication scheme adopted by existing MapReduce implementations woefully inadequate for resources with high unavailability. To address this, we propose MOON, short for MapReduce On Opportunistic eNvironments. MOON extends Hadoop, an open-source implementation of MapReduce, with adaptive task and data scheduling algorithms in order to offer reliable MapReduce services on a hybrid resource architecture, where volunteer computing systems are supplemented by a small set of dedicated nodes. Our tests on an emulated volunteer computing system, which uses a 60-node cluster where each node possesses a similar hardware configuration to a typical computer in a student lab, demonstrate that MOON can deliver a three-fold performance improvement to Hadoop in volatile, volunteer computing environments.

Original languageEnglish
Title of host publicationHPDC 2010 - Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing
Pages95-106
Number of pages12
DOIs
Publication statusPublished - 16 Dec 2010
Externally publishedYes
Event19th ACM International Symposium on High Performance Distributed Computing, HPDC 2010 - Chicago, IL, United States
Duration: 21 Jun 201025 Jun 2010

Other

Other19th ACM International Symposium on High Performance Distributed Computing, HPDC 2010
CountryUnited States
CityChicago, IL
Period21/6/1025/6/10

    Fingerprint

Keywords

  • Cloud computing
  • MapReduce
  • Volunteer computing

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

Cite this

Lin, H., Ma, X., Archuleta, J., Feng, W. C., Gardner, M., & Zhang, Z. (2010). MOON: MapReduce on opportunistic eNvironments. In HPDC 2010 - Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing (pp. 95-106) https://doi.org/10.1145/1851476.1851489