RAFTing MapReduce

Fast recovery on the RAFT

Jorge Arnulfo Quiane Ruiz, Christoph Pinkel, Jorg Schad, Jens Dittrich

Research output: Chapter in Book/Report/Conference proceedingConference contribution

41 Citations (Scopus)

Abstract

MapReduce is a computing paradigm that has gained a lot of popularity as it allows non-expert users to easily run complex analytical tasks at very large-scale. At such scale, task and node failures are no longer an exception but rather a characteristic of large-scale systems. This makes fault-tolerance a critical issue for the efficient operation of any application. MapReduce automatically reschedules failed tasks to available nodes, which in turn recompute such tasks from scratch. However, this policy can significantly decrease performance of applications. In this paper, we propose a family of Recovery Algorithms for Fast-Tracking (RAFT) MapReduce. As ease-of-use is a major feature of MapReduce, RAFT focuses on simplicity and also non-intrusiveness, in order to be implementation-independent. To efficiently recover from task failures, RAFT exploits the fact that MapReduce produces and persists intermediate results at several points in time. RAFT piggy-backs checkpoints on the task progress computation. To deal with multiple node failures, we propose query metadata checkpointing. We keep track of the mapping between input key-value pairs and intermediate data for all reduce tasks. Thereby, RAFT does not need to re-execute completed map tasks entirely. Instead RAFT only recomputes intermediate data that were processed for local reduce tasks and hence not shipped to another node for processing. We also introduce a scheduling strategy taking full advantage of these recovery algorithms. We implemented RAFT on top of Hadoop and evaluated it on a 45-node cluster using three common analytical tasks. Overall, our experimental results demonstrate that RAFT outperforms Hadoop runtimes by 23% on average under task and node failures. The results also show that RAFT has negligible runtime overhead.

Original languageEnglish
Title of host publicationProceedings - International Conference on Data Engineering
Pages589-600
Number of pages12
DOIs
Publication statusPublished - 6 Jun 2011
Externally publishedYes
Event2011 IEEE 27th International Conference on Data Engineering, ICDE 2011 - Hannover, Germany
Duration: 11 Apr 201116 Apr 2011

Other

Other2011 IEEE 27th International Conference on Data Engineering, ICDE 2011
CountryGermany
CityHannover
Period11/4/1116/4/11

Fingerprint

Recovery
Fault tolerance
Metadata
Large scale systems
Scheduling
Processing

ASJC Scopus subject areas

  • Information Systems
  • Signal Processing
  • Software

Cite this

Quiane Ruiz, J. A., Pinkel, C., Schad, J., & Dittrich, J. (2011). RAFTing MapReduce: Fast recovery on the RAFT. In Proceedings - International Conference on Data Engineering (pp. 589-600). [5767877] https://doi.org/10.1109/ICDE.2011.5767877

RAFTing MapReduce : Fast recovery on the RAFT. / Quiane Ruiz, Jorge Arnulfo; Pinkel, Christoph; Schad, Jorg; Dittrich, Jens.

Proceedings - International Conference on Data Engineering. 2011. p. 589-600 5767877.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Quiane Ruiz, JA, Pinkel, C, Schad, J & Dittrich, J 2011, RAFTing MapReduce: Fast recovery on the RAFT. in Proceedings - International Conference on Data Engineering., 5767877, pp. 589-600, 2011 IEEE 27th International Conference on Data Engineering, ICDE 2011, Hannover, Germany, 11/4/11. https://doi.org/10.1109/ICDE.2011.5767877
Quiane Ruiz JA, Pinkel C, Schad J, Dittrich J. RAFTing MapReduce: Fast recovery on the RAFT. In Proceedings - International Conference on Data Engineering. 2011. p. 589-600. 5767877 https://doi.org/10.1109/ICDE.2011.5767877
Quiane Ruiz, Jorge Arnulfo ; Pinkel, Christoph ; Schad, Jorg ; Dittrich, Jens. / RAFTing MapReduce : Fast recovery on the RAFT. Proceedings - International Conference on Data Engineering. 2011. pp. 589-600
@inproceedings{9a88678a7c4d4ebc89764916fb58245c,
title = "RAFTing MapReduce: Fast recovery on the RAFT",
abstract = "MapReduce is a computing paradigm that has gained a lot of popularity as it allows non-expert users to easily run complex analytical tasks at very large-scale. At such scale, task and node failures are no longer an exception but rather a characteristic of large-scale systems. This makes fault-tolerance a critical issue for the efficient operation of any application. MapReduce automatically reschedules failed tasks to available nodes, which in turn recompute such tasks from scratch. However, this policy can significantly decrease performance of applications. In this paper, we propose a family of Recovery Algorithms for Fast-Tracking (RAFT) MapReduce. As ease-of-use is a major feature of MapReduce, RAFT focuses on simplicity and also non-intrusiveness, in order to be implementation-independent. To efficiently recover from task failures, RAFT exploits the fact that MapReduce produces and persists intermediate results at several points in time. RAFT piggy-backs checkpoints on the task progress computation. To deal with multiple node failures, we propose query metadata checkpointing. We keep track of the mapping between input key-value pairs and intermediate data for all reduce tasks. Thereby, RAFT does not need to re-execute completed map tasks entirely. Instead RAFT only recomputes intermediate data that were processed for local reduce tasks and hence not shipped to another node for processing. We also introduce a scheduling strategy taking full advantage of these recovery algorithms. We implemented RAFT on top of Hadoop and evaluated it on a 45-node cluster using three common analytical tasks. Overall, our experimental results demonstrate that RAFT outperforms Hadoop runtimes by 23{\%} on average under task and node failures. The results also show that RAFT has negligible runtime overhead.",
author = "{Quiane Ruiz}, {Jorge Arnulfo} and Christoph Pinkel and Jorg Schad and Jens Dittrich",
year = "2011",
month = "6",
day = "6",
doi = "10.1109/ICDE.2011.5767877",
language = "English",
isbn = "9781424489589",
pages = "589--600",
booktitle = "Proceedings - International Conference on Data Engineering",

}

TY - GEN

T1 - RAFTing MapReduce

T2 - Fast recovery on the RAFT

AU - Quiane Ruiz, Jorge Arnulfo

AU - Pinkel, Christoph

AU - Schad, Jorg

AU - Dittrich, Jens

PY - 2011/6/6

Y1 - 2011/6/6

N2 - MapReduce is a computing paradigm that has gained a lot of popularity as it allows non-expert users to easily run complex analytical tasks at very large-scale. At such scale, task and node failures are no longer an exception but rather a characteristic of large-scale systems. This makes fault-tolerance a critical issue for the efficient operation of any application. MapReduce automatically reschedules failed tasks to available nodes, which in turn recompute such tasks from scratch. However, this policy can significantly decrease performance of applications. In this paper, we propose a family of Recovery Algorithms for Fast-Tracking (RAFT) MapReduce. As ease-of-use is a major feature of MapReduce, RAFT focuses on simplicity and also non-intrusiveness, in order to be implementation-independent. To efficiently recover from task failures, RAFT exploits the fact that MapReduce produces and persists intermediate results at several points in time. RAFT piggy-backs checkpoints on the task progress computation. To deal with multiple node failures, we propose query metadata checkpointing. We keep track of the mapping between input key-value pairs and intermediate data for all reduce tasks. Thereby, RAFT does not need to re-execute completed map tasks entirely. Instead RAFT only recomputes intermediate data that were processed for local reduce tasks and hence not shipped to another node for processing. We also introduce a scheduling strategy taking full advantage of these recovery algorithms. We implemented RAFT on top of Hadoop and evaluated it on a 45-node cluster using three common analytical tasks. Overall, our experimental results demonstrate that RAFT outperforms Hadoop runtimes by 23% on average under task and node failures. The results also show that RAFT has negligible runtime overhead.

AB - MapReduce is a computing paradigm that has gained a lot of popularity as it allows non-expert users to easily run complex analytical tasks at very large-scale. At such scale, task and node failures are no longer an exception but rather a characteristic of large-scale systems. This makes fault-tolerance a critical issue for the efficient operation of any application. MapReduce automatically reschedules failed tasks to available nodes, which in turn recompute such tasks from scratch. However, this policy can significantly decrease performance of applications. In this paper, we propose a family of Recovery Algorithms for Fast-Tracking (RAFT) MapReduce. As ease-of-use is a major feature of MapReduce, RAFT focuses on simplicity and also non-intrusiveness, in order to be implementation-independent. To efficiently recover from task failures, RAFT exploits the fact that MapReduce produces and persists intermediate results at several points in time. RAFT piggy-backs checkpoints on the task progress computation. To deal with multiple node failures, we propose query metadata checkpointing. We keep track of the mapping between input key-value pairs and intermediate data for all reduce tasks. Thereby, RAFT does not need to re-execute completed map tasks entirely. Instead RAFT only recomputes intermediate data that were processed for local reduce tasks and hence not shipped to another node for processing. We also introduce a scheduling strategy taking full advantage of these recovery algorithms. We implemented RAFT on top of Hadoop and evaluated it on a 45-node cluster using three common analytical tasks. Overall, our experimental results demonstrate that RAFT outperforms Hadoop runtimes by 23% on average under task and node failures. The results also show that RAFT has negligible runtime overhead.

UR - http://www.scopus.com/inward/record.url?scp=79957796324&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79957796324&partnerID=8YFLogxK

U2 - 10.1109/ICDE.2011.5767877

DO - 10.1109/ICDE.2011.5767877

M3 - Conference contribution

SN - 9781424489589

SP - 589

EP - 600

BT - Proceedings - International Conference on Data Engineering

ER -