Only aggressive elephants are fast elephants

Jens Dittrich, Jorge Arnulfo Quiańe-Ruiz, Stefan Richter, Stefan Schuh, Alekh Jindal, J̈org Schad

Research output: Contribution to journalArticle

67 Citations (Scopus)

Abstract

Yellow elephants are slow. A major reason is that they consume their inputs entirely before responding to an elephant rider's orders. Some clever riders have trained their yellow elephants to only consume parts of the inputs before responding. However, the teaching time to make an elephant do that is high. So high that the teaching lessons often do not pay off. We take a different approach. We make elephants aggressive; only this will make them very fast. We propose HAIL (Hadoop Aggressive Indexing Library), an enhancement of HDFS and Hadoop MapReduce that dramatically improves runtimes of several classes of MapReduce jobs. HAIL changes the upload pipeline of HDFS in order to create different clustered indexes on each data block replica. An interesting feature of HAIL is that we typically create a win-win situation: we improve both data upload to HDFS and the runtime of the actual Hadoop MapReduce job. In terms of data upload, HAIL improves over HDFS by up to 60% with the default replication factor of three. In terms of query execution, we demonstrate that HAIL runs up to 68x faster than Hadoop. In our experiments, we use six clusters including physical and EC2 clusters of up to 100 nodes. A series of scalability experiments also demonstrates the superiority of HAIL.

Original languageEnglish
Pages (from-to)1591-1602
Number of pages12
JournalProceedings of the VLDB Endowment
Volume5
Issue number11
DOIs
Publication statusPublished - Jul 2012
Externally publishedYes

    Fingerprint

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Computer Science(all)

Cite this

Dittrich, J., Quiańe-Ruiz, J. A., Richter, S., Schuh, S., Jindal, A., & Schad, J. (2012). Only aggressive elephants are fast elephants. Proceedings of the VLDB Endowment, 5(11), 1591-1602. https://doi.org/10.14778/2350229.2350272