Towards zero-overhead static and adaptive indexing in Hadoop

Stefan Richter, Jorge Arnulfo Quiane Ruiz, Stefan Schuh, Jens Dittrich

Research output: Contribution to journalArticle

29 Citations (Scopus)

Abstract

Hadoop MapReduce has evolved to an important industry standard for massive parallel data processing and has become widely adopted for a variety of use-cases. Recent works have shown that indexes can improve the performance of selective MapReduce jobs dramatically. However, one major weakness of existing approaches is high index creation costs. We present HAIL (Hadoop Aggressive Indexing Library), a novel indexing approach for HDFS and Hadoop MapReduce. HAIL creates different clustered indexes over terabytes of data with minimal, often invisible costs, and it dramatically improves runtimes of several classes of MapReduce jobs. HAIL features two different indexing pipelines, static indexing and adaptive indexing. HAIL static indexing efficiently indexes datasets while uploading them to HDFS. Thereby, HAIL leverages the default replication of Hadoop and enhances it with logical replication. This allows HAIL to create multiple clustered indexes for a dataset, e.g., one for each physical replica. Still, in terms of upload time, HAIL matches or even improves over the performance of standard HDFS. Additionally, HAIL adaptive indexing allows for automatic, incremental indexing at job runtime with minimal runtime overhead. For example, HAIL adaptive indexing can completely index a dataset as byproduct of only four MapReduce jobs while incurring an overhead as low as 11 % for the very first of those job only. In our experiments, we show that HAIL improves job runtimes by up to 68× over Hadoop. This article is an extended version of the VLDB 2012 paper (Dittrich et al. in PVLDB 5(11):1591-1602, 2012).

Original languageEnglish
Pages (from-to)469-494
Number of pages26
JournalVLDB Journal
Volume23
Issue number3
DOIs
Publication statusPublished - 1 Jan 2014

Fingerprint

Automatic indexing
Byproducts
Costs
Pipelines
Industry
Experiments

Keywords

  • Adaptive indexing
  • Big data
  • Hadoop
  • HDFS
  • Indexing
  • Map reduce
  • Physical design

ASJC Scopus subject areas

  • Hardware and Architecture
  • Information Systems

Cite this

Towards zero-overhead static and adaptive indexing in Hadoop. / Richter, Stefan; Quiane Ruiz, Jorge Arnulfo; Schuh, Stefan; Dittrich, Jens.

In: VLDB Journal, Vol. 23, No. 3, 01.01.2014, p. 469-494.

Research output: Contribution to journalArticle

Richter, Stefan ; Quiane Ruiz, Jorge Arnulfo ; Schuh, Stefan ; Dittrich, Jens. / Towards zero-overhead static and adaptive indexing in Hadoop. In: VLDB Journal. 2014 ; Vol. 23, No. 3. pp. 469-494.
@article{078821257e474d62bb23da39d84a031e,
title = "Towards zero-overhead static and adaptive indexing in Hadoop",
abstract = "Hadoop MapReduce has evolved to an important industry standard for massive parallel data processing and has become widely adopted for a variety of use-cases. Recent works have shown that indexes can improve the performance of selective MapReduce jobs dramatically. However, one major weakness of existing approaches is high index creation costs. We present HAIL (Hadoop Aggressive Indexing Library), a novel indexing approach for HDFS and Hadoop MapReduce. HAIL creates different clustered indexes over terabytes of data with minimal, often invisible costs, and it dramatically improves runtimes of several classes of MapReduce jobs. HAIL features two different indexing pipelines, static indexing and adaptive indexing. HAIL static indexing efficiently indexes datasets while uploading them to HDFS. Thereby, HAIL leverages the default replication of Hadoop and enhances it with logical replication. This allows HAIL to create multiple clustered indexes for a dataset, e.g., one for each physical replica. Still, in terms of upload time, HAIL matches or even improves over the performance of standard HDFS. Additionally, HAIL adaptive indexing allows for automatic, incremental indexing at job runtime with minimal runtime overhead. For example, HAIL adaptive indexing can completely index a dataset as byproduct of only four MapReduce jobs while incurring an overhead as low as 11 {\%} for the very first of those job only. In our experiments, we show that HAIL improves job runtimes by up to 68× over Hadoop. This article is an extended version of the VLDB 2012 paper (Dittrich et al. in PVLDB 5(11):1591-1602, 2012).",
keywords = "Adaptive indexing, Big data, Hadoop, HDFS, Indexing, Map reduce, Physical design",
author = "Stefan Richter and {Quiane Ruiz}, {Jorge Arnulfo} and Stefan Schuh and Jens Dittrich",
year = "2014",
month = "1",
day = "1",
doi = "10.1007/s00778-013-0332-z",
language = "English",
volume = "23",
pages = "469--494",
journal = "VLDB Journal",
issn = "1066-8888",
publisher = "Springer New York",
number = "3",

}

TY - JOUR

T1 - Towards zero-overhead static and adaptive indexing in Hadoop

AU - Richter, Stefan

AU - Quiane Ruiz, Jorge Arnulfo

AU - Schuh, Stefan

AU - Dittrich, Jens

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Hadoop MapReduce has evolved to an important industry standard for massive parallel data processing and has become widely adopted for a variety of use-cases. Recent works have shown that indexes can improve the performance of selective MapReduce jobs dramatically. However, one major weakness of existing approaches is high index creation costs. We present HAIL (Hadoop Aggressive Indexing Library), a novel indexing approach for HDFS and Hadoop MapReduce. HAIL creates different clustered indexes over terabytes of data with minimal, often invisible costs, and it dramatically improves runtimes of several classes of MapReduce jobs. HAIL features two different indexing pipelines, static indexing and adaptive indexing. HAIL static indexing efficiently indexes datasets while uploading them to HDFS. Thereby, HAIL leverages the default replication of Hadoop and enhances it with logical replication. This allows HAIL to create multiple clustered indexes for a dataset, e.g., one for each physical replica. Still, in terms of upload time, HAIL matches or even improves over the performance of standard HDFS. Additionally, HAIL adaptive indexing allows for automatic, incremental indexing at job runtime with minimal runtime overhead. For example, HAIL adaptive indexing can completely index a dataset as byproduct of only four MapReduce jobs while incurring an overhead as low as 11 % for the very first of those job only. In our experiments, we show that HAIL improves job runtimes by up to 68× over Hadoop. This article is an extended version of the VLDB 2012 paper (Dittrich et al. in PVLDB 5(11):1591-1602, 2012).

AB - Hadoop MapReduce has evolved to an important industry standard for massive parallel data processing and has become widely adopted for a variety of use-cases. Recent works have shown that indexes can improve the performance of selective MapReduce jobs dramatically. However, one major weakness of existing approaches is high index creation costs. We present HAIL (Hadoop Aggressive Indexing Library), a novel indexing approach for HDFS and Hadoop MapReduce. HAIL creates different clustered indexes over terabytes of data with minimal, often invisible costs, and it dramatically improves runtimes of several classes of MapReduce jobs. HAIL features two different indexing pipelines, static indexing and adaptive indexing. HAIL static indexing efficiently indexes datasets while uploading them to HDFS. Thereby, HAIL leverages the default replication of Hadoop and enhances it with logical replication. This allows HAIL to create multiple clustered indexes for a dataset, e.g., one for each physical replica. Still, in terms of upload time, HAIL matches or even improves over the performance of standard HDFS. Additionally, HAIL adaptive indexing allows for automatic, incremental indexing at job runtime with minimal runtime overhead. For example, HAIL adaptive indexing can completely index a dataset as byproduct of only four MapReduce jobs while incurring an overhead as low as 11 % for the very first of those job only. In our experiments, we show that HAIL improves job runtimes by up to 68× over Hadoop. This article is an extended version of the VLDB 2012 paper (Dittrich et al. in PVLDB 5(11):1591-1602, 2012).

KW - Adaptive indexing

KW - Big data

KW - Hadoop

KW - HDFS

KW - Indexing

KW - Map reduce

KW - Physical design

UR - http://www.scopus.com/inward/record.url?scp=84900869467&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84900869467&partnerID=8YFLogxK

U2 - 10.1007/s00778-013-0332-z

DO - 10.1007/s00778-013-0332-z

M3 - Article

VL - 23

SP - 469

EP - 494

JO - VLDB Journal

JF - VLDB Journal

SN - 1066-8888

IS - 3

ER -