ReStore: Reusing results of MapReduce jobs in pig

Iman Elghandour, Ashraf Aboulnaga

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Citations (Scopus)

Abstract

Analyzing large scale data has become an important activity for many organizations, and is now facilitated by the MapReduce programming and execution model and its implementations, most notably Hadoop. Query languages such as Pig Latin, Hive, and Jaql make it simpler for users to express complex analysis tasks, and the compilers of these languages translate these complex tasks into workflows of MapReduce jobs. Each job in these workflows reads its input from the distributed file system used by the MapReduce system (e.g., HDFS in the case of Hadoop) and produces output that is stored in this distributed file system. This output is then read as input by the next job in the workflow. The current practice is to delete these intermediate results from the distributed file system at the end of executing the workflow. It would be more useful if these intermediate results can be stored and reused in future workflows. We demonstrate ReStore, an extension to Pig that enables it to manage storage and reuse of intermediate results of the MapReduce workflows executed in the Pig data analysis system. ReStore matches input workflows of MapReduce jobs with previously executed jobs and rewrites these workflows to reuse the stored results of the matched jobs. ReStore also creates additional reuse opportunities by materializing and reserving the output of query execution operators that are executed within a MapReduce job. In this demonstration we showcase the MapReduce jobs and sub-jobs recommended by ReStore for a given Pig query, the rewriting of input queries to reuse stored intermediate results, and a what-if analysis of the effectiveness of reusing stored outputs of previously executed jobs.

Original languageEnglish
Title of host publicationProceedings of the ACM SIGMOD International Conference on Management of Data
Pages701-704
Number of pages4
DOIs
Publication statusPublished - 28 Jun 2012
Externally publishedYes
Event2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12 - Scottsdale, AZ, United States
Duration: 21 May 201224 May 2012

Other

Other2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12
CountryUnited States
CityScottsdale, AZ
Period21/5/1224/5/12

Fingerprint

Query languages
Demonstrations

Keywords

  • data reuse
  • mapreduce
  • pig latin

ASJC Scopus subject areas

  • Software
  • Information Systems

Cite this

Elghandour, I., & Aboulnaga, A. (2012). ReStore: Reusing results of MapReduce jobs in pig. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 701-704) https://doi.org/10.1145/2213836.2213937

ReStore : Reusing results of MapReduce jobs in pig. / Elghandour, Iman; Aboulnaga, Ashraf.

Proceedings of the ACM SIGMOD International Conference on Management of Data. 2012. p. 701-704.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Elghandour, I & Aboulnaga, A 2012, ReStore: Reusing results of MapReduce jobs in pig. in Proceedings of the ACM SIGMOD International Conference on Management of Data. pp. 701-704, 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12, Scottsdale, AZ, United States, 21/5/12. https://doi.org/10.1145/2213836.2213937
Elghandour I, Aboulnaga A. ReStore: Reusing results of MapReduce jobs in pig. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 2012. p. 701-704 https://doi.org/10.1145/2213836.2213937
Elghandour, Iman ; Aboulnaga, Ashraf. / ReStore : Reusing results of MapReduce jobs in pig. Proceedings of the ACM SIGMOD International Conference on Management of Data. 2012. pp. 701-704
@inproceedings{2c79ccc0ac2441969705e50e2f786bd2,
title = "ReStore: Reusing results of MapReduce jobs in pig",
abstract = "Analyzing large scale data has become an important activity for many organizations, and is now facilitated by the MapReduce programming and execution model and its implementations, most notably Hadoop. Query languages such as Pig Latin, Hive, and Jaql make it simpler for users to express complex analysis tasks, and the compilers of these languages translate these complex tasks into workflows of MapReduce jobs. Each job in these workflows reads its input from the distributed file system used by the MapReduce system (e.g., HDFS in the case of Hadoop) and produces output that is stored in this distributed file system. This output is then read as input by the next job in the workflow. The current practice is to delete these intermediate results from the distributed file system at the end of executing the workflow. It would be more useful if these intermediate results can be stored and reused in future workflows. We demonstrate ReStore, an extension to Pig that enables it to manage storage and reuse of intermediate results of the MapReduce workflows executed in the Pig data analysis system. ReStore matches input workflows of MapReduce jobs with previously executed jobs and rewrites these workflows to reuse the stored results of the matched jobs. ReStore also creates additional reuse opportunities by materializing and reserving the output of query execution operators that are executed within a MapReduce job. In this demonstration we showcase the MapReduce jobs and sub-jobs recommended by ReStore for a given Pig query, the rewriting of input queries to reuse stored intermediate results, and a what-if analysis of the effectiveness of reusing stored outputs of previously executed jobs.",
keywords = "data reuse, mapreduce, pig latin",
author = "Iman Elghandour and Ashraf Aboulnaga",
year = "2012",
month = "6",
day = "28",
doi = "10.1145/2213836.2213937",
language = "English",
isbn = "9781450312479",
pages = "701--704",
booktitle = "Proceedings of the ACM SIGMOD International Conference on Management of Data",

}

TY - GEN

T1 - ReStore

T2 - Reusing results of MapReduce jobs in pig

AU - Elghandour, Iman

AU - Aboulnaga, Ashraf

PY - 2012/6/28

Y1 - 2012/6/28

N2 - Analyzing large scale data has become an important activity for many organizations, and is now facilitated by the MapReduce programming and execution model and its implementations, most notably Hadoop. Query languages such as Pig Latin, Hive, and Jaql make it simpler for users to express complex analysis tasks, and the compilers of these languages translate these complex tasks into workflows of MapReduce jobs. Each job in these workflows reads its input from the distributed file system used by the MapReduce system (e.g., HDFS in the case of Hadoop) and produces output that is stored in this distributed file system. This output is then read as input by the next job in the workflow. The current practice is to delete these intermediate results from the distributed file system at the end of executing the workflow. It would be more useful if these intermediate results can be stored and reused in future workflows. We demonstrate ReStore, an extension to Pig that enables it to manage storage and reuse of intermediate results of the MapReduce workflows executed in the Pig data analysis system. ReStore matches input workflows of MapReduce jobs with previously executed jobs and rewrites these workflows to reuse the stored results of the matched jobs. ReStore also creates additional reuse opportunities by materializing and reserving the output of query execution operators that are executed within a MapReduce job. In this demonstration we showcase the MapReduce jobs and sub-jobs recommended by ReStore for a given Pig query, the rewriting of input queries to reuse stored intermediate results, and a what-if analysis of the effectiveness of reusing stored outputs of previously executed jobs.

AB - Analyzing large scale data has become an important activity for many organizations, and is now facilitated by the MapReduce programming and execution model and its implementations, most notably Hadoop. Query languages such as Pig Latin, Hive, and Jaql make it simpler for users to express complex analysis tasks, and the compilers of these languages translate these complex tasks into workflows of MapReduce jobs. Each job in these workflows reads its input from the distributed file system used by the MapReduce system (e.g., HDFS in the case of Hadoop) and produces output that is stored in this distributed file system. This output is then read as input by the next job in the workflow. The current practice is to delete these intermediate results from the distributed file system at the end of executing the workflow. It would be more useful if these intermediate results can be stored and reused in future workflows. We demonstrate ReStore, an extension to Pig that enables it to manage storage and reuse of intermediate results of the MapReduce workflows executed in the Pig data analysis system. ReStore matches input workflows of MapReduce jobs with previously executed jobs and rewrites these workflows to reuse the stored results of the matched jobs. ReStore also creates additional reuse opportunities by materializing and reserving the output of query execution operators that are executed within a MapReduce job. In this demonstration we showcase the MapReduce jobs and sub-jobs recommended by ReStore for a given Pig query, the rewriting of input queries to reuse stored intermediate results, and a what-if analysis of the effectiveness of reusing stored outputs of previously executed jobs.

KW - data reuse

KW - mapreduce

KW - pig latin

UR - http://www.scopus.com/inward/record.url?scp=84862666446&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84862666446&partnerID=8YFLogxK

U2 - 10.1145/2213836.2213937

DO - 10.1145/2213836.2213937

M3 - Conference contribution

AN - SCOPUS:84862666446

SN - 9781450312479

SP - 701

EP - 704

BT - Proceedings of the ACM SIGMOD International Conference on Management of Data

ER -