Optimizing cross-platform data movement

Sebastian Kruse, Zoi Kaoudi, Jorge Arnulfo Quiane Ruiz, Sanjay Chawla, Felix Naumann, Bertty Contreras-Rojas

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Data analytics are moving beyond the limits of a single data processing platform. A cross-platform query optimizer is necessary to enable applications to run their tasks over multiple platforms efficiently and in a platform-agnostic manner. For the optimizer to be effective, it must consider data movement costs across different data processing platforms. In this paper, we present the graph-based data movement strategy used by Rheem, our open-source cross-platform system. In particular, we (i) model the data movement problem as a new graph problem, which we prove to be NP-hard, and (ii) propose a novel graph exploration algorithm, which allows Rheem to discover multiple hidden opportunities for cross-platform data processing.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE 35th International Conference on Data Engineering, ICDE 2019
PublisherIEEE Computer Society
Pages1642-1645
Number of pages4
ISBN (Electronic)9781538674741
DOIs
Publication statusPublished - 1 Apr 2019
Event35th IEEE International Conference on Data Engineering, ICDE 2019 - Macau, China
Duration: 8 Apr 201911 Apr 2019

Publication series

NameProceedings - International Conference on Data Engineering
Volume2019-April
ISSN (Print)1084-4627

Conference

Conference35th IEEE International Conference on Data Engineering, ICDE 2019
CountryChina
CityMacau
Period8/4/1911/4/19

    Fingerprint

Keywords

  • Cross-platform
  • Data movement
  • Polystore
  • Query opimization

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Information Systems

Cite this

Kruse, S., Kaoudi, Z., Quiane Ruiz, J. A., Chawla, S., Naumann, F., & Contreras-Rojas, B. (2019). Optimizing cross-platform data movement. In Proceedings - 2019 IEEE 35th International Conference on Data Engineering, ICDE 2019 (pp. 1642-1645). [8731354] (Proceedings - International Conference on Data Engineering; Vol. 2019-April). IEEE Computer Society. https://doi.org/10.1109/ICDE.2019.00162