Enhancing data migration performance via parallel data compression

Jonghyun Lee, M. Winslett, Xiaosong Ma, Shengke Yu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Citations (Scopus)

Abstract

Scientific simulations often produce large volumes of output that are moved to another platform for visualization or storage. This long-distance migration is slow due to the data size and slow network. Compression can improve migration performance by reducing the data size, but compression is computation-intensive and so can raise costs. In this work, we show how to reduce data migration cost by incorporating compression into migration. We analyze eight scientific data sets, and propose three approaches for parallel compression of scientific data. Our results show that with reasonably fast processors and typical parallel configurations, the compression cost for large scientific data is outweighed by the performance gain obtained by migrating less data. We found that a client-side compression approach (CC) can improve I/O and migration performance by an order of magnitude. In our experiments, CC always matches or outperforms migration without compression when we overlap migration with computation, even for not very compressible dense floating point data. We also present a variant of CC that is well suited for use with implementations of two-phase I/O.

Original languageEnglish
Title of host publicationProceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages444-451
Number of pages8
ISBN (Print)0769515738, 9780769515731
DOIs
Publication statusPublished - 2002
Externally publishedYes
Event16th International Parallel and Distributed Processing Symposium, IPDPS 2002 - Ft. Lauderdale, United States
Duration: 15 Apr 200219 Apr 2002

Other

Other16th International Parallel and Distributed Processing Symposium, IPDPS 2002
CountryUnited States
CityFt. Lauderdale
Period15/4/0219/4/02

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Modelling and Simulation

Fingerprint Dive into the research topics of 'Enhancing data migration performance via parallel data compression'. Together they form a unique fingerprint.

  • Cite this

    Lee, J., Winslett, M., Ma, X., & Yu, S. (2002). Enhancing data migration performance via parallel data compression. In Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002 (pp. 444-451). [1015528] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2002.1015528