Active buffering plus compressed migration

An integrated solution to parallel simulations' data transport needs

Jonghyun Lee, Xiaosong Ma, Marianne Winslett, Shengke Yu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)

Abstract

Scientific simulations running on parallel platforms output intermediate data periodically, typically moving the output to a remote machine for visualization. Due to the large data size and slow network, improvements in output and migration performance can significantly reduce simulation turnaround time. In this paper, we propose a novel simulation execution environment that integrates active buffering and compressed migration to hide and reduce the cost of output and migration. Our implementation removes previous shortcomings of active buffering and of compressed migration, and our experiments with real world data sets show that integrating the two techniques brings comparable application-visible I/O performance and much more flexibility compared to compressed migration without active buffering. The performance advantage will increase with slower-to-write file formats such as HDF.

Original languageEnglish
Title of host publicationProceedings of the International Conference on Supercomputing
Pages156-166
Number of pages11
Publication statusPublished - 1 Jan 2002
Externally publishedYes
EventConference Proceedings of the 2002 International Conference on Supercomputing - New York, NY, United States
Duration: 22 Jun 200226 Jun 2002

Other

OtherConference Proceedings of the 2002 International Conference on Supercomputing
CountryUnited States
CityNew York, NY
Period22/6/0226/6/02

Fingerprint

Turnaround time
Visualization
Costs
Experiments

Keywords

  • Active buffering
  • Collective I/O
  • Data migration
  • Parallel I/O
  • Scientific simulations

ASJC Scopus subject areas

  • Computer Science(all)

Cite this

Lee, J., Ma, X., Winslett, M., & Yu, S. (2002). Active buffering plus compressed migration: An integrated solution to parallel simulations' data transport needs. In Proceedings of the International Conference on Supercomputing (pp. 156-166)

Active buffering plus compressed migration : An integrated solution to parallel simulations' data transport needs. / Lee, Jonghyun; Ma, Xiaosong; Winslett, Marianne; Yu, Shengke.

Proceedings of the International Conference on Supercomputing. 2002. p. 156-166.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lee, J, Ma, X, Winslett, M & Yu, S 2002, Active buffering plus compressed migration: An integrated solution to parallel simulations' data transport needs. in Proceedings of the International Conference on Supercomputing. pp. 156-166, Conference Proceedings of the 2002 International Conference on Supercomputing, New York, NY, United States, 22/6/02.
Lee J, Ma X, Winslett M, Yu S. Active buffering plus compressed migration: An integrated solution to parallel simulations' data transport needs. In Proceedings of the International Conference on Supercomputing. 2002. p. 156-166
Lee, Jonghyun ; Ma, Xiaosong ; Winslett, Marianne ; Yu, Shengke. / Active buffering plus compressed migration : An integrated solution to parallel simulations' data transport needs. Proceedings of the International Conference on Supercomputing. 2002. pp. 156-166
@inproceedings{4047563d984645f58cc496d18b798561,
title = "Active buffering plus compressed migration: An integrated solution to parallel simulations' data transport needs",
abstract = "Scientific simulations running on parallel platforms output intermediate data periodically, typically moving the output to a remote machine for visualization. Due to the large data size and slow network, improvements in output and migration performance can significantly reduce simulation turnaround time. In this paper, we propose a novel simulation execution environment that integrates active buffering and compressed migration to hide and reduce the cost of output and migration. Our implementation removes previous shortcomings of active buffering and of compressed migration, and our experiments with real world data sets show that integrating the two techniques brings comparable application-visible I/O performance and much more flexibility compared to compressed migration without active buffering. The performance advantage will increase with slower-to-write file formats such as HDF.",
keywords = "Active buffering, Collective I/O, Data migration, Parallel I/O, Scientific simulations",
author = "Jonghyun Lee and Xiaosong Ma and Marianne Winslett and Shengke Yu",
year = "2002",
month = "1",
day = "1",
language = "English",
pages = "156--166",
booktitle = "Proceedings of the International Conference on Supercomputing",

}

TY - GEN

T1 - Active buffering plus compressed migration

T2 - An integrated solution to parallel simulations' data transport needs

AU - Lee, Jonghyun

AU - Ma, Xiaosong

AU - Winslett, Marianne

AU - Yu, Shengke

PY - 2002/1/1

Y1 - 2002/1/1

N2 - Scientific simulations running on parallel platforms output intermediate data periodically, typically moving the output to a remote machine for visualization. Due to the large data size and slow network, improvements in output and migration performance can significantly reduce simulation turnaround time. In this paper, we propose a novel simulation execution environment that integrates active buffering and compressed migration to hide and reduce the cost of output and migration. Our implementation removes previous shortcomings of active buffering and of compressed migration, and our experiments with real world data sets show that integrating the two techniques brings comparable application-visible I/O performance and much more flexibility compared to compressed migration without active buffering. The performance advantage will increase with slower-to-write file formats such as HDF.

AB - Scientific simulations running on parallel platforms output intermediate data periodically, typically moving the output to a remote machine for visualization. Due to the large data size and slow network, improvements in output and migration performance can significantly reduce simulation turnaround time. In this paper, we propose a novel simulation execution environment that integrates active buffering and compressed migration to hide and reduce the cost of output and migration. Our implementation removes previous shortcomings of active buffering and of compressed migration, and our experiments with real world data sets show that integrating the two techniques brings comparable application-visible I/O performance and much more flexibility compared to compressed migration without active buffering. The performance advantage will increase with slower-to-write file formats such as HDF.

KW - Active buffering

KW - Collective I/O

KW - Data migration

KW - Parallel I/O

KW - Scientific simulations

UR - http://www.scopus.com/inward/record.url?scp=0036374246&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0036374246&partnerID=8YFLogxK

M3 - Conference contribution

SP - 156

EP - 166

BT - Proceedings of the International Conference on Supercomputing

ER -