Faster collective output through active buffering

Xiaosong Ma, M. Winslett, Jonghyun Lee, Shengke Yu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

26 Citations (Scopus)

Abstract

Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we propose active buffering for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. Active buffering exploits one-sided communication for I/O processors to fetch data from compute processors' buffers and performs actual writing in the background while compute processors are computing. It gracefully adapts as buffers at different level of the hierarchy fill and empty, and as new collective I/O requests arrive. Experimental results with synthetic benchmarks and a real rocket simulation code on the SGI Origin 2000 and IBM SP show that active buffering improves the apparent collective write throughput so that it approaches the local memory bandwidth or the MPI bandwidth under appropriate conditions. These speedups are due entirely to increased parallelism during I/O, and are in addition to any performance improvements that may come from buffering small requests.

Original languageEnglish
Title of host publicationProceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages329-336
Number of pages8
ISBN (Print)0769515738, 9780769515731
DOIs
Publication statusPublished - 2002
Externally publishedYes
Event16th International Parallel and Distributed Processing Symposium, IPDPS 2002 - Ft. Lauderdale, United States
Duration: 15 Apr 200219 Apr 2002

Other

Other16th International Parallel and Distributed Processing Symposium, IPDPS 2002
CountryUnited States
CityFt. Lauderdale
Period15/4/0219/4/02

Fingerprint

Buffer
Bandwidth
Data storage equipment
Output
Rockets
Metadata
Visualization
Restart
Throughput
Parallelism
Communication
Benchmark
Computing
Experimental Results
Simulation
Hierarchy
Background

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Modelling and Simulation

Cite this

Ma, X., Winslett, M., Lee, J., & Yu, S. (2002). Faster collective output through active buffering. In Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002 (pp. 329-336). [1015511] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2002.1015511

Faster collective output through active buffering. / Ma, Xiaosong; Winslett, M.; Lee, Jonghyun; Yu, Shengke.

Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002. Institute of Electrical and Electronics Engineers Inc., 2002. p. 329-336 1015511.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ma, X, Winslett, M, Lee, J & Yu, S 2002, Faster collective output through active buffering. in Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002., 1015511, Institute of Electrical and Electronics Engineers Inc., pp. 329-336, 16th International Parallel and Distributed Processing Symposium, IPDPS 2002, Ft. Lauderdale, United States, 15/4/02. https://doi.org/10.1109/IPDPS.2002.1015511
Ma X, Winslett M, Lee J, Yu S. Faster collective output through active buffering. In Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002. Institute of Electrical and Electronics Engineers Inc. 2002. p. 329-336. 1015511 https://doi.org/10.1109/IPDPS.2002.1015511
Ma, Xiaosong ; Winslett, M. ; Lee, Jonghyun ; Yu, Shengke. / Faster collective output through active buffering. Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002. Institute of Electrical and Electronics Engineers Inc., 2002. pp. 329-336
@inproceedings{79f8f238cdf344019272eccb8080a3cf,
title = "Faster collective output through active buffering",
abstract = "Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we propose active buffering for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. Active buffering exploits one-sided communication for I/O processors to fetch data from compute processors' buffers and performs actual writing in the background while compute processors are computing. It gracefully adapts as buffers at different level of the hierarchy fill and empty, and as new collective I/O requests arrive. Experimental results with synthetic benchmarks and a real rocket simulation code on the SGI Origin 2000 and IBM SP show that active buffering improves the apparent collective write throughput so that it approaches the local memory bandwidth or the MPI bandwidth under appropriate conditions. These speedups are due entirely to increased parallelism during I/O, and are in addition to any performance improvements that may come from buffering small requests.",
author = "Xiaosong Ma and M. Winslett and Jonghyun Lee and Shengke Yu",
year = "2002",
doi = "10.1109/IPDPS.2002.1015511",
language = "English",
isbn = "0769515738",
pages = "329--336",
booktitle = "Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Faster collective output through active buffering

AU - Ma, Xiaosong

AU - Winslett, M.

AU - Lee, Jonghyun

AU - Yu, Shengke

PY - 2002

Y1 - 2002

N2 - Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we propose active buffering for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. Active buffering exploits one-sided communication for I/O processors to fetch data from compute processors' buffers and performs actual writing in the background while compute processors are computing. It gracefully adapts as buffers at different level of the hierarchy fill and empty, and as new collective I/O requests arrive. Experimental results with synthetic benchmarks and a real rocket simulation code on the SGI Origin 2000 and IBM SP show that active buffering improves the apparent collective write throughput so that it approaches the local memory bandwidth or the MPI bandwidth under appropriate conditions. These speedups are due entirely to increased parallelism during I/O, and are in addition to any performance improvements that may come from buffering small requests.

AB - Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we propose active buffering for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. Active buffering exploits one-sided communication for I/O processors to fetch data from compute processors' buffers and performs actual writing in the background while compute processors are computing. It gracefully adapts as buffers at different level of the hierarchy fill and empty, and as new collective I/O requests arrive. Experimental results with synthetic benchmarks and a real rocket simulation code on the SGI Origin 2000 and IBM SP show that active buffering improves the apparent collective write throughput so that it approaches the local memory bandwidth or the MPI bandwidth under appropriate conditions. These speedups are due entirely to increased parallelism during I/O, and are in addition to any performance improvements that may come from buffering small requests.

UR - http://www.scopus.com/inward/record.url?scp=84966480868&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84966480868&partnerID=8YFLogxK

U2 - 10.1109/IPDPS.2002.1015511

DO - 10.1109/IPDPS.2002.1015511

M3 - Conference contribution

SN - 0769515738

SN - 9780769515731

SP - 329

EP - 336

BT - Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002

PB - Institute of Electrical and Electronics Engineers Inc.

ER -