Faster collective output through active buffering

Xiaosong Ma, M. Winslett, Jonghyun Lee, Shengke Yu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

26 Citations (Scopus)

Abstract

Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we propose active buffering for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. Active buffering exploits one-sided communication for I/O processors to fetch data from compute processors' buffers and performs actual writing in the background while compute processors are computing. It gracefully adapts as buffers at different level of the hierarchy fill and empty, and as new collective I/O requests arrive. Experimental results with synthetic benchmarks and a real rocket simulation code on the SGI Origin 2000 and IBM SP show that active buffering improves the apparent collective write throughput so that it approaches the local memory bandwidth or the MPI bandwidth under appropriate conditions. These speedups are due entirely to increased parallelism during I/O, and are in addition to any performance improvements that may come from buffering small requests.

Original languageEnglish
Title of host publicationProceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages329-336
Number of pages8
ISBN (Print)0769515738, 9780769515731
DOIs
Publication statusPublished - 2002
Externally publishedYes
Event16th International Parallel and Distributed Processing Symposium, IPDPS 2002 - Ft. Lauderdale, United States
Duration: 15 Apr 200219 Apr 2002

Other

Other16th International Parallel and Distributed Processing Symposium, IPDPS 2002
CountryUnited States
CityFt. Lauderdale
Period15/4/0219/4/02

    Fingerprint

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Modelling and Simulation

Cite this

Ma, X., Winslett, M., Lee, J., & Yu, S. (2002). Faster collective output through active buffering. In Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2002 (pp. 329-336). [1015511] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2002.1015511