Improving MPI-IO output performance with active buffering plus threads

Xiaosong Ma, Marianne Winslett, Jonghyun Lee, Shengke Yu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

49 Citations (Scopus)

Abstract

Efficient collective output of intermediate results to secondary storage becomes more and more important for scientific simulations as the gap between processing power/interconnection bandwidth and the I/O system bandwidth enlarges. Dedicated servers can offload I/O from compute processors and shorten the execution time, but it is not always possible or easy for an application to use them. We propose the use of active buffering with threads (ABT) for overlapping I/O with computation efficiently and flexibly without dedicated I/O servers. We show that the implementation of ABT in ROMIO, a popular implementation of MPI-IO, greatly reduces the application-visible cost of ROMIO's collective write calls, and improves an application's overall performance by hiding I/O cost and saving implicit synchronization overhead from collective write operations. Further, ABT is high-level, platform-independent, and transparent to users, giving users the benefit of overlapping I/O with other processing tasks even when the file system or parallel I/O library does not support asynchronous I/O.

Original languageEnglish
Title of host publicationProceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Print)0769519261, 9780769519265
DOIs
Publication statusPublished - 2003
Externally publishedYes
EventInternational Parallel and Distributed Processing Symposium, IPDPS 2003 - Nice, France
Duration: 22 Apr 200326 Apr 2003

Other

OtherInternational Parallel and Distributed Processing Symposium, IPDPS 2003
CountryFrance
CityNice
Period22/4/0326/4/03

    Fingerprint

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Theoretical Computer Science
  • Software

Cite this

Ma, X., Winslett, M., Lee, J., & Yu, S. (2003). Improving MPI-IO output performance with active buffering plus threads. In Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003 [1213165] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2003.1213165