Improving MPI-IO output performance with active buffering plus threads

Xiaosong Ma, Marianne Winslett, Jonghyun Lee, Shengke Yu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

48 Citations (Scopus)

Abstract

Efficient collective output of intermediate results to secondary storage becomes more and more important for scientific simulations as the gap between processing power/interconnection bandwidth and the I/O system bandwidth enlarges. Dedicated servers can offload I/O from compute processors and shorten the execution time, but it is not always possible or easy for an application to use them. We propose the use of active buffering with threads (ABT) for overlapping I/O with computation efficiently and flexibly without dedicated I/O servers. We show that the implementation of ABT in ROMIO, a popular implementation of MPI-IO, greatly reduces the application-visible cost of ROMIO's collective write calls, and improves an application's overall performance by hiding I/O cost and saving implicit synchronization overhead from collective write operations. Further, ABT is high-level, platform-independent, and transparent to users, giving users the benefit of overlapping I/O with other processing tasks even when the file system or parallel I/O library does not support asynchronous I/O.

Original languageEnglish
Title of host publicationProceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Print)0769519261, 9780769519265
DOIs
Publication statusPublished - 2003
Externally publishedYes
EventInternational Parallel and Distributed Processing Symposium, IPDPS 2003 - Nice, France
Duration: 22 Apr 200326 Apr 2003

Other

OtherInternational Parallel and Distributed Processing Symposium, IPDPS 2003
CountryFrance
CityNice
Period22/4/0326/4/03

Fingerprint

Thread
Overlapping
Output
Servers
Server
Bandwidth
Parallel I/O
File System
Costs
Processing
Interconnection
Execution Time
Synchronization
Simulation
Libraries

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Theoretical Computer Science
  • Software

Cite this

Ma, X., Winslett, M., Lee, J., & Yu, S. (2003). Improving MPI-IO output performance with active buffering plus threads. In Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003 [1213165] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2003.1213165

Improving MPI-IO output performance with active buffering plus threads. / Ma, Xiaosong; Winslett, Marianne; Lee, Jonghyun; Yu, Shengke.

Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003. Institute of Electrical and Electronics Engineers Inc., 2003. 1213165.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ma, X, Winslett, M, Lee, J & Yu, S 2003, Improving MPI-IO output performance with active buffering plus threads. in Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003., 1213165, Institute of Electrical and Electronics Engineers Inc., International Parallel and Distributed Processing Symposium, IPDPS 2003, Nice, France, 22/4/03. https://doi.org/10.1109/IPDPS.2003.1213165
Ma X, Winslett M, Lee J, Yu S. Improving MPI-IO output performance with active buffering plus threads. In Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003. Institute of Electrical and Electronics Engineers Inc. 2003. 1213165 https://doi.org/10.1109/IPDPS.2003.1213165
Ma, Xiaosong ; Winslett, Marianne ; Lee, Jonghyun ; Yu, Shengke. / Improving MPI-IO output performance with active buffering plus threads. Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003. Institute of Electrical and Electronics Engineers Inc., 2003.
@inproceedings{6f8b5eb6a7a345f7964ea1465d3959a2,
title = "Improving MPI-IO output performance with active buffering plus threads",
abstract = "Efficient collective output of intermediate results to secondary storage becomes more and more important for scientific simulations as the gap between processing power/interconnection bandwidth and the I/O system bandwidth enlarges. Dedicated servers can offload I/O from compute processors and shorten the execution time, but it is not always possible or easy for an application to use them. We propose the use of active buffering with threads (ABT) for overlapping I/O with computation efficiently and flexibly without dedicated I/O servers. We show that the implementation of ABT in ROMIO, a popular implementation of MPI-IO, greatly reduces the application-visible cost of ROMIO's collective write calls, and improves an application's overall performance by hiding I/O cost and saving implicit synchronization overhead from collective write operations. Further, ABT is high-level, platform-independent, and transparent to users, giving users the benefit of overlapping I/O with other processing tasks even when the file system or parallel I/O library does not support asynchronous I/O.",
author = "Xiaosong Ma and Marianne Winslett and Jonghyun Lee and Shengke Yu",
year = "2003",
doi = "10.1109/IPDPS.2003.1213165",
language = "English",
isbn = "0769519261",
booktitle = "Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Improving MPI-IO output performance with active buffering plus threads

AU - Ma, Xiaosong

AU - Winslett, Marianne

AU - Lee, Jonghyun

AU - Yu, Shengke

PY - 2003

Y1 - 2003

N2 - Efficient collective output of intermediate results to secondary storage becomes more and more important for scientific simulations as the gap between processing power/interconnection bandwidth and the I/O system bandwidth enlarges. Dedicated servers can offload I/O from compute processors and shorten the execution time, but it is not always possible or easy for an application to use them. We propose the use of active buffering with threads (ABT) for overlapping I/O with computation efficiently and flexibly without dedicated I/O servers. We show that the implementation of ABT in ROMIO, a popular implementation of MPI-IO, greatly reduces the application-visible cost of ROMIO's collective write calls, and improves an application's overall performance by hiding I/O cost and saving implicit synchronization overhead from collective write operations. Further, ABT is high-level, platform-independent, and transparent to users, giving users the benefit of overlapping I/O with other processing tasks even when the file system or parallel I/O library does not support asynchronous I/O.

AB - Efficient collective output of intermediate results to secondary storage becomes more and more important for scientific simulations as the gap between processing power/interconnection bandwidth and the I/O system bandwidth enlarges. Dedicated servers can offload I/O from compute processors and shorten the execution time, but it is not always possible or easy for an application to use them. We propose the use of active buffering with threads (ABT) for overlapping I/O with computation efficiently and flexibly without dedicated I/O servers. We show that the implementation of ABT in ROMIO, a popular implementation of MPI-IO, greatly reduces the application-visible cost of ROMIO's collective write calls, and improves an application's overall performance by hiding I/O cost and saving implicit synchronization overhead from collective write operations. Further, ABT is high-level, platform-independent, and transparent to users, giving users the benefit of overlapping I/O with other processing tasks even when the file system or parallel I/O library does not support asynchronous I/O.

UR - http://www.scopus.com/inward/record.url?scp=84947207861&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84947207861&partnerID=8YFLogxK

U2 - 10.1109/IPDPS.2003.1213165

DO - 10.1109/IPDPS.2003.1213165

M3 - Conference contribution

AN - SCOPUS:84947207861

SN - 0769519261

SN - 9780769519265

BT - Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003

PB - Institute of Electrical and Electronics Engineers Inc.

ER -