Flexible and efficient parallel I/O for large-scale multi-component simulations

Xiaosong Ma; Xiangmin Jiao; Michael Campbell; Marianne Winslett

doi:10.1109/IPDPS.2003.1213462

Flexible and efficient parallel I/O for large-scale multi-component simulations

Xiaosong Ma, Xiangmin Jiao, Michael Campbell, Marianne Winslett

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Citations (Scopus)

Abstract

In this paper, we discuss our experience of providing high performance parallel I/O for a large-scale, on-going, multi-disciplinary simulation project for solid propellant rockets. We describe the performance and data management issues observed in this project and present our solutions, including (1) support for relatively fine-grained distribution of irregular datasets in parallel I/O, (2) a flexible data management facility for inter-module communication, and (3) two schemes to overlap computation with I/O. Performance results obtained from both development and production platforms show that our I/O optimizations can dramatically reduce the simulation's visible I/O cost, as well as the number of disk files, and significantly improve the overall performance. Meanwhile, our data management facility helps to provide simulation developers with simple user interfaces for parallel I/O.

Original language	English
Title of host publication	Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Print)	0769519261, 9780769519265
DOIs	https://doi.org/10.1109/IPDPS.2003.1213462
Publication status	Published - 2003
Externally published	Yes
Event	International Parallel and Distributed Processing Symposium, IPDPS 2003 - Nice, France Duration: 22 Apr 2003 → 26 Apr 2003

Other

Other	International Parallel and Distributed Processing Symposium, IPDPS 2003
Country	France
City	Nice
Period	22/4/03 → 26/4/03

Fingerprint

ASJC Scopus subject areas

Computational Theory and Mathematics
Theoretical Computer Science
Software

Cite this

Ma, X., Jiao, X., Campbell, M., & Winslett, M. (2003). Flexible and efficient parallel I/O for large-scale multi-component simulations. In Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003 [1213462] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2003.1213462

Flexible and efficient parallel I/O for large-scale multi-component simulations. / Ma, Xiaosong; Jiao, Xiangmin; Campbell, Michael; Winslett, Marianne.

Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003. Institute of Electrical and Electronics Engineers Inc., 2003. 1213462.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Ma, X, Jiao, X, Campbell, M & Winslett, M 2003, Flexible and efficient parallel I/O for large-scale multi-component simulations. in Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003., 1213462, Institute of Electrical and Electronics Engineers Inc., International Parallel and Distributed Processing Symposium, IPDPS 2003, Nice, France, 22/4/03. https://doi.org/10.1109/IPDPS.2003.1213462

@inproceedings{da780a988f7b49ec8356abed2a87f157,

title = "Flexible and efficient parallel I/O for large-scale multi-component simulations",

abstract = "In this paper, we discuss our experience of providing high performance parallel I/O for a large-scale, on-going, multi-disciplinary simulation project for solid propellant rockets. We describe the performance and data management issues observed in this project and present our solutions, including (1) support for relatively fine-grained distribution of irregular datasets in parallel I/O, (2) a flexible data management facility for inter-module communication, and (3) two schemes to overlap computation with I/O. Performance results obtained from both development and production platforms show that our I/O optimizations can dramatically reduce the simulation's visible I/O cost, as well as the number of disk files, and significantly improve the overall performance. Meanwhile, our data management facility helps to provide simulation developers with simple user interfaces for parallel I/O.",

author = "Xiaosong Ma and Xiangmin Jiao and Michael Campbell and Marianne Winslett",

year = "2003",

doi = "10.1109/IPDPS.2003.1213462",

language = "English",

isbn = "0769519261",

booktitle = "Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

note = "International Parallel and Distributed Processing Symposium, IPDPS 2003 ; Conference date: 22-04-2003 Through 26-04-2003",

}

TY - GEN

T1 - Flexible and efficient parallel I/O for large-scale multi-component simulations

AU - Ma, Xiaosong

AU - Jiao, Xiangmin

AU - Campbell, Michael

AU - Winslett, Marianne

PY - 2003

Y1 - 2003

N2 - In this paper, we discuss our experience of providing high performance parallel I/O for a large-scale, on-going, multi-disciplinary simulation project for solid propellant rockets. We describe the performance and data management issues observed in this project and present our solutions, including (1) support for relatively fine-grained distribution of irregular datasets in parallel I/O, (2) a flexible data management facility for inter-module communication, and (3) two schemes to overlap computation with I/O. Performance results obtained from both development and production platforms show that our I/O optimizations can dramatically reduce the simulation's visible I/O cost, as well as the number of disk files, and significantly improve the overall performance. Meanwhile, our data management facility helps to provide simulation developers with simple user interfaces for parallel I/O.

AB - In this paper, we discuss our experience of providing high performance parallel I/O for a large-scale, on-going, multi-disciplinary simulation project for solid propellant rockets. We describe the performance and data management issues observed in this project and present our solutions, including (1) support for relatively fine-grained distribution of irregular datasets in parallel I/O, (2) a flexible data management facility for inter-module communication, and (3) two schemes to overlap computation with I/O. Performance results obtained from both development and production platforms show that our I/O optimizations can dramatically reduce the simulation's visible I/O cost, as well as the number of disk files, and significantly improve the overall performance. Meanwhile, our data management facility helps to provide simulation developers with simple user interfaces for parallel I/O.

UR - http://www.scopus.com/inward/record.url?scp=84947257456&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84947257456&partnerID=8YFLogxK

U2 - 10.1109/IPDPS.2003.1213462

DO - 10.1109/IPDPS.2003.1213462

M3 - Conference contribution

AN - SCOPUS:84947257456

SN - 0769519261

SN - 9780769519265

BT - Proceedings - International Parallel and Distributed Processing Symposium, IPDPS 2003

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - International Parallel and Distributed Processing Symposium, IPDPS 2003

Y2 - 22 April 2003 through 26 April 2003

ER -

Access to Document

10.1109/IPDPS.2003.1213462