RFS: Efficient and flexible remote file access for MPI-IO

Jonghyun Lee, Xiaosong Ma, Robert Ross, Rajeev Thakur, Marianne Winslett

Research output: Chapter in Book/Report/Conference proceedingConference contribution

21 Citations (Scopus)

Abstract

Scientific applications often need to access remote file systems. Because of slow networks and large data size, however, remote I/O can become an even more serious performance bottleneck than local I/O performance. In this work, we present RFS, a high-performance remote I/O facility for ROMIO, which is a well-known MPI-IO implementation. Our simple, portable, and flexible design eliminates the shortcomings of previous remote I/O efforts. In particular, RFS improves the remote I/O performance by adopting active buffering with threads (ABT), which hides I/O cost by aggressively buffering the output data using available memory and performing background I/O using threads while computation is taking place. Our experimental results show that RFS with ABT can significantly reduce the remote I/O visible cost, achieving up to 92% of the theoretical peak throughput. The computation slowdown caused by concurrent I/O activities was 0.2-6.2%, which is dwarfed by the overall performance improvement in application turnaround time.

Original languageEnglish
Title of host publication2004 IEEE International Conference on Cluster Computing, ICCC 2004
Pages71-81
Number of pages11
DOIs
Publication statusPublished - 1 Dec 2004
Externally publishedYes
Event2004 IEEE International Conference on Cluster Computing, ICCC 2004 - San Diego, CA, United States
Duration: 20 Sep 200423 Sep 2004

Publication series

NameProceedings - IEEE International Conference on Cluster Computing, ICCC
ISSN (Print)1552-5244

Other

Other2004 IEEE International Conference on Cluster Computing, ICCC 2004
CountryUnited States
CitySan Diego, CA
Period20/9/0423/9/04

    Fingerprint

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Lee, J., Ma, X., Ross, R., Thakur, R., & Winslett, M. (2004). RFS: Efficient and flexible remote file access for MPI-IO. In 2004 IEEE International Conference on Cluster Computing, ICCC 2004 (pp. 71-81). (Proceedings - IEEE International Conference on Cluster Computing, ICCC). https://doi.org/10.1109/CLUSTR.2004.1392604