RFS

Efficient and flexible remote file access for MPI-IO

Jonghyun Lee, Xiaosong Ma, Robert Ross, Rajeev Thakur, Marianne Winslett

Research output: Chapter in Book/Report/Conference proceedingConference contribution

20 Citations (Scopus)

Abstract

Scientific applications often need to access remote file systems. Because of slow networks and large data size, however, remote I/O can become an even more serious performance bottleneck than local I/O performance. In this work, we present RFS, a high-performance remote I/O facility for ROMIO, which is a well-known MPI-IO implementation. Our simple, portable, and flexible design eliminates the shortcomings of previous remote I/O efforts. In particular, RFS improves the remote I/O performance by adopting active buffering with threads (ABT), which hides I/O cost by aggressively buffering the output data using available memory and performing background I/O using threads while computation is taking place. Our experimental results show that RFS with ABT can significantly reduce the remote I/O visible cost, achieving up to 92% of the theoretical peak throughput. The computation slowdown caused by concurrent I/O activities was 0.2-6.2%, which is dwarfed by the overall performance improvement in application turnaround time.

Original languageEnglish
Title of host publicationProceedings - IEEE International Conference on Cluster Computing, ICCC
Pages71-81
Number of pages11
DOIs
Publication statusPublished - 1 Dec 2004
Externally publishedYes
Event2004 IEEE International Conference on Cluster Computing, ICCC 2004 - San Diego, CA, United States
Duration: 20 Sep 200423 Sep 2004

Other

Other2004 IEEE International Conference on Cluster Computing, ICCC 2004
CountryUnited States
CitySan Diego, CA
Period20/9/0423/9/04

Fingerprint

Turnaround time
Costs
Throughput
Data storage equipment

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Lee, J., Ma, X., Ross, R., Thakur, R., & Winslett, M. (2004). RFS: Efficient and flexible remote file access for MPI-IO. In Proceedings - IEEE International Conference on Cluster Computing, ICCC (pp. 71-81) https://doi.org/10.1109/CLUSTR.2004.1392604

RFS : Efficient and flexible remote file access for MPI-IO. / Lee, Jonghyun; Ma, Xiaosong; Ross, Robert; Thakur, Rajeev; Winslett, Marianne.

Proceedings - IEEE International Conference on Cluster Computing, ICCC. 2004. p. 71-81.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lee, J, Ma, X, Ross, R, Thakur, R & Winslett, M 2004, RFS: Efficient and flexible remote file access for MPI-IO. in Proceedings - IEEE International Conference on Cluster Computing, ICCC. pp. 71-81, 2004 IEEE International Conference on Cluster Computing, ICCC 2004, San Diego, CA, United States, 20/9/04. https://doi.org/10.1109/CLUSTR.2004.1392604
Lee J, Ma X, Ross R, Thakur R, Winslett M. RFS: Efficient and flexible remote file access for MPI-IO. In Proceedings - IEEE International Conference on Cluster Computing, ICCC. 2004. p. 71-81 https://doi.org/10.1109/CLUSTR.2004.1392604
Lee, Jonghyun ; Ma, Xiaosong ; Ross, Robert ; Thakur, Rajeev ; Winslett, Marianne. / RFS : Efficient and flexible remote file access for MPI-IO. Proceedings - IEEE International Conference on Cluster Computing, ICCC. 2004. pp. 71-81
@inproceedings{6d254dad4b6647c18e38213ec4cbac69,
title = "RFS: Efficient and flexible remote file access for MPI-IO",
abstract = "Scientific applications often need to access remote file systems. Because of slow networks and large data size, however, remote I/O can become an even more serious performance bottleneck than local I/O performance. In this work, we present RFS, a high-performance remote I/O facility for ROMIO, which is a well-known MPI-IO implementation. Our simple, portable, and flexible design eliminates the shortcomings of previous remote I/O efforts. In particular, RFS improves the remote I/O performance by adopting active buffering with threads (ABT), which hides I/O cost by aggressively buffering the output data using available memory and performing background I/O using threads while computation is taking place. Our experimental results show that RFS with ABT can significantly reduce the remote I/O visible cost, achieving up to 92{\%} of the theoretical peak throughput. The computation slowdown caused by concurrent I/O activities was 0.2-6.2{\%}, which is dwarfed by the overall performance improvement in application turnaround time.",
author = "Jonghyun Lee and Xiaosong Ma and Robert Ross and Rajeev Thakur and Marianne Winslett",
year = "2004",
month = "12",
day = "1",
doi = "10.1109/CLUSTR.2004.1392604",
language = "English",
isbn = "0780386949",
pages = "71--81",
booktitle = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",

}

TY - GEN

T1 - RFS

T2 - Efficient and flexible remote file access for MPI-IO

AU - Lee, Jonghyun

AU - Ma, Xiaosong

AU - Ross, Robert

AU - Thakur, Rajeev

AU - Winslett, Marianne

PY - 2004/12/1

Y1 - 2004/12/1

N2 - Scientific applications often need to access remote file systems. Because of slow networks and large data size, however, remote I/O can become an even more serious performance bottleneck than local I/O performance. In this work, we present RFS, a high-performance remote I/O facility for ROMIO, which is a well-known MPI-IO implementation. Our simple, portable, and flexible design eliminates the shortcomings of previous remote I/O efforts. In particular, RFS improves the remote I/O performance by adopting active buffering with threads (ABT), which hides I/O cost by aggressively buffering the output data using available memory and performing background I/O using threads while computation is taking place. Our experimental results show that RFS with ABT can significantly reduce the remote I/O visible cost, achieving up to 92% of the theoretical peak throughput. The computation slowdown caused by concurrent I/O activities was 0.2-6.2%, which is dwarfed by the overall performance improvement in application turnaround time.

AB - Scientific applications often need to access remote file systems. Because of slow networks and large data size, however, remote I/O can become an even more serious performance bottleneck than local I/O performance. In this work, we present RFS, a high-performance remote I/O facility for ROMIO, which is a well-known MPI-IO implementation. Our simple, portable, and flexible design eliminates the shortcomings of previous remote I/O efforts. In particular, RFS improves the remote I/O performance by adopting active buffering with threads (ABT), which hides I/O cost by aggressively buffering the output data using available memory and performing background I/O using threads while computation is taking place. Our experimental results show that RFS with ABT can significantly reduce the remote I/O visible cost, achieving up to 92% of the theoretical peak throughput. The computation slowdown caused by concurrent I/O activities was 0.2-6.2%, which is dwarfed by the overall performance improvement in application turnaround time.

UR - http://www.scopus.com/inward/record.url?scp=20444440714&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=20444440714&partnerID=8YFLogxK

U2 - 10.1109/CLUSTR.2004.1392604

DO - 10.1109/CLUSTR.2004.1392604

M3 - Conference contribution

SN - 0780386949

SP - 71

EP - 81

BT - Proceedings - IEEE International Conference on Cluster Computing, ICCC

ER -