One optimized I/O configuration per HPC application

Leveraging the configurability of cloud

Mingliang Liu, Jidong Zhai, Yan Zhai, Xiaosong Ma, Wenguang Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)

Abstract

There is a trend to migrate HPC (High Performance Computing) applications to cloud platforms, such as the Amazon EC2 Cluster Compute Instances (CCIs). While existing research has mainly focused on the performance impact of virtualized environments and interconnect technologies on parallel programs, we suggest that the configurability enabled by clouds is another important dimension to explore. Unlike on traditional HPC platforms, on a cloud-resident virtual cluster it is easy to change the I/O configurations, such as the choice of file systems, the number of I/O nodes, and the types of virtual disks, to fit the I/O requirements of different applications. In this paper, we discuss how cloud platforms can be employed to form customized and balanced I/O subsystems for individual I/O-intensive MPI applications. Through our preliminary evaluation, we demonstrate that different applications will benefit from individually tailored I/O system configurations. For a given I/O-intensive application, different I/O settings may lead to significant overall application performance or cost difference (up to 2.5-fold). Our exploration indicates that customized system configuration for HPC applications in the cloud is important and non-trivial.

Original languageEnglish
Title of host publicationProceedings of the 2nd Asia-Pacific Workshop on Systems, APSys'11
DOIs
Publication statusPublished - 1 Dec 2011
Externally publishedYes
Event2nd Asia-Pacific Workshop on Systems, APSys'11 - Shanghai, China
Duration: 11 Jul 201112 Jul 2011

Other

Other2nd Asia-Pacific Workshop on Systems, APSys'11
CountryChina
CityShanghai
Period11/7/1112/7/11

Fingerprint

Costs

ASJC Scopus subject areas

  • Control and Systems Engineering

Cite this

Liu, M., Zhai, J., Zhai, Y., Ma, X., & Chen, W. (2011). One optimized I/O configuration per HPC application: Leveraging the configurability of cloud. In Proceedings of the 2nd Asia-Pacific Workshop on Systems, APSys'11 https://doi.org/10.1145/2103799.2103818

One optimized I/O configuration per HPC application : Leveraging the configurability of cloud. / Liu, Mingliang; Zhai, Jidong; Zhai, Yan; Ma, Xiaosong; Chen, Wenguang.

Proceedings of the 2nd Asia-Pacific Workshop on Systems, APSys'11. 2011.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Liu, M, Zhai, J, Zhai, Y, Ma, X & Chen, W 2011, One optimized I/O configuration per HPC application: Leveraging the configurability of cloud. in Proceedings of the 2nd Asia-Pacific Workshop on Systems, APSys'11. 2nd Asia-Pacific Workshop on Systems, APSys'11, Shanghai, China, 11/7/11. https://doi.org/10.1145/2103799.2103818
Liu M, Zhai J, Zhai Y, Ma X, Chen W. One optimized I/O configuration per HPC application: Leveraging the configurability of cloud. In Proceedings of the 2nd Asia-Pacific Workshop on Systems, APSys'11. 2011 https://doi.org/10.1145/2103799.2103818
Liu, Mingliang ; Zhai, Jidong ; Zhai, Yan ; Ma, Xiaosong ; Chen, Wenguang. / One optimized I/O configuration per HPC application : Leveraging the configurability of cloud. Proceedings of the 2nd Asia-Pacific Workshop on Systems, APSys'11. 2011.
@inproceedings{8d641b0bfab64f3c8f49ee33861a2666,
title = "One optimized I/O configuration per HPC application: Leveraging the configurability of cloud",
abstract = "There is a trend to migrate HPC (High Performance Computing) applications to cloud platforms, such as the Amazon EC2 Cluster Compute Instances (CCIs). While existing research has mainly focused on the performance impact of virtualized environments and interconnect technologies on parallel programs, we suggest that the configurability enabled by clouds is another important dimension to explore. Unlike on traditional HPC platforms, on a cloud-resident virtual cluster it is easy to change the I/O configurations, such as the choice of file systems, the number of I/O nodes, and the types of virtual disks, to fit the I/O requirements of different applications. In this paper, we discuss how cloud platforms can be employed to form customized and balanced I/O subsystems for individual I/O-intensive MPI applications. Through our preliminary evaluation, we demonstrate that different applications will benefit from individually tailored I/O system configurations. For a given I/O-intensive application, different I/O settings may lead to significant overall application performance or cost difference (up to 2.5-fold). Our exploration indicates that customized system configuration for HPC applications in the cloud is important and non-trivial.",
author = "Mingliang Liu and Jidong Zhai and Yan Zhai and Xiaosong Ma and Wenguang Chen",
year = "2011",
month = "12",
day = "1",
doi = "10.1145/2103799.2103818",
language = "English",
isbn = "9781450311793",
booktitle = "Proceedings of the 2nd Asia-Pacific Workshop on Systems, APSys'11",

}

TY - GEN

T1 - One optimized I/O configuration per HPC application

T2 - Leveraging the configurability of cloud

AU - Liu, Mingliang

AU - Zhai, Jidong

AU - Zhai, Yan

AU - Ma, Xiaosong

AU - Chen, Wenguang

PY - 2011/12/1

Y1 - 2011/12/1

N2 - There is a trend to migrate HPC (High Performance Computing) applications to cloud platforms, such as the Amazon EC2 Cluster Compute Instances (CCIs). While existing research has mainly focused on the performance impact of virtualized environments and interconnect technologies on parallel programs, we suggest that the configurability enabled by clouds is another important dimension to explore. Unlike on traditional HPC platforms, on a cloud-resident virtual cluster it is easy to change the I/O configurations, such as the choice of file systems, the number of I/O nodes, and the types of virtual disks, to fit the I/O requirements of different applications. In this paper, we discuss how cloud platforms can be employed to form customized and balanced I/O subsystems for individual I/O-intensive MPI applications. Through our preliminary evaluation, we demonstrate that different applications will benefit from individually tailored I/O system configurations. For a given I/O-intensive application, different I/O settings may lead to significant overall application performance or cost difference (up to 2.5-fold). Our exploration indicates that customized system configuration for HPC applications in the cloud is important and non-trivial.

AB - There is a trend to migrate HPC (High Performance Computing) applications to cloud platforms, such as the Amazon EC2 Cluster Compute Instances (CCIs). While existing research has mainly focused on the performance impact of virtualized environments and interconnect technologies on parallel programs, we suggest that the configurability enabled by clouds is another important dimension to explore. Unlike on traditional HPC platforms, on a cloud-resident virtual cluster it is easy to change the I/O configurations, such as the choice of file systems, the number of I/O nodes, and the types of virtual disks, to fit the I/O requirements of different applications. In this paper, we discuss how cloud platforms can be employed to form customized and balanced I/O subsystems for individual I/O-intensive MPI applications. Through our preliminary evaluation, we demonstrate that different applications will benefit from individually tailored I/O system configurations. For a given I/O-intensive application, different I/O settings may lead to significant overall application performance or cost difference (up to 2.5-fold). Our exploration indicates that customized system configuration for HPC applications in the cloud is important and non-trivial.

UR - http://www.scopus.com/inward/record.url?scp=84863173594&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84863173594&partnerID=8YFLogxK

U2 - 10.1145/2103799.2103818

DO - 10.1145/2103799.2103818

M3 - Conference contribution

SN - 9781450311793

BT - Proceedings of the 2nd Asia-Pacific Workshop on Systems, APSys'11

ER -