ACIC: Automatic cloud I/O configurator for HPC applications

Mingliang Liu, Ye Jin, Jidong Zhai, Yan Zha, Qianqian Shi, Xiaosong Ma, Wenguang Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Citations (Scopus)

Abstract

The cloud has become a promising alternative to tradi-tional HPC centers or in-house clusters. This new environ-ment highlights the I/O bottleneck problem, typically with top-of-the-line compute instances but sub-par communica-tion and I/O facilities. It has been observed that changing cloud I/O system configurations leads to significant varia-tion in the performance and cost efficiency of I/O intensive HPC applications. However, storage system configuration is tedious and error-prone to do manually, even for experts. This paper proposes ACIC, which takes a given applica-tion running on a given cloud platform, and automatically searches for optimized I/O system configurations. ACIC utilizes machine learning models to perform black-box per-formance/cost predictions. To tackle the high-dimensional parameter exploration space unique to cloud platforms, we enable affordable, reusable, and incremental training guided by Plackett and Burman Matrices. Results with four repre-sentative applications indicate that ACIC consistently iden-tiffes near-optimal configurations among a large group of candidate settings.

Original languageEnglish
Title of host publicationInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC
PublisherIEEE Computer Society
ISBN (Print)9781450323789
DOIs
Publication statusPublished - 1 Jan 2013
Externally publishedYes
Event2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013 - Denver, CO, United States
Duration: 17 Nov 201322 Nov 2013

Other

Other2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013
CountryUnited States
CityDenver, CO
Period17/11/1322/11/13

Fingerprint

Learning systems
Costs

Keywords

  • Cloud Computing
  • Modeling
  • Performance
  • Storage

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Software

Cite this

Liu, M., Jin, Y., Zhai, J., Zha, Y., Shi, Q., Ma, X., & Chen, W. (2013). ACIC: Automatic cloud I/O configurator for HPC applications. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC [38] IEEE Computer Society. https://doi.org/10.1145/2503210.2503216

ACIC : Automatic cloud I/O configurator for HPC applications. / Liu, Mingliang; Jin, Ye; Zhai, Jidong; Zha, Yan; Shi, Qianqian; Ma, Xiaosong; Chen, Wenguang.

International Conference for High Performance Computing, Networking, Storage and Analysis, SC. IEEE Computer Society, 2013. 38.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Liu, M, Jin, Y, Zhai, J, Zha, Y, Shi, Q, Ma, X & Chen, W 2013, ACIC: Automatic cloud I/O configurator for HPC applications. in International Conference for High Performance Computing, Networking, Storage and Analysis, SC., 38, IEEE Computer Society, 2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013, Denver, CO, United States, 17/11/13. https://doi.org/10.1145/2503210.2503216
Liu M, Jin Y, Zhai J, Zha Y, Shi Q, Ma X et al. ACIC: Automatic cloud I/O configurator for HPC applications. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC. IEEE Computer Society. 2013. 38 https://doi.org/10.1145/2503210.2503216
Liu, Mingliang ; Jin, Ye ; Zhai, Jidong ; Zha, Yan ; Shi, Qianqian ; Ma, Xiaosong ; Chen, Wenguang. / ACIC : Automatic cloud I/O configurator for HPC applications. International Conference for High Performance Computing, Networking, Storage and Analysis, SC. IEEE Computer Society, 2013.
@inproceedings{d0f7ff56e2c144098e8bc6b39aa57521,
title = "ACIC: Automatic cloud I/O configurator for HPC applications",
abstract = "The cloud has become a promising alternative to tradi-tional HPC centers or in-house clusters. This new environ-ment highlights the I/O bottleneck problem, typically with top-of-the-line compute instances but sub-par communica-tion and I/O facilities. It has been observed that changing cloud I/O system configurations leads to significant varia-tion in the performance and cost efficiency of I/O intensive HPC applications. However, storage system configuration is tedious and error-prone to do manually, even for experts. This paper proposes ACIC, which takes a given applica-tion running on a given cloud platform, and automatically searches for optimized I/O system configurations. ACIC utilizes machine learning models to perform black-box per-formance/cost predictions. To tackle the high-dimensional parameter exploration space unique to cloud platforms, we enable affordable, reusable, and incremental training guided by Plackett and Burman Matrices. Results with four repre-sentative applications indicate that ACIC consistently iden-tiffes near-optimal configurations among a large group of candidate settings.",
keywords = "Cloud Computing, Modeling, Performance, Storage",
author = "Mingliang Liu and Ye Jin and Jidong Zhai and Yan Zha and Qianqian Shi and Xiaosong Ma and Wenguang Chen",
year = "2013",
month = "1",
day = "1",
doi = "10.1145/2503210.2503216",
language = "English",
isbn = "9781450323789",
booktitle = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",
publisher = "IEEE Computer Society",

}

TY - GEN

T1 - ACIC

T2 - Automatic cloud I/O configurator for HPC applications

AU - Liu, Mingliang

AU - Jin, Ye

AU - Zhai, Jidong

AU - Zha, Yan

AU - Shi, Qianqian

AU - Ma, Xiaosong

AU - Chen, Wenguang

PY - 2013/1/1

Y1 - 2013/1/1

N2 - The cloud has become a promising alternative to tradi-tional HPC centers or in-house clusters. This new environ-ment highlights the I/O bottleneck problem, typically with top-of-the-line compute instances but sub-par communica-tion and I/O facilities. It has been observed that changing cloud I/O system configurations leads to significant varia-tion in the performance and cost efficiency of I/O intensive HPC applications. However, storage system configuration is tedious and error-prone to do manually, even for experts. This paper proposes ACIC, which takes a given applica-tion running on a given cloud platform, and automatically searches for optimized I/O system configurations. ACIC utilizes machine learning models to perform black-box per-formance/cost predictions. To tackle the high-dimensional parameter exploration space unique to cloud platforms, we enable affordable, reusable, and incremental training guided by Plackett and Burman Matrices. Results with four repre-sentative applications indicate that ACIC consistently iden-tiffes near-optimal configurations among a large group of candidate settings.

AB - The cloud has become a promising alternative to tradi-tional HPC centers or in-house clusters. This new environ-ment highlights the I/O bottleneck problem, typically with top-of-the-line compute instances but sub-par communica-tion and I/O facilities. It has been observed that changing cloud I/O system configurations leads to significant varia-tion in the performance and cost efficiency of I/O intensive HPC applications. However, storage system configuration is tedious and error-prone to do manually, even for experts. This paper proposes ACIC, which takes a given applica-tion running on a given cloud platform, and automatically searches for optimized I/O system configurations. ACIC utilizes machine learning models to perform black-box per-formance/cost predictions. To tackle the high-dimensional parameter exploration space unique to cloud platforms, we enable affordable, reusable, and incremental training guided by Plackett and Burman Matrices. Results with four repre-sentative applications indicate that ACIC consistently iden-tiffes near-optimal configurations among a large group of candidate settings.

KW - Cloud Computing

KW - Modeling

KW - Performance

KW - Storage

UR - http://www.scopus.com/inward/record.url?scp=84899682051&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899682051&partnerID=8YFLogxK

U2 - 10.1145/2503210.2503216

DO - 10.1145/2503210.2503216

M3 - Conference contribution

AN - SCOPUS:84899682051

SN - 9781450323789

BT - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

PB - IEEE Computer Society

ER -