Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters

Shuangcheng Niu, Jidong Zhai, Xiaosong Ma, Xiongchao Tang, Wenguang Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

25 Citations (Scopus)

Abstract

Recent studies have found cloud environments increasingly appealing for executing HPC applications, including tight-ly coupled parallel simulations. While public clouds offer elastic, on-demand resource provisioning and pay-as-you-go pricing, individual users setting up their on-demand virtual clusters may not be able to take full advantage of common cost-saving opportunities, such as reserved instances. In this paper, we propose a Semi-Elastic Cluster (SEC) computing model for organizations to reserve and dynam-ically resize a virtual cloud-based cluster. We present a set of integrated batch scheduling plus resource scaling strategies uniquely enabled by SEC, as well as an online reserved instance provisioning algorithm based on job history. Our trace-driven simulation results show that such a model has a 61.0% cost saving than individual users acquiring and managing cloud resources without causing longer average job wait time. Meanwhile, the overhead of acquiring/maintaining shared cloud instances is shown to take only a few seconds.

Original languageEnglish
Title of host publicationInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC
PublisherIEEE Computer Society
ISBN (Print)9781450323789
DOIs
Publication statusPublished - 1 Jan 2013
Externally publishedYes
Event2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013 - Denver, CO, United States
Duration: 17 Nov 201322 Nov 2013

Other

Other2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013
CountryUnited States
CityDenver, CO
Period17/11/1322/11/13

Fingerprint

Cluster computing
Costs
Scheduling

Keywords

  • Cloud computing
  • Job scheduling
  • Resource provisioning

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Software

Cite this

Niu, S., Zhai, J., Ma, X., Tang, X., & Chen, W. (2013). Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC [56] IEEE Computer Society. https://doi.org/10.1145/2503210.2503236

Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters. / Niu, Shuangcheng; Zhai, Jidong; Ma, Xiaosong; Tang, Xiongchao; Chen, Wenguang.

International Conference for High Performance Computing, Networking, Storage and Analysis, SC. IEEE Computer Society, 2013. 56.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Niu, S, Zhai, J, Ma, X, Tang, X & Chen, W 2013, Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters. in International Conference for High Performance Computing, Networking, Storage and Analysis, SC., 56, IEEE Computer Society, 2013 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013, Denver, CO, United States, 17/11/13. https://doi.org/10.1145/2503210.2503236
Niu S, Zhai J, Ma X, Tang X, Chen W. Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC. IEEE Computer Society. 2013. 56 https://doi.org/10.1145/2503210.2503236
Niu, Shuangcheng ; Zhai, Jidong ; Ma, Xiaosong ; Tang, Xiongchao ; Chen, Wenguang. / Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters. International Conference for High Performance Computing, Networking, Storage and Analysis, SC. IEEE Computer Society, 2013.
@inproceedings{fd69d310fa7848a6879e825cf25aa5b7,
title = "Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters",
abstract = "Recent studies have found cloud environments increasingly appealing for executing HPC applications, including tight-ly coupled parallel simulations. While public clouds offer elastic, on-demand resource provisioning and pay-as-you-go pricing, individual users setting up their on-demand virtual clusters may not be able to take full advantage of common cost-saving opportunities, such as reserved instances. In this paper, we propose a Semi-Elastic Cluster (SEC) computing model for organizations to reserve and dynam-ically resize a virtual cloud-based cluster. We present a set of integrated batch scheduling plus resource scaling strategies uniquely enabled by SEC, as well as an online reserved instance provisioning algorithm based on job history. Our trace-driven simulation results show that such a model has a 61.0{\%} cost saving than individual users acquiring and managing cloud resources without causing longer average job wait time. Meanwhile, the overhead of acquiring/maintaining shared cloud instances is shown to take only a few seconds.",
keywords = "Cloud computing, Job scheduling, Resource provisioning",
author = "Shuangcheng Niu and Jidong Zhai and Xiaosong Ma and Xiongchao Tang and Wenguang Chen",
year = "2013",
month = "1",
day = "1",
doi = "10.1145/2503210.2503236",
language = "English",
isbn = "9781450323789",
booktitle = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",
publisher = "IEEE Computer Society",

}

TY - GEN

T1 - Cost-effective cloud HPC resource provisioning by building semi-elastic virtual clusters

AU - Niu, Shuangcheng

AU - Zhai, Jidong

AU - Ma, Xiaosong

AU - Tang, Xiongchao

AU - Chen, Wenguang

PY - 2013/1/1

Y1 - 2013/1/1

N2 - Recent studies have found cloud environments increasingly appealing for executing HPC applications, including tight-ly coupled parallel simulations. While public clouds offer elastic, on-demand resource provisioning and pay-as-you-go pricing, individual users setting up their on-demand virtual clusters may not be able to take full advantage of common cost-saving opportunities, such as reserved instances. In this paper, we propose a Semi-Elastic Cluster (SEC) computing model for organizations to reserve and dynam-ically resize a virtual cloud-based cluster. We present a set of integrated batch scheduling plus resource scaling strategies uniquely enabled by SEC, as well as an online reserved instance provisioning algorithm based on job history. Our trace-driven simulation results show that such a model has a 61.0% cost saving than individual users acquiring and managing cloud resources without causing longer average job wait time. Meanwhile, the overhead of acquiring/maintaining shared cloud instances is shown to take only a few seconds.

AB - Recent studies have found cloud environments increasingly appealing for executing HPC applications, including tight-ly coupled parallel simulations. While public clouds offer elastic, on-demand resource provisioning and pay-as-you-go pricing, individual users setting up their on-demand virtual clusters may not be able to take full advantage of common cost-saving opportunities, such as reserved instances. In this paper, we propose a Semi-Elastic Cluster (SEC) computing model for organizations to reserve and dynam-ically resize a virtual cloud-based cluster. We present a set of integrated batch scheduling plus resource scaling strategies uniquely enabled by SEC, as well as an online reserved instance provisioning algorithm based on job history. Our trace-driven simulation results show that such a model has a 61.0% cost saving than individual users acquiring and managing cloud resources without causing longer average job wait time. Meanwhile, the overhead of acquiring/maintaining shared cloud instances is shown to take only a few seconds.

KW - Cloud computing

KW - Job scheduling

KW - Resource provisioning

UR - http://www.scopus.com/inward/record.url?scp=84899702246&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899702246&partnerID=8YFLogxK

U2 - 10.1145/2503210.2503236

DO - 10.1145/2503210.2503236

M3 - Conference contribution

SN - 9781450323789

BT - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

PB - IEEE Computer Society

ER -