Skel: Generative software for producing skeletal I/O applications

Jeremy Logan, Scott Klasky, Jay Lofstead, Hasan Abbasi, Stéphane Ethier, Ray Grout, Seung Hoe Ku, Qing Liu, Xiaosong Ma, Manish Parashar, Norbert Podhorszki, Karsten Schwan, Matthew Wolf

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Citations (Scopus)

Abstract

Massively parallel computations consist of a mixture of computation, communication, and I/O. Of these three components, implementing an effective parallel I/O solution has often been overlooked by application scientists and has typically been added to large scale simulations only when existing serial techniques have failed. As scientists' teams scaled their codes to run on hundreds of processors, it was common to call on an I/O expert to implement a set of more scalable I/O routines. These routines were easily separated from the calculations and communication, and in many cases, an I/O kernel was derived from the application which could be used for testing I/O performance independent of the application. These I/O kernels developed a life of their own used as a broad measure for comparing different I/O techniques. Unfortunately, as years passed and computation and communication changes required changes to the I/O, the separate I/O kernel used for benchmarking remained static, no longer providing an accurate indicator of the I/O performance of the simulation, and making I/O research less relevant for the application scientists. In this paper we describe a new approach to this problem where I/O kernels are replaced with skeletal I/O applications that are automatically generated from an abstract set of simulation I/O parameters. We realize this abstraction by leveraging the ADIOS [1] middleware's XML I/O specification with additional runtime parameters. Skeletal applications offer all of the benefits of I/O kernels including allowing I/O optimizations to focus on useful I/O patterns. Moreover, since they are automatically generated, it is easy to produce an updated I/O skeleton whenever the simulation's I/O changes. In this paper we analyze the performance of automatically generated I/O skeletal applications for the S3D and GTS codes. We show that these skeletal applications achieve performance comparable to that of the production applications. We wrap up the paper with a discussion of future changes to make the skeletal application better approximate the actual I/O performed in the simulation.

Original languageEnglish
Title of host publicationProceedings - 7th IEEE International Conference on e-Science Workshops, eScienceW 2011
Pages191-198
Number of pages8
DOIs
Publication statusPublished - 1 Dec 2011
Externally publishedYes
Event7th IEEE International Conference on e-Science Workshops, eScienceW 201 - Stockholm, Sweden
Duration: 5 Dec 20118 Dec 2011

Other

Other7th IEEE International Conference on e-Science Workshops, eScienceW 201
CountrySweden
CityStockholm
Period5/12/118/12/11

Fingerprint

Communication
Software
Benchmarking
Middleware
XML
Kernel
Simulation
Specifications
Testing
Serials

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Management Science and Operations Research

Cite this

Logan, J., Klasky, S., Lofstead, J., Abbasi, H., Ethier, S., Grout, R., ... Wolf, M. (2011). Skel: Generative software for producing skeletal I/O applications. In Proceedings - 7th IEEE International Conference on e-Science Workshops, eScienceW 2011 (pp. 191-198). [6130712] https://doi.org/10.1109/eScienceW.2011.26

Skel : Generative software for producing skeletal I/O applications. / Logan, Jeremy; Klasky, Scott; Lofstead, Jay; Abbasi, Hasan; Ethier, Stéphane; Grout, Ray; Ku, Seung Hoe; Liu, Qing; Ma, Xiaosong; Parashar, Manish; Podhorszki, Norbert; Schwan, Karsten; Wolf, Matthew.

Proceedings - 7th IEEE International Conference on e-Science Workshops, eScienceW 2011. 2011. p. 191-198 6130712.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Logan, J, Klasky, S, Lofstead, J, Abbasi, H, Ethier, S, Grout, R, Ku, SH, Liu, Q, Ma, X, Parashar, M, Podhorszki, N, Schwan, K & Wolf, M 2011, Skel: Generative software for producing skeletal I/O applications. in Proceedings - 7th IEEE International Conference on e-Science Workshops, eScienceW 2011., 6130712, pp. 191-198, 7th IEEE International Conference on e-Science Workshops, eScienceW 201, Stockholm, Sweden, 5/12/11. https://doi.org/10.1109/eScienceW.2011.26
Logan J, Klasky S, Lofstead J, Abbasi H, Ethier S, Grout R et al. Skel: Generative software for producing skeletal I/O applications. In Proceedings - 7th IEEE International Conference on e-Science Workshops, eScienceW 2011. 2011. p. 191-198. 6130712 https://doi.org/10.1109/eScienceW.2011.26
Logan, Jeremy ; Klasky, Scott ; Lofstead, Jay ; Abbasi, Hasan ; Ethier, Stéphane ; Grout, Ray ; Ku, Seung Hoe ; Liu, Qing ; Ma, Xiaosong ; Parashar, Manish ; Podhorszki, Norbert ; Schwan, Karsten ; Wolf, Matthew. / Skel : Generative software for producing skeletal I/O applications. Proceedings - 7th IEEE International Conference on e-Science Workshops, eScienceW 2011. 2011. pp. 191-198
@inproceedings{d77753b8f76d47e1aa1dc563f50dc468,
title = "Skel: Generative software for producing skeletal I/O applications",
abstract = "Massively parallel computations consist of a mixture of computation, communication, and I/O. Of these three components, implementing an effective parallel I/O solution has often been overlooked by application scientists and has typically been added to large scale simulations only when existing serial techniques have failed. As scientists' teams scaled their codes to run on hundreds of processors, it was common to call on an I/O expert to implement a set of more scalable I/O routines. These routines were easily separated from the calculations and communication, and in many cases, an I/O kernel was derived from the application which could be used for testing I/O performance independent of the application. These I/O kernels developed a life of their own used as a broad measure for comparing different I/O techniques. Unfortunately, as years passed and computation and communication changes required changes to the I/O, the separate I/O kernel used for benchmarking remained static, no longer providing an accurate indicator of the I/O performance of the simulation, and making I/O research less relevant for the application scientists. In this paper we describe a new approach to this problem where I/O kernels are replaced with skeletal I/O applications that are automatically generated from an abstract set of simulation I/O parameters. We realize this abstraction by leveraging the ADIOS [1] middleware's XML I/O specification with additional runtime parameters. Skeletal applications offer all of the benefits of I/O kernels including allowing I/O optimizations to focus on useful I/O patterns. Moreover, since they are automatically generated, it is easy to produce an updated I/O skeleton whenever the simulation's I/O changes. In this paper we analyze the performance of automatically generated I/O skeletal applications for the S3D and GTS codes. We show that these skeletal applications achieve performance comparable to that of the production applications. We wrap up the paper with a discussion of future changes to make the skeletal application better approximate the actual I/O performed in the simulation.",
author = "Jeremy Logan and Scott Klasky and Jay Lofstead and Hasan Abbasi and St{\'e}phane Ethier and Ray Grout and Ku, {Seung Hoe} and Qing Liu and Xiaosong Ma and Manish Parashar and Norbert Podhorszki and Karsten Schwan and Matthew Wolf",
year = "2011",
month = "12",
day = "1",
doi = "10.1109/eScienceW.2011.26",
language = "English",
isbn = "9780769545981",
pages = "191--198",
booktitle = "Proceedings - 7th IEEE International Conference on e-Science Workshops, eScienceW 2011",

}

TY - GEN

T1 - Skel

T2 - Generative software for producing skeletal I/O applications

AU - Logan, Jeremy

AU - Klasky, Scott

AU - Lofstead, Jay

AU - Abbasi, Hasan

AU - Ethier, Stéphane

AU - Grout, Ray

AU - Ku, Seung Hoe

AU - Liu, Qing

AU - Ma, Xiaosong

AU - Parashar, Manish

AU - Podhorszki, Norbert

AU - Schwan, Karsten

AU - Wolf, Matthew

PY - 2011/12/1

Y1 - 2011/12/1

N2 - Massively parallel computations consist of a mixture of computation, communication, and I/O. Of these three components, implementing an effective parallel I/O solution has often been overlooked by application scientists and has typically been added to large scale simulations only when existing serial techniques have failed. As scientists' teams scaled their codes to run on hundreds of processors, it was common to call on an I/O expert to implement a set of more scalable I/O routines. These routines were easily separated from the calculations and communication, and in many cases, an I/O kernel was derived from the application which could be used for testing I/O performance independent of the application. These I/O kernels developed a life of their own used as a broad measure for comparing different I/O techniques. Unfortunately, as years passed and computation and communication changes required changes to the I/O, the separate I/O kernel used for benchmarking remained static, no longer providing an accurate indicator of the I/O performance of the simulation, and making I/O research less relevant for the application scientists. In this paper we describe a new approach to this problem where I/O kernels are replaced with skeletal I/O applications that are automatically generated from an abstract set of simulation I/O parameters. We realize this abstraction by leveraging the ADIOS [1] middleware's XML I/O specification with additional runtime parameters. Skeletal applications offer all of the benefits of I/O kernels including allowing I/O optimizations to focus on useful I/O patterns. Moreover, since they are automatically generated, it is easy to produce an updated I/O skeleton whenever the simulation's I/O changes. In this paper we analyze the performance of automatically generated I/O skeletal applications for the S3D and GTS codes. We show that these skeletal applications achieve performance comparable to that of the production applications. We wrap up the paper with a discussion of future changes to make the skeletal application better approximate the actual I/O performed in the simulation.

AB - Massively parallel computations consist of a mixture of computation, communication, and I/O. Of these three components, implementing an effective parallel I/O solution has often been overlooked by application scientists and has typically been added to large scale simulations only when existing serial techniques have failed. As scientists' teams scaled their codes to run on hundreds of processors, it was common to call on an I/O expert to implement a set of more scalable I/O routines. These routines were easily separated from the calculations and communication, and in many cases, an I/O kernel was derived from the application which could be used for testing I/O performance independent of the application. These I/O kernels developed a life of their own used as a broad measure for comparing different I/O techniques. Unfortunately, as years passed and computation and communication changes required changes to the I/O, the separate I/O kernel used for benchmarking remained static, no longer providing an accurate indicator of the I/O performance of the simulation, and making I/O research less relevant for the application scientists. In this paper we describe a new approach to this problem where I/O kernels are replaced with skeletal I/O applications that are automatically generated from an abstract set of simulation I/O parameters. We realize this abstraction by leveraging the ADIOS [1] middleware's XML I/O specification with additional runtime parameters. Skeletal applications offer all of the benefits of I/O kernels including allowing I/O optimizations to focus on useful I/O patterns. Moreover, since they are automatically generated, it is easy to produce an updated I/O skeleton whenever the simulation's I/O changes. In this paper we analyze the performance of automatically generated I/O skeletal applications for the S3D and GTS codes. We show that these skeletal applications achieve performance comparable to that of the production applications. We wrap up the paper with a discussion of future changes to make the skeletal application better approximate the actual I/O performed in the simulation.

UR - http://www.scopus.com/inward/record.url?scp=84863078107&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84863078107&partnerID=8YFLogxK

U2 - 10.1109/eScienceW.2011.26

DO - 10.1109/eScienceW.2011.26

M3 - Conference contribution

AN - SCOPUS:84863078107

SN - 9780769545981

SP - 191

EP - 198

BT - Proceedings - 7th IEEE International Conference on e-Science Workshops, eScienceW 2011

ER -