High-level buffering for hiding periodic output cost in scientific simulations

Xiaosong Ma, Jonghyun Lee, Marianne Winslett

Research output: Contribution to journalArticle

20 Citations (Scopus)

Abstract

Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we present active buffering, a high-level transparent buffering scheme for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. It utilizes idle memory on the processors, yet makes no assumption regarding runtime memory availability. Active buffering can perform background I/O while the computation is going on, is extensible to remote I/O for more efficient data migration, and can be implemented in a portable style in today's parallel I/O libraries. It can also mask performance problems of scientific data formats used by many scientists. Performance experiments with both synthetic benchmarks and real simulation codes on multiple platforms show that active buffering can greatly reduce the visible I/O cost from the application's point of view.

Original languageEnglish
Pages (from-to)193-204
Number of pages12
JournalIEEE Transactions on Parallel and Distributed Systems
Volume17
Issue number3
DOIs
Publication statusPublished - 1 Mar 2006
Externally publishedYes

Fingerprint

Data storage equipment
Output
Costs
Parallel I/O
Simulation
Restart
Metadata
Mask
Migration
Buffer
Masks
Visualization
Availability
Benchmark
Experiment
Experiments
Libraries
Style
Background
Hierarchy

Keywords

  • Experimentation
  • Parallel I/O library design
  • Performance optimization

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Theoretical Computer Science
  • Computational Theory and Mathematics

Cite this

High-level buffering for hiding periodic output cost in scientific simulations. / Ma, Xiaosong; Lee, Jonghyun; Winslett, Marianne.

In: IEEE Transactions on Parallel and Distributed Systems, Vol. 17, No. 3, 01.03.2006, p. 193-204.

Research output: Contribution to journalArticle

@article{cf52a1f660184b469a172937239e4a5e,
title = "High-level buffering for hiding periodic output cost in scientific simulations",
abstract = "Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we present active buffering, a high-level transparent buffering scheme for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. It utilizes idle memory on the processors, yet makes no assumption regarding runtime memory availability. Active buffering can perform background I/O while the computation is going on, is extensible to remote I/O for more efficient data migration, and can be implemented in a portable style in today's parallel I/O libraries. It can also mask performance problems of scientific data formats used by many scientists. Performance experiments with both synthetic benchmarks and real simulation codes on multiple platforms show that active buffering can greatly reduce the visible I/O cost from the application's point of view.",
keywords = "Experimentation, Parallel I/O library design, Performance optimization",
author = "Xiaosong Ma and Jonghyun Lee and Marianne Winslett",
year = "2006",
month = "3",
day = "1",
doi = "10.1109/TPDS.2006.36",
language = "English",
volume = "17",
pages = "193--204",
journal = "IEEE Transactions on Parallel and Distributed Systems",
issn = "1045-9219",
publisher = "IEEE Computer Society",
number = "3",

}

TY - JOUR

T1 - High-level buffering for hiding periodic output cost in scientific simulations

AU - Ma, Xiaosong

AU - Lee, Jonghyun

AU - Winslett, Marianne

PY - 2006/3/1

Y1 - 2006/3/1

N2 - Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we present active buffering, a high-level transparent buffering scheme for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. It utilizes idle memory on the processors, yet makes no assumption regarding runtime memory availability. Active buffering can perform background I/O while the computation is going on, is extensible to remote I/O for more efficient data migration, and can be implemented in a portable style in today's parallel I/O libraries. It can also mask performance problems of scientific data formats used by many scientists. Performance experiments with both synthetic benchmarks and real simulation codes on multiple platforms show that active buffering can greatly reduce the visible I/O cost from the application's point of view.

AB - Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we present active buffering, a high-level transparent buffering scheme for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. It utilizes idle memory on the processors, yet makes no assumption regarding runtime memory availability. Active buffering can perform background I/O while the computation is going on, is extensible to remote I/O for more efficient data migration, and can be implemented in a portable style in today's parallel I/O libraries. It can also mask performance problems of scientific data formats used by many scientists. Performance experiments with both synthetic benchmarks and real simulation codes on multiple platforms show that active buffering can greatly reduce the visible I/O cost from the application's point of view.

KW - Experimentation

KW - Parallel I/O library design

KW - Performance optimization

UR - http://www.scopus.com/inward/record.url?scp=32044451992&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=32044451992&partnerID=8YFLogxK

U2 - 10.1109/TPDS.2006.36

DO - 10.1109/TPDS.2006.36

M3 - Article

VL - 17

SP - 193

EP - 204

JO - IEEE Transactions on Parallel and Distributed Systems

JF - IEEE Transactions on Parallel and Distributed Systems

SN - 1045-9219

IS - 3

ER -