High-level buffering for hiding periodic output cost in scientific simulations

Xiaosong Ma, Jonghyun Lee, Marianne Winslett

Research output: Contribution to journalArticle

20 Citations (Scopus)

Abstract

Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we present active buffering, a high-level transparent buffering scheme for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. It utilizes idle memory on the processors, yet makes no assumption regarding runtime memory availability. Active buffering can perform background I/O while the computation is going on, is extensible to remote I/O for more efficient data migration, and can be implemented in a portable style in today's parallel I/O libraries. It can also mask performance problems of scientific data formats used by many scientists. Performance experiments with both synthetic benchmarks and real simulation codes on multiple platforms show that active buffering can greatly reduce the visible I/O cost from the application's point of view.

Original languageEnglish
Pages (from-to)193-204
Number of pages12
JournalIEEE Transactions on Parallel and Distributed Systems
Volume17
Issue number3
DOIs
Publication statusPublished - 1 Mar 2006

    Fingerprint

Keywords

  • Experimentation
  • Parallel I/O library design
  • Performance optimization

ASJC Scopus subject areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

Cite this