Scalable I/O tracing and analysis

Karthik Vijayakumar, Frank Mueller, Xiaosong Ma, Philip C. Roth

Research output: Chapter in Book/Report/Conference proceedingConference contribution

34 Citations (Scopus)

Abstract

As supercomputer performance approached and then surpassed the petaflop level, I/O performance has become a major performance bottleneck for many scientific applications. Several tools exist to collect I/O traces to assist in the analysis of I/O performance problems. However, these tools either produce extremely large trace files that complicate performance analysis, or sacrifice accuracy to collect high-level statistical information. We propose a multi-level trace generator tool, ScalaIOTrace, that collects traces at several levels in the HPC I/O stack. ScalaIOTrace features aggressive trace compression that generates trace files of near constant size for regular I/O patterns and orders of magnitudes smaller for less regular ones. This enables the collection of I/O and communication traces of applications running on thousands of processors. Our contributions also include automated trace analysis to collect selected statistical information of I/O calls by parsing the compressed trace on-the-fly and time-accurate replay of communication events with MPI-IO calls. We evaluated our approach with the Parallel Ocean Program (POP) climate simulation and the FLASH parallel I/O benchmark. POP uses NetCDF as an I/O library while FLASH I/O uses the parallel HDF5 I/O library, which internally maps onto MPI-IO. We collected MPI-IO and low-level POSIX I/O traces to study application I/O behavior. Our results show constant size trace files of only 145KB irrespective of the number of nodes for FLASH I/O benchmark, which exhibits regular I/O and communication pattern. For POP, we observe up to two orders of magnitude reduction in trace file sizes compared to flat traces. Statistical information gathered reveals insight on the number of I/O and communication calls issued in the POP and FLASH I/O. Such concise traces are unprecedented for isolated I/O and combined I/O plus communication tracing.

Original languageEnglish
Title of host publicationProceedings of the 4th Annual Petascale Data Storage Workshop, PDSW '09, Held in conjunction with Supercomputing '09
Pages26-31
Number of pages6
DOIs
Publication statusPublished - 1 Dec 2009
Externally publishedYes
Event4th Annual Petascale Data Storage Workshop, PDSW '09, Held in conjunction with Supercomputing '09 - Portland, OR, United States
Duration: 15 Nov 200915 Nov 2009

Publication series

NameProceedings of the 4th Annual Petascale Data Storage Workshop, PDSW '09, Held in conjunction with Supercomputing '09

Other

Other4th Annual Petascale Data Storage Workshop, PDSW '09, Held in conjunction with Supercomputing '09
CountryUnited States
CityPortland, OR
Period15/11/0915/11/09

    Fingerprint

ASJC Scopus subject areas

  • Hardware and Architecture
  • Software

Cite this

Vijayakumar, K., Mueller, F., Ma, X., & Roth, P. C. (2009). Scalable I/O tracing and analysis. In Proceedings of the 4th Annual Petascale Data Storage Workshop, PDSW '09, Held in conjunction with Supercomputing '09 (pp. 26-31). (Proceedings of the 4th Annual Petascale Data Storage Workshop, PDSW '09, Held in conjunction with Supercomputing '09). https://doi.org/10.1145/1713072.1713080