Characterizing I/O and Storage Activity on the K Computer for Post-Processing Purposes

Eduardo C. Inacio, Jorji Nonaka, Kenji Ono, Mario A.R. Dantas, Fumiyoshi Shoji

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    1 Citation (Scopus)

    Abstract

    An increasing volume of data is produced by computational science applications executing on flagship-class supercomputers, such as the K computer. Most of these huge datasets would later pass through post-processing for visualization and analysis in order to derive meaningful information. Particular characteristics of the computing environment, application, and the dataset itself, can make efficiently exploring the performance capabilities of large-scale storage systems supporting these supercomputer a challenging task. This paper presents a characterization of the I/O and storage activity of jobs executed on the K computer focusing on post-processing purposes, based upon nine months of production operation recorded. Results demonstrate the intensive data demand of K computer applications, both in terms of volume of file I/O carried out during job execution, amount of data staged-in and staged-out, and number of files produced per job. These aspects shed light on challenges and opportunities for specialized data management libraries for posthoc data visualization and analysis.

    Original languageEnglish
    Title of host publication2018 IEEE Symposium on Computers and Communications, ISCC 2018
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages730-735
    Number of pages6
    Volume2018-June
    ISBN (Electronic)9781538669501
    DOIs
    Publication statusPublished - Nov 15 2018
    Event2018 IEEE Symposium on Computers and Communications, ISCC 2018 - Natal, Brazil
    Duration: Jun 25 2018Jun 28 2018

    Publication series

    NameProceedings - IEEE Symposium on Computers and Communications
    Volume2018-June
    ISSN (Print)1530-1346

    Conference

    Conference2018 IEEE Symposium on Computers and Communications, ISCC 2018
    CountryBrazil
    CityNatal
    Period6/25/186/28/18

    All Science Journal Classification (ASJC) codes

    • Software
    • Signal Processing
    • Mathematics(all)
    • Computer Science Applications
    • Computer Networks and Communications

    Fingerprint Dive into the research topics of 'Characterizing I/O and Storage Activity on the K Computer for Post-Processing Purposes'. Together they form a unique fingerprint.

    Cite this