Data I/O management approach for the post-hoc visualization of big simulation data results

Jorji Nonaka, Eduardo C. Inacio, Kenji Ono, Mario A.R. Dantas, Yasuhiro Kawashima, Tomohiro Kawanabe, Fumiyoshi Shoji

    研究成果: Contribution to journalArticle査読

    2 被引用数 (Scopus)


    Leading-edge supercomputers, such as the K computer, have generated a vast amount of simulation results, and most of these datasets were stored on the file system for the post-hoc analysis such as visualization. In this work, we first investigated the data generation trends of the K computer by analyzing some operational log data files. We verified a tendency of generating large amounts of distributed files as simulation outputs, and in most cases, the number of files has been proportional to the number of utilized computational nodes, that is, each computational node producing one or more files. Considering that the computational cost of visualization tasks is usually much smaller than that required for large-scale numerical simulations, a flexible data input/output (I/O) management mechanism becomes highly useful for the post-hoc visualization and analysis. In this work, we focused on the xDMlib data management library, and its flexible data I/O mechanism in order to enable flexible data loading of big computational climate simulation results. In the proposed approach, a pre-processing is executed on the target distributed files for generating a light-weight metadata necessary for the elaboration of the data assignment mapping used in the subsequent data loading process. We evaluated the proposed approach by using a 32-node visualization cluster, and the K computer. Besides the inevitable performance penalty associated with longer data loading time, when using smaller number of processes, there is a benefit for avoiding any data replication via copy, conversion, or extraction. In addition, users will be able to freely select any number of nodes, without caring about the number of distributed files, for the post-hoc visualization and analysis purposes.

    ジャーナルInternational Journal of Modeling, Simulation, and Scientific Computing
    出版ステータス出版済み - 6 1 2018

    All Science Journal Classification (ASJC) codes

    • モデリングとシミュレーション
    • コンピュータ サイエンスの応用


    「Data I/O management approach for the post-hoc visualization of big simulation data results」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。