Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets

Kenji Ono, Jorji Nonaka, Hiroyuki Yoshikawa, Takeshi Nanri, Yoshiyuki Morie, Tomohiro Kawanabe, Fumiyoshi Shoji

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents an in situ framework focused on time-varying simulations, and uses a novel temporal buffer for storing simulation results sampled at user-defined intervals. This framework has been designed to provide flexible data processing and visualization capabilities in modern HPC operational environments composed of powerful front-end systems, for pre-and post-processing purposes, along with traditional back-end HPC systems. The temporal buffer is implemented using the functionalities provided by Open Address Space (OpAS) library, which enables asynchronous one-sided communication from outside processes to any exposed memory region on the simulator side. This buffer can store time-varying simulation results, and can be processed via in situ approaches with different proximities. We present a prototype of our framework, and code integration process with a target simulation code. The proposed in situ framework utilizes separate files to describe the initialization and execution codes, which are in the form of Python scripts. This framework also enables the runtime modification of these Python-based files, thus providing greater flexibility to the users, not only for data processing, such as visualization and analysis, but also for the simulation steering.

Original languageEnglish
Title of host publicationHigh Performance Computing - ISC High Performance 2018 International Workshops, Revised Selected Papers
EditorsMichèle Weiland, Sadaf Alam, Rio Yokota, John Shalf
PublisherSpringer Verlag
Pages243-257
Number of pages15
ISBN (Print)9783030024642
DOIs
Publication statusPublished - Jan 1 2018
EventInternational Conference on High Performance Computing, ISC High Performance 2018 - Frankfurt, Germany
Duration: Jun 28 2018Jun 28 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11203 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceInternational Conference on High Performance Computing, ISC High Performance 2018
CountryGermany
CityFrankfurt
Period6/28/186/28/18

Fingerprint

Data visualization
Data Visualization
Buffer
Time-varying
Python
Simulation
Visualization
Simulators
Data storage equipment
Communication
Processing
Initialization
Post-processing
Proximity
Preprocessing
Simulator
Flexibility
Design
Framework
Prototype

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Ono, K., Nonaka, J., Yoshikawa, H., Nanri, T., Morie, Y., Kawanabe, T., & Shoji, F. (2018). Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets. In M. Weiland, S. Alam, R. Yokota, & J. Shalf (Eds.), High Performance Computing - ISC High Performance 2018 International Workshops, Revised Selected Papers (pp. 243-257). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11203 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-02465-9_17

Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets. / Ono, Kenji; Nonaka, Jorji; Yoshikawa, Hiroyuki; Nanri, Takeshi; Morie, Yoshiyuki; Kawanabe, Tomohiro; Shoji, Fumiyoshi.

High Performance Computing - ISC High Performance 2018 International Workshops, Revised Selected Papers. ed. / Michèle Weiland; Sadaf Alam; Rio Yokota; John Shalf. Springer Verlag, 2018. p. 243-257 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11203 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ono, K, Nonaka, J, Yoshikawa, H, Nanri, T, Morie, Y, Kawanabe, T & Shoji, F 2018, Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets. in M Weiland, S Alam, R Yokota & J Shalf (eds), High Performance Computing - ISC High Performance 2018 International Workshops, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11203 LNCS, Springer Verlag, pp. 243-257, International Conference on High Performance Computing, ISC High Performance 2018, Frankfurt, Germany, 6/28/18. https://doi.org/10.1007/978-3-030-02465-9_17
Ono K, Nonaka J, Yoshikawa H, Nanri T, Morie Y, Kawanabe T et al. Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets. In Weiland M, Alam S, Yokota R, Shalf J, editors, High Performance Computing - ISC High Performance 2018 International Workshops, Revised Selected Papers. Springer Verlag. 2018. p. 243-257. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-02465-9_17
Ono, Kenji ; Nonaka, Jorji ; Yoshikawa, Hiroyuki ; Nanri, Takeshi ; Morie, Yoshiyuki ; Kawanabe, Tomohiro ; Shoji, Fumiyoshi. / Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets. High Performance Computing - ISC High Performance 2018 International Workshops, Revised Selected Papers. editor / Michèle Weiland ; Sadaf Alam ; Rio Yokota ; John Shalf. Springer Verlag, 2018. pp. 243-257 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{a5b19c86ace34952ac03fa6e7114a2d3,
title = "Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets",
abstract = "This paper presents an in situ framework focused on time-varying simulations, and uses a novel temporal buffer for storing simulation results sampled at user-defined intervals. This framework has been designed to provide flexible data processing and visualization capabilities in modern HPC operational environments composed of powerful front-end systems, for pre-and post-processing purposes, along with traditional back-end HPC systems. The temporal buffer is implemented using the functionalities provided by Open Address Space (OpAS) library, which enables asynchronous one-sided communication from outside processes to any exposed memory region on the simulator side. This buffer can store time-varying simulation results, and can be processed via in situ approaches with different proximities. We present a prototype of our framework, and code integration process with a target simulation code. The proposed in situ framework utilizes separate files to describe the initialization and execution codes, which are in the form of Python scripts. This framework also enables the runtime modification of these Python-based files, thus providing greater flexibility to the users, not only for data processing, such as visualization and analysis, but also for the simulation steering.",
author = "Kenji Ono and Jorji Nonaka and Hiroyuki Yoshikawa and Takeshi Nanri and Yoshiyuki Morie and Tomohiro Kawanabe and Fumiyoshi Shoji",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/978-3-030-02465-9_17",
language = "English",
isbn = "9783030024642",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "243--257",
editor = "Mich{\`e}le Weiland and Sadaf Alam and Rio Yokota and John Shalf",
booktitle = "High Performance Computing - ISC High Performance 2018 International Workshops, Revised Selected Papers",
address = "Germany",

}

TY - GEN

T1 - Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets

AU - Ono, Kenji

AU - Nonaka, Jorji

AU - Yoshikawa, Hiroyuki

AU - Nanri, Takeshi

AU - Morie, Yoshiyuki

AU - Kawanabe, Tomohiro

AU - Shoji, Fumiyoshi

PY - 2018/1/1

Y1 - 2018/1/1

N2 - This paper presents an in situ framework focused on time-varying simulations, and uses a novel temporal buffer for storing simulation results sampled at user-defined intervals. This framework has been designed to provide flexible data processing and visualization capabilities in modern HPC operational environments composed of powerful front-end systems, for pre-and post-processing purposes, along with traditional back-end HPC systems. The temporal buffer is implemented using the functionalities provided by Open Address Space (OpAS) library, which enables asynchronous one-sided communication from outside processes to any exposed memory region on the simulator side. This buffer can store time-varying simulation results, and can be processed via in situ approaches with different proximities. We present a prototype of our framework, and code integration process with a target simulation code. The proposed in situ framework utilizes separate files to describe the initialization and execution codes, which are in the form of Python scripts. This framework also enables the runtime modification of these Python-based files, thus providing greater flexibility to the users, not only for data processing, such as visualization and analysis, but also for the simulation steering.

AB - This paper presents an in situ framework focused on time-varying simulations, and uses a novel temporal buffer for storing simulation results sampled at user-defined intervals. This framework has been designed to provide flexible data processing and visualization capabilities in modern HPC operational environments composed of powerful front-end systems, for pre-and post-processing purposes, along with traditional back-end HPC systems. The temporal buffer is implemented using the functionalities provided by Open Address Space (OpAS) library, which enables asynchronous one-sided communication from outside processes to any exposed memory region on the simulator side. This buffer can store time-varying simulation results, and can be processed via in situ approaches with different proximities. We present a prototype of our framework, and code integration process with a target simulation code. The proposed in situ framework utilizes separate files to describe the initialization and execution codes, which are in the form of Python scripts. This framework also enables the runtime modification of these Python-based files, thus providing greater flexibility to the users, not only for data processing, such as visualization and analysis, but also for the simulation steering.

UR - http://www.scopus.com/inward/record.url?scp=85066148704&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066148704&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-02465-9_17

DO - 10.1007/978-3-030-02465-9_17

M3 - Conference contribution

AN - SCOPUS:85066148704

SN - 9783030024642

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 243

EP - 257

BT - High Performance Computing - ISC High Performance 2018 International Workshops, Revised Selected Papers

A2 - Weiland, Michèle

A2 - Alam, Sadaf

A2 - Yokota, Rio

A2 - Shalf, John

PB - Springer Verlag

ER -