Abstract
Thanks to the supercomputer, more and more complicated simulations are successfully achieved. On the other hand, to analyze and understand the intrinsic properties of the big datasets from the simulations is an urgent research for scientists. However, the explosive size of the big datasets makes such kind of task difficult. Therefore, reduction of the size of the big datasets becomes an important topic, in which data compression and parallel computing are the two key techniques. In this paper, we presented a parallel data compression approach to reduce the size of time-varying big datasets. Firstly, we employ the proper orthogonal decomposition (POD) method for compression. The POD method can extract the underlying features of datasets to greatly reduce the size of big datasets. Meanwhile, the compressed datasets can be decompressed linearly. This feature can help scientists to interactively visualize big datasets for analysis. Then, we introduced a novel m-swap method to effectively parallelize the POD compression algorithm. The m-swap method can reach a high performance through fully using all parallel computing processors. In another word, no idle processors exist in the parallel compression process. Furthermore, the m-swap method can greatly reduce the cost of interprocessor communication. This is achieved by controlling the data transfer among 2m processors to obtain the best balance of computation cost of these processors. Finally, the effectiveness of our method will be demonstrated through compressing several time-varying big datasets on the K computer with ten thousands of processors.
Original language | English |
---|---|
Title of host publication | Proceedings - 2014 IEEE International Congress on Big Data, BigData Congress 2014 |
Publisher | Institute of Electrical and Electronics Engineers Inc. |
Pages | 438-445 |
Number of pages | 8 |
ISBN (Electronic) | 9781479950577 |
DOIs | |
Publication status | Published - Sep 22 2014 |
Externally published | Yes |
Event | 3rd IEEE International Congress on Big Data, BigData Congress 2014 - Anchorage, United States Duration: Jun 27 2014 → Jul 2 2014 |
Other
Other | 3rd IEEE International Congress on Big Data, BigData Congress 2014 |
---|---|
Country/Territory | United States |
City | Anchorage |
Period | 6/27/14 → 7/2/14 |
All Science Journal Classification (ASJC) codes
- Computer Science Applications