Large-scale flow simulations using lattice Boltzmann method with AMR following free-surface on multiple GPUs

Free-surface flow simulations require high-resolution grids to capture phenomena at the interface as well as a long computational time. In this paper, we propose a numerical method for realizing large-scale free-surface flow simulations using the lattice Boltzmann method and multiple GPUs. By introducing the adaptive mesh refinement (AMR) method, which adapts high-resolution grids to free surfaces, to the lattice Boltzmann method, the number of lattice points can be greatly reduced. In the calculation of the AMR method, the spatial distribution of a computational load changes with time; therefore, the number of lattice points assigned to each GPU is kept equal by dynamic domain partitioning using a space-filling curve. We measured the weak scalability of our AMR code on the TSUBAME3.0 supercomputer at the Tokyo Institute of Technology. By hiding GPU–GPU communication overheads by the overlapping method, the performance increased 1.29 times that of the naïve implementation, and we achieved the fairly high performance of 14,570 MLUPS using 256 GPUs. We demonstrate large-scale simulations for the dam breaking problem and show a reduction in computational cost with the AMR method.