"Enabling High Fidelity Neutron Transport Simulations on Petascle Architectures"
D. Kaushik, M. Smith, A. Wollaber, B. Smith, A. Siegel, and W. S. Yang
Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, vol. 67, Portland, Oregon, ACM, . Also Preprint ANL/MCS-P1608-0409
Preprint Version: [pdf]
The UNIC code is being developed as part of the DOE’s Nuclear Energy Advanced Modeling and Simulation (NEAMS) program. UNIC is an unstructured, deterministic neutron transport code that allows a highly detailed description of a nuclear reactor core in our numerical simulations. The goal of our simulation efforts is to reduce the uncertainties and biases in reactor design calculations by progressively replacing existing multi-level averaging (homogenization) techniques with more direct solution methods based on first principles. Since the neutron transport equation is seven dimensional (three in space, two in angle, one in energy, and one in time), these simulations are among the most memory and computationally
intensive in all of computational science. To model the complex geometry of a reactor core, billions of spatial elements, hundreds of angles, and thousands of energy groups are necessary, which leads to problem sizes with petascale degrees of freedom. Therefore, these calculations exhaust memory resources on current and even next-generation architectures. In this paper, we present UNIC simulation results for two important representative problems in reactor design/analysis - PHENIX and ZPR. In each case, UNIC shows excellent weak scalability on up to 163,840 cores of BlueGene/P (Argonne) and 131,072 cores of XT5 (ORNL). While our current per processor performance is not ideal, we demonstrate a clear ability to effectively utilize the leadership computing platforms. Over the coming months, we aim to improve the per-processor performance while maintaining the high parallel efficiency by employing better algorithms (such as spatial p-refinement, optimized matrix-tensor operations, and weighted partitioning for load balancing). Combining
these additional algorithmic improvements with larger parallel machines in the near future should allow us to realize our long term goal of explicit geometry coupled multiphysics reactor simulations. In the long run, these high fidelity simulations will be able to replace expensive mockup experiments and reduce the uncertainty in crucial reactor design and operational parameters.