When Brian Romero began his doctoral research in the Advanced Fluids Lab, led by Professor Svetlana Poroseva at The University of New Mexico’s Mechanical Engineering Department, he quickly realized that the tool he needed to complete his work had not yet been developed. He needed a computational fluid dynamics solver capable of simulating supersonic flows with shock waves but “couldn’t find one that I liked, that was fast enough, or that was open source and available to me” Romero said.
With oversight from Poroseva, and with partial support from Defense Threat Reduction Agency (DTRA) and Los Alamos National Laboratory (LANL) contracts, Romero turned to UNM’s Center for Advanced Research Computing (CARC) and set out to construct the necessary tool.
“CARC was the only cluster I had access to when I first started coding,” Romero explained. “I was lucky enough to have the CARC Xena GPU [Graphics Processing Unit] cluster to work with. I couldn’t have developed and tested without that cluster. FIESTA© [Fast Interface Evolution, Shocks and Transition in the Atmosphere] would not have been possible without CARC. CARC is where FIESTA© was born.”
Romero designed FIESTA©, he designed it with the future in mind. Thinking beyond the use for his own work, he wanted to develop code that was user and developer friendly, and could accommodate researchers’ ever evolving use of GPUs, High Performance Computing (HPC) systems, and exascale supercomputers.
“FIESTA© is a product of my Ph.D. It is a tool that I designed and built out of necessity to get my work done,” Romero observed. “I wanted to make it easy for users and future developers to use. I designed it in a modular way, so other types of physics can be easily implemented in it. It is designed to do more than it can do now.”
Although initially designed for atmospheric simulations, as well as some smaller laboratory scale experiments, this future thinking built into the code allows FIESTA© to do a wide range of simulations.
“I designed FIESTA© for exascale systems with neat technology built in to enable that. I used a framework called Kokkos, from Sandia Labs. Kokkos is a performance portable ecosystem, which enables me to run FIESTA© on any GPU machines. Another popular GPU technology, CUDA®, can also get a code running on GPUs, but is limited to NVIDIA® architectures. Kokkos lets me run FIESTA© on GPUs from a variety vendors supported by the Kokkos team, while still being able to run FIESTA© on traditional CPUs [Central Processing Unit]. This makes the code extremely flexible. [In the past] we would need to have two codes: one to run on GPUs and one to run on CPUs. But now with Kokkos, we have one code that can run anywhere. And Kokkos has long-term support from the Department of Energy, so FIESTA© will continue to run on new GPUs that come out in the future. This makes FIESTA© ready for the next generation of exascale hardware.”
FIESTA© also has advanced features for parallel file systems using HDF5® and a unique input file system using an embedded scripting language called Lua.
“I haven’t seen Lua in other CFD [computational fluid dynamics] codes, this is [a] really useful feature I’m proud of,” Romero said.
Romero is working closely with UNM’s Shock Tube Lab, an experimental group in UNM’s Mechanical Engineering department led by Professor Peter Vorobieff, to verify data and check the accuracy of FIESTA©. Recently, FIESTA© simulations were compared to shock-tube experiments to verify the mechanism behind the shock-driven Kelvin-Helmholtz instability (SDKHI), a phenomenon useful for mixing fuel and air in certain types of scramjet engines. Romero explained, “Past experiments were not able to reveal necessary details to confidently explain this mechanism. Simulations with FIESTA© were able to confirm this mechanism.”
This work, done in collaboration with Poroseva, Vorobieff, and LANL scientist Jon Reisner, has been published in the Physics of Fluids Journal and has been presented at five conferences in the last two years.
Romero explained, “…FIESTA© was run on GPU clusters at Los Alamos National Laboratory and Lawrence Livermore National Laboratory. Running simulations on GPUs produced simulation results over 10 times faster than if we ran on a CPU cluster… Since we could run so quickly on GPUs we were able to do multiple simulations to explore the effects of different parameters on the shock-driven Kelvin-Helmholtz instability.”
Romero and Poroseva named the code FIESTA© in recognition of the connection to New Mexico; the chile pepper logo is a nod to Romero’s love of New Mexican cuisine. FIESTA© is a GPU capable high-order compressible flow solver designed by Romero and Poroseva and is owned and copyrighted by UNM. Although Romero remains the primary user, he notes others have started to pick it up and use it, including others in his department as well as in UNM’s PSAAP center.
According to Professor Patrick Bridges, CARC Director and Principal Investigator at UNM’s PSAAP center, “FIESTA© is a great example of modern scientific applications design in both its state-of-the-art numerical algorithms and its use of Kokkos. As a result, it’s been a great example for us to use to understand and optimize the communication requirements of next-generation high-performance applications.”
Although Romero plans to graduate this summer, his vision is for FIESTA© to remain a UNM asset for years to come.
“We are looking for students to continue to develop FIESTA©. It has the potential to be [a] problem-solving code for researchers from anywhere, not just UNM,” Romero concluded. FIESTA© will be made available publicly as open-source software in Spring 2022. Meanwhile, Romero continues to utilize CARC resources as he finishes his doctoral requirements.
Image: FIESTA results from Shock Tube Lab shock-driven Kelvin-Helmholtz instability (SDKHI) experiment: Vertical and horizontal cross section of a dense gas column at different times during interaction with a shock-wave. Images courtesy of Brian Romero
