Generative Pre-Trained Physics-Informed Neural Networks toward meta-learning of parametric PDEs
Board Location: #99
Discipline: Mathematics and Statistics
Subcategory: Mathematics and Statistics
Session: 3
Shawn Koohy - University of Massachusetts Dartmouth, North Dartmouth, MA
Co-Author(s): Yanlai Chen, University of Massachusetts Dartmouth, North Dartmouth, MA
Physics-Informed Neural Networks (PINNs) have proven themselves a powerful tool for numerically solving nonlinear partial differential equations (PDEs) that arise in many applications of mathematics, physics, and engineering. PINNs leverage the expressivity of deep neural networks, increased data availability, free open-source computing software, and GPU-accelerated computation. Unlike traditional solvers relying heavily on domain discretization, PINNs define a surrogate function across the entire spatial-temporal domain. The physical understanding of a system and its properties are typically not considered when using these traditional methods. PINNs however, build a physical intuition of a given problem by integrating known physical laws (the PDE form), along with boundary and initial conditions, directly into the learning process. This is done in the form of minimizing residuals that approximate the error estimate against the analytical solution. In the context of real-time simulations depicting the evolution of parametric systems, training PINNs is time-intensive, and the neural network is often excessively overparameterized. We propose a Generative Pre-Trained PINN (GPT-PINN) to mitigate both challenges in the setting of parametric PDEs. GPT-PINN, a network of networks, makes use of a single hidden layer with a significantly reduced number of neurons. Its hidden layer neurons use pre-trained PINNs at adaptively selected system configurations, introducing elements of meta and transfer learning into the PINN. GPT-PINN learns the system’s parametric dependencies and incrementally expands its hidden layer, one neuron at a time. If linear operators exist within the PDE, GPT-PINN becomes very non-intrusive with the ability to precompute and reuse much of the computation needed for training. When using adaptive PINNs as activation functions, the adaptivity is built within the neurons removing the need to sacrifice the simple architecture of GPT-PINN. Ultimately, GPT-PINN accurately and efficiently generates surrogate solutions across the entire parameter domain of the PDE. GPT-PINN was tested on three families of PDEs, the Klein-Gordon, Burgers’, and Allen-Cahn equations. We found the ratio between the time it takes to test the GPT-PINN against the baseline PINN to be 0.0022, 0.009, and 0.0006, respectively to the PDE families. GPT-PINN can at most be as accurate as the baseline PINN, therefore we computed errors between the two and found the worst-case errors are at most 6% and decrease exponentially with the hidden layer’s neuron count. Future work aims to further reduce GPT-PINN’s training time such that the intersection between the time to test a baseline PINN and GPT-PINN occurs at the number of neurons used in GPT-PINN. This will be done by adaptively choosing which data point to train GPT-PINN on instead of training on all possible points within the domain.
Funder Acknowledgement(s): This research was partially supported by NSF grant DMS-2208277, Air Force Office of Scientific Research grant FA9550-23-1-0037, and UMass Dartmouth Marine and UnderSea Technology (MUST) Research Program made possible via an Office of Naval Research grant N00014-20-1-2849.
Faculty Advisor: Yanlai Chen, yanlai.chen@umassd.edu
Role: On this project, I coded and built GPT-PINN using Python and various machine learning frameworks such as PyTorch and TensorFlow. I conducted all numerical experiments including training and testing GPT-PINN, provided the ability to combine PINNs trained with TensorFlow and PyTorch to work in unison, derived the gradient of the loss function which I built my own optimizer for, and collected statistical measurements from various loss and error quantities. I am also a co-author on a paper my mentor and I wrote which is now in the journal “Finite Elements in Analysis & Design”.