Linearized Learning Methods with Multiscale Deep Neural Networks for Stationary Navier-Stokes Equations with Oscillatory Solutions
AA Linearized Learning with Multiscale DeepNeural Network for Stationary Navier-StokesEquations with Oscillatory Solutions
Lizuo Liu ∗ Bo Wang † Wei Cai ‡ February 8, 2021
Summary
In this paper, we combine a linearized iterative method with multi-scale deep neural network to compute oscillatory flows for stationaryNavior-Stokes equation in complex domains. The multiscale neuralnetwork converts the high frequency components in target to low fre-quency components before training, thus accelerating the convergenceof training of neural network for all frequencies. To solve the station-ary nonlinear Navier-Stokes equation, we introduce the idea of lin-earization of Navier- Stokes equation and iterative methods to treatthe nonlinear convection term. Three forms of linearizations will beconsidered. First we will conduct a benchmark problem of the lin-earized schemes in comparison with schemes based directly on thenonlinear PDE. Then, a Navier Stokes flow with high frequency com-ponents in a 2-D domain with a hole are learned by the linearizedmultiscale deep multiscale neural network. The results show that mul-tiscale deep neural network combining with the linearized schemes canbe trained fast and accurately. ∗ Department of Mathematics, Southern Methodist University, Dallas, TX 75275. † LCSM(MOE), School of Mathematics and Statistics, Hunan Normal University,Changsha, Hunan, 410081, PR China. ‡ Corresponding author, Department of Mathematics, Southern Methodist University,Dallas, TX 75275( [email protected] ). a r X i v : . [ m a t h . NA ] F e b Introduction
Deep neural network (DNN) machine learning has been studied as alternativenumerical methods for solving partial differential equations arising from fluiddynamics [1] and wave propagations [2] . DNN has the potential of a flexiblemeshless method to solve governing equations from fluid and solid mechanicsapplications as an alternative method to traditional finite element method.Moreover, it has shown much power in handling high dimensional parabolicPDEs [3, 4].Recent work on DNN have shown a particular frequency de-pendence performance in learning solution of PDEs and fitting of functions.Namely, the lower frequency components of the solution are learned first andquickly than the higher frequency components [5]. Several attempts havebeen made to remove such a frequency bias for the DNN performances. Theidea is to convert the higher frequency content of the solution to a lowerfrequency range so the conventional DNN can learn the solution. One wayto achieve this goal is to use a phase shift while the other way is to introducea multiscale structure of the DNN where neural network with different scalewill target different range of the frequency in the solutions. The PhaseDNNhas been shown to be very effect for high frequency wave propagations whilethe MscaleDNN has been used to learn highly oscillatory Stokes flow solu-tions in complex domains.In this paper, we will study how to extend the MscaleDNN for the non-linear Navier Stokes equations. A direct application of MscaleDNN to thenonlinear PDEs does not produce the dramatic improvement for the linearStoke equations over the conventional DNN. The learning of the solution oflinear PDEs via least square residual of the PDEs is in some sense equivalentto the fitting problems in the frequency domain from the Parsevel equalityof the Fourier transform. So it is understandable that a multiscale DNN’sperformance for a fitting problem also holds for the solution of linear PDEs.Taking this cue for the MscaleDNN’s improvement for the linear Stokes prob-lem, we will develope a linearized learning procedure for the Navier-Stokesequations by integrating a linearization of the Navier-Stokes equation in theloss function of the MscaleDNN and dynamically updating the linearizationas the learning is being carried out. Numerical results have shown the con-vergence of this approach and it produces highly accurate approximation tooscillatory solutions of the Navier-Stokes equations.The rest of the paper will be organized as follows. Section 2 will intro-duce the multiscale DNN structure and then three linearized learning for the2avier-Stokes equation will be proposed in Section 3. Numerical tests of thelinearized learning schemes will be conducted for 2-D oscillatory flows in adomain containing a cylinder in Section 4. Finally, conclusion and futurework will be discussed in Section 5.
In order to improve the capability of the DNN to represent functions withmultiple scales, we will apply the MscaleDNN [6], which consists of a series ofparallel normal sub-neural networks. Each of the sub-networks will receive ascaled version of the input and their outputs will then be combined to makethe final out-put of the MscaleDNN (refer to Fig. 1). The individual sub-network in the MscaleDNN with a scaled input is designed to approximatea segment of frequency content of the targeted function and the effect of thescaling is to convert a specific high frequency content to a lower frequencyrange so the learning can be accomplished much quickly. Due to the radialscaling used in the MscaleDNN, it is specially fitting for approximation ofhigh dimensional parabolic equations.Figure 1: Illustration of a MscaleDNN.Fig. 1 shows the schematics of a MscaleDNN consisting of n networks.Each scaled input passing through a sub-network of L hidden layers can be3xpressed in the following formula f θ ( x ) = W [ L − σ ◦ ( · · · ( W [1] σ ◦ ( W [0] ( x ) + b [0] ) + b [1] ) · · · ) + b [ L − , (1)where W [1] to W [ L − and b [1] to b [ L − are the weight matrices and bias un-knowns, respectively, to be optimized via the training, σ ( x ) is the activationfunction. In this work, the following plane wave activation function will beused for its localized frequency property [1], σ ( x ) = sin( x ) . (2)For the input scales, we could select the scale for the i -th sub-network tobe i (as shown in Fig. 1) or 2 i − . Mathematically, a MscaleDNN solution f ( x ) is represented by the following sum of sub-networks f θ ni with networkparameters denoted by θ n i (i.e., weight matrices and bias) f ( x ) ∼ M (cid:88) i =1 f θ ni ( α i x ) , (3)where α i is the chosen scale for the i -th sub-network in Fig. 1. For moredetails on the design and discussion of the MscaleDNN, please refer to [6]. The following two dimensional stationary Navier-Stokes equation will besolved by the deep neural networks, ( u · ∇ ) u − ν ∆ u + ∇ p = f in Ω , ∇ · u = 0 in Ω , u = g on Γ D , (4)where Ω is an open bounded domain in R . The MscaleDnn solution willbe found as in the traditional least square finite element method where thesolution is obtained by minimizing a loss function in terms of the residual ofNavier Stokes equation. 4o solve the NS stokes in a least square approach, as usual, the PDEs (4)is written in a system of first order equations in [1]. A velocity-gradient ofvelocity-pressure formulation will be used for the Navier Stokes equations. − ν ∇ · U + U · u + ∇ p = f in Ω (5a) U − ( ∇ u ) (cid:62) = 0 in Ω (5b) ∇ · u = 0 in Ω (5c)To obtain the governing equation for the pressure, we take divergence onboth sides of the Navier Stokes equation to arrive at the following Poissonequation, ∆ p + 2( − u x v y + u y v x ) = ∇ · f . (6)Then, a loss function can be defined as L V gV P ( θ u , θ p , θ U ) := (cid:107) ν ∇ · U − U · u − ∇ p + f (cid:107) + α (cid:107) ∆ p + 2( − u x v y + u y v x ) − ∇ · f (cid:107) Ω + (cid:107)∇ u − U (cid:107) + (cid:107)∇ · u (cid:107) + β (cid:107) u − g (cid:107) ∂ Ω . (7)However, it will be shown that the training of the network based on theformulation 5 including the Nonlinear first order system and the Poissonequation 6 converges slowly if at all, as shown by the numerical results in insection 4.1. To speed up the convergence of the training, we propose to introduce aniterative solution procedure for the nonlinear NS equation so each step of thetraining is done for a linearized version of the Navier-Stokes equation andthe linearization [7] will be updated as the training progresses.Assume the current learned solutions are ( u n , v n ), the next functions tobe learnt are denoted as ( u n +1 , v n +1 ), then we can linearize the Navier-Stokesequation in the following three schemes.5 Scheme 1:
Gradients of velocity are fixed. − ν ∆ u n +1 + u nx u n +1 + u ny v n +1 + p x = f (8a) − ν ∆ v n +1 + v nx u n +1 + v ny v n +1 + p y = f (8b)∆ p + 2( − u n +1 x v n +1 y + u n +1 y v n +1 x ) = ∇ · f (8c)In this scheme, we use the currently learned solutions with a few epochstraining to replace the gradients in the nonlinear term. • Scheme 2:
Velocity is fixed. − ν ∆ u n +1 + u n +1 x u n + u n +1 y v n + p x = f (9a) − ν ∆ v n +1 + v n +1 x u n + v n +1 y v n + p y = f (9b)∆ p + 2( − u n +1 x v n +1 y + u n +1 y v n +1 x ) = ∇ · f (9c)In this scheme, we fix the velocity term in nonlinear term to be thecurrent learned velocity. • Scheme 3:
Mixed form. − ν ∆ u n +1 + 12 ( u n +1 x u n + u n +1 y v n + u nx u n +1 + u ny v n +1 ) + p x = f (10a) − ν ∆ v n +1 + 12 ( v n +1 x u n + v n +1 y v n + v nx u n +1 + v ny v n +1 ) + p y = f (10b)∆ p + 2( − u n +1 x v n +1 y + u n +1 y v n +1 x ) = ∇ · f (10c)A average of the previous two schemes is taken to replace the nonlinearconvection term in the NS equation for this scheme.Based on equations (8),(9),(10), we can design several loss functions asfollows. 6 oss function for Scheme 1: Loss ∇ u = Res u + Res v + Bdry u + Res p , Res u = (cid:90) Ω (cid:0) − ν ∆ u n +1 + u nx u n +1 + u ny v n +1 + p x − f (cid:1) dxdy, Res v = (cid:90) Ω (cid:0) − ν ∆ v n +1 + v nx u n +1 + v ny v n +1 + p y − f (cid:1) dxdy, Res p = (cid:90) Ω (cid:0) ∆ p + 2( − u n +1 x v n +1 y + u n +1 y v n +1 x ) − ∇ · f (cid:1) dxdy, Bdry u = (cid:90) ∂ Ω (cid:0) u n +1 − u (cid:1) dS. (11) Loss function for Scheme 2:
Loss u = Res u + Res v + Bdry u + Res p , Res u = (cid:90) Ω (cid:0) − ν ∆ u n +1 + u n +1 x u n + u n +1 y v n + p x − f (cid:1) dxdy, Res v = (cid:90) Ω (cid:0) − ν ∆ v n +1 + v n +1 x u n + v n +1 y v n + p y − f (cid:1) dxdy, Res p = (cid:90) Ω (cid:0) ∆ p + 2( − u n +1 x v n +1 y + u n +1 y v n +1 x ) − ∇ · f (cid:1) dxdy, Bdry u = (cid:90) ∂ Ω (cid:0) u n +1 − u (cid:1) dS. (12) Loss function for Scheme 3:
Loss mix = Res u + Res v + Bdry u + Res p , Res u = (cid:90) Ω (cid:18) − ν ∆ u n +1 + 12 ( u n +1 x u n + u n +1 y v n + u nx u n +1 + u ny v n +1 ) + p x − f (cid:19) dxdy, Res v = (cid:90) Ω (cid:18) − ν ∆ v n +1 + 12 ( v n +1 x u n + v n +1 y v n + v nx u n +1 + v ny v n +1 ) + p y − f (cid:19) dxdy, Res p = (cid:90) Ω (cid:0) ∆ p + 2( − u n +1 x v n +1 y + u n +1 y v n +1 x ) − ∇ · f (cid:1) dxdy, Bdry u = (cid:90) ∂ Ω (cid:0) u n +1 − u (cid:1) dS. (13)7 .3 Linearized Learning Algorithms with MscaleDNN Our linearized MscaleDNN learning algorithms for the NS equation are de-signed with the following steps. • Initialize u , u temp and p • i = 1 • threshold = 1e12 • While i < Epochs – fix u temp ,using loss one of the loss function in (11) or (12) or (13)to train u i and p – Every 5 epoch check the loss. If the loss is less than the threshold ∗ u temp = u i ∗ threshold = Loss – i = i+1 We first consider the problem in a rectangle domain Ω = [0 , × [0 ,
1] withone cylinder hole centered at 0 . , . u = 1 − e λx cos (2 nπx + 2 mπx ) (cid:1) ,u = λ mπ e λx sin (2 nπx + 2 mπx ) + nm e λx cos (2 nπx + 2 mπx ) ,p = e λx , λ = Re2 − (cid:113) Re + 4 π , Re = v . (14)We will compare the decay of the loss functions with a fully connectedNN based on the nonlinear NS equation (7) and the loss of the 3 linearizedschemes (11) or (12) or (13).As a benchmark problem, we consider the frequency n = 1 , m = 2. Thesource term f is obtained by substituting the exact solution into the NavierStokes equation 4. We set the viscosity ν = 0 .
05 and compare the perfor-mance of algorithms using the VgVP formulation with the loss function in (7)8nd three linearized learning schemes. We randomly sample 160000 pointsinside Ω and 16000 points inside ∂ Ω during each epoch. In the learningprocess, we set batch size to be 3200 inside domain and 320 on boundaryfor each step. The deep neural network solutions obtained by minimizingdifferent loss functions in (11) − (13) are compared in Figure 2. The re-sults show that all three linearized learning neural networks converge in 300epochs for all schemes while learning using the loss function (7) for the non-linear Navier-Stokes equations fails to (top line in Figure 2 ). All the schemesuse 4 hidden layer, 100 hidden neurons, fully connected neural networks for u , p . The comparisons of the x component of velocity and pressure along line y = 0 . Mix LossFix grad LossFix V LossVgVP Loss
Figure 2: Losses (bottom 3 lines) of three linearized learning schemes (11) − (13) and loss (top line) based on nonlinear Navier-Stokes equation (7). In our previous work, it has been shown that a multiscale DNN will beneeded to learn oscillatory solutions for the Stokes equations. So the lin-earized learning schemes with mscale deep neural network will be also usedfor Navier-Stokes equations with oscillatory solutions. Here we use the samedomain like the benchmark problem but the frequencies now are taken to be9 .00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 x v x exactMSDNN (a) Fix V in nonlinearterm x v x exactMSDNN (b) Mixed form for non-linear term x v x exactMSDNN (c) Fix gradient of V innonlinear term Figure 3: The x components of velocity after 300 epoch training for benchmark problem along line y = 0 . p exactMSDNN (a) Fix V in nonlinearterm p exactMSDNN (b) Mixed form for non-linear term p exactMSDNN (c) Fix gradient of V innonlinear term Figure 4: The pressures after 300 epoch training for bench mark problemalong line y = 0 . .0 0.5 1.0 1.5 2.0x0.00.10.20.30.40.5 p exactMSDNN (a) Pressure of the oscillatory case after1000 epoch training alone line y = 0 . x v x exactMSDNN (b) Velocity of the first component after1000 epoch training along line y = 0 . Figure 5: The results of the oscillatory case using multi-scale neural networks n = 15 , m = 20, much higher than the benchmark problems and, a trainingalgorithm based on the VGVP scheme using the loss function (7) for thenonlinear NS does not coverge within the same number of epoches.We consider Scheme 2 of the linearized learning algorithms where theprevious velocity are used to linearized the convection velocity. Figure 5and 8 show the results of network after 1000 epoch training. The multiscaledeep neural networks has 8 scales: { x, x, x, x, x, x, x } , whose sub-networks contain 4 hidden layers and 128 hidden neurons in each layer. Ascomparison, we use a 4-layer fully connected neural network with 1024 hiddenneurons combining scheme 2 for this oscillating flows. The results are shownin Fig 6. Fig 7 shows the errors of these 2 different neural network structuresalong line y = 0 .
7. The MscaleDNN improves the accuracy of the pressurefield compared with the fully connected neural network while both DNN per-forms similarly for the velocity fields. We expect that the MscaleDNN willperform substantially better for higher frequency flow fields.
In this paper we proposed three linearized learning schemes to solve thestationary highly oscillatory Navier-Stokes flows with multiscale deep neu-ral networks and showed the acceleration of convergence of the schemes aresubstantial, which demonstrate the capability of the Mscale deep neural net-11 .0 0.5 1.0 1.5 2.0x0.00.10.20.30.40.5 p exactFCN (a) Pressure of the oscillatory case after1000 epoch training alone line y = 0 . x v x exactFCN (b) Velocity of the first component after1000 epoch training along line y = 0 . Figure 6: The results of the oscillatory case using fully connected neuralnetworks FCN solution errorMSNN solution error (a) Error of two different models w.r.t.pressure of the oscillatory case after1000 epoch training alone line y = 0 . FCN solution errorMSNN solution error (b) Error of two different models w.r.t.velocity of the first component after1000 epoch training along line y = 0 . Figure 7: The errors of the oscillatory case for two different network archi-tectures 12 .00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.000.250.000.250.500.751.001.25 0.0000.2250.4500.6750.9001.1251.3501.5751.8002.025
Figure 8: The x component of velocityworks and the effectiveness of the linearized schemes to solve the nonlinearNavier Stokes equations. These schemes shed some lights on the practicalapplications of neural network machine learning algorithms to the nonlinearequations, which are time-consuming using traditional finite element meth-ods. The deep neural network based methods offer an alternative that doesn’trequire meshes without the need to solve large-scale linear systems, in con-strast to traditional numerical methods.On the other hand, there are much more works to be done for these lin-earized learning methods, among them the most important one is to under-stand the convergence property of these schemes. The applications of theseschemes to other nonlinear PDEs should be considered. Another challengingfuture work is to consider the time dependent Navier-Stokes equation, whichwill be explored in the near future.
References [1] Bo Wang, Wenzhong Zhang, and Wei Cai. Multi-scale deep neural net-work (mscalednn) methods for oscillatory stokes flows in complex do-mains.
Communications in Computational Physics , 28(5):2139–2157,2020.[2] Wei Cai, Xiaoguang Li, and Lizuo Liu. Phasednn - a parallel phase shiftdeep neural network for adaptive wideband learning, 2019.133] Maziar Raissi. Forward-backward stochastic neural networks: Deep learn-ing of high-dimensional partial differential equations, 2018.[4] Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensionalpartial differential equations using deep learning.
Proceedings of the Na-tional Academy of Sciences , 115(34):8505–8510, 2018.[5] Zhi-Qin John Xu. Frequency principle: Fourier analysis sheds lighton deep neural networks.
Communications in Computational Physics ,28(5):1746–1767, Jun 2020.[6] Ziqi Liu, Wei Cai, and Zhi-Qin John Xu. Multi-scale deep neural network(mscalednn) for solving poisson-boltzmann equation in complex domains.
Communications in Computational Physics , 28(5):1970–2001, 2020.[7] Yinnian He and Jian Li. Convergence of three iterative methods basedon the finite element discretization for the stationary navier–stokesequations.