Implicit kinetic schemes

Transkript

1 Implicit kinetic schemes David Coulette, E. Franck, P. Helluy, M. Mehrenberger, L. Navoret IRMA Strasbourg & Inria TONUS, France October /41

2 Outline Kinetic schemes High order implicit kinetic scheme Parallel transport solver implementation Parallel performance tests 1/41

3 Section 1 Kinetic schemes 2/41

4 Kinetic framework[chen et al., 1994] Distribution function: f (x,v,t), x R d, v R d, t [0,T ]. Microscopic collision vector K(v) R m. Macroscopic conserved data w(x,t) = f (x,v,t)k(v)dv. v Microscopic entropy s and associated Maxwellian M w (v): v M w K = w, v s(m w ) = max v fk=w { v } s(f ). Kinetic-BGK equation (a = a(x, t) is the acceleration): t f + v x f + a v f = 1 τ (M w f ). 3/41

5 Kinetic schemes When the relaxation time τ is small, the kinetic equation provides an approximation of the hyperbolic conservative system t w + F (w) + Π(w) = 0, with F i (w) = v i M w (v)k(v)dv. v Π(w) = a v M w (v)k(v) = a M w (v) v K(v). v v Main idea: numerical solvers for the linear scalar transport equation lead to natural solvers for the non-linear hyberbolic system [Deshpande, 1986]. Micro or macro approach. 4/41

6 Lattice Boltzmann method (LBM) discrete measure in v. Low number of velocities D1Q D2Q D3Q LBM-BGK : t f i + v i f i = 1 τ (M i(f ) f i ) splitted scheme : advection + relaxation space grid (lattice) generated by velocity set for a given time step exact advection. (from lattice gas automata) 5/41

7 Lattice Boltzmann method (LBM) Basic features collision vector K built from low order multivariate polynomials -> macro quantities are moments. splitting scheme : linear advection + local nonlinear relaxation. typically τ t (over-relaxation) : fast oscillations of f around the quasi-equilibrium manifold. t and x are linked for exact transport (integer cfl). Extensions relaxation can be more complex than BGK ([Dubois 2008,2009]) MRT : complement K to get a full basis in moment space and relax on non-conserved moments. not restricted to Euler/Navier-Stokes. Example : MHD kinetic model for both fluid flow and magnetic field components [Dellar, 2002, Dellar, 2011, Dellar, 2013]) 6/41

8 Project guidelines MHD for Magnetic Fusion Devices ( Tokamaks, Stellarators ) keep the good numerical features of the LBM approach (linear transport, non linearity stays local ) unstructured curved mesh "large" time steps : not limited by cfl type condition. Main features of our approach non-exact transport on unstructured mesh : Discontinous-Galerkin (DG) high order in space, h,p refinement, local coupling. no cfl implicit solvers make the best use of the parallel properties of the kinetic approach. 7/41

9 Section 2 High order implicit kinetic scheme 8/41

10 Toy problem: D1Q3 model v i = iλ,i { 1,0,1}. K(v) = (1,v) T. Acceleration a = 0. w = (ρ,q). Density ρ, momentum q = ρu, and fluid velocity u. When τ 0, w is solution of the isothermal Euler equations M is a Maxwellian state if: we obtain t ρ + x (q) = 0, t (q) + x (q 2 /ρ + c 2 ρ) = 0. ρ = i M i q = i M iv i ρu 2 + c 2 = i M iv 2 i M 0 = ρ(λ 2 u 2 c 2 )/λ 2 M i = ρ ( 2 λu + u 2 + c 2) /λ 2 9/41

11 Stability The D1Q3 model can also be written t ρ + x q = 0, t q + x z = 0, t z + λ 2 x q = 1 τ (q2 ρ + c2 ρ z). Chapman-Enskog method shows that for small τ we have t ρ + x q = 0, ( ) q 2 t q + x ρ + c2 ρ = τ x ((λ 2 c 2 3u 2) ( x q + 2u u 2 c 2) ) x ρ. The viscosity terms are not entropy dissipative small Mach flows. Different from the Jin-Xin relaxation [Jin and Xin, 1995, Dubois, 2013]. 10/41

12 Implicit upwind approximation 1D finite volume approximation for the transport equation space step x and time step t. constant approximation of f in each cell C i =]i x,(i + 1) x[, i Z. If x C i and t [n t,(n + 1) t[ then f (x,v,t) f n i (v). Upwind scheme v > 0, v < 0, f n i f n i f n 1 i t f n 1 i t + v f i n fi 1 n = 0 x + v f i+1 n f i n = 0 x 11/41

13 Comments Unconditionally stable. CFL number= t/v x can be arbitrary. Implicit scheme but upper or lower triangular linear system. The viscosity of the implicit transport solver counterbalances the (wrong) Chapman-Enskog viscosity when t τ. Parallelism: uncoupled transport equations, local relaxation. Higher order in x: Discontinuous Galerkin (DG) method (polynomial approximation of f in cells C i ) Issue: higher order in time? 12/41

14 Second order extension Spatial approximation leads to a differential equation v (t) = g(v(t)), v(0) = v 0. First order numerical method ϕ( t)v 0 = v( t) + O( t 2 ). From the first order method we can build a time-symmetric method ψ( t) = ϕ( t/2) ϕ 1 ( t/2) which is second order accurate (Crank-Nicolson) 13/41

15 Multistep symmetric splitting (1) Bibliography: [McLachlan and Quispel, 2002] Example : D2Q9 model Euler stationary state in a constant gravity field g = ge y. Analytical solution ρ = ρ 0 e gy/t Splitting schemes made of symmetric building blocks : Transport (T) - Macroscopic source (S) (gravity) BGK Relaxation (R). 1 st order T ( t) S( t) R( t) 2 nd order T ( t/2) R( t/2) S( t) R( t/2) T ( t/2) 14/41

16 Multistep symmetric splitting (2) : convergence log 2 (ɛ/ɛref) st order 1st order t1 2nd order full 2nd order collapsed log 2 ( t/ t ref ) The second order collapsed scheme is obtained by fusing the last transport substep of an iteration with the first transport substep of the next one. 15/41

17 Multistep symmetric splitting (3): Dynamical test D2Q9 model on a square : 8x8 elements, 10x10 subcells, 3 rd order Initial condition : narrow gaussian density bump ρ = exp( 40 (x 2 + y 2 ). Convergence evaluated from highly time-resolved solution. log 2 (ɛ/ɛref) st order 2nd order full 2nd order collapsed log 2 ( t/ t ref) 16/41

18 Section 3 Parallel transport solver implementation 17/41

19 3D mesh We consider a coarse mesh made of hexahedral curved macrocells Each macrocell is itself split into smaller subcells of size h. In each subcell L we consider polynomial basis functions ψ L i of degree p. Possible non-conformity in h and p. We need a conservative scheme. We want to avoid CFL condition. 18/41

20 DG approximation of the transport equation Implicit DG approximation scheme: L, i L fl n f L n 1 ψi L v ψi L f n ( L + v n + fl n + v n f n ) R ψ L t i = 0. L L R denotes the neighbor cells along L. v n + = max(v n,0), v n = min(v n,0). n LR is the unit normal vector on L oriented from L to R. L R L n LR R 19/41

21 Upwind numbering L is upwind with respect to R if v n LR > 0 on L R. In a cell L, the solution depends only on the values of f in the upwind macrocells. No assembly and factorization of the global system /41

22 Dependency graph For a given velocity v we can build a dependency graph. Vertices are associated to macrocells and edges to macrocells interfaces or boundaries. We consider two fictitious additional vertices: the upwind vertex and the downwind vertex. 9 NA NA 10 21/41

23 Algorithm Bibliography: [Duff and Reid, 1978, Johnson et al., 1984, Wang and Xu, 1999, Natvig and Lie, 2008] Transport algorithm for one velocity Topological ordering of the dependency graph. For each cell (in topological order): Compute volumic terms contributions. Compute fluxes from upwind interfaces/boundaries. Assembly and LU factorization of the local linear system (KLU) Solve local system Extract the results to the downwind cells interfaces. Parallelization? across velocities (transport step) across macrocells. 22/41

24 StarPU parallelization StarPU is a library developed at Inria Bordeaux [Augonnet et al., 2012]: Task-based parallelism. Task description: codelets, inputs (R), outputs (W or RW). The user submits tasks in a correct sequential order. StarPU schedules the tasks in parallel if possible. 23/41

25 Implementation : SCHNAPS/StarPU + LBM = KIRSCH Initial sequential code SCHNAPS : DG - general hyperbolic systems. SCHNAPS/StarPU : migration of schnaps kernel operations into StartPu Tasks. Several implementations of the same task are possible (CPU, optimized CPU, GPU I, GPU II, MIC, etc.) specialized version for DG-LBM : KIRSCH SCHNAPS : Solveur Conservatif Hyperbolique Non-lineaire Applique aux PlaSmas KIRSCH : KInetic Representation for SCHnaps 24/41

26 Section 4 Parallel performance tests 25/41

27 What StarPU does for us Task graph for 2DQ9 model a single 2D macrocell a single time step of the first order scheme ( T + R) field_set_to_zero_global Pretty simple... 26/41

28 field_set_to_zero_global field_set_to_zero_global field_set_to_zero_global field_set_to_zero_global What StarPU does for us 4 macrocells in a 2D square a single time step of the first order scheme ( T + R) Slightly more complicated... 27/41

29 D2Q9 multithread performance Full D2Q9 scheme on square grids. Constant dof number per macrocell. Number N of macrocells N from 1 to 64 = 8x8. for 1 macrocell : saturation at n core = n v. This is expected. efficiency grows with N due to topological parallelism. speedup x1x3375 D2Q9 2x2x3375 D2Q9 3x3x3375 D2Q9 4x4x3375 D2Q9 8x8x3375 D2Q9 ideal cores 28/41

30 Gantt diagram D2Q9 model - 12 threads Low overhead - low sleep/idle time (in red) 29/41

31 Gantt diagram 8x8 square - D2Q9 model - 24 threads No enough work to do! 30/41

32 D3Q* multithread performance D3Q15,D3Q19,D3Q27 models on a cube with 4x4x4 elements and 8000 dof per elements with eager scheduler. speedup x4x4x8000 nv 15 4x4x4x8000 nv 19 4x4x4x8000 nv 27 ideal cores Saturation at 80% efficiency even when n v > n cores. 31/41

33 MPI data distribution : StarPU/MPI static distribution of macrocell elements across MPI processes. one scheduler instance per MPI process. each process deals only with the task graph of the elements it owns reduction of scheduling complexity necessary MPI communications are handled automagically (!) by StarPU. subdomain parallelism only : partitioning is made in order to keep nodes busy (one should avoid internal subdomains far from external boundaries) 32/41

34 Preliminary MPI scalings : D2Q9 Irma Atlas : 4 nodes - 24 CPU per node D2Q9 model square 8x8 elts x3200 dof Wall time (s) for 100 iterations Nmpi/Nthreads /41

35 Gantt diagram D2Q9 model - 2x6 threads Low overhead/sleep time 34/41

36 Gantt diagram 8x8 square - D2Q9 model - 2x12 threads Again : not enough work to do! 35/41

37 MPI scalings : D3Q15 D3Q15-4x4x4x8000 cubic macromesh - 50 iterations -Wall time (s) Nthreads/Nmpi D3Q15 8x8x8x4096 cubic macromesh 50 iterations - Wall time (s) Nthreads/Nmpi /41

38 37/41 What StarPU does for us (again) Toroidal mesh : 720 macroelements interfaces Toroidal mesh - transport graph for (1, 0, 0) velocity

39 MPI Scaling : D3Q15 in a torus Toroidal mesh : 720 macroelements x 3335 dof 2064 interfaces boundary faces Wall time in sec for 100 interations. Nthreads/Nmpi /41

40 Conclusion and prospects Implicit kinetic schemes for fluids: no CFL condition. low memory footprint : no storage of large linear systems (preconditioning, factorization...) Second order time integrator compatible with τ 0. Task-based Parallelization of an implicit transport solver. Work in progress/prospects higher order time integration. beyond BGK : multiple-relaxation times (MRT) relaxation ( t f i = 1 τ j Ω ji(f eq j f j ) ) 3D Magneto-Hydrodynamics (MHD), non isothermal, visco-resistive... kernel optimization / porting to GPU (CUDA) other kinetic models (Vlasov) 39/41

41 Bibliography I [Augonnet et al., 2012] Augonnet, C., Aumage, O., Furmento, N., Namyst, R., and Thibault, S. (2012). StarPU-MPI: Task Programming over Clusters of Machines Enhanced with Accelerators. In Jesper Larsson Träff, S. B. and Dongarra, J., editors, EuroMPI 2012, volume 7490 of LNCS. Springer. Poster Session. [Chen et al., 1994] Chen, G. Q., Levermore, C. D., and Liu, T. P. (1994). Hyperbolic conservation laws with stiff relaxation terms and entropy. Communications on Pure and Applied Mathematics, 47(6): [Dellar, 2002] Dellar, P. J. (2002). Lattice kinetic schemes for magnetohydrodynamics. Journal of Computational Physics, 179(1): [Dellar, 2011] Dellar, P. J. (2011). Lattice boltzmann formulation for braginskii magnetohydrodynamics. Computers & Fluids, 46(1): [Dellar, 2013] Dellar, P. J. (2013). Moment-based boundary conditions for lattice boltzmann magnetohydrodynamics. In Numerical Mathematics and Advanced Applications 2011, pages Springer. [Deshpande, 1986] Deshpande, S. (1986). Kinetic theory based new upwind methods for inviscid compressible flows. In 24th AIAA Aerospace Sciences Meeting, volume 1. [Dubois, 2013] Dubois, F. (2013). Stable lattice boltzmann schemes with a dual entropy approach for monodimensional nonlinear waves. Computers & Mathematics with Applications, 65(2): [Duff and Reid, 1978] Duff, I. S. and Reid, J. K. (1978). An implementation of tarjan s algorithm for the block triangularization of a matrix. ACM Transactions on Mathematical Software (TOMS), 4(2): /41

42 Bibliography II [Jin and Xin, 1995] Jin, S. and Xin, Z. (1995). The relaxation schemes for systems of conservation laws in arbitrary space dimensions. Communications on pure and applied mathematics, 48(3): [Johnson et al., 1984] Johnson, C., Nävert, U., and Pitkäranta, J. (1984). Finite element methods for linear hyperbolic problems. Computer methods in applied mechanics and engineering, 45(1): [McLachlan and Quispel, 2002] McLachlan, R. I. and Quispel, G. R. W. (2002). Splitting methods. Acta Numerica, 11: [Natvig and Lie, 2008] Natvig, J. R. and Lie, K.-A. (2008). Fast computation of multiphase flow in porous media by implicit discontinuous galerkin schemes with optimal ordering of elements. Journal of Computational Physics, 227(24): [Wang and Xu, 1999] Wang, F. and Xu, J. (1999). A crosswind block iterative method for convection-dominated problems. SIAM Journal on Scientific Computing, 21(2): /41