Parallelization of Three-Dimensional Discrete Particle Model for Dense Gas-Solid Flows On Unstructured Mesh

        A parallel code for three-dimensional discrete particle/element model (DPM or DEM) coupled with the computational fluid dynamics (CFD) model has been developed and used to simulate dense gas-solid flows in fluidized bed reactors. The gas-phase hydrodynamics is described in the Eulerian framework while the dispersed phase is in the Lagrangian one. The finite volume method is applied to discretize the continuity and momentum equations of the gas phase on 3D unstructured mesh. The fractional time-marching scheme is designed for the two-phase momentum coupling. The particle motion is tracked by directly solving the dynamical equations of each individual particle. Particle-particle and particle-wall interactions are taken into account by a soft-sphere model. The numerical solution procedure in a time step begins by calculating the particle contact forces and integrating the particle momentum equations over fractional DPM time steps, followed by solving the gas-phase hydrodynamics with SIMPLE algorithm. During integration of the particle momentum equations, the hydrodynamic forces on particles are calculated and the corresponding source terms of the gas momentum are integrated.

        The coupled numerical solution of the DPM-CFD problem, described above, poses new challenges in parallelization. The total load on the DPM part of the code is directly related to the number of particles, which in our design is targeted to be in the order of millions of particles. The computational load in the CFD part can be distributed evenly on many processors using domain decomposition within the framework of MPI. However, if we use the same structure for DPM, the heterogeneous spatial distribution of particles will result in poor load balancing among the many processors.

        The hybrid parallel computing technique is used in handling the coupled CFD-DPM model to partly overcome the difficulties identified above and achieve efficient load balancing. Firstly at the coarse-grain level, the solution domain is decomposed into partitions using bisection algorithm to minimize the number of faces at the partition boundaries while keeping almost equal number of cells in each partition. The solution of the gas-phase governing equations is performed on these partitions. Particles and the solution of their dynamics are associated with partitions according to their hosting cells. This makes no data exchange between processors for calculating the hydrodynamic forces on particles. By introducing proper data mapping between partitions, the cell void fraction is calculated accurately even if a particle is shared by several decomposed partitions. Neighboring partitions are grouped by a gross evaluation before simulation, with each group having similar number of particles. The computation task of a group of partitions is assigned to a compute node, which has multi-cores or multi-processors with a shared memory. Each core or processor in a node takes the computation of the gas governing equations in one partition. Processors communicate and exchange data through Message Passing Interface (MPI) at the coarse-grain parallelism.

        Secondly, the multithreading technique is used to parallelize the computation of the dynamics of the particles in each partition. The number of compute threads is determined according to the number of particles in partitions and the number of cores or processors in compute node. Thread pooling is employed to assign computing threads to cores or processors. In such a way there is almost no waiting of the processors in a compute node. Since the particle numbers in all compute nodes are almost the same, the above strategy yields an efficient load balancing among compute nodes. Test numerical experiments on HPC cluster show that the developed model is scalable up to 128 CPUs and can be used to simulate dense gas-solid flows in fluidized bed with more than 10 millions of particles.