Thunderhead Engineering Forum

Public Forum => PyroSim => Topic started by: Dapke36 on March 10, 2020, 08:20:13 am

Title: OpenMP Threads and Threads MPI
Post by: Dapke36 on March 10, 2020, 08:20:13 am
I'm still trying to wrap my brain around when to use OpenMP and when to use OpenMP Threads MPI (~"MPI) in FDS/Pyrosim.  I read the FDS users guide section 3.3.1, but still struggling on application...and i'm hoping someone can help!

The current machine I am utilizing for smaller runs is 12 GB ram with a Xeon e5 with 4 physical cores reported in that 1 cpu. When I read the user's guide, I am getting confused when they are talking about mapping to cores.

Using that hardware above, lets say I have a model with 4 meshes.  Should I be doing an MPI thread per core, i.e.:

Open MP Threads: 1
Open MP Threads MPI: 4
Open MP Stack: 200M

Or should I be doing 1 mpi thread per physical CPU and 1 OpenMP thread per core of that cpu? i.e.:

Open MP Threads: 4
Open MP Threads MPI: 1
Open MP Stack: 200M

Is MPI supposed to be per core available (parallel or cluster, whatever's linked), or per CPU? 

If there's one MPI per core, I should try to get my # of meshes to match the # of cores available, right? (with exceptions for dense meshes and distribution, understandably) Or is the OpenMP Thread supposed to match the # of cores?

If I double the CPU's on the machine to 2 physical CPU's, each with 4 cores (8 cores total), would it then be 8 MPI processes?

Any insight that can be offered is appreciated!
Title: Re: OpenMP Threads and Threads MPI
Post by: chang on September 08, 2020, 04:56:14 pm
Did you ever figur this out? Very curious myself.
Title: Re: OpenMP Threads and Threads MPI
Post by: kazukikaze on December 30, 2020, 10:44:42 pm
Try and play with those and monitor through task manager or intel/ryzen cpu monitoring software.

I did some experiment myself and manage to run only 8 cores on my 16cores processor using the below parameters.

Open MP threads : 8
Open MP threads MPI : 1
Open MP Stack : 16M

I tried changing the MP threads MPI to 8 but it pushes all 16 cores to work. From what I understand the MPI means the number of Processor that is running. So usually it should be 1 since most computers have only 1 cpu. The Open MP threads is the one that dedicate your cores to your simulation.
Title: Re: OpenMP Threads and Threads MPI
Post by: elchanan on January 03, 2021, 05:43:56 am
MPI uses parallel computing between physical cores (virtual are not advisable). the speed up is in relation to the number of cores used. so theoretically you could have 10 meshes on 10 different cores and the simulation would run much faster than one mesh one one core.
Open MP on the other hand, does parallel computations by dividing the task between a few threads. so the speedup is much more limited.
If you use 2 MPI processes, you will get almost double the speed. If you use 2 OMP, you will get about 20-30% speedup. if you jump to 8, MPI will give about 6-7 times faster simulation and OMP will only double the speed.
So in conclusion, use MPI according to available physical cores. you can try to add OMP if you have available virtual cores. experiment a little with both of them to better understand.