Using different number of CPUs and GPUs and generating databases

[olboyd-usgs]

Hello,
I am working on a system with several CPU cores but one GPU. If I use a single CPU (NPROC = 1) to create the databases and run an elastic simulation with 3.6e6 spectral elements for 6e5 timesteps, it takes 85 hours to create the databases and 1 hour to solve the simulation. If I use NPROC = 10), it uses 10 times as much memory on the GPU and takes the same amount of time to solve.

I would like to test many velocity models and prefer not have to rerun xgenerate_databases, since it takes so long to run on a single CPU.

Is there a way to use many CPUs to build the databases and then combine them as if it were produced using NPROC = 1? Also, is there a way to revise the velocity model without having to rerun xgenerate_databases?

Your help is greatly appreciated,
Oliver

[aakash10gupta]

Hi @olboyd-usgs, I will share my take on the CPU side of the story, but others please add / correct if needed.

I'm assuming that you are using the cartesian version. As far as I understand, suppose your mesh is sliced into 300 slices, you will use 300 cores to generate the databases, and then also to run your simulations. The only case I will use 1 core to generate the database is when my mesh is not sliced at all, and then I will use 1 core for the simulation too. Is there a reason for you to use 1 CPU core when you say that you have several CPU cores? Maybe I didn't follow something.

Also, my understanding is that, it is the generate database step where the model properties (velocity model, density, etc.) are being assigned to the mesh. So, every time you change your model properties, you have to generate the database again.

Let me know if this is helpful or not, and I or someone can try again.

P.S.: If your query is specific to specfem3d_cartesian, probably better to create an issue on the specfem3d_cartesian issues page, just to be sure that we are on the same "page".

[olboyd-usgs]

Yes, I am using the cartesian version, but wasn't sure if my question was applicable to other versions. I am using a GPU for the solver, which is significantly faster than the CPUs, but I only have one GPU. Regarding updating the velocity model, I was hoping that there was a quick way to update the database with a new velocity model without having to do unrelated tasks, assuming there are no changes to the mesh.

[aakash10gupta]

Ok, I follow your question now.

I think this would mainly fall in the GPU based simulation realm, and I would let someone more experienced with that respond.

My speculation -

• I don't think there is a quick way to update the database with a new velocity model. I believe it needs to be built from scratch each time the input model is changed, even when the mesh is the same.
• I'm not sure if there is a way to combine the sliced database files into one, to make it conducive for simulations on one GPU. Maybe GPU users have some insight if they have faced similar issues.
• If it makes sense for your use case, may be use several CPUs for both database generation and the simulation. Also, if memory is not a constraint, stick to your 10 CPU then 1 GPU approach.

[olboyd-usgs]

Most of the time spent running xgenerate_databases is, in my case, calculating adjacency information. I've modified setup_mesh_adjacency.f90 to save and resuse adjacency information if the mesh hasn't changed as well as to use multiple threads. Adjacency information is already being saved in external_mesh.bin, but it was easier for me to have it in its own file. Making it multithreaded requires the compiler flag -fopenmp and revisions to src/shared/search_kdtree.f90. Feel free to contact me if you are interested in the revised code.