Merging multiple mpco files

[bozturk]

Dear STKO Team,

I am using mpiexec.hydra.exe to run my model in parallel in order to speed up my analyses. This process generates multiple MPCO files, with names such as "model.part-0.mpco", "model.part-1.mpco", "model.part-2.mpco", "model.part-3.mpco", and "model.part-4.mpco", based on the number of processors used (in this case, 5 processors were specified in the command line).

However, I believe that the data in these files is randomly partitioned, so they are not useful on their own. Is there a way to merge all of these files into a single file? I have tried using the HD5 library in Python to do this, but have not yet found a solution.

For the note, I am using the latest version of OpenSees (v3.4.0), compiled by Prof. Nicola Tarque.

Regards,
Burak O.

[STKO Team]

Dear user,

I am using mpiexec.hydra.exe to run my model in parallel in order to speed up my analyses. This process generates multiple MPCO files, with names such as "model.part-0.mpco", "model.part-1.mpco", "model.part-2.mpco", "model.part-3.mpco", and "model.part-4.mpco", based on the number of processors used (in this case, 5 processors were specified in the command line).

That's correct. If you are using the command line, it means you did not use the STKO pre-processor.
Are you using OpenSeesSP or OpenSeesMP?

However, I believe that the data in these files is randomly partitioned, so they are not useful on their own.

No, they are not random. "model.part-N.mpco" will contain only the partition of the model in processor "N"

Is there a way to merge all of these files into a single file? I have tried using the HD5 library in Python to do this, but have not yet found a solution.

It's not necessary. When you load one of them in the STKO post processor, it will automatically merge "virtually" all of them as if they were a single MPCO file.
If it does not do that, it means that there is something wrong with your model (see the red error message in the STKO's terminal).

Could you share your input file (STKO pre-processor file or Tcl files)?

[bozturk]

Dear user,

I am using mpiexec.hydra.exe to run my model in parallel in order to speed up my analyses. This process generates multiple MPCO files, with names such as "model.part-0.mpco", "model.part-1.mpco", "model.part-2.mpco", "model.part-3.mpco", and "model.part-4.mpco", based on the number of processors used (in this case, 5 processors were specified in the command line).

That's correct. If you are using the command line, it means you did not use the STKO pre-processor.
Are you using OpenSeesSP or OpenSeesMP?

However, I believe that the data in these files is randomly partitioned, so they are not useful on their own.

No, they are not random. "model.part-N.mpco" will contain only the partition of the model in processor "N"

Is there a way to merge all of these files into a single file? I have tried using the HD5 library in Python to do this, but have not yet found a solution.

It's not necessary. When you load one of them in the STKO post processor, it will automatically merge "virtually" all of them as if they were a single MPCO file.
If it does not do that, it means that there is something wrong with your model (see the red error message in the STKO's terminal).

Could you share your input file (STKO pre-processor file or Tcl files)?

I am using OpenSeesSP and executing the command: "mpiexec.hydra.exe -n 2 openseessp.exe model.tcl" to run my model.

The model runs as expected, without any errors, but when I attempt to divide the work among multiple cores, it becomes stuck and do not return any error messages or warnings. (still utilizes CPU cores for indefinite period)

I have also tried running the script using a single core as "mpiexec.hydra.exe -n 1 openseessp.exe model.tcl", and it worked without issue. I believe the problem may lie with the DomainPartitioner.

To further investigate the issue, I commented out lines in my script one by one to see where it became stuck and discovered that commenting out the "system Mumps -ICNTL14 200" line solved the analysis and generated mpco files.
(of course I cannot just comment out system definition but just wanted to see where it goes.)

However, when I attempted to import those MPCO files into STKO post-processor, I encountered error messages stating that:

Partition "PATH/model.part-*.mpco" has at least 1 incompatible model stage (it will be skipped).

(During the analysis, opensees did not crush or failed to converge.)

I have included a simplified version of my model for reference.

https://drive.google.com/file/d/1qBzp2X ... sp=sharing

Thank you so much for taking the time to review this issue. I'd appreciate if you could share with me your thoughts about this.

[STKO Team]

Here's a file that works fine now:

model.zip

(85.38 KiB) Downloaded 21 times

This is what I had to change:

1. Solve the issue of Stage 3 not running: use a wipeAnalysis before setting any new analysis. In this way, you make sure you clear any previous analysis component. In your case, it was necessary before the third stage, because adding new elements in an already partitioned domain, causes these elements to go only on the first domain, and this creates problems with the existing analysis components. So, as a rule, you can always use wipeAnalsyis before starting a new analysis (in STKO in fact we automatically write it before ANY analysis)
2. Solve the inconsistency in MPCO recorders: use a wipeAnalysis before setting any new analysis. Since you added new elements (piles) in an already partitioned domain, they will go only on the first sub-domain. Therefore OpenSees will mark only that domain as changed. The MPCO Recorder assumes that all sub-domains are consistent with their changes (same number of model_stages).
   One way to do this, is to call domainChange just before any analysis. Note: Thanks to this example we noticed an issue in OpenSeesSP (in fact, with STKO we prefer to use OpenSeesMP for parallel computing). The domainChange command is not propagated to the sub-domains. So I had to write a "brutal" workaround to make your example work.
   (see the description of the procedure "customDomainChange" I wrote at the beginning of the model.tcl file).
   In the meantime I will try to fix this issue in the OpenSees source code so that in future the domainChange method will work

[bozturk]

Thank you very much, I can confirm this has resolved the problem.

Are there any drawbacks of using customDomainChange procedure for my analyses at this time?

[STKO Team]

There should be no drawback in terms of results (in fact you add and remove an sp load without doing any analysis in between).
The only drawback could be about performance when you have many nodes... but it's the only solution now. We will soon make a Pull Request to fix this issue.
I will try to fix the issue directly in the domainChange method of the PartitionedDomain class. If the OpenSees' maintainers will not accept it (because it will be hard to do all the necessary tests to make sure this change will not affect any other part of the code), we will make a workaround directly in the MPCORecorder class