Make code on one thread more efficient

[NoraHagmeyer]

E.g. during the active set strategy a bunch of matrices are created (and not destroyed?) within each iteration. This seems to lead to an excessive RAM use and thus an error within the simulation for large numbers of DoFs (n= 8, ~65000 DoFs, direct solver, dense matrices). The MATLAB code still runs through for the same number parameters. So, the code can surely be implemented in a more efficient manner.

[NoraHagmeyer]

In GitLab by @tsarikahin on Jul 1, 2021, 17:41

Can you give more explanations?

For instance, positions variable should be deallocated?

[NoraHagmeyer]

I'm also guessing the matrices might be a problem

[mayrmt]

@NoraHagmeyer, we are about to tackle this issue know with the help of our student @PhilipOesterlePekrun.

Do you know where we can find the original Matlab implementation for comparison?

[NoraHagmeyer]

@NoraHagmeyer, we are about to tackle this issue know with the help of our student @PhilipOesterlePekrun.

Do you know where we can find the original Matlab implementation for comparison?

I am pretty sure we had a repo und gitlab. But from your question I am guessing it didn't make it to Github because we didn't think it was important anymore. Since I don't have access to my old email address or hardware anymore, I am afraid I don't have a local copy anymore either. Maybe @RShaw026 has one?
However, the code has been designed according to the Matlab code which is the main issue. In Matlab, efficient code looks differently than in C++. I'd hope that at this point, MIRCO is suffiently well tested to make changes without breaking anything.

[mayrmt]

@NoraHagmeyer I have found the Matlab code on our old gitlab instance. I pushed it into a private repo in my GitHub account. You (and @PhilipOesterlePekrun) should have received invitations to this repository, in case you want to have a look. (I do not think, that it is super helpful though.)

[PhilipOesterlePekrun]

@mayrmt Thank you!

[PhilipOesterlePekrun]

I do not get any errors for Resolution = 8 (which would be (2^8+1)^2=66049dofs if I understand correctly). I do get an error for Resolution > 8, but that is because there are no shape factors available for that.

So, is that still an issue or is it just the overall efficiency? But how are we comparing it to the Matlab code? Which Matlab .m script is being compared to running main.cpp? There seems to be no overall/main Matlab script that runs everything.

Thanks in advance!

[mayrmt]

@PhilipOesterlePekrun, we do not have to compare to Matlab. We just have to improve performance of the C++ code.

If you don't run into memory problems with n=8, then you probably have enough memory on your machine. Though that's not the case for everyone --- and even with enough memory, we still copy massive amounts of data, which costs execution time.

[PhilipOesterlePekrun]

Ok, I will try with less memory to find the memory issue.

Regarding the execution time, it seemed mostly evenly spread out among the functions when I analyzed it with "perf"--though, Teuchos::SerialDenseMatrix<int, double>::checkIndex(int, int) accounts for a surprisingly high amount of the time (37% in serial). I think there is a way to turn off bounds checking for Teuchos::SerialDenseMatrix. But I'll look into the critical memory issue first.

Also, I think Matlab can be surprisingly efficient with matrix operations because its algorithms also use vectorization. This is something that Kokkos could of course bring us.

[mayrmt]

If we have good performance except for everything related to Teuchos::SerialDenseVector/Matrix/Solver, then I am happy, because I intend to replace them by some Kokkos anyways.

[PhilipOesterlePekrun]

Regarding the memory, I have inspected the maximum amount of memory used with two examples. I tested these in serial and with OMP_NUM_THREADS=24 with pretty much no difference, as expected.

• For input_sup8.xml with random topology with Delta=12.0, the largest (predicted) potential contact set is 4293 (systemsize), so the largest A has a size of 4293^2*8B = 140MB, while the maximum overall memory usage was 166MB and the maximum overall heap allocation was 144MB.
• For Delta=15.0, the largest potential contact set is 7328, so the largest A has a size of 7328^2*8B = 410MB, while the maximum overall memory usage was 445MB.

So, to me it doesn't seem like there is a critical memory issue, but rather just that the matrices inherently use a lot of memory, which makes sense.