Memory bottleneck

[efertitta]

I have been trying to push Vayesta to calculate molecules with ~200 atoms with >2000 cGTO, but seem to a wall mostly due to memory issues. The BNO construction is not an issue, but the construction of cluster eri is. DF brought me a bit further, but this projection still is a bottleneck
Digging in the code and in pyscf, this seems to be done via rotation of the full eri which is of course very memory consuming. Could an 'integral-direct' approach help here?

[ghb24]

So I have no doubt that this could be done, but doing it efficiently would take a bit of care and thought. At one level, we would just do a first 'quarter-transform' of the three-index DF integrals into the cluster basis in a direct fashion. This seems feasible, but getting is working for both solids and molecules may be difficult - I guess we just start with molecules. It does of course mean generating all integrals for each cluster, rather than reusing them. This would increase the prefactor in the (leading-order) N^4 step of the algorithm for the ERI projection...but could get the memory overhead down to N^2.

A longer-term option (I guess in conjunction with this) would be to start thinking about whether we can define a compact support of the cluster orbitals over the AOs, perhaps in a domain of AOs on atoms up to a certain distance away. This is obviously the first step towards a fully linear-scaling approach.

[ghb24]

@maxnus any thoughts...?

[maxnus]

What's the cluster size here? Do you have an example script? I'm not aware that we project full system ERIs in any solver, as this would scale N^4 and anyways require too much memory.

[efertitta]

Hi @maxnus
I am not sure about the PBC case, but for molecules core.eris calls ao2mo in pyscf to get the cluster hamiltonian
line 147 >mf.with_df.ao2mo(mo_coeff, compact=compact)
line 151 >pyscf.ao2mo.kernel(emb.mf._eri, mo_coeff, compact=compact)
line 149 >mol.ao2mo(mo_coeff, compact=compact)

Under the hood these make a projection of the full system ERI into the cluster orbitlas, mo_coeff if I am not mistaken

The cluster size is not really the problem: I have add troubles even to construct the DMET bath for an hydrogen ring with 402 atoms with cc-pvdz. That is only 2 cluster orbitals, but contracting the full system ERI (2010^4) seem to be the issue...?

I attach my input
Actually on a related memory issues: here I had remove mf = mpi.scf(mf) and run the scf on each task instead. The reason is the the DF eri is so large that cannot be casted from master to all ranks (at least my architecture/memory settings won't allow it)

from pyscf import gto
from pyscf import scf
import vayesta
import vayesta.ewf
from vayesta.mpi import mpi
from vayesta.misc.molecules import ring

mol = gto.Mole()
mol.atom = ring("H", 402, 2.0)
mol.basis = 'cc-pvdz'
mol.output = 'pyscf-mpi%d.out' % mpi.rank
mol.verbose = 4
mol.build()

mf = scf.RHF(mol).density_fit()
mf.max_cycle = 200
#mf = mpi.scf(mf)
mf.kernel()

emb = vayesta.ewf.EWF(mf, solver='CCSD', bath_options=dict(bathtype='dmet'))
with emb.iao_fragmentation() as f:
f.add_all_atomic_fragments()

emb.kernel()

[maxnus]

The cluster size is not really the problem: I have add troubles even to construct the DMET bath for an hydrogen ring with 402 atoms with cc-pvdz. That is only 2 cluster orbitals, but contracting the full system ERI (2010^4) seem to be the issue...?

Hi Edoardo, this is weird, it should not construct the full tensor when using DF. Unfortunately, I won't be able to look into it before next week.

[efertitta]

Quick update: following @ghb24 's suggestions I tried to reduce PYSCF_MAX_MEMORY to force ao2mo into outcore and not store all intermediates on memory. Unfortunately this not seem to affect the total memory allocation used and I seem to hit the same wall. Only the calculations are noticibly slower. I am sure if this seems that it is still running in incore despite the lower max_memory

[maxnus]

line 147 >mf.with_df.ao2mo(mo_coeff, compact=compact)
line 151 >pyscf.ao2mo.kernel(emb.mf._eri, mo_coeff, compact=compact)
line 149 >mol.ao2mo(mo_coeff, compact=compact)

Of these three transformation routines, only the first one should be encountered if density-fitting is used. The second is for systems without df and the third is a backup for the case that ERIs have been deleted somehow in the meantime.
I tried your example with reduced atoms and indeed only the df.ao2mo version was used.
Where exactly does your calculation run out of memory?

On an unrelated note:
mpi.scf does not speed up the mean-field calculation, but only keeps all ranks other than the master idle.
MPI does help to parallelize the embedding calculations, but in your case, you have a fully rotationally symmetric system, so as long as the mean field obeys this symmetry, it would be better to make use of this instead of using MPI.

[efertitta]

Hi Max,
How many atoms have you tried with? Up to 200sh runs fine, but it won't work with 400sh like the example I show here (I think even 300 is problematic). I am not really interested in this system in particular, I just wanted to see how the method behaves with basis set size.
I am studying some enzymes with 200+ atoms and was aiming to use a basis set with >3000 GTOs. SCF converges, the BNOs gets constructed, but when it's time to build the local hamiltonian it goes OOM
I can share my inputs/outputs if it helps