Add section on preventing too much parallism/high memory use to the docs

[bouweandela]

The default Dask settings may result in too much parallism and/or high memory use when running multiple executions in parallel on the same machine.

ESMValTool

ESMValCore works best with the Dask Distributed scheduler. I would also recommend setting max_parallel_tasks (an ESMValCore setting), to a low number, e.g. 1, 2 or 3, because only one ESMValCore preprocessing task will submit jobs to the Distributed scheduler at a time to avoid overloading the workers. Therefore, I would recommend running with the following settings for ESMValCore:
~/.config/esmvaltool/config.yaml:

max_parallel_tasks: 2
dask:
  use: local_distributed
  profiles:
    local_distributed:
      cluster:
        type: distributed.LocalCluster
        n_workers: 2
        threads_per_worker: 2
        memory_limit: 4GiB

where the total memory use will be n_workers * memory_limit (= 8GB in the example above), so you can reduce n_workers if you need it to use less memory. I would recommend using at least 4 GB or RAM per Dask Distributed worker. Some jobs might be able to run with 2GB, but probably not all of them. You can tune the total memory / CPU use by specifying the number of workers. The number of CPU cores used will be n_workers * theads_per_worker.

For more information on configuring ESMValCore, see https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html

PMP and ilamb

Both PMP and ilamb also use Dask through Xarray, but we've seen crashes because of that, e.g. #394 and #385 (comment). For PMP and ilamb I would recommend using the following global Dask configuration, to avoid the Dask crashes:
~/.config/dask/config.yaml

scheduler: synchronous

or, if you want faster runs, at least control the number of workers that gets started (by default the is the number of CPU cores, which is way too much if you are running other executions on the same machine because each job will use this many):
~/.config/dask/config.yaml

scheduler: threads
num_workers: 4

Note that the ilamb diagnostics are currently restricted to the synchronous scheduler by

climate-ref/packages/climate-ref-ilamb/src/climate_ref_ilamb/standard.py

Lines 286 to 287 in 25ffb6e

	# Run ILAMB in a single-threaded mode to avoid issues with multithreading (#394)
	with dask.config.set(scheduler="synchronous"):

, so it will not respect the global Dask settings.

For more information on configuring Dask, see:

• https://docs.dask.org/en/latest/configuration.html
• https://docs.dask.org/en/latest/scheduling.html

[bouweandela]

@lewisjared I could work on this next week. Any suggestion of where to put this?

[lewisjared]

Great!

Any suggestion of where to put this?

Maybe its a how-to guide for now, but we should link to it from the getting started material and the executor docs

To add to this I also set the following env variables to limit the size of the PMP jobs:

• OPENBLAS_NUM_THREADS
• MKL_NUM_THREADS
• OMP_NUM_THREADS

I don't know if only one of them is used, but I didn't dive into the code @lee1043 might know

[minxu74]

I just found that under dask, the OPENBLAS_NUM_THREADS is not respected by the code. I got errors of oversubscribed threads and OPENBLAS wanted to use 128 threads that exceeded the default 64 threads built for numpy.

[bouweandela]

I've added instructions on configuring Dask in #591. Feel free to extend those with environmental variables needed for other libraries.