Delta right now imports CuPy whether it's running on the CPU or GPU. This is a problem when running on the CPU but in the Delta container where CuPy is installed:
stephey@nid00218:/global/cscratch1/sd/stephey/delta/delta> OMP_NUM_THREADS=16 srun -n 16 -c 2 --cpu-bind=cores shifter python3 -m mpi4py.futures processor.py --config configs/hackathon_test.json --transport transport_tx --num_ranks_preprocess=1 --num_ranks_analysis=16 --num_queue_threads=1 --run_id=test_25259_GT
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/cupy/__init__.py", line 20, in <module>
from cupy import core # NOQA
File "/usr/local/lib/python3.8/dist-packages/cupy/core/__init__.py", line 1, in <module>
from cupy.core import core # NOQA
ImportError: libcuda.so.1: cannot open shared object file: No such file or directory
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/mpi4py/futures/__main__.py", line 72, in <module>
main()
File "/usr/local/lib/python3.8/dist-packages/mpi4py/futures/__main__.py", line 60, in main
run_command_line()
File "/usr/local/lib/python3.8/dist-packages/mpi4py/run.py", line 47, in run_command_line
run_path(sys.argv[0], run_name='__main__')
File "/usr/lib/python3.8/runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "processor.py", line 28, in <module>
from analysis.task_list import tasklist
File "/global/cscratch1/sd/stephey/delta/delta/analysis/task_list.py", line 6, in <module>
from analysis.helpers import get_analysis_task
File "/global/cscratch1/sd/stephey/delta/delta/analysis/helpers.py", line 10, in <module>
from analysis.task_spectral_cu import task_coherence_cu, task_crosscorr_cu, task_crossphase_cu, task_crosspower_cu
File "/global/cscratch1/sd/stephey/delta/delta/analysis/task_spectral_cu.py", line 4, in <module>
from analysis.kernels_spectral_cu import kernel_crosscorr_cu, kernel_coherence_cu, kernel_crosspower_cu, kernel_crossphase_cu
File "/global/cscratch1/sd/stephey/delta/delta/analysis/kernels_spectral_cu.py", line 7, in <module>
import cupy as cp
File "/usr/local/lib/python3.8/dist-packages/cupy/__init__.py", line 41, in <module>
raise ImportError(_msg) from e
ImportError: CuPy is not correctly installed.
Delta right now imports CuPy whether it's running on the CPU or GPU. This is a problem when running on the CPU but in the Delta container where CuPy is installed:
I suggest that Delta have an
--archor--modeflag that a user can set to CPU or GPU depending on what they need. Then Delta could handle the imports accordingly.