This repository was archived by the owner on May 23, 2024. It is now read-only.
Description I have CUDA_VISIBLE_DEVICES=0,1,2,3 in the shell. Running:
mpiexec -np 4 hpcbind --distribute=4 --output-prefix=test --output-mode=all --lstopo -- ./test
shows basically the same GPU setting for each test.hpcbind.[0-3]:
[HPCBIND]
HPCBIND_HAS_HWLOC=1
HPCBIND_HAS_NVIDIA=1
HPCBIND_HWLOC_CPUSET=0x00001001
HPCBIND_HWLOC_DISTRIBUTE=4
HPCBIND_HWLOC_DISTRIBUTE_PARTITION=0
HPCBIND_HWLOC_PARENT_CPUSET=0x00555555
HPCBIND_HWLOC_PROC_BIND=all
HPCBIND_HWLOC_VERSION=2.2.0
HPCBIND_NUM_CORES=1
HPCBIND_NUM_NUMAS=1
HPCBIND_NUM_PUS=2
HPCBIND_NUM_SOCKETS=1
HPCBIND_NVIDIA_ENABLE_GPU_MAPPING=1
HPCBIND_NVIDIA_VISIBLE_GPUS=0,1,2,3
HPCBIND_OPENMP_RATIO=1/1
HPCBIND_OPENMP_VERSION=4.0
HPCBIND_QUEUE_MAPPING=0
HPCBIND_QUEUE_NAME=openmpi
HPCBIND_QUEUE_RANK=0
HPCBIND_QUEUE_SIZE=4
[HWLOC]
[CUDA]
CUDA_HOME=/home/aznb/spack/var/spack/environments/trilinos-cudacc61/.spack-env/view
CUDA_LAUNCH_BLOCKING=1
CUDA_VISIBLE_DEVICES=0,1,2,3
[OPENMP]
OMP_NESTED=false
OMP_NUM_THREADS=2
OMP_PLACES=threads
OMP_PROC_BIND=spread
[GOMP] (gcc, g++, and gfortran)
[KMP] (icc, icpc, and ifort)
[XLSMPOPTS] (xlc, xlc++, and xlf)
[BINDINGS]
Machine (128GB total)
Package L#0
NUMANode L#0 (P#0 64GB)
L3 L#0 (15MB) + L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
PU L#0 (P#0)
PU L#1 (P#12)
Package L#1
NUMANode L#1 (P#1 64GB)
and my app './test' report the same GPU id for all processes.
However, if I run
mpiexec -np 4 hpcbind --distribute=4 --output-prefix=test --output-mode=all --lstopo -- ./test --kokkos-num-devices=4
then each process reports a unique GPU id
Reactions are currently unavailable
I have
CUDA_VISIBLE_DEVICES=0,1,2,3in the shell. Running:shows basically the same GPU setting for each test.hpcbind.[0-3]:
and my app './test' report the same GPU id for all processes.
However, if I run
then each process reports a unique GPU id