Bus error (core dumped) in deepmd multitask training

[ChiahsinChu]

Summary

When performing multi-task training with deepmd v2.2.11 in Zeus HPC, bus error (core dumped) is thrown during neighbour stat (i.e., before training starts).

Software

DeepMD-kit v2.2.11 + Tensorflow v2.14.0
(installed from the official off-line package)

Details

Run command:

#!/bin/bash

#SBATCH -J deepmd
#SBATCH -N 1
#SBATCH --ntasks-per-node=1
#SBATCH -t 96:00:00
#SBATCH --partition=gpu1
#SBATCH --gres=gpu:1
#SBATCH --mem=16G

module load miniconda/3
module load cuda/11.8
source activate deepmd-v2.2.11

dp train input.json 1>>dp_train.stderr 2>>dp_train.stdout

Error:

/var/spool/slurmd/job716526/slurm_script: line 15: 20357 Bus error               (core dumped) dp train input.json >> dp_train.stderr 2>> dp_train.stdout

Input and output files are attached:
tf-v2-train.tar.gz

[link89]

You may need to find out which node the job was running on, then ssh to that node and run dmesg -T to collect more log about this issue.