CPU underutilization (25%) on Bunya HPC

[JoeIbrahim]

I'm attempting to run some UW2 models on UQ's Bunya HPC. I am using the ompi version of UW with Apptainer. Bunya uses a Slurm scheduler and it's my first time using Slurm, so maybe I'm making some rookie mistakes. The model runs without any errors and I get some output, but the overall CPU utilization is 25%.

I normally run these models on Gadi and have not noticed any issues, so I suspect it might have something to do with how I've set everything up on Bunya. This model is definitely running slower than it does on Gadi with the same number of CPU's - far less outputs generated in 12hrs. Is there a way to check the CPU efficiency on Gadi? If so, I can compare. Here are the summary statistics for the job:

================================================================================
                              Slurm Job Statistics
================================================================================
         Job ID: 17583095
  NetID/Account: yousephibrahim/a_ibrahim
       Job Name: 800C_nomelt
          State: TIMEOUT
          Nodes: 1
      CPU Cores: 96
     CPU Memory: 50GB (520.8MB per CPU-core)
  QOS/Partition: normal/general
        Cluster: bunya
     Start Time: Wed Oct 15, 2025 at 4:47 PM
       Run Time: 12:30:13
     Time Limit: 12:00:00
                              Overall Utilization
================================================================================
  CPU utilization  [||||||||||||                                   25%]
  CPU memory usage [||||||||||||||||||||||||||||||||               65%]
                              Detailed Utilization
================================================================================
  CPU utilization per node (CPU time used/run time)
      bun019.hpc.net.uq.edu.au: 12-09:38:24/50-00:20:48 (efficiency=24.8%)
  CPU memory usage per node - used/allocated
      bun019.hpc.net.uq.edu.au: 32.7GB/50.0GB (348.3MB/533.3MB per core of 96)
                                     Notes
================================================================================
  * The overall CPU utilization of this job is 25%. This value is low compared
    to the target range of 80% and above. Please investigate the reason for
    the low efficiency. For instance, have you conducted a scaling analysis?
    For more info:
      https://github.com/UQ-RCC/hpc-docs/blob/main/guides/Bunya-User-Guide.md

This is the SLURM script I use:

#!/bin/bash --login
#SBATCH --nodes=1
#SBATCH --ntasks=48
#SBATCH --cpus-per-task=1
#SBATCH --mem=50G
#SBATCH --job-name=800C_nomelt
#SBATCH --time=12:00:00
#SBATCH --qos=normal
#SBATCH --partition=general
#SBATCH --account=a_ibrahim
#SBATCH -o slurm-%j.out
#SBATCH -e slurm-%j.err

module load openmpi/4.1.4

export singularityDir=/home/yousephibrahim/Underworld
export containerImage=$singularityDir/UNDERWORLD_ompi.sif
SCRIPT="800C.py"

# execute
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS -c $SLURM_CPUS_PER_TASK apptainer exec $containerImage python3 $SCRIPT
#======START=====
echo "The current job ID is $SLURM_JOB_ID"
echo "Running on $SLURM_JOB_NUM_NODES nodes"
echo "Using $SLURM_NTASKS_PER_NODE tasks per node"
echo "A total of $SLURM_NTASKS tasks is used"
echo "Node list:"
sacct --format=JobID,NodeList%100 -j $SLURM_JOB_ID

Thanks for your help!!

[JoeIbrahim]

Some figures of the cpu usage in case it helps:

[julesghub]

Hi @JoeIbrahim,

Great to hear you are trying on Bunya with Apptainer!
I'd say the under utilisation issue is because the container's MPI setup isn't compatible with Bunya's configuration. So parallel performance reverts to basic operations.

The singularity container we prepared for Gadi is quite specific to the GADI configuration, so I doubt it will work with Bunya without modifications. Can you point me to any information on Bunya's HPC apptainer setup.

[JoeIbrahim]

Hey Julian! Thanks for the quick response!

Bunya is currently using version 1.4.1-1.el8 of Apptainer. Theres some info in the HPC docs here: https://github.com/UQ-RCC/hpc-docs/blob/main/guides/Bunya-User-Guide.md#interactive-jobs and here: https://github.com/UQ-RCC/hpc-docs/blob/main/guides/Basics-apptainer.md

[julesghub]

Thanks @JoeIbrahim.

Where / how did you get UNDERWORLD_ompi.sif?
Did you make it yourself?

This NCI tool / page helped to build the container on GADI.
https://opus.nci.org.au/spaces/Help/pages/132382818/Tool+to+check+singularity+containers
Does a similar page exist for Bunya?

[JoeIbrahim]

I made it with apptainer using:

pull --dir mydirectory UNDERWORLD_ompi.sif docker://underworldcode/underworld2:2.16.4-ompi

I can't find anything like the singularity-check-container tool on Bunya. I'll reach out to their support just to double check.

[JoeIbrahim]

Bunya support confirms no tool like that on Bunya, but they say we can probably get a similar report to singularity-check-container through a series of apptainer exec commands.

[JoeIbrahim]

Maybe this is helpful?

Heres output from apptainer exec command reporting CPU flags:

apptainer exec UNDERWORLD_ompi.sif bash -c 'lscpu | grep Flags'

=== CPU Flags ===
Architecture:        x86_64

Model name:          AMD EPYC 9454 48-Core Processor

Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d

Heres what I get when I run lscpu | grep Flags on the compute node:

Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni vnmi umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities