flecsi: New hangs with HPX 1.11.0

[rbberger]

We're observing new, consistent hangs in FleCSI in the data-index unit test when switching from HPX 1.10.0 to 1.11.0.

Our Spack environment:

spack:
  specs:
  - flecsi
  view:
    default:
      exclude:
      - gcc-runtime
      root: .spack-env/view
  packages:
    boost:
      require:
      - +debug
      - cxxstd=17 +program_options
      prefer:
      - '@1.88.0'
    c:
      prefer:
      - gcc
    cxx:
      prefer:
      - gcc
    flecsi:
      require:
      - spec: backend=hpx build_type=Debug caliper_detail=low +flog +graphviz %gcc
        when: '@2.4:'
    fortran:
      prefer:
      - gcc
    hpx:
      require:
      - build_type=Debug
      prefer:
      - '@1.11.0'
    kokkos:
      require:
      - +serial~openmp~cuda~rocm
      prefer:
      - '@4.6.01'
    mpi:
      require:
      - mpich
    cmake:
      prefer:
      - '@3.31.6'
    graphviz:
      prefer:
      - '@12.1.0'
      require:
      - +poppler+pangocairo %gcc
    py-sphinx:
      prefer:
      - '@8.2.0'
    hdf5:
      prefer:
      - '@1.14.6'
    metis:
      require:
      - '@5.1.0'
    parmetis:
      require:
      - '@4.0.3'
    mpich:
      prefer:
      - '@4.3.0'
      require:
      - +hydra +slurm +fortran device=ch4 netmod=ucx %c,cxx,fortran=gcc
    ucx:
      prefer:
      - '@1.18.0'
      require:
      - +cma+dc+rc+rdmacm+thread_multiple+ud+verbs+debug %c,cxx=gcc

Concretized:

$ spack spec
 -   flecsi@2.4.develop~cuda~doc+flog~format+graphviz~hdf5~ipo~rocm+shared backend=hpx build_system=cmake build_type=Debug caliper_detail=low commit=77f5313b54f6782ec2a57002ce1a9f4f35641da8 generator=make arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]      ^boost@1.88.0+atomic~charconv+chrono~clanglibcpp~cobalt~container~context~contract~coroutine+date_time+debug+exception~fiber+filesystem+graph~graph_parallel+icu+iostreams~json+locale+log+math~mpi~mqtt5+multithreaded~nowide~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+stacktrace+system~taggedlayout+test+thread+timer~type_erasure~url~versionedlayout+wave build_system=generic cxxstd=17 patches:=a440f96 visibility=hidden arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-rhel8-broadwell %c=gcc@13.2.0
[e]              ^diffutils@3.6 build_system=autotools arch=linux-rhel8-broadwell
[+]          ^icu4c@76.1 build_system=autotools cxxstd=17 arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]          ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]      ^caliper@2.12.1+adiak~cuda~fortran+gotcha~ipo+kokkos+libdw~libpfm+libunwind+mpi+papi~python~rocm+sampler+shared~sosflow~tests+tools~variorum~vtune build_system=cmake build_type=Release generator=make arch=linux-rhel8-broadwell %c,cxx,fortran=gcc@13.2.0
[+]          ^adiak@0.4.1~ipo+mpi+shared build_system=cmake build_type=Release commit=7ac997111785bee6d9391664b1d18ebc2b3c557b generator=make arch=linux-rhel8-broadwell %c,cxx,fortran=gcc@13.2.0
[+]          ^elfutils@0.192~debuginfod+exeprefix+nls build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]                  ^tar@1.35 build_system=autotools zip=pigz arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                      ^pigz@2.8 build_system=makefile arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^m4@1.4.20+sigsegv build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]                  ^libsigsegv@2.14 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]          ^libunwind@1.8.1~block_signals~conservative_checks~cxx_exceptions~debug~debug_frame+docs~pic+tests+weak_backtrace~xz~zlib build_system=autotools components:=none libs:=shared,static arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^papi@7.1.0~cuda~debug+example~infiniband~lmsensors~nvml~powercap~rapl+rdpmc~rocm~rocm_smi~sde+shared~static_tools build_system=autotools patches:=48cb202 arch=linux-rhel8-broadwell %c,cxx,fortran=gcc@13.2.0
[+]          ^python@3.11.11+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches:=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^expat@2.7.1+libbsd build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]                  ^libbsd@0.12.2 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                      ^libmd@1.1.0 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^gdbm@1.23 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^libffi@3.4.8 build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^libxcrypt@4.4.38~obsolete_api build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[e]              ^openssl@1.1.1k~docs+shared build_system=generic certs=mozilla arch=linux-rhel8-broadwell
[+]              ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^sqlite@3.46.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^util-linux-uuid@2.41 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]      ^cmake@3.31.6~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[e]          ^curl@7.61.1~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-rhel8-broadwell
[+]          ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]      ^compiler-wrapper@1.0 build_system=generic arch=linux-rhel8-broadwell
[e]      ^gcc@13.2.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-rhel8-broadwell
[+]      ^gcc-runtime@13.2.0 build_system=generic arch=linux-rhel8-broadwell
[e]      ^glibc@2.28 build_system=autotools arch=linux-rhel8-broadwell
[+]      ^gmake@4.4.1~guile build_system=generic arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]      ^graphviz@12.1.0~doc~expat~ghostscript~gtkplus~gts~java~libgd+pangocairo+poppler~qt~quartz~x build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^autoconf@2.72 build_system=autotools arch=linux-rhel8-broadwell
[+]              ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^automake@1.16.5 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]          ^bison@3.8.2~color build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^cairo@1.17.4~X+fc+ft+gobject+pdf+pic+png+shared~svg build_system=autotools patches:=7097196 arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^pixman@0.44.0+shared~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^which@2.21 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]          ^flex@2.6.3+lex~nls build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^fontconfig@2.15.0~pic build_system=autotools arch=linux-rhel8-broadwell %c,fortran=gcc@13.2.0
[+]              ^font-util@1.4.1 build_system=autotools fonts:=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                  ^bdftopcf@1.1.1 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                      ^fontsproto@2.1.3 build_system=autotools arch=linux-rhel8-broadwell
[+]                      ^libxfont@1.5.4 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                          ^xtrans@1.6.0 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                      ^xproto@7.0.31 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                  ^mkfontdir@1.0.7 build_system=autotools arch=linux-rhel8-broadwell
[+]                  ^mkfontscale@1.2.3 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                      ^libfontenc@1.1.8 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^gperf@3.1 build_system=autotools patches:=3dd36db arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^freetype@2.13.2+pic+shared build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]          ^glib@2.78.3~libmount~strip build_system=meson buildtype=release default_library:=shared tracing:=none arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^meson@1.7.0 build_system=python_pip patches:=0f0b1bd arch=linux-rhel8-broadwell
[+]                  ^py-pip@25.1.1 build_system=generic arch=linux-rhel8-broadwell
[+]                  ^py-setuptools@80.9.0 build_system=generic arch=linux-rhel8-broadwell
[+]                  ^py-wheel@0.45.1 build_system=generic arch=linux-rhel8-broadwell
[+]              ^pcre2@10.44~jit+multibyte+pic build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^python-venv@1.0 build_system=generic arch=linux-rhel8-broadwell
[+]          ^libpng@1.6.47~ipo~pic build_system=cmake build_type=Release generator=make libs:=shared,static arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^libtool@2.4.7 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]          ^pango@1.54.0~X~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^fribidi@1.0.12 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^gobject-introspection@1.78.1~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^harfbuzz@10.2.0~graphite2~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^json-glib@1.6.6~strip build_system=meson buildtype=release default_library:=shared arch=linux-rhel8-broadwell
[+]          ^pkgconf@2.3.0 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]          ^poppler@23.04.0~boost~cms~cpp~glib~gobject~gpgme~ipo~jpeg~libcurl~nss~openjpeg~png~qt~tiff~zlib build_system=cmake build_type=Release generator=make arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^poppler-data@0.4.12~ipo build_system=cmake build_type=Release generator=make arch=linux-rhel8-broadwell
[+]          ^sed@4.9 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]      ^hpx@1.11.0~apex~async_cuda~async_mpi~cuda~examples~generic_coroutines~ipo~rocm~tools build_system=cmake build_type=Debug cxxstd=17 generator=ninja instrumentation:=none malloc=system max_cpu_count=auto networking:=mpi arch=linux-rhel8-broadwell %cxx=gcc@13.2.0
[+]          ^asio@1.32.0~boost_coroutine~boost_regex~separate_compilation build_system=autotools cxxstd=17 arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[e]          ^git@2.39.3+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-rhel8-broadwell
[+]          ^hwloc@2.11.1~cairo~cuda~gl~level_zero~libudev+libxml2~nvml~opencl+pci~rocm build_system=autotools libs:=shared,static arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]          ^ninja@1.12.1+re2c build_system=generic patches:=93f4bb3 arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^re2c@3.1 build_system=autotools arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]      ^kokkos@4.6.01~aggressive_vectorization~cmake_lang~compiler_warnings+complex_align~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hip_relocatable_device_code~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]      ^mpich@4.3.0~argobots~cuda+fortran~hcoll+hwloc+hydra~level_zero+libxml2+pci~rocm+romio+slurm~vci~verbs+wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ucx pmi=default arch=linux-rhel8-broadwell %c,cxx,fortran=gcc@13.2.0
[+]          ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]          ^libpciaccess@0.17 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]              ^util-macros@1.20.1 build_system=autotools arch=linux-rhel8-broadwell
[+]          ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[e]          ^slurm@24-05-5-1~cgroup~gtk~hdf5~hwloc~mariadb~nvml~pam~pmix+readline~restd~rsmi build_system=autotools sysconfdir=PREFIX/etc arch=linux-rhel8-broadwell
[+]          ^ucx@1.18.0~assertions~backtrace_detail+cma~cuda+dc+debug~dm+examples~gdrcopy~gtest~ib_hw_tm~java~knem~logging~mlx5_dv+openmp+optimizations~parameter_checking+pic+rc+rdmacm~rocm+thread_multiple~ucg+ud+verbs~vfs~xpmem build_system=autotools libs:=shared,static opt=3 simd:=auto arch=linux-rhel8-broadwell %c,cxx=gcc@13.2.0
[+]              ^rdma-core@57.0~ipo+man_pages+pyverbs+static build_system=cmake build_type=Release generator=make arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                  ^libnl@3.3.0 build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0
[+]                  ^py-docutils@0.20.1 build_system=python_pip arch=linux-rhel8-broadwell
[+]          ^yaksa@0.3~cuda~level_zero~rocm build_system=autotools arch=linux-rhel8-broadwell %c=gcc@13.2.0

[rajj28]

@hkaiser i would like to work on this issue

[hkaiser]

@hkaiser i would like to work on this issue

Please go ahead. This might be a tricky one.

[rbberger]

@hkaiser there seems to be a commit with a reproducer, but I'm not seeing a fix. What is the state of this?

Also, what are the chances of having a backport of 9c11915 for something like a 1.10.1? 1.10.0 works mostly for us, except in some configuration cases where the Boost detection fails. This is the program_options detection failure you might have heard about. HPX 1.10.0 does CMake MODULE detection by default, which doesn't compose with clients like FleCSI. This is why preferring CONFIG mode is better.

[hkaiser]

@rbberger https://github.com/sponsors/STEllAR-GROUP ;-)