CBC/CBCD with minimally sized FLUDS

[eappen-nelluvelil]

PR Checklist

• I have updated the user guide and/or Python API documentation if necessary.

Summary

The scope is limited to the latest three commits: 68ff405, 45e29dc, and a6b6905. The first 3 commits are from PR #1008. This PR will be updated to remove those commits once they are merged into main.

Commit 1

CBC_SPDS now computes an exact static cell-face slot plan for local psi banks.

• Builds the local task DAG in topological-rank space.
• Computes the reachability relation between local producer and consumer cells for directed local faces.
• Reduces safe face-slot reuse to a minimum chain-cover / maximum matching problem on the associated bipartite split graph, solved with Hopcroft-Karp.
• Extracts and verifies a static per-face slot assignment for local psi banks, with conservative fallback to one slot per directed local face if verification rejects the computed plan.

Commit 2

CBC_FLUDS uses the CBC_SPDS slot plan to minimize host local psi storage.

• Sizes the host local psi buffer by the computed slot count.
• Sizes each slot by the maximum face-node extent of the local directed faces assigned to that slot.

Commit 3

CBCD uses the same static slot plan to minimize device local psi storage, and replaces the earlier per-angleset communicators with one aggregated communicator per groupset.

• Adds CBCD_AsynchronousCommunicator with:
  • one communication thread per groupset;
  • bounded per-destination multi-producer, single-consumer outgoing queues;
  • aggregated angleset outgoing MPI sends; and
  • per-angleset incoming mailboxes.
• Extends CBCD_AngleSet, CBCD_FLUDS, and CBCDSweepChunk to:
  • launch ready-cell batches on the device;
  • publish and consume compact non-local face batches;
  • map received non-local data by receiver-local face index; and
  • minimally size the local psi bank from the CBC_SPDS slot plan.

The plot below shows strong-scaling comparisons on Tuo between CBCD V1 and CBCD V2 on 32k and 268k cell mesh problems, both with 448 directions and 1 groupset with 64 groups.

Some grind times are unavailable due to earlier binary instabilities on Tuo with Cray MPICH and ROCm. Once bank access has been restored on Tuo, the results will be updated.

[eappen-nelluvelil]

@wdarylhawkins @quocdang1998 The c5g7_restart.py host CBC test fails since refactored CBC cannot read the restart files in c5g7_restart/. The test passes without using those restart files.

Note: This regression test now passes.

[eappen-nelluvelil]

@wdarylhawkins @quocdang1998 Switching this to a draft PR until I get back access to Tuo compute banks to update scaling studies.