Using semaphore to sync with all peer processes in finalization stage

[rocm-devops]

[rocprofv3] Implement synchronization using POSIX semaphore in finalization

PR Details

Associated Jira Ticket Number/Link

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update
• Continuous Integration

Technical details

This PR is to fix https://ontrack-internal.amd.com/browse/SWDEV-539022. In some workload, during the exiting stage, the framework uses a fast-fail pattern: if one process finishes, it kills all peer processes. E.g. in vLLM:
vllm/executor/multiproc_worker_utils.py:127:

 # Cleanup any remaining workers
            if logger:
                logger.info("Killing local vLLM worker processes")
            for worker in self.workers:
                worker.kill_worker()

This fix is using a POSIX semaphore to sync with peer processes, wait all peers finish and keep executing.

Added/updated tests?

• Yes
• No, Does not apply to this PR.

Updated CHANGELOG?

• Yes
• No, Does not apply to this PR.

Added/Updated documentation?

• Yes
• No, Does not apply to this PR.

[rocm-devops]

Manual review required for ce3efa
Assigned Auditors: @udaykikr

[rocm-devops]

Manual review required for 4794bd
Assigned Auditors: @yuxuanli

[rocm-devops]

My concerns lie in the while loop (e.g. is it possible for one process to get beyond this function but still exist when another process reads <ppid>/children -- resulting in an infinite loop) and whether it is possible that the parent PID could have unrelated child processes that have daemon-like runtimes -- resulting in a hang. At the very least, there needs to be a rocprofv3 option, e.g. --process-sync, which enables this. I am extremely hesitant to have this be the default and for it to be unconditional because apps do strange things.

I need some more time to review but first, the --process-sync option should be added

Thanks for the comments. The --process-sync is added and default is false (not sync).

[rocm-devops]

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

[jayhawk-commits]

Imported to ROCm/rocm-systems.