[rocpd] Refactor to use python to convert rocpd to CSV + add CSV tests + remove old cpp implementation by systems-assistant[bot] · Pull Request #159 · ROCm/rocm-systems

systems-assistant · 2025-08-07T19:15:14Z

PR Details

Associated Jira Ticket Number/Link

https://ontrack-internal.amd.com/browse/SWDEV-543953

What type of PR is this? (check all applicable)

Technical details

Added/updated tests?

Yes
No, Does not apply to this PR.

Updated CHANGELOG?

Yes
No, Does not apply to this PR.

Added/Updated documentation?

Yes
No, Does not apply to this PR.

🔁 Imported from ROCm/rocprofiler-sdk#102
🧑‍💻 Originally authored by @rocm-devops

* Adding Roctx usage doc * updated CHANGELOG * dpc update * Fixing Related Pages issue * updating doc * updating docs * Adding Resource naming section * Fixed Formatting * format fix * format fix * Fixing build due to incorrect indentation [ROCm/rocprofiler-sdk commit: 12508b9]

…inx (ROCm#159) Bumps [rocm-docs-core[api_reference]](https://github.com/ROCm/rocm-docs-core) from 1.18.1 to 1.18.2. - [Release notes](https://github.com/ROCm/rocm-docs-core/releases) - [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md) - [Commits](ROCm/rocm-docs-core@v1.18.1...v1.18.2) --- updated-dependencies: - dependency-name: rocm-docs-core[api_reference] dependency-version: 1.18.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [ROCm/rocprofiler-systems commit: 7f2b198]

…e new API (#159)

…evice Heap kernels (#159)

ammarwa · 2025-08-12T11:15:07Z

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Copilot

Pull Request Overview

This pull request refactors the rocpd CSV generation functionality from C++ to Python, adding comprehensive CSV testing and removing the old C++ implementation. The refactor aims to simplify the codebase by leveraging Python for CSV generation while maintaining the same output format.

Key Changes:

Replaced C++ CSV generation with Python implementation using SQL queries
Added comprehensive CSV validation tests that compare CSV output with JSON data
Removed old C++ CSV-related files and CMake configurations

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
projects/rocprofiler-sdk/tests/rocprofv3/rocpd/validate.py	Added test_csv_data function for validating CSV output against JSON data
projects/rocprofiler-sdk/tests/rocprofv3/rocpd/conftest.py	Added csv_data fixture to handle CSV input files for testing
projects/rocprofiler-sdk/tests/rocprofv3/rocpd/CMakeLists.txt	Updated test command to include CSV files in validation
projects/rocprofiler-sdk/tests/pytest-packages/tests/rocprofv3.py	Added comprehensive CSV validation logic with time sanity checks and data matching
projects/rocprofiler-sdk/source/lib/python/rocpd/source/csv.hpp	Removed C++ CSV header file
projects/rocprofiler-sdk/source/lib/python/rocpd/source/csv.cpp	Removed C++ CSV implementation file
projects/rocprofiler-sdk/source/lib/python/rocpd/source/CMakeLists.txt	Updated source lists to exclude CSV C++ files
projects/rocprofiler-sdk/source/lib/python/rocpd/libpyrocpd.cpp	Removed C++ write_csv function binding
projects/rocprofiler-sdk/source/lib/python/rocpd/csv.py	Added complete Python CSV generation implementation with SQL-based queries

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

…e new API (#159) [ROCm/clr commit: b006380]

…evice Heap kernels (#159) [ROCm/hip-tests commit: 327edf9]

…kae_csv_python_writer

CSV tests + remove old cpp implementation (#159) * Write agent info to CSV * Write kernel to CSV * Write memory copy to CSV * Write memory allocation to CSV * Write hip api to CSV * Write hsa api to CSV * Write marker api to CSV * Write counters to CSV * Write scratch memory to CSV * Write rccl api to CSV * Write rocdecode api to CSV * Write rocjpeg api to CSV * Remove info_process joins * Format agent id * Compose full file name is sql writer function * Add missing fields to kernel traces csv * Rename vgpr_count to arch_vgpr_count * Fix kernel name * Skip empty query results * Format csv.py * Delete c++ CSV writer * Add CSV header comparison test * Fix comment spacing in csv.py * Change ALLOC to ALLOCATE in memory allocation writer * Do not append trace to agent info file name * Revert changes for VGPR_Count * Fix csv validation test * Add sorting by guid * Use EXISTS to check query results are not empty * Merge API-specific queries * Optimize regions query * Column name mapping for agent info * Pass config to sql writer * Move agent id string building to a separate function * add titled_headers argument * Remove titled-columns argument * Improvements for regions csv * fix CSV validation test * improve CSV validation test * remove roctxMarkA from csv validation test * fix capability field titles in agent info * remove filter.py from query as that is still experimental * Remove some aliases, now that query will auto-title the column headers [rocm-systems] ROCm/rocm-systems#159 (commit 3f001b0)

…s + remove old cpp implementation (#159) * Write agent info to CSV * Write kernel to CSV * Write memory copy to CSV * Write memory allocation to CSV * Write hip api to CSV * Write hsa api to CSV * Write marker api to CSV * Write counters to CSV * Write scratch memory to CSV * Write rccl api to CSV * Write rocdecode api to CSV * Write rocjpeg api to CSV * Remove info_process joins * Format agent id * Compose full file name is sql writer function * Add missing fields to kernel traces csv * Rename vgpr_count to arch_vgpr_count * Fix kernel name * Skip empty query results * Format csv.py * Delete c++ CSV writer * Add CSV header comparison test * Fix comment spacing in csv.py * Change ALLOC to ALLOCATE in memory allocation writer * Do not append trace to agent info file name * Revert changes for VGPR_Count * Fix csv validation test * Add sorting by guid * Use EXISTS to check query results are not empty * Merge API-specific queries * Optimize regions query * Column name mapping for agent info * Pass config to sql writer * Move agent id string building to a separate function * add titled_headers argument * Remove titled-columns argument * Improvements for regions csv * fix CSV validation test * improve CSV validation test * remove roctxMarkA from csv validation test * fix capability field titles in agent info * remove filter.py from query as that is still experimental * Remove some aliases, now that query will auto-title the column headers --------- Co-authored-by: Aleksei Tumakaev <atumakae@amd.com> Co-authored-by: Young Hui <young.hui@amd.com> Co-authored-by: Young Hui - AMD <145490163+yhuiYH@users.noreply.github.com>

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com> [ROCm/amdsmi commit: de6f49b]

* Refactor `Barrier_all` and `Sync_all` to use default context - Removed context-specific implementations of barrier_all and sync_all - Added barrier_all and sync_all to the default context implementation - Updated functional tests to use the default context for barrier_all and sync_all * Update `Barrier_all` and `Sync_all` API usage in documentation * Update `CHANGELOG` --------- Co-authored-by: Yiltan <ytemucin@amd.com>

* Refactor `Barrier_all` and `Sync_all` to use default context - Removed context-specific implementations of barrier_all and sync_all - Added barrier_all and sync_all to the default context implementation - Updated functional tests to use the default context for barrier_all and sync_all * Update `Barrier_all` and `Sync_all` API usage in documentation * Update `CHANGELOG` --------- Co-authored-by: Yiltan <ytemucin@amd.com> [ROCm/rocshmem commit: bf48bca]

…159) * Improve the memory pool logic to enable the reuse of VA-API surfaces. * Update the version

…159) * Improve the memory pool logic to enable the reuse of VA-API surfaces. * Update the version [ROCm/rocjpeg commit: 6046873]

[ROCm/hipFile commit: 302b5d0]

systems-assistant Bot added the imported pr label Aug 7, 2025

systems-assistant Bot requested review from jrmadsen and t-tye as code owners August 7, 2025 19:15

systems-assistant Bot added the project: rocprofiler-sdk label Aug 7, 2025

jayhawk-commits assigned amd-atumakae Aug 7, 2025

jayhawk-commits added the organization: ROCm label Aug 7, 2025

jayhawk-commits added rocpd WIP labels Aug 7, 2025

systems-assistant Bot pushed a commit that referenced this pull request Aug 10, 2025

SWDEV-525653 - Make hipGetDeviceProperties and hipChooseDevice use th…

b006380

…e new API (#159)

systems-assistant Bot pushed a commit that referenced this pull request Aug 10, 2025

SWDEV-513197-[catch2][dtest]PerfTest-Improve launch performance for D…

327edf9

…evice Heap kernels (#159)

amd-atumakae force-pushed the import/develop/ROCm_rocprofiler-sdk/atumakae_csv_python_writer branch from 547b8cf to 14fd46e Compare August 12, 2025 10:50

amd-atumakae requested a review from yhuiYH August 12, 2025 14:05

yhuiYH reviewed Aug 12, 2025

View reviewed changes

yhuiYH reviewed Aug 13, 2025

View reviewed changes

Comment thread projects/rocprofiler-sdk/source/lib/python/rocpd/csv.py Outdated

yhuiYH added ready for 2nd review and removed WIP labels Aug 13, 2025

yhuiYH changed the title ~~[rocpd] CSV python writer~~ [rocpd] Refactor to use python to convert rocpd to CSV + add CSV tests + remove old cpp implementation Aug 13, 2025

yhuiYH previously approved these changes Aug 13, 2025

View reviewed changes

yhuiYH assigned jrmadsen Aug 13, 2025

jayhawk-commits requested a review from Copilot August 13, 2025 22:14

Copilot AI reviewed Aug 13, 2025

View reviewed changes

jrmadsen requested changes Aug 13, 2025

View reviewed changes

Comment thread projects/rocprofiler-sdk/source/lib/python/rocpd/csv.py

Comment thread projects/rocprofiler-sdk/source/lib/python/rocpd/csv.py Outdated

Comment thread projects/rocprofiler-sdk/tests/rocprofv3/rocpd/CMakeLists.txt Outdated

amd-atumakae force-pushed the import/develop/ROCm_rocprofiler-sdk/atumakae_csv_python_writer branch from 36bfe80 to ae50865 Compare August 14, 2025 18:45

yhuiYH reviewed Aug 14, 2025

View reviewed changes

Comment thread projects/rocprofiler-sdk/source/lib/python/rocpd/csv.py Outdated

amd-atumakae requested a review from jrmadsen August 15, 2025 12:20

jayhawk-commits pushed a commit that referenced this pull request Aug 18, 2025

SWDEV-525653 - Make hipGetDeviceProperties and hipChooseDevice use th…

d427574

…e new API (#159) [ROCm/clr commit: b006380]

jayhawk-commits pushed a commit that referenced this pull request Aug 18, 2025

SWDEV-513197-[catch2][dtest]PerfTest-Improve launch performance for D…

ff14fb3

…evice Heap kernels (#159) [ROCm/hip-tests commit: 327edf9]

amd-atumakae and others added 13 commits September 16, 2025 11:41

Optimize regions query

7537508

Column name mapping for agent info

25c2030

Pass config to sql writer

2ff015d

Move agent id string building to a separate function

b816cf9

add titled_headers argument

bcaae83

Remove titled-columns argument

9ba6370

Improvements for regions csv

ed5c4d1

fix CSV validation test

90bcbe0

improve CSV validation test

698841a

remove roctxMarkA from csv validation test

16a6d0a

fix capability field titles in agent info

4c016d5

remove filter.py from query as that is still experimental

a5bed0b

Remove some aliases, now that query will auto-title the column headers

c324883

jrmadsen force-pushed the import/develop/ROCm_rocprofiler-sdk/atumakae_csv_python_writer branch from 73dba93 to c324883 Compare September 16, 2025 16:41

yhuiYH added ready to merge and removed ready for 2nd review labels Sep 16, 2025

Merge branch 'develop' into import/develop/ROCm_rocprofiler-sdk/atuma…

0947c7e

…kae_csv_python_writer

yhuiYH merged commit 3f001b0 into develop Sep 19, 2025
28 of 39 checks passed

yhuiYH deleted the import/develop/ROCm_rocprofiler-sdk/atumakae_csv_python_writer branch September 19, 2025 14:15

ammallya pushed a commit that referenced this pull request Nov 21, 2025

Merge amd-staging into amd-mainline 20250307 (#159)

de6f49b

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>

ammallya pushed a commit that referenced this pull request Nov 21, 2025

Merge amd-staging into amd-mainline 20250307 (#159)

0f8bf24

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com> [ROCm/amdsmi commit: de6f49b]

ammallya pushed a commit that referenced this pull request Jan 30, 2026

Improve the memory pool logic to enable the reuse of VA-API surfaces (#…

6046873

…159) * Improve the memory pool logic to enable the reuse of VA-API surfaces. * Update the version

ammallya pushed a commit that referenced this pull request Jan 30, 2026

Improve the memory pool logic to enable the reuse of VA-API surfaces (#…

3b10d19

…159) * Improve the memory pool logic to enable the reuse of VA-API surfaces. * Update the version [ROCm/rocjpeg commit: 6046873]

ammallya pushed a commit that referenced this pull request Apr 9, 2026

docs: Document how to use fio with hipfile (#159)

302b5d0

ammallya pushed a commit that referenced this pull request Apr 9, 2026

docs: Document how to use fio with hipfile (#159)

a622bab

[ROCm/hipFile commit: 302b5d0]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rocpd] Refactor to use python to convert rocpd to CSV + add CSV tests + remove old cpp implementation#159

[rocpd] Refactor to use python to convert rocpd to CSV + add CSV tests + remove old cpp implementation#159
yhuiYH merged 44 commits into
developfrom
import/develop/ROCm_rocprofiler-sdk/atumakae_csv_python_writer

systems-assistant Bot commented Aug 7, 2025

Uh oh!

ammarwa commented Aug 12, 2025 •

edited

Loading

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

systems-assistant Bot commented Aug 7, 2025

PR Details

Associated Jira Ticket Number/Link

What type of PR is this? (check all applicable)

Technical details

Added/updated tests?

Updated CHANGELOG?

Added/Updated documentation?

Uh oh!

ammarwa commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes:

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ammarwa commented Aug 12, 2025 •

edited

Loading