[rocpd] Adding merge and package submodules for rocpd by systems-assistant[bot] · Pull Request #164 · ROCm/rocm-systems

systems-assistant · 2025-08-07T19:16:11Z

PR Details

Add merge.py and package.py python modules for rocpd to try to reduce the number of databases to manage, as well as consolidating the databases into a folder, to move around easier.

Associated Jira Ticket Number/Link

SWDEV-547026

What type of PR is this? (check all applicable)

Technical details

Adding ability to merge multiple databases into one.
Defines a new .rpdb folder aka package, which contains an index.yaml metadatafile.
Creation of package
- default, create index.yaml in current location that simply references where the files relative to current folder (use case: don't need to copy large files, just reference them where they are)
- If --consolidate is requested, creates a new unique folder and copies the DBs into that folder/package.
- Default is to MOVE DBs when --consolidate is passed in, provided user with a --copy option to allow user to keep original data unmodified.
Use of package
- Provides function to parse/flatten the yaml file back to the list of database files
- When flattening yaml file, if the number of databases provided exceed IDEAL_NUMBER_OF_DATABASE_FILES (currently 5), auto merge them. UPDATE: Jonathan wants to always auto-merge to 1 DB. So we created ability for user to pass in parameter --automerge-limit to raise this limit.
- Auto-merge will create new package/folder and merge the DBs into suitable number, and inform the user DBs were auto-merged and provide them with new DB path they CAN use
- Auto-merge is disabled for merge and package submodules if the user is using it directly, because they often don't want auto-merge algorithm interfering with what they want to do.

Added/updated tests?

Yes
No, Does not apply to this PR.

Updated CHANGELOG?

Yes
No, Does not apply to this PR.

Added/Updated documentation?

Yes
No, Does not apply to this PR.

🔁 Imported from ROCm/rocprofiler-sdk#106
🧑‍💻 Originally authored by @rocm-devops

use internal cluster for ci [ROCm/aqlprofile commit: fdedcfc]

) [ROCm/rocprofiler-systems commit: 807a622]

ammarwa · 2025-08-16T02:41:39Z

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

[ROCm/clr commit: e92ea15]

…erge when using sub-commands merge & package

- Improve handling more corner cases for bad input files when parsing input parameters (bad yaml files, bad .rpdb folder, folders as input) - Changed to use UUID in merged filename instead of the time, in auto-merge algorithm

…rams to a function

…, recreating indexes, foreign key check (disabled for now, due to processing time)

- add_args function returns a functor which accepts input and args - time_window functor returned from add_args automatically applies time windowing of input

- copying can be enabled by passing "copy=True" or --copy cmdline argument

… 1. Tests updated to use query, much quicker

* adding ROCpd database merge * adding ROCpd database merge concatenating all tables * update merge script - copy all tables from files * fix merge format * Add package submodule, initial POC. Need to refine * Minor fixes and clean up duplicated code in package.py * Revamp metadata layout, add wildcard and .rpdb parsing * Add auto merge & package when > 5 DBs, add examples, don't use auto_merge when using sub-commands merge & package * - Extend package/yaml inputs to all rocpd modules - Improve handling more corner cases for bad input files when parsing input parameters (bad yaml files, bad .rpdb folder, folders as input) - Changed to use UUID in merged filename instead of the time, in auto-merge algorithm * Minor text fixes for consistancy between modules * Add more wildcard support and add package, merge tests * Make changes based on review suggestions * Move parsing packages into importer.py, simplified adding required params to a function * fix package test by flattening input list before processing * Integrate merge.py changes from Jonathan to add name-collision checks, recreating indexes, foreign key check (disabled for now, due to processing time) * Rework rocpd.<submodule>.{add_args,process_args} - add_args function returns a functor which accepts input and args - time_window functor returned from add_args automatically applies time windowing of input * change merge&package limit to 1, merge should create data views * Move files by default instead of making copies - copying can be enabled by passing "copy=True" or --copy cmdline argument * refactor package to make the logic cleaner, set merge limit back to 5 * Allow automerge-limit param to override limit, change default back to 1. Tests updated to use query, much quicker * Update --help instructions for package [rocm-systems] ROCm/rocm-systems#164 (commit 061948a)

* rocjpeg - CMake Find updates * Version file - fix * Changelog - updates * Cleanup * Test - cmakelists updates * header - minor fix --------- Co-authored-by: Kiriti <kiriti@santiago.amd.com>

* rocjpeg - CMake Find updates * Version file - fix * Changelog - updates * Cleanup * Test - cmakelists updates * header - minor fix --------- Co-authored-by: Kiriti <kiriti@santiago.amd.com> [ROCm/rocjpeg commit: bdb12b5]

Updates ROCm version used in the CI container to 7.1.1 AMDGPU repo removed as we don't need any packages from that repository (mostly kernel packages).

Updates ROCm version used in the CI container to 7.1.1 AMDGPU repo removed as we don't need any packages from that repository (mostly kernel packages). [ROCm/hipFile commit: 05eff1c]

systems-assistant Bot requested a review from jrmadsen as a code owner August 7, 2025 19:16

systems-assistant Bot added the imported pr label Aug 7, 2025

systems-assistant Bot requested a review from t-tye as a code owner August 7, 2025 19:16

systems-assistant Bot added the project: rocprofiler-sdk label Aug 7, 2025

jayhawk-commits assigned yhuiYH Aug 7, 2025

jayhawk-commits added the organization: ROCm label Aug 7, 2025

ammallya pushed a commit that referenced this pull request Aug 7, 2025

[CI] use internal cluster for ci (#164)

126e461

use internal cluster for ci [ROCm/aqlprofile commit: fdedcfc]

jayhawk-commits added rocpd WIP labels Aug 7, 2025

ywang103-amd pushed a commit to ywang103-amd/rocm-systems that referenced this pull request Aug 7, 2025

Change the default value of ROCPROFSYS_SAMPLING_CPUS to "none" (ROCm#164

36f7de2

) [ROCm/rocprofiler-systems commit: 807a622]

systems-assistant Bot pushed a commit that referenced this pull request Aug 10, 2025

SWDEV-518317 - Remove Redundant Error Message in removeFatBinary (#164)

e92ea15

yhuiYH marked this pull request as draft August 13, 2025 17:52

jayhawk-commits pushed a commit that referenced this pull request Aug 18, 2025

SWDEV-518317 - Remove Redundant Error Message in removeFatBinary (#164)

9c1d61f

[ROCm/clr commit: e92ea15]

yhuiYH force-pushed the import/develop/ROCm_rocprofiler-sdk/yhui_rocpd-merge-and-package branch 2 times, most recently from 6612e41 to 27edabf Compare August 26, 2025 05:53

yhuiYH added rocprofiler-sdk: Ready for 1st code review and removed WIP labels Aug 26, 2025

yhuiYH assigned amd-atumakae, mcrnobrn-amd and bgopesh and unassigned yhuiYH Aug 26, 2025

yhuiYH marked this pull request as ready for review August 26, 2025 05:56

bgopesh reviewed Aug 26, 2025

View reviewed changes

Comment thread projects/rocprofiler-sdk/source/lib/python/rocpd/merge.py Outdated

Comment thread projects/rocprofiler-sdk/source/lib/python/rocpd/package.py

Comment thread projects/rocprofiler-sdk/source/lib/python/rocpd/merge.py Outdated

bgopesh approved these changes Aug 29, 2025

View reviewed changes

bgopesh assigned jrmadsen Aug 29, 2025

bgopesh added rocprofiler-sdk: Ready for 2nd code review and removed rocprofiler-sdk: Ready for 1st code review labels Aug 29, 2025

bgopesh force-pushed the import/develop/ROCm_rocprofiler-sdk/yhui_rocpd-merge-and-package branch from 3203549 to 58fff90 Compare August 29, 2025 04:48

yhuiYH force-pushed the import/develop/ROCm_rocprofiler-sdk/yhui_rocpd-merge-and-package branch from 0981616 to 2264461 Compare September 9, 2025 00:48

yhuiYH and others added 14 commits November 10, 2025 12:15

Minor fixes and clean up duplicated code in package.py

672a3ae

Revamp metadata layout, add wildcard and .rpdb parsing

069d6f3

Add auto merge & package when > 5 DBs, add examples, don't use auto_m…

8f87f67

…erge when using sub-commands merge & package

- Extend package/yaml inputs to all rocpd modules

a344a69

- Improve handling more corner cases for bad input files when parsing input parameters (bad yaml files, bad .rpdb folder, folders as input) - Changed to use UUID in merged filename instead of the time, in auto-merge algorithm

Minor text fixes for consistancy between modules

34e4fd2

Add more wildcard support and add package, merge tests

df3e95a

Make changes based on review suggestions

1b0a897

Move parsing packages into importer.py, simplified adding required pa…

25e8c2e

…rams to a function

fix package test by flattening input list before processing

f93922f

Integrate merge.py changes from Jonathan to add name-collision checks…

cc2c17a

…, recreating indexes, foreign key check (disabled for now, due to processing time)

Rework rocpd.<submodule>.{add_args,process_args}

20d5cf4

- add_args function returns a functor which accepts input and args - time_window functor returned from add_args automatically applies time windowing of input

change merge&package limit to 1, merge should create data views

2b41546

Move files by default instead of making copies

be9d13d

- copying can be enabled by passing "copy=True" or --copy cmdline argument

refactor package to make the logic cleaner, set merge limit back to 5

3c72c4c

jrmadsen force-pushed the import/develop/ROCm_rocprofiler-sdk/yhui_rocpd-merge-and-package branch from 26181a6 to 3c72c4c Compare November 10, 2025 18:15

Allow automerge-limit param to override limit, change default back to…

73b6a0d

… 1. Tests updated to use query, much quicker

yhuiYH removed the WIP label Nov 11, 2025

Update --help instructions for package

f428cc9

yhuiYH added ready for 2nd review rocprofiler-sdk: Ready for 2nd code review labels Nov 11, 2025

bwelton self-assigned this Nov 11, 2025

amd-atumakae self-requested a review November 12, 2025 18:35

amd-atumakae approved these changes Nov 12, 2025

View reviewed changes

yhuiYH merged commit 061948a into develop Nov 12, 2025
28 of 43 checks passed

yhuiYH deleted the import/develop/ROCm_rocprofiler-sdk/yhui_rocpd-merge-and-package branch November 12, 2025 22:07

ammallya pushed a commit that referenced this pull request Apr 9, 2026

CI: Update CI images to use ROCm 7.1.1 (#164)

05eff1c

Updates ROCm version used in the CI container to 7.1.1 AMDGPU repo removed as we don't need any packages from that repository (mostly kernel packages).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rocpd] Adding merge and package submodules for rocpd#164

[rocpd] Adding merge and package submodules for rocpd#164
yhuiYH merged 21 commits into
developfrom
import/develop/ROCm_rocprofiler-sdk/yhui_rocpd-merge-and-package

systems-assistant Bot commented Aug 7, 2025 •

edited by yhuiYH

Loading

Uh oh!

ammarwa commented Aug 16, 2025

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Conversation

systems-assistant Bot commented Aug 7, 2025 • edited by yhuiYH Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Details

Associated Jira Ticket Number/Link

What type of PR is this? (check all applicable)

Technical details

Added/updated tests?

Updated CHANGELOG?

Added/Updated documentation?

Uh oh!

ammarwa commented Aug 16, 2025

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

systems-assistant Bot commented Aug 7, 2025 •

edited by yhuiYH

Loading