[rocpd] Schema Updates by systems-assistant[bot] · Pull Request #347 · ROCm/rocm-systems

systems-assistant · 2025-08-14T14:28:58Z

PR Details

Associated Jira Ticket Number/Link

What type of PR is this? (check all applicable)

Technical details

Added/updated tests?

Yes
No, Does not apply to this PR.

Updated CHANGELOG?

Yes
No, Does not apply to this PR.

Added/Updated documentation?

Yes
No, Does not apply to this PR.

🔁 Imported from ROCm/rocprofiler-sdk#118
🧑‍💻 Originally authored by @rocm-devops

bwelton

This may have been on the original PR, but can you add some context about why this is being changed and what the effect of the change is?

yhuiYH · 2025-09-17T22:36:20Z

+                for name in tbls:
+                    log(f"Inserting rows into {name} from {alias}.{name}")
+                    conn.execute(f'INSERT INTO "{name}" SELECT * FROM {alias}."{name}"')


Suggested change

for name in tbls:

log(f"Inserting rows into {name} from {alias}.{name}")

conn.execute(f'INSERT INTO "{name}" SELECT * FROM {alias}."{name}"')

for name in tbls:

log(f"Inserting rows into {name} from {alias}.{name}")

rows = conn.execute(f'SELECT * FROM {alias}."{name}"').fetchall()

if rows:

col_count = len(rows[0])

placeholders = ", ".join(["?"] * col_count)

conn.executemany(

f'INSERT INTO "{name}" VALUES ({placeholders})',

rows

)

Looking at merge.py, @bgopesh suggested I change this to a batch operation in my PR. Doing that in your implementation will halve the merge time! I'm looking at ways of incorporating your merge.py with the merge.py that the Silo team wrote in my PR. But this is a good change to pickup.

yhuiYH · 2025-09-17T22:43:16Z

+        # Optional: enforce integrity
+        try:
+            conn.execute("PRAGMA quick_check;")
+        except sqlite3.DatabaseError as e:
+            log(f"SQLite3 quick_check reported an issue: {e}")


This integrity check (I see you have it marked as optional here in the comments) will add a 66%-175% time hit to merge time. It may be better to skip it, or have it as a user-opt-in check (but what user will really want to turn this on?).
ie, merging 2 13MB DBs, merge time goes from 3.67 seconds to 1.33 seconds when we skip this check.
merging 16 13MB DBs, merge time goes from 37.03 seconds to 22.23 second when we skip this check.

- support multi-process stable folder naming convention

- add merge submodule - combine multiple databases into one - add functions submodule - defines (in Python) such as STDDEV_SAMP - add/update bindings submodule - support using rocpd without Python bindings - not all features are supported in pure python yet

- original schema is v3.0.0 - new schema is v4.0.0

- all python classes are CamelCase - fully implement bindings.py - fix issues with counter collection in csv.py - use lds_block_size instead of lds_size - support using rocpd in RocpdReader - handle .rpdb files in attachment tests - cleanup .rpdb files generated in tests - remove generation of database in tests/rocprofv3/counter-collection/list_metrics - fix test_csv_data to not ignore roctxMarkA

@abouteiller

* Change to enable ibgda bitcode compilation * Apply suggestion from @abouteiller --------- Co-authored-by: Aurelien Bouteiller <aurelien.bouteiller@amd.com>

@abouteiller

* Change to enable ibgda bitcode compilation * Apply suggestion from @abouteiller --------- Co-authored-by: Aurelien Bouteiller <aurelien.bouteiller@amd.com> [ROCm/rocshmem commit: fbe5730]

github-actions · 2026-05-14T17:36:30Z

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

Add new commits
Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.

yhuiYH · 2026-05-14T17:46:19Z

Do NOT close yet, we are still referencing this PR to move the necessary pieces into develop.

github-actions · 2026-06-09T00:40:40Z

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

Add new commits
Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.

yhuiYH · 2026-06-10T13:42:03Z

BUMP, still referencing this PR for schema updates.

systems-assistant Bot added the imported pr label Aug 14, 2025

systems-assistant Bot requested review from SrirakshaNag, ammarwa, bwelton, jrmadsen and t-tye as code owners August 14, 2025 14:29

ammarwa assigned jrmadsen Aug 14, 2025

github-actions Bot added the project: rocprofiler-sdk label Aug 14, 2025

systems-assistant Bot added the external contribution label Aug 14, 2025

jrmadsen force-pushed the import/develop/ROCm_rocprofiler-sdk/jomadsen_rocprofiler-sdk-rocpd-updates branch from c303613 to 7ddd5da Compare August 14, 2025 16:21

jrmadsen requested review from dmitrii-galantsev and oliveiradan as code owners August 14, 2025 16:21

github-actions Bot added project: rocm-smi-lib github actions Pull requests that update GitHub Actions code project: clr project: hipother labels Aug 14, 2025

jrmadsen force-pushed the import/develop/ROCm_rocprofiler-sdk/jomadsen_rocprofiler-sdk-rocpd-updates branch from 7ddd5da to c235cff Compare August 14, 2025 21:31

jayhawk-commits added organization: ROCm and removed project: rocm-smi-lib github actions Pull requests that update GitHub Actions code external contribution project: clr project: hipother labels Aug 19, 2025

jrmadsen force-pushed the import/develop/ROCm_rocprofiler-sdk/jomadsen_rocprofiler-sdk-rocpd-updates branch from c235cff to f5335e5 Compare August 22, 2025 18:05

jrmadsen requested a review from bgopesh as a code owner August 22, 2025 18:05

bwelton reviewed Aug 26, 2025

View reviewed changes

sadikarmagan assigned yhuiYH Sep 16, 2025

yhuiYH reviewed Sep 17, 2025

View reviewed changes

jrmadsen force-pushed the import/develop/ROCm_rocprofiler-sdk/jomadsen_rocprofiler-sdk-rocpd-updates branch 2 times, most recently from be09959 to f1b9797 Compare September 22, 2025 18:41

jrmadsen added 11 commits September 29, 2025 14:08

rocpd merge fixes (types.hpp + data_views.sql)

d2da1c7

[SEPARATE PR] C++ public API: get_{callback,buffer}_tracing_operations

427777e

rocprofv3 ROCpd output folder (.rpdb)

84075f5

- support multi-process stable folder naming convention

rocprofiler-sdk-rocpd schema versioning

a7de35e

- original schema is v3.0.0 - new schema is v4.0.0

Fix output_keys query for USER/LOGNAME

75af8c6

Fix bindings.py (thread -> Thread, etc.)

0ae1644

generateRocpd.cpp fixes after rebase

2d041a5

Fix C++ header include in sql.h

ba7b78b

Bump version to 1.1.0

da074da

jrmadsen force-pushed the import/develop/ROCm_rocprofiler-sdk/jomadsen_rocprofiler-sdk-rocpd-updates branch from b3840d6 to da074da Compare September 29, 2025 19:08

jrmadsen added the WIP label Sep 30, 2025

Write rocpd data via bind instead of stringify

cef953e

jayhawk-commits requested review from a team as code owners February 26, 2026 21:20

anujshuk-amd mentioned this pull request Apr 23, 2026

[rocpdsna] Add v4 schema support with versioned writer #5347

Closed

5 tasks

github-actions Bot added the Stale PR has no activity for 25+ days label May 14, 2026

github-actions Bot removed the Stale PR has no activity for 25+ days label May 15, 2026

github-actions Bot added the Stale PR has no activity for 25+ days label Jun 9, 2026

github-actions Bot removed the Stale PR has no activity for 25+ days label Jun 11, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rocpd] Schema Updates#347

[rocpd] Schema Updates#347
systems-assistant[bot] wants to merge 32 commits into
developfrom
import/develop/ROCm_rocprofiler-sdk/jomadsen_rocprofiler-sdk-rocpd-updates

systems-assistant Bot commented Aug 14, 2025

Uh oh!

bwelton left a comment

Uh oh!

yhuiYH Sep 17, 2025

Uh oh!

yhuiYH Sep 17, 2025

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

yhuiYH commented May 14, 2026

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

yhuiYH commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

-                for name in tbls:
-                    log(f"Inserting rows into {name} from {alias}.{name}")
-                    conn.execute(f'INSERT INTO "{name}" SELECT * FROM {alias}."{name}"')
+                for name in tbls:
+                    log(f"Inserting rows into {name} from {alias}.{name}")
+                    rows = conn.execute(f'SELECT * FROM {alias}."{name}"').fetchall()
+                    if rows:
+                        col_count = len(rows[0])
+                        placeholders = ", ".join(["?"] * col_count)
+                        conn.executemany(
+                            f'INSERT INTO "{name}" VALUES ({placeholders})',
+                            rows
+                        )

Conversation

systems-assistant Bot commented Aug 14, 2025

PR Details

Associated Jira Ticket Number/Link

What type of PR is this? (check all applicable)

Technical details

Added/updated tests?

Updated CHANGELOG?

Added/Updated documentation?

Uh oh!

bwelton left a comment

Choose a reason for hiding this comment

Uh oh!

yhuiYH Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

yhuiYH Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 14, 2026

Uh oh!

yhuiYH commented May 14, 2026

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

yhuiYH commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants