Bake provenance JSON into images and auto-populate post_results.py args#338
Open
misiugodfrey wants to merge 2 commits into
Open
Bake provenance JSON into images and auto-populate post_results.py args#338misiugodfrey wants to merge 2 commits into
misiugodfrey wants to merge 2 commits into
Conversation
Writes /opt/velox-testing/provenance.json into each Presto image at build time (both native_build.dockerfile and provenance_labels.dockerfile), so run_context.py can read it from the container filesystem in both Docker and SLURM/Enroot environments. run_context.py: reads the provenance file and merges presto_sha/branch/repo and velox_sha/branch/repo into the benchmark context dict. post_results.py: auto-populates --velox-branch, --velox-repo, --presto-branch, --presto-repo from the benchmark_result.json context section; CLI args still take precedence. BenchmarkMetadata gains 6 optional provenance fields. presto-build.yml: replaces the labels: input on the deps and coordinator build steps with a second provenance_labels.dockerfile wrapper step, so CI and local builds both write the provenance file (buildx labels: applies OCI metadata only and does not execute RUN steps).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/opt/velox-testing/provenance.jsoninto each Presto image at build time (bothnative_build.dockerfileandprovenance_labels.dockerfile), sorun_context.pycan read it from the container filesystem in both Docker and SLURM/Enroot environments (Enroot.sqshimages cannot expose OCI metadata like labels, but the baked-in file is accessible via the normal filesystem)run_context.pyreads the provenance file and mergespresto_sha/branch/repoandvelox_sha/branch/repointo the benchmark context dict written tobenchmark_result.jsonpost_results.pyauto-populates--velox-branch,--velox-repo,--presto-branch,--presto-repofrom thebenchmark_result.jsoncontext; CLI args still take precedence; older result files without provenance fields are handled gracefullypresto-build.yml: replaces thelabels:input on the deps and coordinator build steps with a secondprovenance_labels.dockerfilewrapper step — buildxlabels:applies OCI metadata only and does not executeRUNsteps, so the provenance file would not have been written via that pathTest plan
presto/scripts/start_native_cpu_presto.shdocker run --rm presto-native-worker-cpu:$USER cat /opt/velox-testing/provenance.jsonpresto/scripts/run_benchmark.sh -b tpch -s bench_sf1benchmark_result.jsoncontext hasvelox_branch,velox_repo,presto_branch,presto_repopopulatedpost_results.py --dry-runwithout branch/repo args; confirm payloadengine_configshows values auto-populated from the result file--velox-branch override; confirm CLI value wins over the labelengine_configwithout errorpresto-build.yml; pull the resulting image into a SLURM/Enroot context and run a quick benchmark to confirm provenance fields are populated via the filesystem path