CI/CD - Fix Image build for cuda11.1.1 and py unit test pipelines #771

polarG · 2026-01-06T22:55:48Z

Description

When building the CUDA 11.1.1 image, pip (Python 3.8) cannot find a pre-built wheel for the latest wandb release (v0.23.1). As a result, pip attempts to build wandb from source. However, the build fails because the image does not have Go installed, which is required for building wandb from source. Then the error appears.
Deprecated API arguments, 'interpolation', when numpy >v1.22.0. And it was removed since v2.4.0.

Solution

For the CUDA 11.1.1 build, install the required build tools (e.g., Go, Rust, and Cargo) needed for wandb.
Update the NumPy API usage to support NumPy 1.22.0+ while maintaining backward compatibility with earlier versions.

codecov · 2026-01-06T23:42:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.69%. Comparing base (c99380b) to head (c4213bf).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #771   +/-   ##
=======================================
  Coverage   85.69%   85.69%           
=======================================
  Files         102      102           
  Lines        7699     7699           
=======================================
  Hits         6598     6598           
  Misses       1101     1101

Flag	Coverage Δ
cpu-python3.10-unit-test	`70.97% <ø> (ø)`
cpu-python3.7-unit-test	`70.42% <ø> (ø)`
cuda-unit-test	`83.61% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

superbench/benchmarks/base.py

abuccts

pip (Python 3.8) cannot find a pre-built wheel for the latest wandb release (v0.23.1)

According to wandb v0.23.1, it's compatible with py3.8. Maybe you need to upgrade pip/wheel?

However, the build fails because the image does not have Go installed, which is required for building wandb from source.

Why not just add an apt-get install -y golang?

polarG · 2026-01-12T22:27:18Z

pip (Python 3.8) cannot find a pre-built wheel for the latest wandb release (v0.23.1)

According to wandb v0.23.1, it's compatible with py3.8. Maybe you need to upgrade pip/wheel?

However, the build fails because the image does not have Go installed, which is required for building wandb from source.

Why not just add an apt-get install -y golang?

Added build-essential packages to the CUDA 11.1.1 Dockerfile to enable building wandb.

This reverts commit 059daa9. modified: .github/workflows/build-image.yml new file: dockerfile/cuda11.1.1.dockerfile

This reverts commit 4f23173. modified: dockerfile/cuda11.1.1.dockerfile

…using version control.

dockerfile/cuda11.1.1.dockerfile

superbench/benchmarks/base.py

Copilot

Pull request overview

This PR fixes CI/CD issues affecting the CUDA 11.1.1 Docker image build and Python unit test pipelines by addressing two main problems: missing build dependencies for wandb installation and deprecated NumPy API usage.

Changes:

Added Go and Rust build tools to the CUDA 11.1.1 Dockerfile to support wandb v0.23.1 source builds
Updated NumPy percentile API usage to support NumPy 2.x while maintaining backward compatibility with 1.x

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
superbench/benchmarks/base.py	Updated `np.percentile` API to use try-except for backward compatibility, preferring `method='nearest'` for NumPy 2.x and falling back to `interpolation='nearest'` for older versions
dockerfile/cuda11.1.1.dockerfile	Added Go 1.24, Rust, and Cargo installations with required dependencies to enable wandb source builds

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

superbench/benchmarks/base.py

polarG · 2026-01-14T23:57:37Z

pip (Python 3.8) cannot find a pre-built wheel for the latest wandb release (v0.23.1)

According to wandb v0.23.1, it's compatible with py3.8. Maybe you need to upgrade pip/wheel?

However, the build fails because the image does not have Go installed, which is required for building wandb from source.

Why not just add an apt-get install -y golang?

The default go version is not enough for wandb-0.23.1.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dockerfile/cuda11.1.1.dockerfile

abuccts

Can you update the title and descriptions accordingly?

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dockerfile/cuda11.1.1.dockerfile

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-22T05:10:05Z

dockerfile/cuda11.1.1.dockerfile

 WORKDIR ${SB_HOME}

+# Install Rust for wandb build (required by megatron_lm target)
+RUN curl https://sh.rustup.rs -sSf | sh -s -- -y


Installing Rust via curl https://sh.rustup.rs | sh is an unverified remote script execution pattern. To reduce supply-chain risk and improve reproducibility, prefer installing rustc/cargo from apt (if available) or pin the rustup/toolchain version and verify the installer (e.g., checksum/signature) before executing it.

dockerfile/cuda11.1.1.dockerfile

Copilot · 2026-01-22T05:10:06Z

dockerfile/cuda11.1.1.dockerfile

    && \
+    add-apt-repository -y ppa:longsleep/golang-backports && \
+    apt-get update && \
+    apt-get install -y golang-1.24-go=1.24* && \


Adding the longsleep/golang-backports PPA introduces an external repository dependency that can make builds less reproducible/available over time. If possible, prefer an official Go tarball with a pinned checksum (or another pinned source) to reduce CI flakiness from PPA changes/outages.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Anchor wandb version when python is 3.8.

05a89a6

polarG self-assigned this Jan 6, 2026

polarG added the enhancement New feature or request label Jan 6, 2026

polarG requested a review from a team as a code owner January 6, 2026 22:55

polarG added CI/CD Continuous integration or deployment dependencies Pull requests that update a dependency file labels Jan 6, 2026

Hongtao Zhang added 3 commits January 7, 2026 01:57

Adopt new api.

af676fc

Fix lint issues.

5a4d01e

Fix lint issues.

22fd5f5

polarG changed the title ~~CI/CD - Fix Image build for cuda11.1.1~~ CI/CD - Fix Image build for cuda11.1.1 and update numpy API Jan 7, 2026

polarG requested review from abuccts and guoshzhao January 7, 2026 18:33

guoshzhao reviewed Jan 9, 2026

View reviewed changes

superbench/benchmarks/base.py Outdated Show resolved Hide resolved

abuccts reviewed Jan 9, 2026

View reviewed changes

Hongtao Zhang added 2 commits January 12, 2026 21:25

Revert wandb changes.

0080d92

Remove cuda11.1.1.

059daa9

polarG changed the title ~~CI/CD - Fix Image build for cuda11.1.1 and update numpy API~~ CI/CD - Remove Image build for cuda11.1.1 and update numpy API Jan 12, 2026

polarG enabled auto-merge (squash) January 12, 2026 22:25

Hongtao Zhang added 6 commits January 13, 2026 18:08

Revert "Remove cuda11.1.1."

7f6de0c

This reverts commit 059daa9. modified: .github/workflows/build-image.yml new file: dockerfile/cuda11.1.1.dockerfile

Install go in cuda11.1.1 dockerfile.

4f23173

Revert "Install go in cuda11.1.1 dockerfile."

5bf14d4

This reverts commit 4f23173. modified: dockerfile/cuda11.1.1.dockerfile

Anchor wandb to 0.22.3 when python < 3.12.

0687eec

Add wandb version control for python >= 3.12.

ae05143

Add build dependencies of wandb in 11.1.1 to build wandb, instead of …

1ecbcda

…using version control.

polarG changed the title ~~CI/CD - Remove Image build for cuda11.1.1 and update numpy API~~ CI/CD - Fix Image build for cuda11.1.1 and unit test pipelines Jan 14, 2026

polarG changed the title ~~CI/CD - Fix Image build for cuda11.1.1 and unit test pipelines~~ CI/CD - Fix Image build for cuda11.1.1 and py unit test pipelines Jan 14, 2026

abuccts reviewed Jan 14, 2026

View reviewed changes

dockerfile/cuda11.1.1.dockerfile Outdated Show resolved Hide resolved

dockerfile/cuda11.1.1.dockerfile Outdated Show resolved Hide resolved

dockerfile/cuda11.1.1.dockerfile Outdated Show resolved Hide resolved

superbench/benchmarks/base.py Outdated Show resolved Hide resolved

abuccts requested a review from Copilot January 14, 2026 23:30

Copilot started reviewing on behalf of abuccts January 14, 2026 23:31 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

superbench/benchmarks/base.py Outdated Show resolved Hide resolved

Hongtao Zhang added 2 commits January 15, 2026 00:10

Fix comments.

8e8d286

Fix comments.

4d7d6c4

Copilot AI review requested due to automatic review settings January 21, 2026 23:08

Copilot started reviewing on behalf of polarG January 21, 2026 23:09 View session

Copilot AI reviewed Jan 21, 2026

View reviewed changes

abuccts approved these changes Jan 21, 2026

View reviewed changes

dockerfile/cuda11.1.1.dockerfile Outdated Show resolved Hide resolved

abuccts reviewed Jan 21, 2026

View reviewed changes

Fix comments.

82734d4

polarG requested a review from Copilot January 22, 2026 00:51

Copilot started reviewing on behalf of polarG January 22, 2026 00:52 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

dockerfile/cuda11.1.1.dockerfile Show resolved Hide resolved

dockerfile/cuda11.1.1.dockerfile Outdated Show resolved Hide resolved

dockerfile/cuda11.1.1.dockerfile Show resolved Hide resolved

Fix go install location.

db9e12b

polarG requested a review from Copilot January 22, 2026 05:06

Copilot started reviewing on behalf of polarG January 22, 2026 05:07 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

Hongtao Zhang added 2 commits January 22, 2026 05:21

Reorder installation steps for proper dependency handling.

709680b

Revert PATH change.

c4213bf

Copilot AI reviewed Jan 22, 2026

View reviewed changes

CI/CD - Fix Image build for cuda11.1.1 and py unit test pipelines #771

Are you sure you want to change the base?

CI/CD - Fix Image build for cuda11.1.1 and py unit test pipelines #771

Uh oh!

Conversation

polarG commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

abuccts left a comment

Choose a reason for hiding this comment

Uh oh!

polarG commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

polarG commented Jan 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abuccts left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

polarG commented Jan 6, 2026 •

edited

Loading

codecov bot commented Jan 6, 2026 •

edited

Loading

polarG commented Jan 12, 2026 •

edited

Loading