Build fixes and enhancements by jakub-kocka · Pull Request #63 · espressif/idf-python-wheels

jakub-kocka · 2026-03-23T08:10:26Z

idf-component-manager v3.0.0 does not support Python <= 3.10 https://pypi.org/project/idf-component-manager/3.0.0/ adding to the exclude_list.yaml because of some dependency has propagated

Description

Because of the failing wheels built due to the idf-component-manager, the package was added to exclude_list.yaml.

idf-component-manager v3.0.0 does not support Python <= 3.10 - https://pypi.org/project/idf-component-manager/3.0.0/

Also, during the test builds an issue about broken wheels has pop up - fixed the BadZipFile / "Bad magic number" errors - PEP 427 (https://github.com/espressif/idf-python-wheels/actions/runs/23427493077/job/68191831315)

As part of this PR, the esptool dependencies are added into the requirements assembly - the packages that will be built

This is the fix for the failing esptool pipeline because of missing dependencies and pip's mistaken resolver (https://github.com/espressif/esptool/actions/runs/22508432922/job/65212183798)

This step is mostly for the esptool master builds when using an extra index, because even if the package itself is excluded, its dependencies are downloaded with the package as well, and later on the desired ones are discarded

The PyPI package Python version test before build has been implemented as a new feature

Testing

Build workflow: https://github.com/espressif/idf-python-wheels/actions/runs/24186702486

Checklist

Before submitting a Pull Request, please ensure the following:

🚨 This PR does not introduce breaking changes.
All CI checks (GH Actions) pass.
Documentation is updated as needed.
Tests are updated or added as necessary.
Code is well-commented, especially in complex areas.
Git history is clean — commits are squashed to the minimum necessary.

github-actions · 2026-03-23T08:25:59Z

	Messages
📖	You might consider squashing your 4 commits (simplifying branch history).

👋 Hello jakub-kocka, we appreciate your contribution to this project!

Click to see more instructions ...

This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- Addressing info messages (📖) is strongly recommended; they're less critical but valuable.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...

We do welcome contributions in the form of bug reports, feature requests and pull requests.

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
4. If the change is approved and passes the tests it is merged into the default branch.

Generated by 🚫 dangerJS against 233fcbd

Copilot

Pull request overview

Updates the wheel build/test pipeline to avoid known incompatible dependencies, improve resilience to corrupt wheel artifacts, and stabilize dependency resolution during wheel builds.

Changes:

Extend exclude/marker handling (notably platform ∩ python exclusion without a version) and add new exclude entries.
Detect and discard corrupt/non-zip “.whl” artifacts during repair and install verification steps.
Add esptool dependencies into the assembled requirements and introduce a --force-interpreter-binary build mode for dependent wheels.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`exclude_list.yaml`	Adds new exclusions (pydantic_core on Win/macOS + py3.14; idf-component-manager 3.0.0 on older Pythons).
`yaml_list_adapter.py`	Adds intersection-style exclude marker generation for platform+python (no version) and refactors python marker fragment building.
`test_build_wheels.py`	Adds unit tests covering the new platform∩python intersection exclude behavior.
`test_wheels_install.py`	Adds preflight zip validation + pip error classification to discard corrupt wheel artifacts.
`repair_wheels.py`	Adds zip validation before repair and validates repaired output; deletes invalid artifacts.
`build_wheels.py`	Adds tomllib/tomli parsing and appends esptool `pyproject.toml` dependencies to assembled requirements.
`build_requirements.txt`	Adds `tomli` for Python < 3.11 to support TOML parsing fallback.
`build_wheels_from_file.py`	Adds `--force-interpreter-binary` support and per-requirement `--no-binary` injection (non-Windows).
`_helper_functions.py`	Makes `print_color` safer on non-UTF8 Windows consoles by sanitizing output text.
`.github/workflows/unit-tests.yml`	Switches unit-test dependency install to `-r build_requirements.txt`.
`.github/workflows/build-wheels-python-dependent.yml`	Enables `--force-interpreter-binary`, sets `PYO3_USE_ABI3_FORWARD_COMPATIBILITY`, and adds an ARMv7 artifact permission fix.
`.github/workflows/build-wheels-platforms.yml`	Adds an ARMv7 artifact permission fix before upload.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

peterdragun

There is a typo in one of the commit messages PYthon -> Python.

This MR has a lot of commits which does not seem to be related. Please try to avoid making such big MRs with unrelated changes; it is harder to review and easier to miss some important details.

There are two commits that try to address seemingly the same issue - BadZipFile - why are those two separate commits? Would it make sense to merge into one?

The last commit message is also quite vague and addresses some issues that were introduced in earlier commits in this MR. Try to keep the commits clean and MRs as simple as possible.

jakub-kocka · 2026-04-09T08:53:31Z

There is a typo in one of the commit messages PYthon -> Python.

This MR has a lot of commits which does not seem to be related. Please try to avoid making such big MRs with unrelated changes; it is harder to review and easier to miss some important details.

There are two commits that try to address seemingly the same issue - BadZipFile - why are those two separate commits? Would it make sense to merge into one?

The last commit message is also quite vague and addresses some issues that were introduced in earlier commits in this MR. Try to keep the commits clean and MRs as simple as possible.

Thank you, Peter. I will try to polish this.

Actually, it started as a component manager constraint fix, and with the tests, issues started to pop up - I have changed the name of the PR and will reorder it as well as possible ... you are right, that the esptool requirements addition could be in a different PR tho

…n macOS - Add idf-component-manager to exclude_list for unsupported Python versions - Correct platform+python intersection handling in YAMLListAdapter - Exclude dbus-python for Python 3.11 on macOS

- Fetch esptool dependencies when assembling requirements - Align cPython / workflow builds with interpreter environments - Fix permissions for root and non-root CI runners

- Detect and handle invalid or corrupt wheels (BadZip / PEP 427) - Dedupe wheel paths; tolerate missing files when pruning bad zips - Keep wheels when auditwheel reports InvalidLibc on Linux ARMv7 - Refinements from review Made-with: Cursor

jakub-kocka · 2026-04-09T09:48:13Z

@peterdragun, I have updated the commit history and added the PyPI check as you suggested for the package metadata of supported Pythons before the build.

Hope it is cleaner now, I am sorry for the accidentally grown in size PR - the main reason was to have successful builds (which was not exactly true by letting some issues go to other PR) 🙏

peterdragun

LGTM, my understanding of this repo is quite limited, but as far as I can tell, this seems correct. Thank you for applying my suggestions.

Skip pip wheel for exact == pins when PyPI metadata Requires-Python excludes the current interpreter (build_wheels and build_wheels_from_file). Add helpers, tests, and SKIP_PYPI_REQUIRES_PYTHON_CHECK escape hatch. Remove redundant exclude_list row for idf-component-manager==3.0.0; covered by the preflight for that release (Requires-Python >=3.10,<4). Made-with: Cursor

jakub-kocka · 2026-04-10T08:30:51Z

Thank you for the review. Since the builds still succeed, I am merging this to have working builds for scheduled runs.

jakub-kocka self-assigned this Mar 23, 2026

jakub-kocka force-pushed the fix/component_manager branch 7 times, most recently from 1f4bbeb to 9b91cdf Compare March 30, 2026 07:49

jakub-kocka requested a review from Copilot March 30, 2026 12:18

Copilot started reviewing on behalf of jakub-kocka March 30, 2026 12:19 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

Comment thread exclude_list.yaml Outdated

Comment thread exclude_list.yaml Outdated

Comment thread build_wheels.py Outdated

Comment thread yaml_list_adapter.py Outdated

Comment thread test_build_wheels.py Outdated

jakub-kocka mentioned this pull request Mar 31, 2026

fix: Use image from wheels for ARMv7 builds espressif/python-binary-action#10

Merged

jakub-kocka force-pushed the fix/component_manager branch 3 times, most recently from 620715a to 1fda564 Compare April 1, 2026 12:17

jakub-kocka force-pushed the fix/component_manager branch from 59f7ab7 to d023d68 Compare April 8, 2026 07:28

jakub-kocka requested a review from peterdragun April 9, 2026 07:14

peterdragun reviewed Apr 9, 2026

View reviewed changes

Comment thread test_wheels_install.py Outdated

Comment thread build_wheels.py Outdated

Comment thread exclude_list.yaml Outdated

jakub-kocka changed the title ~~Added idf-component-manager to exclude_list~~ Build fixes and enhancements Apr 9, 2026

jakub-kocka added 3 commits April 9, 2026 11:15

fix(exclude): idf-component-manager, YAML intersection, dbus-python o…

7593ea0

…n macOS - Add idf-component-manager to exclude_list for unsupported Python versions - Correct platform+python intersection handling in YAMLListAdapter - Exclude dbus-python for Python 3.11 on macOS

feat(ci): esptool deps, interpreter matrix, runner permissions

5723676

- Fetch esptool dependencies when assembling requirements - Align cPython / workflow builds with interpreter environments - Fix permissions for root and non-root CI runners

jakub-kocka force-pushed the fix/component_manager branch from 9706f42 to 4ea168c Compare April 9, 2026 09:39

jakub-kocka force-pushed the fix/component_manager branch from 4ea168c to 2d80b1e Compare April 9, 2026 10:09

peterdragun reviewed Apr 9, 2026

View reviewed changes

Comment thread _helper_functions.py Outdated

peterdragun reviewed Apr 9, 2026

View reviewed changes

Comment thread _helper_functions.py

peterdragun approved these changes Apr 9, 2026

View reviewed changes

jakub-kocka force-pushed the fix/component_manager branch from 2d80b1e to 233fcbd Compare April 9, 2026 10:59

jakub-kocka merged commit 36a69f8 into main Apr 10, 2026
119 checks passed

jakub-kocka deleted the fix/component_manager branch April 10, 2026 08:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build fixes and enhancements#63

Build fixes and enhancements#63
jakub-kocka merged 4 commits into
mainfrom
fix/component_manager

jakub-kocka commented Mar 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

peterdragun left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakub-kocka commented Apr 9, 2026

Uh oh!

jakub-kocka commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

peterdragun left a comment

Uh oh!

jakub-kocka commented Apr 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jakub-kocka commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related

Testing

Checklist

Uh oh!

github-actions Bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

peterdragun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakub-kocka commented Apr 9, 2026

Uh oh!

jakub-kocka commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

peterdragun left a comment

Choose a reason for hiding this comment

Uh oh!

jakub-kocka commented Apr 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jakub-kocka commented Mar 23, 2026 •

edited

Loading

github-actions Bot commented Mar 23, 2026 •

edited

Loading