Skip to content

Upgrade Python from 3.11 to 3.12 across all services#11

Open
devin-ai-integration[bot] wants to merge 5 commits intodevelopfrom
devin/1763505758-upgrade-python-3.13
Open

Upgrade Python from 3.11 to 3.12 across all services#11
devin-ai-integration[bot] wants to merge 5 commits intodevelopfrom
devin/1763505758-upgrade-python-3.13

Conversation

@devin-ai-integration
Copy link
Copy Markdown

@devin-ai-integration devin-ai-integration bot commented Nov 18, 2025

Upgrade Python from 3.11 to 3.12 across all services

Summary

This PR upgrades Python from version 3.11 to 3.12 across the entire XFD codebase, including Docker images, GitHub Actions workflows, pre-commit configuration, AWS Lambda runtime, and Python dependencies.

Key changes:

  • Updated all Docker base images to Python 3.12 (worker, pe, python)
  • Updated GitHub Actions workflows to use Python 3.12
  • Updated AWS Lambda runtime to python3.12 in serverless.yml
  • Upgraded numpy from 1.24.3 to 1.26.4 (required for Python 3.12 compatibility)
  • Fixed bandit pre-commit hook by adding pbr as additional dependency
  • Updated pre-commit language_version to python3.12

Note: This PR initially targeted Python 3.13 but pivoted to 3.12 due to ecosystem compatibility issues with numpy and zstandard/cffi packages. Python 3.12 provides better package ecosystem support with prebuilt wheels.

Review & Testing Checklist for Human

  • Verify all CI checks pass, especially the build_worker job which previously failed with numpy compatibility issues
  • Test worker Docker builds in staging/integration environments to ensure scanning functionality works correctly with numpy 1.26.4
  • Verify AWS Lambda python3.12 runtime is available and working in target regions (especially GovCloud for production)
  • Run integration tests against staging API to verify no breaking changes from numpy upgrade
  • Test pre-commit hooks locally by running pre-commit run --all-files to ensure bandit works with pbr dependency

Test Plan

  1. Deploy to integration environment and verify all Lambda functions start successfully
  2. Run a full scan cycle (all scan types) to verify worker functionality with new Python version
  3. Check CloudWatch logs for any Python 3.12-specific errors or warnings
  4. Verify vulnerability scanning data is processed correctly with numpy 1.26.4

Notes

  • The branch name references Python 3.13 but the actual upgrade is to Python 3.12 (due to the pivot during development)
  • The bandit fix uses additional_dependencies: [pbr] as a workaround rather than upgrading bandit itself (keeping changes minimal)
  • CONTRIBUTING.md and setup-env script did not require updates as they use generic Python version references
  • Django 5.2.6 and FastAPI 0.116.2 are both compatible with Python 3.12

Link to Devin run: https://app.devin.ai/sessions/5bff2523f83540728718336abe92c688
Requested by: Zoheb Munshi (@zohebmunshi)

- Update Docker base images (Dockerfile.worker, Dockerfile.pe)
- Update GitHub Actions workflows (backend.yml, regression.yml, build.yml, security.yml)
- Update pre-commit configuration
- All changes maintain compatibility with Django 5.2.6 and FastAPI 0.116.2

Co-Authored-By: Zoheb Munshi <zohebmunshi@gmail.com>
@devin-ai-integration
Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Initial attempt to upgrade to Python 3.13 encountered multiple package compatibility issues:
- numpy 1.24.3 failed with pkgutil.ImpImporter removal in Python 3.13
- zstandard/cffi failed with undefined symbol errors in Python 3.13

Python 3.12 provides better package ecosystem support and meets the 'latest compatible version' requirement.

Changes:
- Updated all Dockerfiles to use Python 3.12 (worker, python, pe)
- Updated all GitHub Actions workflows to python-version: 3.12
- Updated .pre-commit-config.yaml to python3.12
- Updated backend/serverless.yml runtime to python3.12
- Updated numpy from 1.24.3 to 1.26.4 (compatible with Python 3.12)
- Verified worker Docker build succeeds with Python 3.12

Note: Frontend npm audit failure is pre-existing and unrelated to Python upgrade.
Co-Authored-By: Zoheb Munshi <zohebmunshi@gmail.com>
@devin-ai-integration devin-ai-integration bot changed the title Upgrade Python version from 3.11 to 3.13 Upgrade Python from 3.11 to 3.12 across all environments Nov 18, 2025
Co-Authored-By: Zoheb Munshi <zohebmunshi@gmail.com>
@devin-ai-integration devin-ai-integration bot changed the title Upgrade Python from 3.11 to 3.12 across all environments Upgrade Python from 3.11 to 3.12 across all services Nov 18, 2025
devin-ai-integration bot and others added 2 commits November 18, 2025 23:13
- Patch matplotlib==3.3.4 to matplotlib>=3.8,<3.9 before installing ATC-Framework
- matplotlib 3.3.4 fails to build on Python 3.12 due to SafeConfigParser removal
- matplotlib 3.8+ supports Python 3.12

Co-Authored-By: Zoheb Munshi <zohebmunshi@gmail.com>
- Previous sed pattern didn't match 'matplotlib == 3.3.4' (with spaces) in setup.py
- Updated to use sed -E with \s* to match optional whitespace
- Added verification steps to print matplotlib lines before/after patching
- pip install . uses install_requires from setup.py, not requirements.txt files

Co-Authored-By: Zoheb Munshi <zohebmunshi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants