Rewrite tests to python for windows runtime testing by julek-wolfssl · Pull Request #215 · wolfSSL/wolfCLU

julek-wolfssl · 2026-04-01T14:56:01Z

No description provided.

- Add shared test helper and cross-platform test runner - Extract binary lookup and run_wolfssl() into tests/wolfclu_test.py so all test files share the same logic for finding the wolfssl binary across Linux (./wolfssl) and Windows (x64/Debug/wolfssl.exe etc.). - Add tests/run_tests.py which discovers and runs all *-test.py files, intended for Windows where `make check` is not available. - Enable PKCS7 and CRL in MSVC - Fix three Windows bugs uncovered by the test: - Add StartTCP() (WSAStartup) in client and server setup so Winsock is initialized before gethostbyname/connect calls - Pass SNI hostname (-S flag) to the underlying client_test so modern TLS servers accept the connection - Implement checkStdin() for Windows using WaitForSingleObject so s_client exits promptly when stdin is a closed pipe - Update user_settings.h with defines required for TLS 1.3 and full wolfCLU functionality: WOLFSSL_TLS13, HAVE_HKDF, WC_RSA_PSS, HAVE_SUPPORTED_CURVES, HAVE_FFDHE_2048, HAVE_SNI. - Fix run_wolfssl() to use stdin=DEVNULL when no input is provided, preventing subprocesses from blocking on inherited stdin (e.g. over SSH/network sessions on Windows). - Also add WOLFCLU_SKIP_SLOW_TESTS env var to skip slow tests on Windows, and optimize large file creation to a single write. - Add HAVE_PKCS12 to Windows user_settings.h. Skip binary DER stdin test on Windows where pipe binary mode is unreliable. - Combine ocsp-test.sh and ocsp-interop-test.sh into a single Python test module that tests all client/responder combinations (wolfssl and openssl). - Add StartTCP() in OCSP setup for Winsock initialization on Windows. - Add HAVE_OCSP and HAVE_OCSP_RESPONDER to Windows user_settings.h. - Rewrite base64 test from bash to Python unittest - Rewrite bench test from bash to Python unittest - Rewrite client test to Python and fix Windows networking bugs - Replace tests/client/client-test.sh with a cross-platform Python unittest. - Rewrite dgst test from bash to Python unittest - Rewrite dh test from bash to Python unittest - Rewrite dsa test from bash to Python unittest - Rewrite enc test from bash to Python unittest - Rewrite genkey sign/verify test from bash to Python unittest - Rewrite hash test from bash to Python unittest - Rewrite pkcs7/pkcs8/pkcs12 tests from bash to Python unittest - Rewrite pkey/rsa/ecparam tests from bash to Python unittest - Rewrite rand test from bash to Python unittest - Rewrite server test from bash to Python unittest - Rewrite encdec test from bash to Python unittest - Rewrite x509/CRL tests from bash to Python unittest - Rewrite OCSP tests from bash to Python unittest - Rewrite OCSP SCGI test from bash to Python, drop nginx dependency - Replace nginx + bash with a pure-Python HTTP-to-SCGI proxy using stdlib http.server and raw sockets for the SCGI netstring protocol. No external dependencies needed. - Remove nginx from CI apt-get installs since it is no longer required for testing.

Replace fsanitize-check.yml, macos-check.yml, and ubuntu-check.yml with a single ci.yml that uses a matrix of OS, config, and sanitizer dimensions. Update all checkout actions to v4 and setup-msbuild to v2.

Simplify windows-check.yml: remove ${{env.GITHUB_WORKSPACE}} indirection, drop PlatformToolset/WindowsTargetPlatformVersion overrides, add timeout, pin wolfSSL ref, consolidate env vars. Update VS project to v145 toolset, add missing source files to filters, enable multi-processor compilation.

Copilot

Pull request overview

This PR migrates the wolfCLU test suite from shell scripts to Python unittest modules to enable reliable runtime testing on Windows, while also updating Windows build configuration and CI workflows.

Changes:

Replaced many tests/**/*.sh scripts with equivalent tests/**/*-test.py Python unittest modules and added a Windows-friendly tests/run_tests.py runner.
Updated Automake integration to recognize .py tests and added Python detection in configure.ac.
Updated Windows Visual Studio project/solution and CI workflows; added Windows networking initialization and SNI handling in the client setup.

Reviewed changes

Copilot reviewed 93 out of 94 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
wolfclu/clu_header_main.h	Undefines CRL reason macros before wolfSSL headers (Windows build hygiene).
wolfCLU.vcxproj.filters	Adds new source files to the Visual Studio filters list.
wolfCLU.vcxproj	Updates toolset and enables multiprocessor compilation.
wolfclu.sln	Updates VS version metadata and fixes x64 config mapping.
tests/x509/x509-verify-test.sh	Removed (replaced by Python unittest).
tests/x509/x509-verify-test.py	New Python unittest for `wolfssl verify`.
tests/x509/x509-req-test.sh	Removed (replaced by Python unittest).
tests/x509/x509-process-test.sh	Removed (replaced by Python unittest).
tests/x509/x509-ca-test.sh	Removed (replaced by Python unittest).
tests/x509/include.am	Registers Python x509 tests in Automake.
tests/x509/CRL-verify-test.sh	Removed (replaced by Python unittest).
tests/x509/CRL-verify-test.py	New Python unittest for `wolfssl crl`.
tests/wolfclu_test.py	Shared helper to locate `wolfssl` binary and run commands.
tests/testEncDec/test_aesctr.sh	Removed (replaced by Python unittest).
tests/testEncDec/test_aescbc_3des_cam.sh	Removed (replaced by Python unittest).
tests/testEncDec/README.md	Removed (test approach changed to self-contained Python).
tests/testEncDec/include.am	Registers new enc/dec Python test in Automake.
tests/testEncDec/encdec-test.py	New encrypt/decrypt round-trip tests for multiple ciphers.
tests/server/server-test.sh	Removed (replaced by Python unittest).
tests/server/server-test.py	New Python unittest for `s_server`/`s_client` handshake.
tests/server/include.am	Registers Python server test in Automake.
tests/run_tests.py	New Windows-oriented Python test runner discovering `*-test.py`.
tests/rand/rand-test.sh	Removed (replaced by Python unittest).
tests/rand/rand-test.py	New Python unittest for `wolfssl rand`.
tests/rand/include.am	Registers Python rand test in Automake.
tests/pkey/rsa-test.sh	Removed (replaced by Python unittest).
tests/pkey/rsa-test.py	New Python unittest for RSA key conversion/validation.
tests/pkey/pkey-test.sh	Removed (replaced by Python unittest).
tests/pkey/pkey-test.py	New Python unittest for `pkey` public/private conversions.
tests/pkey/include.am	Registers Python pkey tests in Automake.
tests/pkey/ecparam-test.sh	Removed (replaced by Python unittest).
tests/pkey/ecparam-test.py	New Python unittest for curve enumeration and key generation.
tests/pkcs/pkcs8-test.sh	Removed (replaced by Python unittest).
tests/pkcs/pkcs8-test.py	New Python unittest for PKCS8 conversion and error cases.
tests/pkcs/pkcs7-test.sh	Removed (replaced by Python unittest).
tests/pkcs/pkcs7-test.py	New Python unittest for PKCS7 conversions and printing certs.
tests/pkcs/pkcs12-test.sh	Removed (replaced by Python unittest).
tests/pkcs/pkcs12-test.py	New Python unittest for PKCS12 extraction options.
tests/pkcs/include.am	Registers Python pkcs tests in Automake.
tests/ocsp/ocsp-test.sh	Removed (replaced by Python unittest).
tests/ocsp/ocsp-test.py	New combined OCSP interop test suite in Python.
tests/ocsp/include.am	Registers Python OCSP test in Automake.
tests/ocsp-scgi/scgi_params	Removed (nginx dependency eliminated).
tests/ocsp-scgi/ocsp-scgi-test.py	New Python HTTP-to-SCGI proxy test replacing nginx.
tests/ocsp-scgi/include.am	Registers Python OCSP-SCGI test in Automake.
tests/hash/include.am	Switches hash tests from `.sh` to `.py`.
tests/hash/hash-test.sh	Removed (replaced by Python unittest).
tests/hash/hash-test.py	New Python unittest for `-hash` and shortcut hash commands.
tests/genkey_sign_ver/include.am	Switches genkey/sign/verify test registration to Python.
tests/genkey_sign_ver/genkey-sign-ver-test.sh	Removed (replaced by Python unittest).
tests/genkey_sign_ver/genkey-sign-ver-test.py	New Python unittest for keygen + sign/verify workflows incl. PQC.
tests/encrypt/include.am	Switches encrypt tests from `.sh` to `.py`.
tests/encrypt/enc-test.sh	Removed (replaced by Python unittest).
tests/encrypt/enc-test.py	New Python unittest for enc/dec, interop, and legacy names.
tests/dsa/include.am	Switches DSA tests from `.sh` to `.py`.
tests/dsa/dsa-test.sh	Removed (replaced by Python unittest).
tests/dsa/dsa-test.py	New Python unittest for dsaparam behaviors.
tests/dh/include.am	Switches DH tests from `.sh` to `.py`.
tests/dh/dh-test.sh	Removed (replaced by Python unittest).
tests/dh/dh-test.py	New Python unittest for dhparam behaviors.
tests/dgst/include.am	Switches dgst tests from `.sh` to `.py`.
tests/dgst/dgst-test.sh	Removed (replaced by Python unittest).
tests/dgst/dgst-test.py	New Python unittest for signature verification + large-file cases.
tests/client/include.am	Switches client test from `.sh` to `.py`.
tests/client/client-test.sh	Removed (replaced by Python unittest).
tests/client/client-test.py	New Python unittest for external TLS connection + x509 extraction.
tests/bench/include.am	Switches bench test from `.sh` to `.py`.
tests/bench/bench-test.sh	Removed (replaced by Python unittest).
tests/bench/bench-test.py	New Python unittest for `-bench` smoke tests.
tests/base64/include.am	Switches base64 test from `.sh` to `.py`.
tests/base64/base64-test.sh	Removed (replaced by Python unittest).
tests/base64/base64-test.py	New Python unittest for base64 encode/decode + stdin coverage.
src/x509/clu_x509_sign.c	Adjusts unsupported-hash handling in cert signing code path.
src/tools/clu_funcs.c	Gates AES-CTR help output on wolfSSL version macro.
src/server/clu_server_setup.c	Calls `StartTCP()` before running server test (Windows networking).
src/ocsp/clu_ocsp.c	Calls `StartTCP()` before OCSP client/responder execution.
src/crypto/clu_decrypt.c	Adjusts file-read loop behavior for decrypt path.
src/client/clu_client_setup.c	Adds SNI args and calls `StartTCP()` before running client test.
src/client/client.c	Implements Windows `checkStdin()` logic and enables it on Windows.
Makefile.am	Adds `.py` test extensions and includes new testEncDec include.am.
ide/winvs/user_settings.h	Enables SNI/TLS13/PKCS/CRL/OCSP features for Windows builds.
configure.ac	Adds Python discovery/substitution for Automake test harness.
.gitignore	Ignores Visual Studio build outputs and Python cache dirs.
.github/workflows/windows-check.yml	Modernizes Windows build + runs new Python test runner.
.github/workflows/ubuntu-check.yml	Removed (replaced by consolidated CI workflow).
.github/workflows/macos-check.yml	Removed (replaced by consolidated CI workflow).
.github/workflows/fsanitize-check.yml	Removed (replaced by consolidated CI workflow matrix).
.github/workflows/ci.yml	New consolidated Ubuntu/macOS matrix build + `make check`.
.gitattributes	Enforces LF for PEM/config files; marks DER as binary.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/x509/clu_x509_sign.c

tests/ocsp/ocsp-test.py

tests/run_tests.py

tests/hash/hash-test.py

wolfclu.sln

Copilot

Pull request overview

Copilot reviewed 93 out of 94 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

wolfCLU.vcxproj

.github/workflows/windows-check.yml

configure.ac

Makefile.am

tests/wolfclu_test.py

tests/testEncDec/encdec-test.py

tests/pkey/rsa-test.py

tests/pkcs/pkcs12-test.py

tests/server/server-test.py

src/client/clu_client_setup.c

- Use $(DefaultPlatformToolset) in vcxproj instead of hard-coded v145 - Remove toolset detection step from Windows CI (no longer needed) - Disable Python tests when Python 3 not found instead of failing build - Add -noservername flag to s_client (SNI on by default, matching openssl) - Fix wolfclu_test.py paths to be __file__-relative for location independence - Add trailing newline to pkcs12 stdin password inputs - Redirect s_server stdout/stderr to DEVNULL to prevent pipe deadlock

Copilot

Pull request overview

Copilot reviewed 94 out of 95 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

src/crypto/clu_decrypt.c:166

The read loop now treats only XFREAD(...) > 0 as success and logs "Input file does not exist" for a 0-byte read. Because feof(inFile) is checked before attempting the read, EOF will typically be detected only after XFREAD returns 0; with this change that becomes a hard error instead of being handled as EOF. Consider checking feof()/ferror() after the read (or handling ret == 0 && feof(inFile) separately) so valid EOF conditions don’t get reported as a missing file.

            }
            else {
                ret = (int)XFREAD(input, 1, MAX_LEN, inFile);
                if (ret > 0) {
                    tempMax = ret;
                    ret = 0; /* success */
                }
                else {
                    wolfCLU_LogError("Input file does not exist.");
                    ret = FREAD_ERROR;
                }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/client/clu_client_setup.c

Makefile.am

tests/server/server-test.py

tests/ocsp-scgi/ocsp-scgi-test.py

Add test_main() helper in wolfclu_test.py that runs unittest with exit=False and translates the result: all-skipped/no-tests -> exit 77 (automake SKIP), failures -> exit 1, otherwise -> exit 0.

julek-wolfssl added 9 commits March 31, 2026 14:13

windows setup

db30ae8

Add SM3 hash type in wolfCLU_CertSign function

cb5d98a

Run tests in windows CI

75e45d1

Undef CRL_REASON_* because it clashes with other definitions

2b1c91c

Fix test failures

b28b79d

Consolidate CI workflows and update action versions

be40be0

Replace fsanitize-check.yml, macos-check.yml, and ubuntu-check.yml with a single ci.yml that uses a matrix of OS, config, and sanitizer dimensions. Update all checkout actions to v4 and setup-msbuild to v2.

Use vswhere to detect PlatformToolset

28939d3

julek-wolfssl self-assigned this Apr 1, 2026

Copilot AI review requested due to automatic review settings April 1, 2026 14:56

Copilot started reviewing on behalf of julek-wolfssl April 1, 2026 14:58 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

src/x509/clu_x509_sign.c Show resolved Hide resolved

tests/ocsp/ocsp-test.py Show resolved Hide resolved

tests/run_tests.py Outdated Show resolved Hide resolved

tests/hash/hash-test.py Show resolved Hide resolved

wolfclu.sln Show resolved Hide resolved

julek-wolfssl added 2 commits April 1, 2026 17:17

Fix test failures

4847855

Address code review

f5d97d0

Copilot AI review requested due to automatic review settings April 1, 2026 15:36

Copilot started reviewing on behalf of julek-wolfssl April 1, 2026 15:38 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings April 2, 2026 08:38

julek-wolfssl force-pushed the windows-ci branch from 774573b to e7e1d5d Compare April 2, 2026 08:38

Copilot started reviewing on behalf of julek-wolfssl April 2, 2026 08:38 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

src/client/clu_client_setup.c Show resolved Hide resolved

Makefile.am Show resolved Hide resolved

tests/server/server-test.py Show resolved Hide resolved

tests/ocsp-scgi/ocsp-scgi-test.py Show resolved Hide resolved

Exit 77 for skipped Python tests so automake reports SKIP

bcd3742

Add test_main() helper in wolfclu_test.py that runs unittest with exit=False and translates the result: all-skipped/no-tests -> exit 77 (automake SKIP), failures -> exit 1, otherwise -> exit 0.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite tests to python for windows runtime testing#215

Rewrite tests to python for windows runtime testing#215
julek-wolfssl wants to merge 13 commits intowolfSSL:mainfrom
julek-wolfssl:windows-ci

julek-wolfssl commented Apr 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

julek-wolfssl commented Apr 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants