Skip to content

[CI][C++] MinGW jobs fail every run after MSYS2 toolchain updates #49930

@tadeja

Description

@tadeja

Describe the bug, including details regarding any error messages, version, and platform.

The MINGW64 and CLANG64 MinGW jobs in .github/workflows/cpp.yml have failed every run since 2026 April 30 - first failure on main: https://github.com/apache/arrow/actions/runs/25213414021/job/73928546151#step:12:2644

The following tests FAILED:
	 41 - arrow-async-utility-test (Exit code 0xc0000374)   arrow-tests unittest
	 62 - arrow-dataset-dataset-writer-test (SEGFAULT)      arrow_dataset unittest
	 65 - arrow-dataset-file-test (Failed)                  arrow_dataset unittest
Error: Process completed with exit code 8.

Recent failure 2026 May 05 - https://github.com/apache/arrow/actions/runs/25379694834/job/74424280352#step:12:2468

The following tests FAILED:
	 41 - arrow-async-utility-test (Exit code 0xc0000374)   arrow-tests unittest
	 44 - arrow-threading-utility-test (Timeout)            arrow-tests unittest
	 62 - arrow-dataset-dataset-writer-test (Failed)        arrow_dataset unittest
	 65 - arrow-dataset-file-test (Failed)                  arrow_dataset unittest
	 76 - arrow-s3fs-test (Failed)                          arrow-tests filesystem unittest
Error: Process completed with exit code 8.

My empirical isolation (PR #49931 commit fb4d7a0, all pins removed, MinIO bumped to RELEASE.2025-01-20T14-49-07Z) - CI run https://github.com/apache/arrow/actions/runs/25442601074/job/74637722454#step:13:1327

The following tests FAILED:
	 41 - arrow-async-utility-test (Exit code 0xc0000374)   arrow-tests unittest
	 44 - arrow-threading-utility-test (Timeout)            arrow-tests unittest
	 62 - arrow-dataset-dataset-writer-test (Failed)        arrow_dataset unittest
	 65 - arrow-dataset-file-test (Failed)                  arrow_dataset unittest
	 76 - arrow-s3fs-test (Timeout)                         arrow-tests filesystem unittest

CLANG64 in the same run: 100% pass. Confirms gcc-16 (not the gcc-libs cascade or aws-sdk-cpp behavior) is what triggers the cluster.


Likely triggered by two MSYS2 updates:

  1. MINGW64: gcc 15.2 -> 16.1 triggers a cluster of failures in 5 tests.
    Latest CI status from PR 49462's MINGW64 CI - https://github.com/apache/arrow/actions/runs/25458933029/job/74695767495?pr=49462

Per-test status:

  • arrow-json-test : MultipleChunksParallel crash - Fixed by GH-49272: [C++][CI] Fix intermittent segfault in arrow-json-test with MinGW #49462 (Win32 TLS workaround)
  • arrow-async-utility-test : Heap corruption (0xc0000374) - Unfixed; root cause unclear
  • arrow-threading-utility-test : SegFault then 300s timeout - Unfixed; could be another __emutls site or a gcc-16 codegen issue
  • arrow-dataset-dataset-writer-test : std::bad_weak_ptr in BatchWriteConcurrent - Unfixed; possibly an Arrowshared_from_this() lifetime bug exposed by gcc-16
  • arrow-dataset-file-test : SegFault - Unfixed; possibly the same race as dataset-writer-test

PR 49462 is necessary but not enough. The other 4 need their own fixes: possibly in Arrow, possibly in MSYS2/gcc, possibly some mix.

  1. MINGW64 + CLANG64: aws-sdk-cpp 1.11.479 -> 1.11.801 stopped sending Content-Md5 on DeleteObjects; bundled MinIO (RELEASE.2024-09-13) still requires it. Resolved in GH-49930: [CI][C++] Pin MinGW MSYS2 packages to unblock CI #49931 by bumping bundled MinIO.

Temporary workaround: pin MSYS2 packages to last known good versions on MINGW64 in #49931

This issue tracks reverting to current upstream MSYS2 packages once
all 5 failing tests pass on current upstream MSYS2 (without temporary pins of #49931!). See point (1.) for TEST - STATUS

Component(s)

Continuous Integration, C++

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions