It upgrades Elastic search client and urlib3 libraries.

[fressi-elastic]

This PR upgrades the Elasticsearch client libraries and urllib3 to newer major versions, updating from Elasticsearch 8.x to 9.x and urllib3 1.x to 2.x. The changes include API parameter updates, refactoring of header handling logic, and deprecation of certain track dependencies.

Changes:

• It sets the minimal Elasticsearch server version to 8.0.0 and update documentation.
• It update python package requirements:
  • elasticsearch[async] from 8.6.1 to 9.2.1
  • elastic-transport from 8.4.1 to 9.2.1
  • urllib3 from 1.26.19 to 2.6.3
  • docker from 6.0.0 to 7.1.0
• It renames maxsize parameter to connections_per_node in client initialization to match the new API
• It refactors header handling logic into reusable functions and improved MIME type compatibility handling
• It replaces obsolete docker-compose command with docker compose to avoid conflicts with new urlib3 version.
• It fixes it/docker_dev_image_test.py and update elasticsearch server to version 9.2.4.

This PR replaces the now obsolete #2003 and it requires PR elastic/rally-tracks#1000

[Copilot]

Pull request overview

This PR upgrades the Elasticsearch client libraries and urllib3 to newer major versions, updating from Elasticsearch 8.x to 9.x and urllib3 1.x to 2.x. The changes include API parameter updates, refactoring of header handling logic, and deprecation of certain track dependencies.

Changes:

• Updated elasticsearch[async] from 8.6.1 to 9.2.1, elastic-transport from 8.4.1 to 9.2.1, and urllib3 from 1.26.19 to 2.6.3
• Renamed maxsize parameter to connections_per_node in client initialization to match the new API
• Refactored header handling logic into reusable functions and improved MIME type compatibility handling

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pyproject.toml	Updated dependency versions for elasticsearch, elastic-transport, urllib3, and pip
esrally/client/factory.py	Changed maxsize to connections_per_node parameter
tests/client/factory_test.py	Updated test to use connections_per_node parameter
esrally/client/common.py	Refactored header combining and MIME type compatibility logic into dedicated functions
esrally/client/synchronous.py	Updated to use new header handling functions and added endpoint_id/path_parts parameters
esrally/client/asynchronous.py	Updated RallyAsyncElasticsearch signature and header handling to use new functions
esrally/track/loader.py	Added deprecation handling for elasticsearch and elastic-transport dependencies in tracks
esrally/utils/versions.py	Added type hints to is_version_identifier function
it/init.py	Updated test distributions to include 9.2.4 and removed 6.8.0/ARM logic
tests/client/common_test.py	Removed test file (likely obsolete after refactoring)
tests/driver/runner_test.py	Added typing import and @typing.no_type_check decorator to test

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[fressi-elastic]

I am waiting for review for the following Elasticsearch PR fix that should block this PR build failures: elastic/elasticsearch#140947

It should correct the following error:
https://github.com/elastic/rally/actions/runs/21173420503/job/60895685787?pr=2024#step:11:530

[fressi-elastic]

PR elastic/elasticsearch#140947 has been merged. Now current Elasticsearch should compile again.

[gareth-ellis]

LGTM - Have you run any comparisons with the old and new clients, and validated whether performance is equivalent?

[gbanasiak]

A few areas of concern:

• How does this change affect the range of supported Elasticsearch versions? This needs to be reflected in Rally documentation.
• If the intention is to drop support for ES 6.8 and 7.x as part of client upgrade, we should release one more Rally version before merging this update IMO. It's been a while since we released.
• What is the test plan here? Go to nightly staging, and then nightly production, and revert if things go south? I would prefer to test this more extensively before the merge (e.g. by running all nightlies in test mode).
• Why _ProductChecker is kept in synchronous.py? This client upgrade is an opportunity to simplify, because the following no longer holds:

Because Rally needs to support versions of ES >= 6.8.0, we resurrect the previous
logic for determining the authenticity of the server, which does not rely exclusively
on this header.

[fressi-elastic]

• How does this change affect the range of supported Elasticsearch versions? This needs to be reflected in Rally documentation.

It should now match versions supported by Elasticsearch Python client 9.x

• If the intention is to drop support for ES 6.8 and 7.x as part of client upgrade, we should release one more Rally version before merging this update IMO. It's been a while since we released.

OK.

• What is the test plan here? Go to nightly staging, and then nightly production, and revert if things go south? I would prefer to test this more extensively before the merge (e.g. by running all nightlies in test mode).

It would be nice to push this to a branch used by the nightly staging CI jobs, so we would eventually see it working or not. Is master branch used by both staging and production nightlies?

• Why _ProductChecker is kept in synchronous.py? This client upgrade is an opportunity to simplify, because the following no longer holds:

Because Rally needs to support versions of ES >= 6.8.0, we resurrect the previous
logic for determining the authenticity of the server, which does not rely exclusively
on this header.

I removed the old logic and matched the code in Rally as much as possible with the official client, with the only exception of the support for serverless and the automatic addition of application/*headers when body is given. It also helps with Elasticsearch 9 integration adding the x-ndjson mime type in a bulk request as it is being more strict that the version 8 on this aspect.