Add optional SQL proxy sidecar for OpenShift compatibility

[tomach]

Description

This PR introduces an optional sidecar container (crate-control) for CrateDB pods that acts as a lightweight SQL proxy for operator actions. The goal is to replace the operator's current use of pod_exec for administrative SQL operations. pod_exec is restricted or unsupported in OpenShift, which prevents the operator from performing required bootstrap and maintenance tasks.

With this change, the operator can communicate with CrateDB via an in-cluster HTTP endpoint instead of exec into pods. This mechanism is fully backwards compatible: existing clusters without the sidecar continue to use pod_exec.

High-level design

Sidecar

Each CrateDB pod gets an additional container exposing a small HTTP API:

• Endpoint: POST /exec
• Internal port: 5050
• No external exposure (only reachable inside the cluster)

The sidecar:

• Validates incoming requests using a bootstrap token provided via environment variable
• Forwards SQL statements to the local CrateDB HTTP endpoint (http://localhost:4200/_sql)
• Returns the raw CrateDB response to the operator

Because requests originate from localhost, existing host-based authentication remains unchanged:

"-Cauth.host_based.config.0.user": "crate",
"-Cauth.host_based.config.0.address": "_local_",
"-Cauth.host_based.config.0.method": "trust",

Authentication between operator and sidecar is handled exclusively via the injected bootstrap token.

The implementation uses Bottle + waitress + Python stdlib (urllib) to keep the memory footprint small.

Operator integration

When enabled, the operator:

• Injects the sidecar container into the CrateDB StatefulSet
• Creates:
  • a headless Service (clusterIP: None) for reaching sidecars
  • a Kubernetes Secret containing the bootstrap token
• Uses the sidecar for all SQL-based administrative tasks, including:
  • bootstrapping the system user
  • configuring JWT authentication
  • restoring snapshots / backups
  • any operation previously relying on pod_exec

The sidecar image version is aligned with the operator version (same tag/commit via CI).

For clusters created before this feature (or when the sidecar is not present), the operator automatically falls back to the existing pod_exec behavior.

OpenShift testing guide

The following steps describe how to manually verify the operator and sidecar on an OpenShift cluster.

Prerequisites

Ensure the cluster has:

• CrateDB CRD installed
• crate-operator RBAC and storage classes applied

1. Build and deploy the operator

From the crate/crate-operator repository:

docker build -t <registry>/crate/crate-operator:<tag> .
docker push <registry>/crate/crate-operator:<tag>

Deploy the operator using the newly built image.

2. Build and publish the sidecar image

From the sidecar directory:

docker build -t <registry>/crate/crate-control:<tag> .
docker push <registry>/crate/crate-control:<tag>

Configure the operator to use this image:

- name: CRATEDB_OPERATOR_CRATE_CONTROL_IMAGE
  value: "<registry>/crate/crate-control:<tag>"

Restart the operator after updating this env variable.

3. Deploy a CrateDB cluster

Create a new CrateDB custom resource.

Verify:

• Each CrateDB pod contains the crate-control sidecar
• A headless Service for the sidecar exists
• A Secret containing the bootstrap token is created

4. Verify logs and behavior

Check operator and sidecar logs

Confirm:

• System user is bootstrapped successfully
• JWT settings are applied
• No pod_exec calls are used when the sidecar is present
• /health endpoint reports ready
• SQL requests flow through /exec

Notes

• The sidecar is only reachable inside the cluster.
• No external ports are exposed.
• Existing clusters remain unaffected unless the sidecar is enabled.

Checklist

• Link to issue this PR refers to: https://github.com/crate/cloud/issues/2823
• Relevant changes are reflected in CHANGES.rst
• Added or changed code is covered by tests
• Documentation has been updated if necessary
• Changed code does not contain any breaking changes (or this is a major version change)