Skip to content

K8SPXC-1754 Updated the update certificates doc with automated steps #307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nastena1606 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

K8SPXC-1754 Updated the update certificates doc with automated steps #307

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
213 changes: 108 additions & 105 deletions docs/tls-update.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,160 +2,163 @@

How your TLS certificates are updated depends on how they were created:

* Certificates generated by the Operator are long-term. If you need to rotate them, you must do it manually.
* Certificates generated by the Operator are long-term. If you need to update them, you must provide new certificates to the Operator. Jump to the sections below for the steps.

* Certificates issued by the cert-manager are short-term. They are valid for 3 months. The cert-manager automatically reissues the certificates on schedule and without downtime.

* Certificates manually generated by you are not renewed automatically. It is your responsibility to timely update them. Use the steps in the following sections for how to do it.

This document describes how to update the internal certificate.
This document describes how to update the internal certificates. Use the same steps to update external certificates. Read more about internal and external certificates in the [About TLS](TLS.md) section.

## Check your certificates for expiration
## Before you start

If you [use cert-manager](tls-cert-manager.md):
Export the namespace where your cluster is deployed as an environment variable:

1. Check the necessary secrets names (`cluster1-ssl` and
`cluster1-ssl-internal` by default):
```bash
export NAMESPACE=<my-namespace>
```

```bash
kubectl get certificate
```
## Check your certificates for expiration

You will have the following response:
=== "With cert-manager"

``` {.text .no-copy}
NAME READY SECRET AGE
cluster1-ca-cert True cluster1-ca-cert 49m
cluster1-ssl True cluster1-ssl 49m
cluster1-ssl-internal True cluster1-ssl-internal 49m
```
If you [use cert-manager](tls-cert-manager.md) to generate certificates, do the following.

2. Optionally you can also check that the certificates issuer is up and running:
1. Check the necessary secrets names (`cluster1-ssl` and
`cluster1-ssl-internal` by default):

```bash
kubectl get issuer
```
```bash
kubectl get certificate -n $NAMESPACE
```

The response should be as follows:
??? example "Expected output"

``` {.text .no-copy}
NAME READY AGE
cluster1-pxc-ca-issuer True 49m
cluster1-pxc-issuer True 49m
```
``` {.text .no-copy}
NAME READY SECRET AGE
cluster1-ca-cert True cluster1-ca-cert 49m
cluster1-ssl True cluster1-ssl 49m
cluster1-ssl-internal True cluster1-ssl-internal 49m
```

!!! note
2. Optionally you can also check that the certificates issuer is up and running:

If you don't use cert-manager, list your secrets:
```bash
kubectl get issuer -n $NAMESPACE
```

??? example "Expected output"

``` {.text .no-copy}
NAME READY AGE
cluster1-pxc-ca-issuer True 49m
cluster1-pxc-issuer True 49m
```
3. Use the following command to find out the certificates validity dates,
substituting Secret names if necessary:

```bash
kubectl get secrets -n $NAMESPACE
{
kubectl -n $NAMESPACE get secret/cluster1-ssl-internal -o jsonpath='{.data.tls\.crt'} | base64 --decode | openssl x509 -inform pem -noout -text | grep -E "Not Before|Not After"
kubectl -n $NAMESPACE get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep -E "Not Before|Not After"
}
```

Then either use the default ones or the one you created
??? example "Sample output"

```{.text .no-copy}
notBefore=Nov 7 10:54:00 2025 GMT
notAfter=Nov 7 10:54:00 2026 GMT
```
Comment on lines +68 to +71
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sample output (notBefore/notAfter) doesn’t match the command shown (openssl ... -text | grep "Not After"), which would typically only print a Not After line (and not notBefore). Update the command (e.g., use -noout -dates and/or adjust grep) or adjust the sample output to match.

Copilot uses AI. Check for mistakes.

3. Use the following command to find out the certificates validity dates,
substituting Secrets names if necessary:
=== "Operator-generated or manual TLS certificates"

```bash
{
kubectl get secret/cluster1-ssl-internal -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep "Not After"
kubectl get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep "Not After"
}
```
If the Operator created TLS certificates or you created them yourself, do the following.

??? example "Sample output"
1. List Secrets in your namespace and note the names used for external and internal TLS. By default these are `cluster1-ssl` and `cluster1-ssl-internal`. Otherwise the Secret names match the values of the `sslSecretName` and `sslInternalSecretName` options in your cluster Custom Resource.

```{.text .no-copy}
notBefore=Nov 7 10:54:00 2025 GMT
notAfter=Nov 7 10:54:00 2026 GMT
```bash
kubectl get secrets -n $NAMESPACE
```

### Update certificates without downtime
2. Optionally, confirm that both TLS Secrets exist and are of type `kubernetes.io/tls`:

If you don’t use cert-manager and have *created certificates manually*,
you can follow the next steps to perform a no-downtime update of these
certificates *if they are still valid*.
```bash
kubectl get secrets -n $NAMESPACE -o custom-columns=NAME:.metadata.name,TYPE:.type | grep -E 'NAME|ssl'
```

!!! note
Adjust the filter or Secret names to match your cluster.

For already expired certificates, follow the alternative way.
3. Use the following command to find out the certificates validity dates,
substituting Secret names if necessary:

Having non-expired certificates, you can roll out new certificates (both CA and TLS) with the Operator
as follows.
```bash
{
kubectl -n $NAMESPACE get secret/cluster1-ssl-internal -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep -E "Not Before|Not After"
kubectl -n $NAMESPACE get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep -E "Not Before|Not After"
}
```

1. Generate a new CA certificate (`ca.pem`), a new TLS certificate (server.pem) and a key for it (server-key.pem).
??? example "Sample output"

```{.text .no-copy}
Not Before=Nov 7 10:44:00 2025 GMT
Not After=Nov 7 10:44:00 2026 GMT
Not Before=Nov 7 10:54:00 2025 GMT
Not After=Nov 7 10:54:00 2026 GMT
```

2. Get the current CA (`ca.pem.old`) and TLS (`tls.pem.old`) certificates
and the TLS certificate key (`tls.key.old`):
### Update valid certificates without downtime

```bash
kubectl get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.pem.old
kubectl get secret/cluster1-ssl -o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.pem.old
kubectl get secret/cluster1-ssl -o jsonpath='{.data.tls\.key}' | base64 --decode > tls.key.old
```
This section explains how to update your certificates that are still valid, without downtime. For already expired certificates, refer to [Update expired certificates](#update-expired-certificates-with-downtime) section.

3. Combine new and current `ca.pem` into a `ca.pem.combined` file:
**When to use this method:**

```bash
cat ca.pem ca.pem.old >> ca.pem.combined
```
- Your certificates were generated by the Operator, but *not* using cert-manager
- You generated the certificates manually or provided pre-existing certificates

4. Create a new Secrets object with the *old* TLS certificate (`tls.pem.old`) and key (`tls.key.old`), but a *new combined* `ca.pem` (`ca.pem.combined`):
To roll out new certificates (both CA and TLS) with the Operator, follow these steps.

``` bash
kubectl create secret generic cluster1-ssl \
--from-file=tls.crt=server.pem.old \
--from-file=tls.key=server-key.pem.old \
--from-file=ca.crt=ca.pem.combined \
--type=kubernetes.io/tls -o yaml --dry-run=client | kubectl apply -f -
```

5. The cluster will go through a rolling restart. This process will not cause issues, because every node has the old TLS certificate/key, and both new
and old CA certificates.
1. Generate a new CA certificate (`ca.pem`), a new TLS certificate (`server.pem`) and a key for it (`server-key.pem`). Refer to the [Generate certificates manually](tls-manual.md) tutorial for the steps.

2. Create a Secret object and provide the new certificates within it. The Secret name must be in the format `<existing-secret>-new`.

For example, if the existing Secret name is `cluster1-ssl-internal`, the new Secret name is `cluster1-ssl-internal-new`.

In the following command:

6. Create a new Secrets object again. This time use a new TLS certificate (`server.pem` in the example) and a new TLS key (`server-key.pem`), and again the combined CA certificate (`ca.pem.combined`):
* `ca.pem` is added to the Secret as `ca.crt`
* `server.pem` is added to the Secret as `tls.crt`
* `server-key.pem` is added to the Secret as `tls.key`

``` bash
kubectl create secret generic cluster1-ssl \
```bash
kubectl -n $NAMESPACE create secret generic cluster1-ssl-internal-new \
--from-file=ca.crt=ca.pem \
--from-file=tls.crt=server.pem \
--from-file=tls.key=server-key.pem \
--from-file=ca.crt=ca.pem.combined \
--type=Opague -o yaml --dry-run=client | kubectl apply -f -
--from-file=tls.key=server-key.pem
```

7. The cluster will go through a rolling restart. This process will not cause issues, because every node already has a new CA certificate (as a part
of the combined CA certificate), and can successfully allow joiners with new
TLS certificate to join. A joiner node also has a combined CA certificate, so
it can authenticate against older TLS certificate.
On the next reconcile, the Operator detects the Secret and automatically updates the certificates. This triggers the rolling restart of the database Pods.

8. Create a final Secrets object: use the new TLS certificate (`server.pmm`) and
its key (`server-key.pem`), and only the new CA certificate (`ca.pem`):
??? note "What happens under the hood"

When the Operator reconciles the cluster and detects the Secret with the new TLS certificates, it does the following:

* Combines a new and current CA certificates into a single file
* Updates the current Secret to first append the combined CA certificate and then replace the TLS server and key certificates with new ones
* Drops the old CA certificate leaving only the new one

On every step, the Operator performs the rolling restart of the database Pods to ensure the cluster availability and inter-node communication during the update.

``` bash
kubectl create secret generic cluster1-ssl \
--from-file=tls.crt=server.pem \
--from-file=tls.key=server-key.pem \
--from-file=ca.crt=ca.pem \
--type=Opague -o yaml --dry-run=client | kubectl apply -f -
```

9. The cluster will go through a rolling restart, but it will do it
without problems: the old CA certificate is removed, and every node is
already using new TLS certificate and no nodes rely on the old CA
certificate any more.
### Update expired certificates with downtime

### Update certificates with downtime
This section explains how to update the certificates that have already expired. To check certificate validity, see [Check your certificates for expiration](#check-your-certificates-for-expiration).

If your certificates have been already expired (or if you continue to use the
Operator version prior to 1.9.0), you should move through the
*pause - update Secrets - unpause* route as follows.
If your certificates have been already expired, follow the *pause - update Secrets - unpause* procedure below:

1. Pause the cluster [in a standard way](pause.md), and make
sure it has reached its paused state.
1. [Pause the cluster](pause.md), and make sure it has reached its paused state.

2. If cert-manager is used, delete issuer
and TLS certificates:
2. If cert-manager is used, delete issuer and TLS certificates:

```bash
{
Expand All @@ -167,7 +170,7 @@ Operator version prior to 1.9.0), you should move through the
3. Delete Secrets to force the SSL reconciliation:

```bash
kubectl delete secret/cluster1-ssl secret/cluster1-ssl-internal
kubectl -n $NAMESPACE delete secret/cluster1-ssl secret/cluster1-ssl-internal
```

4. Check certificates to make sure reconciliation have succeeded.
Expand Down