From df42e1a51f4ba3b89ddebb792277a067921faf54 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Thu, 30 Apr 2026 17:58:31 +0200 Subject: [PATCH] K8SPXC-1754 Updated the update certificates doc with automated steps Enhanced instructions for checking certificate expiration and validity, and improved overall wording for better clarity. --- docs/tls-update.md | 213 +++++++++++++++++++++++---------------------- 1 file changed, 108 insertions(+), 105 deletions(-) diff --git a/docs/tls-update.md b/docs/tls-update.md index 6e46073..25209c7 100644 --- a/docs/tls-update.md +++ b/docs/tls-update.md @@ -2,160 +2,163 @@ How your TLS certificates are updated depends on how they were created: -* Certificates generated by the Operator are long-term. If you need to rotate them, you must do it manually. +* Certificates generated by the Operator are long-term. If you need to update them, you must provide new certificates to the Operator. Jump to the sections below for the steps. * Certificates issued by the cert-manager are short-term. They are valid for 3 months. The cert-manager automatically reissues the certificates on schedule and without downtime. * Certificates manually generated by you are not renewed automatically. It is your responsibility to timely update them. Use the steps in the following sections for how to do it. -This document describes how to update the internal certificate. +This document describes how to update the internal certificates. Use the same steps to update external certificates. Read more about internal and external certificates in the [About TLS](TLS.md) section. -## Check your certificates for expiration +## Before you start -If you [use cert-manager](tls-cert-manager.md): +Export the namespace where your cluster is deployed as an environment variable: -1. Check the necessary secrets names (`cluster1-ssl` and - `cluster1-ssl-internal` by default): +```bash +export NAMESPACE= +``` - ```bash - kubectl get certificate - ``` +## Check your certificates for expiration - You will have the following response: +=== "With cert-manager" - ``` {.text .no-copy} - NAME READY SECRET AGE - cluster1-ca-cert True cluster1-ca-cert 49m - cluster1-ssl True cluster1-ssl 49m - cluster1-ssl-internal True cluster1-ssl-internal 49m - ``` + If you [use cert-manager](tls-cert-manager.md) to generate certificates, do the following. -2. Optionally you can also check that the certificates issuer is up and running: + 1. Check the necessary secrets names (`cluster1-ssl` and + `cluster1-ssl-internal` by default): - ```bash - kubectl get issuer - ``` + ```bash + kubectl get certificate -n $NAMESPACE + ``` - The response should be as follows: + ??? example "Expected output" - ``` {.text .no-copy} - NAME READY AGE - cluster1-pxc-ca-issuer True 49m - cluster1-pxc-issuer True 49m - ``` + ``` {.text .no-copy} + NAME READY SECRET AGE + cluster1-ca-cert True cluster1-ca-cert 49m + cluster1-ssl True cluster1-ssl 49m + cluster1-ssl-internal True cluster1-ssl-internal 49m + ``` - !!! note + 2. Optionally you can also check that the certificates issuer is up and running: - If you don't use cert-manager, list your secrets: + ```bash + kubectl get issuer -n $NAMESPACE + ``` + + ??? example "Expected output" + + ``` {.text .no-copy} + NAME READY AGE + cluster1-pxc-ca-issuer True 49m + cluster1-pxc-issuer True 49m + ``` + 3. Use the following command to find out the certificates validity dates, + substituting Secret names if necessary: ```bash - kubectl get secrets -n $NAMESPACE + { + kubectl -n $NAMESPACE get secret/cluster1-ssl-internal -o jsonpath='{.data.tls\.crt'} | base64 --decode | openssl x509 -inform pem -noout -text | grep -E "Not Before|Not After" + kubectl -n $NAMESPACE get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep -E "Not Before|Not After" + } ``` - Then either use the default ones or the one you created + ??? example "Sample output" + + ```{.text .no-copy} + notBefore=Nov 7 10:54:00 2025 GMT + notAfter=Nov 7 10:54:00 2026 GMT + ``` -3. Use the following command to find out the certificates validity dates, - substituting Secrets names if necessary: +=== "Operator-generated or manual TLS certificates" - ```bash - { - kubectl get secret/cluster1-ssl-internal -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep "Not After" - kubectl get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep "Not After" - } - ``` + If the Operator created TLS certificates or you created them yourself, do the following. - ??? example "Sample output" + 1. List Secrets in your namespace and note the names used for external and internal TLS. By default these are `cluster1-ssl` and `cluster1-ssl-internal`. Otherwise the Secret names match the values of the `sslSecretName` and `sslInternalSecretName` options in your cluster Custom Resource. - ```{.text .no-copy} - notBefore=Nov 7 10:54:00 2025 GMT - notAfter=Nov 7 10:54:00 2026 GMT + ```bash + kubectl get secrets -n $NAMESPACE ``` -### Update certificates without downtime + 2. Optionally, confirm that both TLS Secrets exist and are of type `kubernetes.io/tls`: -If you don’t use cert-manager and have *created certificates manually*, -you can follow the next steps to perform a no-downtime update of these -certificates *if they are still valid*. + ```bash + kubectl get secrets -n $NAMESPACE -o custom-columns=NAME:.metadata.name,TYPE:.type | grep -E 'NAME|ssl' + ``` -!!! note + Adjust the filter or Secret names to match your cluster. - For already expired certificates, follow the alternative way. + 3. Use the following command to find out the certificates validity dates, + substituting Secret names if necessary: -Having non-expired certificates, you can roll out new certificates (both CA and TLS) with the Operator -as follows. + ```bash + { + kubectl -n $NAMESPACE get secret/cluster1-ssl-internal -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep -E "Not Before|Not After" + kubectl -n $NAMESPACE get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode | openssl x509 -inform pem -noout -text | grep -E "Not Before|Not After" + } + ``` -1. Generate a new CA certificate (`ca.pem`), a new TLS certificate (server.pem) and a key for it (server-key.pem). + ??? example "Sample output" + + ```{.text .no-copy} + Not Before=Nov 7 10:44:00 2025 GMT + Not After=Nov 7 10:44:00 2026 GMT + Not Before=Nov 7 10:54:00 2025 GMT + Not After=Nov 7 10:54:00 2026 GMT + ``` -2. Get the current CA (`ca.pem.old`) and TLS (`tls.pem.old`) certificates - and the TLS certificate key (`tls.key.old`): +### Update valid certificates without downtime - ```bash - kubectl get secret/cluster1-ssl -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.pem.old - kubectl get secret/cluster1-ssl -o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.pem.old - kubectl get secret/cluster1-ssl -o jsonpath='{.data.tls\.key}' | base64 --decode > tls.key.old - ``` +This section explains how to update your certificates that are still valid, without downtime. For already expired certificates, refer to [Update expired certificates](#update-expired-certificates-with-downtime) section. -3. Combine new and current `ca.pem` into a `ca.pem.combined` file: +**When to use this method:** - ```bash - cat ca.pem ca.pem.old >> ca.pem.combined - ``` +- Your certificates were generated by the Operator, but *not* using cert-manager +- You generated the certificates manually or provided pre-existing certificates -4. Create a new Secrets object with the *old* TLS certificate (`tls.pem.old`) and key (`tls.key.old`), but a *new combined* `ca.pem` (`ca.pem.combined`): +To roll out new certificates (both CA and TLS) with the Operator, follow these steps. - ``` bash - kubectl create secret generic cluster1-ssl \ - --from-file=tls.crt=server.pem.old \ - --from-file=tls.key=server-key.pem.old \ - --from-file=ca.crt=ca.pem.combined \ - --type=kubernetes.io/tls -o yaml --dry-run=client | kubectl apply -f - - ``` - -5. The cluster will go through a rolling restart. This process will not cause issues, because every node has the old TLS certificate/key, and both new - and old CA certificates. +1. Generate a new CA certificate (`ca.pem`), a new TLS certificate (`server.pem`) and a key for it (`server-key.pem`). Refer to the [Generate certificates manually](tls-manual.md) tutorial for the steps. + +2. Create a Secret object and provide the new certificates within it. The Secret name must be in the format `-new`. + + For example, if the existing Secret name is `cluster1-ssl-internal`, the new Secret name is `cluster1-ssl-internal-new`. + + In the following command: -6. Create a new Secrets object again. This time use a new TLS certificate (`server.pem` in the example) and a new TLS key (`server-key.pem`), and again the combined CA certificate (`ca.pem.combined`): + * `ca.pem` is added to the Secret as `ca.crt` + * `server.pem` is added to the Secret as `tls.crt` + * `server-key.pem` is added to the Secret as `tls.key` - ``` bash - kubectl create secret generic cluster1-ssl \ + ```bash + kubectl -n $NAMESPACE create secret generic cluster1-ssl-internal-new \ + --from-file=ca.crt=ca.pem \ --from-file=tls.crt=server.pem \ - --from-file=tls.key=server-key.pem \ - --from-file=ca.crt=ca.pem.combined \ - --type=Opague -o yaml --dry-run=client | kubectl apply -f - + --from-file=tls.key=server-key.pem ``` -7. The cluster will go through a rolling restart. This process will not cause issues, because every node already has a new CA certificate (as a part - of the combined CA certificate), and can successfully allow joiners with new - TLS certificate to join. A joiner node also has a combined CA certificate, so - it can authenticate against older TLS certificate. +On the next reconcile, the Operator detects the Secret and automatically updates the certificates. This triggers the rolling restart of the database Pods. -8. Create a final Secrets object: use the new TLS certificate (`server.pmm`) and - its key (`server-key.pem`), and only the new CA certificate (`ca.pem`): +??? note "What happens under the hood" + + When the Operator reconciles the cluster and detects the Secret with the new TLS certificates, it does the following: + + * Combines a new and current CA certificates into a single file + * Updates the current Secret to first append the combined CA certificate and then replace the TLS server and key certificates with new ones + * Drops the old CA certificate leaving only the new one + + On every step, the Operator performs the rolling restart of the database Pods to ensure the cluster availability and inter-node communication during the update. - ``` bash - kubectl create secret generic cluster1-ssl \ - --from-file=tls.crt=server.pem \ - --from-file=tls.key=server-key.pem \ - --from-file=ca.crt=ca.pem \ - --type=Opague -o yaml --dry-run=client | kubectl apply -f - - ``` -9. The cluster will go through a rolling restart, but it will do it - without problems: the old CA certificate is removed, and every node is - already using new TLS certificate and no nodes rely on the old CA - certificate any more. +### Update expired certificates with downtime -### Update certificates with downtime +This section explains how to update the certificates that have already expired. To check certificate validity, see [Check your certificates for expiration](#check-your-certificates-for-expiration). -If your certificates have been already expired (or if you continue to use the -Operator version prior to 1.9.0), you should move through the -*pause - update Secrets - unpause* route as follows. +If your certificates have been already expired, follow the *pause - update Secrets - unpause* procedure below: -1. Pause the cluster [in a standard way](pause.md), and make - sure it has reached its paused state. +1. [Pause the cluster](pause.md), and make sure it has reached its paused state. -2. If cert-manager is used, delete issuer - and TLS certificates: +2. If cert-manager is used, delete issuer and TLS certificates: ```bash { @@ -167,7 +170,7 @@ Operator version prior to 1.9.0), you should move through the 3. Delete Secrets to force the SSL reconciliation: ```bash - kubectl delete secret/cluster1-ssl secret/cluster1-ssl-internal + kubectl -n $NAMESPACE delete secret/cluster1-ssl secret/cluster1-ssl-internal ``` 4. Check certificates to make sure reconciliation have succeeded.