TLS for Postgres on Kubernetes: OpenSSL CVE-2021-3450 Edition

Not too long ago I wrote a blog post about how to deploy TLS for Postgres on Kubernetes in attempt to provide a helpful guide from bringing your own TLS/PKI setup to Postgres clusters on Kubernetes. In part, I also wanted a personal reference for how to do it!

However, some things have changed since I first wrote that post. OpenSSL released a fix for CVE-2021-3450 (courtesy to my colleague Tom Swartz for reminding me of this) that prevents users from bypassing some of the x509 certificate verifications. This rolled out in various updates to OpenSSL and has found its way into PGO, the open source Postgres Operator for Kubernetes from Crunchy Data.

For PGO functionality, the OpenSSL fix has zero effect: PGO is already following proper x509 enforcement guidance. However, you can see the effects of this when using Postgres itself: Postgres uses OpenSSL to handle TLS and certificate-based authentication. An improper certificate will now fail verifications.

Given I have now seen this in the wild, I thought it would be good to update the "how to setup Postgres TLS in Kubernetes" entry for a CVE-2021-3450 world.

How to Setup TLS for Postgres in Kubernetes: CVE-2021-3450 Edition

Before reading this, I suggest making yourself familiar with the previous blog post in this unintended series. Like the previous blog post, I am going to generate a ECDSA certificate.

Unlike the previous blog post, to demonstrate the difference, I am also going to set up certificate-based authentication between Postgres instances using PGO, the Postgres Operator. Using the previous recipe with newer versions of OpenSSL, you would not be able to set this up.

The first step to get this working is to copy your template openssl.cnf file. This file is used to provide information about the x509 v3 extensions, which will be required later on in the tutorial. As of this writing on EL8 (RHEL 8 / CentOS 8), you can find this file at /etc/pki/tls/openssl.cnf. I copied it into my working directory:

cp /etc/pki/tls/openssl.cnf ./openssl.cnf

Open up this file. You are going to want to check two things:

1. x509_extensions is uncommented and set to v3_ca.
2. req_extensions is uncommented and set to v3_req.

Also check that the [ v3_req ] block contains at least these values:

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

Generating the Certificate Authority (CA)

From here, let's create a CA. Borrowing from the previous recipe, you can run this command to create your CA, with a few slight modifications:

openssl req \
 -x509 \
 -nodes \
 -newkey ec \
 -pkeyopt ec_paramgen_curve:prime256v1 \
 -pkeyopt ec_param_enc:named_curve \
 -sha384 \
 -keyout ca.key \
 -out ca.crt \
 -days 3650 \
 -extensions v3_ca \
 -subj "/CN=\*"

Based on your OpenSSL configuration the -extensions v3_ca may be superfluous. If you inspect the CA certificate with:

openssl x509 -noout -text -in ca.crt

You should see something similar to:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:c0:65:62:89:55:ff:3f:38:db:88:37:1f:2e:2e:aa:8e:68:ce:92
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: CN = *
        Validity
            Not Before: May 11 15:10:16 2021 GMT
            Not After : May  9 15:10:16 2031 GMT
        Subject: CN = *
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:0b:cc:86:af:44:de:cd:d4:f4:9f:96:01:d2:11:
                    5e:c2:be:46:6a:79:3b:88:cd:a0:94:38:ac:0a:f9:
                    13:51:e9:1c:70:a1:68:a5:2d:03:8d:01:b8:27:d8:
                    83:95:ec:ef:c1:ea:74:e5:b1:65:55:84:18:9e:64:
                    7e:f2:19:44:48
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                6D:A9:B1:FD:13:18:13:DD:7D:58:ED:47:AA:26:FD:43:4C:3B:5C:B3
            X509v3 Authority Key Identifier:
                keyid:6D:A9:B1:FD:13:18:13:DD:7D:58:ED:47:AA:26:FD:43:4C:3B:5C:B3

            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: ecdsa-with-SHA384
         30:45:02:20:08:93:06:bf:b0:94:a1:8d:8e:09:3a:4d:75:13:
         8c:b7:d0:b3:a1:f4:54:50:9e:94:c7:20:c9:f9:51:77:1c:c7:
         02:21:00:b1:c9:30:d4:90:dd:ea:38:0b:9c:bd:05:1b:e9:71:
         75:49:ec:c1:af:d4:83:23:d9:a2:2d:3e:83:ac:27:fb:8a

The main thing to note is:

X509v3 Basic Constraints: critical
    CA:TRUE

which indicates that this certificate represents a certificate authority (CA).

Generating the Server Certificate

Now, let's generate the server certificate. This is where the recipe diverges from the last time.

First, generate the certificate signing request (CSR) that we will use for our CA to sign the certificate:

openssl req \
 -new \
 -newkey ec \
 -nodes \
 -pkeyopt ec_paramgen_curve:prime256v1 \
 -pkeyopt ec_param_enc:named_curve \
 -sha384 \
 -keyout server.key \
 -out server.csr \
 -days 365 \
 -subj "/CN=hippo.pgo"

Now, let's have the CA create the certificate. This time, we will explicitly pass in the v3_req extension to ensure that the key uses are present:

openssl x509 \
 -req \
 -in server.csr \
 -days 365 \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -sha384 \
 -extfile openssl.cnf \
 -extensions v3_req \
 -out server.crt

Let's inspect the server certificate:

openssl x509 -noout -text -in server.crt

Should yield something similar to:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            3f:c7:d8:57:76:3e:05:c6:ea:dc:e2:b9:58:08:3f:a3:de:8d:02:97
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: CN = *
        Validity
            Not Before: May 11 15:15:35 2021 GMT
            Not After : May 11 15:15:35 2022 GMT
        Subject: CN = hippo.pgo
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:7a:f5:54:b5:0f:b2:ae:c7:b4:55:f6:65:97:19:
                    f2:eb:ab:bf:c5:2b:ef:ee:12:4e:82:3b:4b:5f:90:
                    96:89:9c:4a:64:3f:9b:e6:c1:19:bc:18:80:cf:ab:
                    e6:0e:a9:7c:35:bf:4a:8e:e6:3d:78:32:ec:1a:8e:
                    32:25:ac:75:8b
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment
    Signature Algorithm: ecdsa-with-SHA384
         30:46:02:21:00:ab:0c:3d:cb:e2:38:e9:66:86:3c:ee:5f:ca:
         c7:d4:3d:db:e0:f4:e7:cd:64:e8:ff:dc:cf:53:58:8b:3c:fe:
         81:02:21:00:b6:2f:a2:20:f5:5a:73:c5:49:9d:8c:5b:e4:52:
         28:bb:9c:d3:13:e9:57:6f:d9:05:45:68:37:65:54:13:22:0d

Again, here is the main part to notice:

X509v3 Basic Constraints: CA:FALSE
X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment

The main part to notice is the x509 v3 key usages section. The previous recipe did not set this; a main component of CVE 2021-3450 is that OpenSSL is ensuring that the key usages are checked when strict verification is enabled (which is a good idea). By using the above method, we can ensure that we are generating certificates that are prepared for newer versions of OpenSSL.

To really see this in action in the context of PGO and Postgres, let's generate a certificate for replica authentication.

Generating the Replica Authentication Certificate

The formula for generating the replica authentication certificate is similar for the server certificate. The two noticeable differences:

1. The name of the file is prefixed with replica
2. The CN will have the name primaryuser; this is to conveniently map to what PGO configures in Postgres for certificate-based authentication between replicas.

Here is the recipe below:

openssl req \
 -new \
 -newkey ec \
 -nodes \
 -pkeyopt ec_paramgen_curve:prime256v1 \
 -pkeyopt ec_param_enc:named_curve \
 -sha384 \
 -keyout replicas.key \
 -out replicas.csr \
 -days 365 \
 -subj "/CN=primaryuser"

openssl x509 \
 -req \
 -in replicas.csr \
 -days 365 \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -sha384 \
 -extfile openssl.cnf \
 -extensions v3_req \
 -out replicas.crt

Feel free to inspect the certificate with openssl x509 -noout -text -in replicas.crt to ensure that the appropriate x509 v3 key usages are set.

Let's now tie this into creating a Postgres cluster.

Creating a TLS Enforced Postgres Cluster with Certificate Authentication Between Replicas

(Say that section heading ten times fast!)

This recipe will assume that you have already deployed PGO to Kubernetes and you are working in a namespace named pgo.

First, we need to create a few Secrets for the CA, the server certificate, and the replica certificate. We can do that with the following commands:

kubectl create secret generic -n pgo postgres-ca --from-file=ca.crt=ca.crt
kubectl create secret tls -n pgo hippo.tls --key=server.key --cert=server.crt
kubectl create secret tls -n pgo hippo-replicas.tls --key=replicas.key
--cert=replicas.crt

Once those are created, let's create a cluster that enforces TLS and includes TLS authentication between replicas:

pgo create cluster hippo \
 --tls-only \
 --replica-count=1 \
 --server-ca-secret=postgres-ca \
 --server-tls-secret=hippo.tls \
 --replication-tls-secret=hippo-replicas.tls

You may need to wait a few moments for everything to provision.

How will we know that everything worked? For one, your replica instance will be connected, healthy, and receiving changes from the primary. We can verify this in a few ways.

First, inspect the logs on the replica:

kubectl -n jkatz logs --selector=pg-cluster=hippo,role=replica --tail=100

You should see something similar to this (some output truncated)

# ...

pg_hba:

- hostssl replication primaryuser 0.0.0.0/0 cert

# ...

2021-05-11 15:27:37,204 INFO: replica has been created using pgbackrest
2021-05-11 15:27:37,209 INFO: bootstrapped from leader 'hippo-778c9bd7b5-hnz44'

# ...

2021-05-11 15:27:40,488 INFO: establishing a new patroni connection to the
postgres cluster
2021-05-11 15:27:40,565 INFO: no action. i am a secondary and i am following a
leader

# ...

This logs indicate that we are able to connect to a primary, authenticated via certificates.

Next Steps

Open source and all of its dependencies can present fun challenges to ensuring your environment still works when software updates occur, particularly when the update deals with closing a CVE. It's important to continuously test your environment to ensure it is still working as expected...just as its important for recipe writers to continuously test their recipes!