Deploy TLS for PgBouncer in Kubernetes

TLS allows for the secure transmission of data between systems and is also a requirement of many production environments. Part of setting up TLS is ensuring anything communicating over a network within your system also has TLS. If you are not encrypting traffic between all your endpoints, you open yourself up to snooping.

An earlier post describes how to set up PostgreSQL clusters with TLS on Kubernetes using the Crunchy Data PostgreSQL Operator. This setup works well for creating encrypted connections between your application and /database. The Postgres Operator also supports the PgBouncer connection pooler, which begs the question: how do you set up TLS connections for PgBouncer in Kubernetes?

Since PgBouncer sits between your application and PostgreSQL, setting up TLS for PgBouncer is a more involved process. The Postgres Operator simplifies this process, but it is helpful to understand what is going on behind the scenes to ensure communications encrypt throughout.

Setting up TLS for PostgreSQL

To set up TLS for PgBouncer in Kubernetes, you must first set up TLS for PostgreSQL. This is a requirement for several reasons:

• Good design: it ensures that all communications are encrypted.
• Sharing a common certificate authority: this helps for more advanced setups that want to provide both verification and encrypted communication.

Note that once TLS is set up for PgBouncer, all connections to PgBouncer must be over TLS. This helps reduce the complexity of the overall setup. You are also guaranteed that you get the desired behavior: encrypted communication.

The previous blog post describes in detail how to set up TLS for a PostgreSQL cluster, so I encourage you to use that as a reference. For the purposes of this example, please ensure that you have created a TLS-enabled PostgreSQL cluster named hippo that looks similar to this:

pgo create cluster hippo --tls-only \
--server-ca-secret=postgresql-ca \
--server-tls-secret=hippo.tls

Note that this creates a PostgreSQL cluster that requires all connections to be over TLS. This is not a requirement for deploying PgBouncer over TLS, but is good practice.

Deploy PgBouncer with TLS

Once you have your PostgreSQL cluster up and running with TLS, you can deploy a PgBouncer with TLS! You can enable TLS for PgBouncer with the pgo create pgbouncer command and the --tls-secret flag, If you want to get set up quickly, you could reuse the TLS keypair that you generated for the PostgreSQL cluster (hippo.tls), but instead let's create a new key/certificate pair for the PgBouncer instance.

The Postgres Operator creates a Kubernetes Service for PgBouncer by using the name of the cluster (e.g. hippo) and affixing the -pgbouncer suffix to it. This means that for our hippo PostgreSQL cluster, the name of the PgBouncer Service would be hippo-pgbouncer.

We know that the DNS for the cluster for this deployment would be "hippo-pgbouncer.pgo" and we can create a certificate with that name. Use the certificate authority created in the previous blog post to generate the certificate:

# generate the private key and certificate signing request (CSR)
openssl req \
  -new \
  -newkey ec \
  -nodes \
  -pkeyopt ec_paramgen_curve:prime256v1 \
  -pkeyopt ec_param_enc:named_curve \
  -sha384 \
  -keyout pgbouncer.key \
  -out pgbouncer.csr \
  -days 365 \
  -subj "/CN=hippo-pgbouncer.pgo"

# generate the certificate
openssl x509 \
  -req \
  -in pgbouncer.csr \
  -days 365 \
  -CA ca.crt \
  -CAkey ca.key \
  -CAcreateserial \
  -sha384 \
  -out pgbouncer.crt

Use the output from the above commands to create a TLS Secret that the PgBouncer deployment will use:

kubectl create secret tls hippo-pgbouncer.tls -n pgo --cert=pgbouncer.crt --key=pgbouncer.key

With the TLS Secret in place, you can now deploy PgBouncer with TLS enabled:

pgo create pgbouncer hippo --tls-secret=hippo-pgbouncer.tls

Wait a few moments for PgBouncer to initialize and then test your connection. Similar to the previous blog post, in a different window set up a port forward to your local machine to the PgBouncer service:

kubectl -n pgo port-forward svc/hippo-pgbouncer 5432:5432

Connect to your Postgres cluster via PgBouncer using the following command:

PGPASSWORD=$(kubectl -n pgo get secrets hippo-postgres-secret -o jsonpath="{.data.password}" | base64 -d) psql -h localhost -U postgres hippo

If you successfully set up TLS communication, you should see something similar to:

psql (13.1)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

hippo=>

Next Steps

The setup demonstrated above also allows for using Postgres' verify-full TLS mode (explained in the previous blog post), though you will need to customize the PgBouncer configuration to set verify-full to work between PgBouncer and PostgreSQL.

The security of your system is only as good as your weakest link, so if communicating over TLS is one of your requirements, you need to ensure that your services are communicating over TLS. Adding TLS support for PgBouncer in the PostgreSQL Operator helps to meet these requirements in an automated fashion.