Database Security Best Practices on Kubernetes

As more data workloads shift to running on Kubernetes, one of the important topics to consider is security of your data. Kubernetes brings many conveniences for securing workloads with the ability to extend security functionality databases through the use of the Operator pattern. Database security best practices on Kubernetes is a frequent conversation we're having with our customers around deploying PostgreSQL on Kubernetes with PGO, the open source Postgres Operator from Crunchy Data.

Even with security conveniences that Kubernetes provides, you should be aware of some database security best practices when managing your data on Kubernetes. Many Kubernetes Operators help deploy your databases to follow these best practices by default, including as PGO, while also providing ways to add your own customizations to further secure your data.

While we at Crunchy Data hold open source PostgreSQL near-and-dear to our hearts, we wanted to provide a list of best practices for securing databases of any nature on Kubernetes. Let's take a look at what you can to secure your data the cloud-native way!

Run as an Unprivileged User

Containers run as a single user. This is great, unless this user is root or a user with OS superuser privileges: this can potentially wreak havoc on your system as root can install additional programs within the container or, if there is a breakout, access the rest of your filesystem.

There are some simple things you can to mitigate this though!

• Ensure your container image starts as a non-root user (e.g. ensure USER is not 0 or root).
• Set the Pod Security Context to run as non-root. You can do this by setting runAsNonRoot to true.
• Set the Container Security Context to disable privilege escalation by setting allowPrivilegeEscalation to false.

You may not want to explicitly set runAsUser to a specific non-root UID: systems like OpenShift will vary the UIDs each time a Pod is spawned, allowing for additional entropy in the management and protection of the system.

Your database software may also have some protections against running as a privileged user. For example, Postgres will refuse to start up if it detects that the root user is trying to boot up the database.

Encrypt your Data

There are two types of methods of encryption when protecting your data in Kubernetes: encryption-in-transit and encryption-at-rest.

Encryption-in-transit ensures that data is secured over communication channels. This is especially important given the distributed nature of Kubernetes. If you're on an untrusted network or a multi-tenant system, you will want to ensure all communications with your database system are encrypted.

Using TLS (preferably 1.3, but at least 1.2) allows for your applications to securely communicate with your database over an encrypted channel and ensures for verification of connected endpoints. PGO, the open source Postgres Operator from Crunchy Data, enables TLS for Postgres databases on Kubernetes by default which makes it easy to encrypt communication.

It's also best practice to encrypt data on disk, which helps to prevent offline attacks and data tampering. Your database software may provide this through transparent data encryption; if it does not, you should use a Kubernetes storage system that provides encryption or ensure your OS is encrypting your storage layer.

Credential Management

Once your database is up and running, you need to consider how your applications can access your data. Credential management is an important step in this process.

Credentials are commonly stored in Kubernetes Secrets, which provide a way to pass sensitive information into running Pods. These include credentials like passwords, TLS certifications, access tokens, and more.

Based upon your compliance requirements, you may want to look into further securing your Kubernetes Secrets. You can encrypt your Secrets using one of Kubernetes built-in methods (secretbox is the current recommendation), a key-management system (KMS) for stronger security, or a 3rd-party system like vault that injects the data into Secrets.

With your database credentials stored in Secrets, you then need to know how you want to allow for your application to access them. For example, PGO makes it easy for applications following 12 factor principles to connect their application to the database without having to know the credentials. This also makes it easier to rotate the credentials, helping to protect the case where the database password is leaked.

Your database may also support various single sign-on (SSO) systems that help with credential management. For example, PostgreSQL supports authentication via TLS certificates. While PGO comes with its own PKI, you can also use PGO with certmanager to manage both TLS and login credentials for Postgres users with full management and rotation!

Keep Database Software Up-to-Date

This may sound simple, but one of the biggest attack vectors on your database is not keeping it up-to-date with the latest security fix releases.

For any database system you are running, you should know what the update schedule is. For example, PostgreSQL schedules update releases once-per-quarter and provides up to a year in advance.

Kubernetes Operators can also help with keeping database software up-to-date by allowing you to roll out updates with little to no downtime. PGO allows you to swap database images using a "rolling update" strategy: the Postgres Operator will first apply updates to its replicas, cuts over from the primary to an updated replica, and finally updates the old primary. This strategy ensures your application remains up and running while PGO applies the database software updates.

Follow Configuration Best Practices

It is one thing to deploy your database on Kubernetes: you then need to configure it for secure use. This can be a daunting task for complex database systems.

Similar to guidance published on securing Kubernetes, you can refer to the DISA Secure Technical Implementation Guides (STIGs) and CIS Benchmarks for securing databases. These include instructions for optimal configurations for security, enabling logging and monitoring, choosing proper authentication methods, and more.

For Postgres, Crunchy Data collaborated on both the PostgreSQL STIG and the CIS Benchmark for PostgreSQL and has built tools to help validate and configure PostgreSQL according to these guidelines.

Limit Where You Can Write

One advantage of running a database in a container is that you can provide an immutable, or unchanging, file system to use. This limits the surface area where unauthorized users or programs can add malicious software to your system or overwrite key system files.

For your databases, you should only allow writes to your data volumes, i.e. your persistent volumes. You can enforce this by setting the Container Security Context setting of readOnlyFileSystem to true (PGO does this for your Postgres databases by default). This is not foolproof: an attacker can still use your database software to write out files to the data volumes, but you will have taken other measures to protect that by keeping your database software up-to-date and following configuration best practices!

Securing The "Weakest Link"

You can be secure in one area, and insecure in another. Even if your database system is secure, you need to check other layers of the system that your data resides on.

If you're running Kubernetes, there can be a lot of different areas you need to check, including:

• Your physical hardware and storage
• Your operating system
• Your applications
• Who has access

and of course, Kubernetes itself. There are many comprehensive guides for how to secure all these different parts of your system, including DISA STIGs and CIS Benchmarks.

You should also think about where your data is transferred. For example, let's say you give your developers access to copies of the production data for debugging purposes. Make sure to include these pseudo-development databases in your security regime, otherwise those databases become the weakest link in your system!

Conclusion

So, given all this, where do you start? On the Kubernetes side, there are open source tools around like KubeLinter and kubescape that can give you an analysis of your Kubernetes environment and your application runtimes.

You should also evaluate your database deployment tooling to ensure that it is following database security best practices. PGO, the open source Postgres Operator from Crunchy Data, was designed with both Kubernetes and Postgres security considerations in mind, and can help get you started towards running secure production Postgres workloads on Kubernetes.

(Interested in seeing PGO in action? Join us for a webinar on Wednesday, Nov 17th.)