Thin Provisioning with Crunchy Postgres for Kubernetes

One of the major changes that the cloud brought to application and database management was the concept of "thin provisioning." With large amounts of compute or storage resources available behind an API, you can provision what you need now and expand your infrastructure as required. Frameworks like 12Factor and Cloud Native emerged to help developers leverage these new paradigms.

In the past, these capabilities were only accessible to those leveraging public clouds. On-premises, with lead times of weeks or months to provision new servers, planning for growth was essential. Now, with technologies like Kubernetes and OpenShift, you can add resources to an application with an API in your private cloud, just like in the public cloud.

While scaling applications is straightforward with load balancers, containers, and stateless apps, containerized databases and advanced storage technologies bring similar capabilities to databases. With Crunchy Postgres for Kubernetes, growing your data storage is as simple as editing your cluster definition. However, monitoring and manually expanding storage can be time-consuming.

With Crunchy Postgres for Kubernetes 5.6, we’ve automated this process. By setting a starting size and maximum size in your cluster YAML, the operator will automatically monitor and grow your Postgres storage within those limits.

Setting Up Autogrow

Let's look at how you can use this new feature. First, ensure your underlying storage supports disk space expansion:

kubectl describe StorageClass | grep -e Name -e Expansion

Not all storage providers support growing disk space. If your storage provider does not support growing disks, you will not be able to use this feature.

If expandable disks are supported, enable the feature flag:

PGO_FEATURE_GATES="AutoGrowVolumes=true"

Next, create a cluster that will auto-grow:

spec:
  postgresVersion: 16
  instances:
    dataVolumeClaimSpec:
      accessModes: [ReadWriteOnce]
      resources:
        requests:
          storage: 10Gi # Start with only the space you need
        limits:
          storage: 5Ti # Set the upper limit on disk growth

Choose a starting size that will comfortably hold your initial data plus some room for growth.

Autogrow in Action

Imagine starting a new social media site. Your database starts with 10Gi of storage. As you invite people to try it out, data grows. When usage reaches 7690Mi, or 75% of capacity, Crunchy Postgres for Kubernetes automatically adds 5120Mi. This increases the disk size to 15Gi and brings usage back down to 50%. It is also done transparently to the database, which means no downtime. This process continues with the database growing each time it hits 75% of capacity. When it reaches the maximum size that you set, you'll get alerts indicating that you need to expand storage further or clean up some data.

Under the Hood

Crunchy Postgres for Kubernetes watches your disk space - collecting metrics about it at regular intervals. When the Operator sees that your usage is over 75% of capacity, it asks the storage provider to increase the disk size by 50%. This lowers usage back down to 50% of capacity.

After requesting the change, the Operator monitors the volume and waits for the storage provider to update the disk. Generally, these updates take a few minutes.

Keep in mind that different underlying storage providers have different rules. For example, some round sizes up to the nearest Gi. Some throttle how often you can request a size change. These affect small disks more than others, so be sure to choose an initial size that can hold all your data and has some room to grow before triggering your first change in size.

Also, to avoid unnecessary resizing events, consider putting the pgdata directory on its own disk. This will prevent other processes from consuming space or IOPS, and can also help with database performance. You can see how to set up dedicated WAL volumes in the documentation.

We have tested this feature with OpenShift, GKS, and EKS. However, any provider that supports growing PVCs will work. Check your storage provider’s documentation to learn details about how it handles these requests.

For more details, check out the documentation on autogrowable-disk for this new feature. If you have questions or want to see how Crunchy Postgres for Kubernetes can automate your database management, contact us at info@crunchydata.com.