In today's landscape of complex systems with numerous observability options, OpenTelemetry has emerged as the standard for collecting logging and metrics. It creates a vendor-agnostic platform that works with almost any source and destination by taking in logs and metrics from all components, standardizing them, and routing them where needed. Though setup requires effort, the payoff is substantial: a unified view of your entire system, even across distributed environments.

With Crunchy Postgres for Kubernetes 5.8, we've automated OpenTelemetry for your Postgres databases and related services. By activating a few settings, your metrics and logs can flow to various OpenTelemetry-compatible systems, letting you focus on maintaining healthy Postgres deployments without the burden of building complex monitoring infrastructure.

Let's explore why we chose OpenTelemetry as our foundation, how to set it up, and ways to extend it to deliver your critical data to external systems.

Observability for Postgres

At its heart, observability means understanding a system's state by analyzing its outputs. This includes examining logs and watching performance metrics to gain broader insights into system behavior and troubleshoot issues. Collecting this data centrally provides comprehensive visibility without requiring deep knowledge of internal implementations.

Example of some of the logs and metrics available from Postgres

Managing Postgres at scale requires viewing all system outputs consistently. The challenge is that each system has its own logging solution with different formats for logs and metrics, and the sheer volume can be overwhelming.

This is where OpenTelemetry comes into play. By providing a consistent standard and framework for managing these external outputs, OpenTelemetry allows you to bring consistency to the various outputs produced by the systems within your Postgres deployments. This consistency not only streamlines and simplifies your ability to view logging and metrics information, but it also allows you to manage those logs and metrics using a variety of OpenTelemetry-compatible services and backends.

This means you can focus on viewing and understanding your system’s outputs, without having to worry about vendor lock-in or reimplementing your solution for collecting logs and metrics when switching and/or trying out new observability solutions and strategies.

Crunchy Postgres for Kubernetes now leverages OpenTelemetry out-of-the-box to unify logs and metrics from all your Postgres databases. This covers not just the database itself, but also all of the supporting systems and services. Since Crunchy Postgres for Kubernetes handles the heavy lifting of collecting, filtering, and transforming data according to OpenTelemetry standards, you can concentrate on analyzing those outputs through your preferred observability platforms and dashboards.

OpenTelemetry in Action using Crunchy Postgres for Kubernetes

Using OpenTelemetry within Crunchy Postgres for Kubernetes is easy! This section will walk you through the steps required to enable OpenTelemetry for both metrics and logging. As you will see, with just a few simple steps you can enhance the observability of your Postgres deployments, and obtain deeper insights into the health of the databases and components comprising those deployments.

Enabling the OpenTelemetry for Logging & Metrics

In order to use OpenTelemetry for logging and metrics, the OpenTelemetryLogging and OpenTelemetryMetrics feature gates must be enabled. Please see the Feature Gate Installation Guide for guidance on how to enable these feature gates using your method of installing Crunchy Postgres for Kubernetes.

Once the feature gates are enabled and your PostgresCluster spec has been updated according to the OpenTelemetry logging and metrics guides, you will see OpenTelemetry collector sidecars deployed alongside the various Pods deployed for any new or existing PostgresCluster’s, as shown in the following diagram:

These sidecars will be responsible for collecting and transforming any logs and metrics, and then exporting them to one or more OpenTelemetry-compatible services or backends.

Configuring OpenTelemetry for Google Cloud Logging

Let’s look at how you can use Crunchy Postgres for Kubernetes and OpenTelemetry to export logs  to Google Cloud Logging. This will provide you with a robust solution for viewing logging information across the various components comprising your Postgres cluster.

Since we will be using Google Cloud Logging, I will be using GKE for my Kubernetes. This will allow me to leverage some of the built-in methods Google provides to streamline and simplify exporting OpenTelemetry logs to Google Cloud Logging.

1. First, make sure that the OpenTelemetryLogging and OpenTelemetryMetrics feature gates are enabled. Then create a PostgresCluster with two Postgres replicas, as well as pgBackRest and PgBouncer deployments:

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: hippo-otel
spec:
  imagePullSecrets:
    - name: crunchy-regcred
  postgresVersion: 17
  instances:
    - name: instance1
      dataVolumeClaimSpec:
        accessModes:
          - 'ReadWriteOnce'
        resources:
          requests:
            storage: 1Gi
  backups:
    pgbackrest:
      repos:
        - name: repo1
          volume:
            volumeClaimSpec:
              accessModes:
                - 'ReadWriteOnce'
              resources:
                requests:
                  storage: 1Gi
  proxy:
    pgBouncer: {}

This will allow us to capture OpenTelemetry logs from the Postgres database, as well as from the Disaster Recovery and connection pooling components.

2. Next, update the PostgresCluster with the following instrumentation section:

instrumentation:
  config:
    detectors:
      - name: gcp
    exporters:
      googlecloud:
        log:
          default_log_name: 'collector-exported-log'
          resource_filters:
            - prefix: 'k8s'
            - prefix: 'db'
  logs:
    exporters: ['googlecloud']

This will add a simple OpenTelemetry exporter for Google Cloud logging.  Note that the googlecloud exporter configuration comes directly from the Google Cloud Exporter configuration reference, which means you can further tune your exporter as needed according to that doc.  And for additional details about the other settings available under the instrumentation sections, see the OpenTelemetry logging guide in the Crunchy Postgres for Kubernetes documentation.

3. Now lets navigate to Google Cloud Logging to view logs for hippo-otel. We will specifically navigate to the “Logs Explorer” view, and then run the following query:

logName=~".*/logs/collector-exported-log"

Running this query will display any logs coming from the PostgresCluster, according to the default_log_name configured for the exporter in the previous step.

And that's it! With just these few steps, you can now start viewing and analyzing logs for your deployment using Google Cloud Logging.

Let's say you want to look at any connection pooling logs coming from the hippo-otel cluster.  Doing this is as simple as adding the following to your query:

resource.labels.container_name="pgbouncer"

And then running the query again.

So as you can clearly see, Crunchy Postgres for Kubernetes puts the various logs for your system directly at your fingertips, allowing you to quickly, easily and seamlessly access the logging information you require to better understand your running deployments.

Setting up OpenTelemetry Metrics

In Crunchy Postgres for Kubernetes v5.8, the switch to OpenTelemertry for metrics is a drop-in replacement for the previous monitoring architecture that used the Prometheus Postgres Exporter.  Therefore, you can still leverage the power of the pgMonitor project to obtain crucial insights into your Postgres deployments, and then visualize those metrics information using Grafana dashboards.  However, instead of the Postgres Exporter running as a sidecar alongside the components making up your PostgresCluster, an OpenTelemetry collector will now be utilized instead.  This means your metrics data will now be exported into Prometheus using the OpenTelemetry standard.

Let’s go ahead and enable OpenTelemetry logging for the hippo-otel PostgresCluster we created in the previous demonstration of OTel logging.

1. If you followed the steps above, the  OpenTelemetryMetrics feature gate should already be enabled, so, configuring OpenTelemetry metrics simply requires the following in your spec:

spec:
  instrumentation: {}

2. From there, install Prometheus and Grafana according to installation instructions to start viewing any OpenTelemetry metrics.  For this example, I will install these components using the crunchy-monitoring Helm chart.

helm install crunchy-monitoring -n postgres-operator \
oci://registry.developers.crunchydata.com/crunchydata/crunchy-monitoring \
--set-json grafana='{"admin":{"username":"admin","password":"admin"}}'

3. Next, port-forward to the Grafana Service, and login using the credentials provided in the previous step.

kubectl -n postgres-operator port-forward service/crunchy-monitoring-grafana
3000:3000

And that's it! You can now view metrics for your database using the data that has been collected using OpenTelemetry.

For instance, just like we viewed the PgBouncer logs, let’s now visualize PgBouncer metrics.  PgBouncer metrics are a new addition to the Crunchy Postgres for Kubernetes Metrics and Monitoring stack for the Crunchy Postgres for Kubernetes v5.8 release.

For a better understanding of the metrics and dashboards available, please see the Monitoring Architecture page in the Crunchy Postgres for Kubernetes documentation.  And while this initial release of OpenTelemetry metrics focuses on exporting data to Prometheus for use with the pgMonitor dashboards, stay tuned for future releases where it will be possible to export your metrics data into a variety of different OpenTelemetry-compatible services and backends by defining your own exporters!

Conclusion

I hope you enjoyed seeing just how easy it is to enhance the observability of your Postgres databases using Crunchy Postgres for Kubernetes and OpenTelemetry, and how these powerful new features better equip you to manage, maintain, and understand your Postgres deployments in Kubernetes. To continue the conversation around anything discussed in this blog, please feel free to reach out to your account team, or via the Crunchy Discord Server. I look forward to hearing about your observability goals, and how Crunchy Postgres for Kubernetes 5.8 will help to simplify, streamline, and enhance the observability of your Postgres deployments!