CloudNativePG in production: HA, backups, and major upgrades without downtime

Standing up CloudNativePG is easy: one Cluster CR and you have a primary with replicas (that’s the intro article). The pain starts on day-2. Automatic failover that actually fires under load; backups you can actually restore from; a major upgrade without an hour of downtime. This is exactly where teams stumble — because “works in the demo” and “survives an incident” are different states. Let’s cover what separates one from the other.

Table of contents

HA isn’t “we have replicas,” it’s “failover fires”

Having two replicas doesn’t by itself give you high availability. HA is when, if the primary dies, the cluster picks a new master on its own — no human — and repoints traffic to it within seconds, without ending up in split-brain, where two pods think they’re primary at once.

CloudNativePG solves this by making the operator the single decision point. There is exactly one master, and only the operator knows which — the pods don’t “negotiate” among themselves (there’s no distributed consensus, which is itself a source of split-brain). When the primary is lost, the operator fences the old instance (stops routing traffic to it and prevents it from returning as master), promotes the most up-to-date replica, and repoints the -rw service to it. An app talking to <cluster>-rw experiences this as a brief connection drop.

Synchronous replication is critical here. With an async replica, some of the last transactions may not have arrived by the time the master died — and they’re lost on promotion. Synchronous mode requires confirmation from at least one replica before a commit is considered successful:

spec:
  instances: 3
  postgresql:
    synchronous:
      method: any
      number: 1              # ≥1 replica confirms every commit
      dataDurability: required

The trade-off is honest: dataDurability: required means that if the quorum of synchronous replicas is lost, writes stall — the database chooses consistency over availability. If for your scenario it’s more important to keep accepting writes even alone, set preferred, and the primary keeps working asynchronously, accepting the risk of losing the last transactions in a failure. It’s your call, but make it deliberately, not by default.

Separate from emergency failover is switchover — a planned change of master (say, before draining a node for maintenance). It’s the same cutover but controlled: the operator waits for the replica to catch up to the primary and only then swaps roles — with a minimal write window. Failover reacts to an incident; switchover prevents one.

A backup you can’t restore from isn’t a backup

The main day-2 mistake is to treat a configured backup as protection. Real protection is a regularly tested restore. A backup no one has ever deployed is a hypothesis, not a guarantee.

Through the Barman Cloud Plugin, CloudNativePG gives you two layers: a periodic base backup and a continuous WAL archive. Together they give you PITR — recovery to any second within the retention window, not just to the moment of the nightly snapshot. The backup schedule is declared declaratively:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: pg-nightly
  namespace: prod
spec:
  schedule: "0 0 2 * * *"       # every night at 02:00 (cron with seconds)
  backupOwnerReference: self
  cluster:
    name: pg
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

The key skill isn’t configuring this — it’s restoring from it. PITR in CNPG is a new cluster that bootstraps from the object store to a given point in time. The old cluster is left untouched — that’s precisely the safe way to test a restore or undo the effects of a mistaken DELETE:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-restore
  namespace: prod
spec:
  instances: 3
  storage:
    size: 50Gi
    storageClass: fast-ssd
  bootstrap:
    recovery:
      source: origin
      recoveryTarget:
        targetTime: "2026-07-06T01:45:00Z"   # 15 minutes before the incident
  externalClusters:
    - name: origin
      plugin:
        name: barman-cloud.cloudnative-pg.io
        parameters:
          barmanObjectName: pg-backups
          serverName: pg                      # name of the source cluster

The operator takes the last base backup before targetTime, replays WAL up to the required second, and brings up a fresh cluster in that state. Make such a run part of your routine — for example, restore prod into a separate namespace once a month and compare checksums. Then on the day of a real incident you’re executing a rehearsed procedure, not reading docs under pressure.

Monitoring: what actually matters

CloudNativePG exposes Prometheus metrics out of the box — an exporter is built into every instance on port 9187. You turn it on via a PodMonitor (or enablePodMonitor: true in the cluster spec if you have the Prometheus Operator installed):

spec:
  monitoring:
    enablePodMonitor: true

There are many metrics, but in prod you really watch a few signals:

• cnpg_pg_replication_lag — replication lag in seconds. Rising means replicas fall behind, and on failover you lose more data (or writes stall in synchronous mode).
• cnpg_pg_wal / WAL archiving — if WAL stops shipping to S3, your PITR archive silently breaks and you only find out when you try to restore. This is a top-priority alert.
• cnpg_backends_total and pool usage — approaching max_connections means it’s time to look at PgBouncer, not to raise the connection limit.
• cnpg_collector_last_failed_backup_timestamp — the time of the last failed backup; if it’s newer than the last successful one, protection has degraded.

PgBouncer from the Pooler exposes its own metrics separately (port 9127, prefix cnpg_pgbouncer_) — from there, pool saturation and the number of waiting clients matter most.

Upgrades: minor is routine, major is by plan

Minor updates (17.2 → 17.4) in CNPG are routine, with no downtime for the app. You change imageName and the operator does a rolling update: it restarts replicas one by one, then does a switchover onto an already-updated replica and updates the former primary. The only window is the seconds of the switchover, and only for writes.

spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:17.4   # was 17.2
  primaryUpdateStrategy: unsupervised                  # switchover on its own

With primaryUpdateStrategy: supervised the operator updates the replicas but leaves the primary cutover to you — you trigger the switchover in a convenient window with kubectl cnpg promote.

A major upgrade (16 → 17) is a different story: the on-disk data format changes, so a rolling update doesn’t apply. There are two approaches:

• Declarative in-place via pg_upgrade. You change the image’s major version, the operator stops the cluster, runs pg_upgrade in place, and brings it back up on the new version. Fast (minutes, and not tied to data volume the way a dump is), but with a short planned downtime for the duration of the upgrade.
• Blue/green via logical replication. You stand up a new cluster on the new major version, set up logical replication from the old one, wait for sync, and switch traffic over with almost no downtime. More expensive to set up, but the write window is the seconds of the cutover.

The choice depends on your SLA: if a few minutes of planned downtime at night is acceptable, pg_upgrade is simpler; if not, blue/green. Either way, rehearse it on a copy restored via PITR — the very restore you should be testing anyway.

Two more day-2 operations: disk resize and rolling restart

The database disk runs out sooner or later. If the StorageClass supports expansion (allowVolumeExpansion: true), growing it is a one-field edit; the operator expands the PVC without recreating the pods:

spec:
  storage:
    size: 100Gi        # was 50Gi — just bump it

A rolling restart (for example, to apply a Postgres parameter change that requires a restart) is also an operator operation: it restarts instances one at a time, starting with replicas, with a switchover at the end. No manual “kill the pod and pray.”

Comparison: “just works” vs “survives an incident”

Bottom line

Stateful on Kubernetes is about day-2, and CloudNativePG closes it honestly: HA with automatic failover and split-brain protection, declarative backups with PITR, built-in metrics, minor and major upgrades without a long downtime. But the operator gives you mechanisms, not guarantees. The guarantee comes from discipline: synchronous replication chosen deliberately, an alert on WAL archiving stopping configured, and PITR restore tested on a schedule — not on the day of the incident. Production Postgres on k8s rests not on a configured backup but on a regularly tested restore — everything else CNPG has already done for you.