Deploy a highly-available PostgreSQL database on GKE

Stay organized with collections Save and categorize content based on your preferences.

Create your cluster infrastructure

In this section, you'll run a Terraform script to create a custom Virtual Private Cloud (VPC), a Artifact Registry repository to store PostgreSQL images, and two regional GKE clusters. One cluster will be deployed in us-central1 and the second cluster for backup will be deployed in us-west1.

To create the cluster, follow these steps:

terraform-chdir=terraform/gke-autopilotinit terraform-chdir=terraform/gke-autopilotapply-varproject_id=$PROJECT_ID

terraform-chdir=terraform/gke-standardinit terraform-chdir=terraform/gke-standardapply-varproject_id=$PROJECT_ID

Deploy PostgreSQL on your cluster

In this section, you'll deploy a PostgreSQL database instance to run on GKE by using a Helm chart.

Install PostgreSQL

To install PostgreSQL on your cluster, follow these steps.

1. Configure Docker access.

   gcloudauthconfigure-dockerus-docker.pkg.dev 

2. Populate Artifact Registry with the required PostgreSQL Docker images.

   ./scripts/gcr.shbitnami/postgresql-repmgr15.1.0-debian-11-r0 ./scripts/gcr.shbitnami/postgres-exporter0.11.1-debian-11-r27 ./scripts/gcr.shbitnami/pgpool4.3.3-debian-11-r28 

   The script pushes the following Bitnami images to the Artifact Registry for Helm to install:

   • postgresql-repmgr: This PostgreSQL cluster solution includes the PostgreSQL replication manager (repmgr), an open-source tool for managing replication and failover on PostgreSQL clusters.
   • postgres-exporter: PostgreSQL Exporter gathers PostgreSQL metrics for Prometheus consumption.
   • pgpool: Pgpool-II is the PostgreSQL proxy. It provides connection pooling and load balancing.

3. Verify that the correct images are stored in the repo.

   gcloudartifactsdockerimageslistus-docker.pkg.dev/$PROJECT_ID/main\--format="flattened(package)"

   The output is similar to the following:

   --- image: us-docker.pkg.dev/[PROJECT_ID]/main/bitnami/pgpool --- image: us-docker.pkg.dev/[PROJECT_ID]/main/bitnami/postgres-exporter --- image: us-docker.pkg.dev/h[PROJECT_ID]/main/bitnami/postgresql-repmgr 

4. Configure kubectl command line access to the primary cluster.

   gcloudcontainerclustersget-credentials$SOURCE_CLUSTER\ --region=$REGION--project=$PROJECT_ID

5. Create a namespace.

   exportNAMESPACE=postgresql kubectlcreatenamespace$NAMESPACE

6. If you are deploying to an Autopilot cluster, configure node provisioning across three zones. You can skip this step if you are deploying to a Standard cluster.

   By default, Autopilot provisions resources in only two zones. The deployment defined in prepareforha.yaml ensures that Autopilot provisions nodes across three zones in your cluster, by setting these values:

   • replicas:3
   • podAntiAffinity with requiredDuringSchedulingIgnoredDuringExecution and topologyKey: "topology.kubernetes.io/zone"

   kubectl-n$NAMESPACEapply-fscripts/prepareforha.yaml 

   apiVersion:apps/v1kind:Deploymentmetadata:name:prepare-three-zone-halabels:app:prepare-three-zone-haapp.kubernetes.io/name:postgresql-haspec:replicas:3selector:matchLabels:app:prepare-three-zone-haapp.kubernetes.io/name:postgresql-hatemplate:metadata:labels:app:prepare-three-zone-haapp.kubernetes.io/name:postgresql-haspec:affinity:podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:-labelSelector:matchExpressions:-key:appoperator:Invalues:-prepare-three-zone-hatopologyKey:"topology.kubernetes.io/zone"nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:-preference:matchExpressions:-key:cloud.google.com/compute-classoperator:Invalues:-"Scale-Out"weight:1nodeSelector:app.stateful/component:postgresqltolerations:-effect:NoSchedulekey:app.stateful/componentoperator:Equalvalue:postgresqlcontainers:-name:prepare-three-zone-haimage:busybox:latestcommand:-"/bin/sh"-"-c"-"whiletrue;dosleep3600;done"resources:limits:cpu:"500m"ephemeral-storage:"10Mi"memory:"0.5Gi"requests:cpu:"500m"ephemeral-storage:"10Mi"memory:"0.5Gi"

7. Update the Helm dependency.

   cdhelm/postgresql-bootstrap helmdependencyupdate 

8. Inspect and verify the charts that Helm will install.

   helm-npostgresqltemplatepostgresql.\--setglobal.imageRegistry="us-docker.pkg.dev/$PROJECT_ID/main"

9. Install the Helm chart.

   helm-npostgresqlupgrade--installpostgresql.\--setglobal.imageRegistry="us-docker.pkg.dev/$PROJECT_ID/main"

   The output is similar to the following:

   NAMESPACE: postgresql STATUS: deployed REVISION: 1 TEST SUITE: None 

10. Verify that the PostgreSQL replicas are running.

    kubectlgetall-n$NAMESPACE

    The output is similar to the following:

    NAME READY STATUS RESTARTS AGE pod/postgresql-postgresql-bootstrap-pgpool-75664444cb-dkl24 1/1 Running 0 8m39s pod/postgresql-postgresql-ha-pgpool-6d86bf9b58-ff2bg 1/1 Running 0 8m39s pod/postgresql-postgresql-ha-postgresql-0 2/2 Running 0 8m39s pod/postgresql-postgresql-ha-postgresql-1 2/2 Running 0 8m39s pod/postgresql-postgresql-ha-postgresql-2 2/2 Running 0 8m38s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/postgresql-postgresql-ha-pgpool ClusterIP 192.168.99.236 <none> 5432/TCP 8m39s service/postgresql-postgresql-ha-postgresql ClusterIP 192.168.90.20 <none> 5432/TCP 8m39s service/postgresql-postgresql-ha-postgresql-headless ClusterIP None <none> 5432/TCP 8m39s service/postgresql-postgresql-ha-postgresql-metrics ClusterIP 192.168.127.198 <none> 9187/TCP 8m39s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/postgresql-postgresql-bootstrap-pgpool 1/1 1 1 8m39s deployment.apps/postgresql-postgresql-ha-pgpool 1/1 1 1 8m39s NAME DESIRED CURRENT READY AGE replicaset.apps/postgresql-postgresql-bootstrap-pgpool-75664444cb 1 1 1 8m39s replicaset.apps/postgresql-postgresql-ha-pgpool-6d86bf9b58 1 1 1 8m39s NAME READY AGE statefulset.apps/postgresql-postgresql-ha-postgresql 3/3 8m39s 

Create a test dataset

In this section, you'll create a database and a table with sample values. The database serves as a test dataset for the failover process you'll test later in this tutorial.

1. Connect to your PostgreSQL instance.

   cd../../ ./scripts/launch-client.sh 

   The output is similar to the following:

   Launching Pod pg-client in the namespace postgresql ... pod/pg-client created waiting for the Pod to be ready Copying script files to the target Pod pg-client ... Pod: pg-client is healthy 

2. Start a shell session.

   kubectlexec-itpg-client-npostgresql--/bin/bash 

3. Create a database and a table, and then insert some test rows.

   psql-h$HOST_PGPOOL-Upostgres-a-q-f/tmp/scripts/generate-db.sql 

4. Verify the number of rows for each table.

   psql-h$HOST_PGPOOL-Upostgres-a-q-f/tmp/scripts/count-rows.sql 

   The output is similar to the following:

   select COUNT(*) from tb01; count -------- 300000 (1 row) select COUNT(*) from tb02; count -------- 300000 (1 row) 

5. Generate test data.

   exportDB=postgres pgbench-i-h$HOST_PGPOOL-Upostgres$DB-s50

   The output is similar to the following:

   dropping old tables... creating tables... generating data (client-side)... 5000000 of 5000000 tuples (100%) done (elapsed 29.85 s, remaining 0.00 s) vacuuming... creating primary keys... done in 36.86 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 31.10 s, vacuum 1.88 s, primary keys 3.86 s). 

6. Exit the postgres client Pod.

   exit

Monitor PostgreSQL

In this section, you'll view metrics and set up alerts for your PostgreSQL instance. You'll use Google Cloud Managed Service for Prometheus to perform monitoring and alerting.

View metrics

Your PostgreSQL deployment includes a postgresql-exporter sidecar container. This container exposes a /metrics endpoint. Google Cloud Managed Service for Prometheus is configured to monitor the PostgreSQL Pods on this endpoint. You can view these metrics through Google Cloud console dashboards.

The Google Cloud console provides a few ways to create and save dashboard configuration:

• Creation and Export: You can create dashboards directly in Google Cloud console, then export and store them in a code repository. To do this, in the dashboard toolbar, open the JSON editor and download the dashboard JSON file.
• Storage and Import: You can import a dashboard from a JSON file by clicking +Create Dashboard and uploading the dashboard's JSON content using the JSON editor menu).

To visualize data from your PostgreSQL application and GKE cluster, follow these steps:

1. Create the following dashboards.

   cdmonitoring gcloudmonitoringdashboardscreate\--config-from-file=dashboard/postgresql-overview.json\--project=$PROJECT_ID gcloudmonitoringdashboardscreate\--config-from-filedashboard/gke-postgresql.json\--project$PROJECT_ID

2. In the Google Cloud console, navigate to the Cloud Monitoring Dashboard. Go to the Cloud Monitoring Dashboard

3. Select Custom from the dashboard list. The following dashboards appear:

   • PostgreSQL Overview: Displays metrics from the PostgreSQL application, including database uptime, database size, and transaction latency.
   • GKE PostgreSQL Cluster: Displays metrics from the GKE cluster that PostgreSQL is running on, including CPU usage, memory usage, and volume utilization.

4. Click on each link to examine the dashboards generated.

Set up alerts

Alerting gives you timely awareness of problems in your applications so you can resolve the problems quickly. You can create an alerting policy to specify the circumstances under which you want to be alerted and how you want to be notified. You can also create notification channels that let you select where alerts are sent.

In this section, you'll use Terraform to configure the following example alerts:

• db_max_transaction: Monitors the max lag of transactions in seconds; an alert will be triggered if the value is greater than 10.
• db_node_up: Monitors the status of database Pods; 0 means a Pod is down and triggers an alert.

To set up alerts, follow these steps:

1. Configure alerts with Terraform.

   EMAIL=YOUR_EMAILcdalerting/terraform terraforminit terraformplan-varproject_id=$PROJECT_ID-varemail_address=$EMAIL terraformapply-varproject_id=$PROJECT_ID-varemail_address=$EMAIL

   Replace the following values:

   • YOUR_EMAIL: your email address.

   The output is similar to the following :

   Apply complete! Resources: 3 added, 0 changed, 0 destroyed. 

2. Connect to the client Pod.

   cd../../../ kubectlexec-it--namespacepostgresqlpg-client--/bin/bash 

3. Generate a load test to test the db_max_transaction alert.

   pgbench-i-h$HOST_PGPOOL-Upostgres-s200postgres 

   The output is similar to the following:

   dropping old tables... creating tables... generating data (client-side)... 20000000 of 20000000 tuples (100%) done (elapsed 163.22 s, remaining 0.00 s) vacuuming... creating primary keys... done in 191.30 s (drop tables 0.14 s, create tables 0.01 s, client-side generate 165.62 s, vacuum 4.52 s, primary keys 21.00 s). 

   The alert triggers and sends an email to YOUR_EMAIL with a subject line that starts with "[ALERT] Max Lag of transaction".

4. In the Google Cloud console, navigate to the Alert Policy page.

   Go to Alert Policy

5. Select db_max_transaction from the listed policies. From the chart, you should see a spike from the load test which exceeds the threshold hold of 10 for the Prometheus metric pg_stat_activity_max_tx_duration/gauge.

6. Exit the postgres client Pod.

   exit

Manage PostgreSQL and GKE upgrades

Version updates for both PostgreSQL and Kubernetes are released on a regular schedule. Follow operational best practices to update your software environment regularly. By default, GKE manages cluster and node pool upgrades for you.

Upgrade PostgreSQL

This section shows how you can perform a version upgrade for PostgreSQL. For this tutorial, you'll use a rolling update strategy for upgrading your Pods, so that at no point all of the Pods are down.

To perform a version upgrade, follow these steps:

1. Push an updated version of the postgresql-repmgr image to Artifact Registry. Define the new version (for example, postgresql-repmgr 15.1.0-debian-11-r1).

   NEW_IMAGE=us-docker.pkg.dev/$PROJECT_ID/main/bitnami/postgresql-repmgr:15.1.0-debian-11-r1 ./scripts/gcr.shbitnami/postgresql-repmgr15.1.0-debian-11-r1 

2. Trigger a rolling update using kubectl.

   kubectlsetimagestatefulset-npostgresqlpostgresql-postgresql-ha-postgresqlpostgresql=$NEW_IMAGE kubectlrolloutrestartstatefulsets-npostgresqlpostgresql-postgresql-ha-postgresql kubectlrolloutstatusstatefulset-npostgresqlpostgresql-postgresql-ha-postgresql 

   You will see the StatefulSet complete a rolling update, starting with the highest ordinal replica to the lowest.

   The output is similar to the following:

   Waiting for 1 pods to be ready... waiting for statefulset rolling update to complete 1 pods at revision postgresql-postgresql-ha-postgresql-5c566ccf49... Waiting for 1 pods to be ready... Waiting for 1 pods to be ready... waiting for statefulset rolling update to complete 2 pods at revision postgresql-postgresql-ha-postgresql-5c566ccf49... Waiting for 1 pods to be ready... Waiting for 1 pods to be ready... statefulset rolling update complete 3 pods at revision postgresql-postgresql-ha-postgresql-5c566ccf49... 

Plan for GKE upgrades on Standard clusters

This section is applicable if you are running Standard clusters. You can take proactive steps and set configurations to mitigate risk and facilitate a smoother cluster upgrade when you are running stateful services, including:

• Follow GKE best practices for upgrading clusters. Choose an appropriate upgrade strategy to ensure the upgrades happen during the period of the maintenance window:

  • Choose surge upgrades if cost optimization is important and if your workloads can tolerate a graceful shutdown in less than 60 minutes.
  • Choose blue-green upgrades if your workloads are less tolerant of disruptions, and a temporary cost increase due to higher resource usage is acceptable.

  To learn more, see Upgrade a cluster running a stateful workload.

• Use the Recommender service to check for deprecation insights and recommendations to avoid service interruptions.

• Use maintenance windows to ensure upgrades happen when you intend them. Before the maintenance window, ensure your database backups are successful.

• Before allowing traffic to the upgraded nodes, use readiness and liveness probes to ensure they are ready for traffic.

• Create Probes that assess whether replication is in sync before accepting traffic. This can be done through custom scripts, depending on the complexity and scale of your database.

Verify database availability during Standard cluster upgrades

This section is applicable if you are running Standard clusters. To verify PostgreSQL availability during upgrades, the general process is to generate traffic against the PostgreSQL database during the upgrade process. Then, use pgbench to check that the database can handle a baseline level of traffic during an upgrade, compared to when the database is fully available.

1. Connect to your PostgreSQL instance.

   ./scripts/launch-client.sh 

   The output is similar to the following:

   Launching Pod pg-client in the namespace postgresql ... pod/pg-client created waiting for the Pod to be ready Copying script files to the target Pod pg-client ... Pod: pg-client is healthy 

2. In Cloud Shell, shell into the client Pod.

   kubectlexec-it-npostgresqlpg-client--/bin/bash 

3. Initialize pgbench .

   pgbench-i-h$HOST_PGPOOL-Upostgrespostgres 

4. Use the following command to get baseline results for confirming that your PostgreSQL application stays highly-available during the time window for an upgrade. To get a baseline result, test with multi-connections via multi jobs (threads) for 30 seconds.

   pgbench-h$HOST_PGPOOL-Upostgrespostgres-c10-j4-T30-R200

   The output looks similar to the following:

   pgbench (14.5) starting vacuum...end. transaction type: <builtin: TPC-B (sort of)> scaling factor: 1 query mode: simple number of clients: 10 number of threads: 4 duration: 30 s number of transactions actually processed: 5980 latency average = 7.613 ms latency stddev = 2.898 ms rate limit schedule lag: avg 0.256 (max 36.613) ms initial connection time = 397.804 ms tps = 201.955497 (without initial connection time) 

5. To ensure availability during upgrades, you can generate some load against your database, and ensure that the PostgreSQL application provides a consistent response rate during the upgrade. To perform this test, generate some traffic against the database, using the pgbench command. The following command will run pgbench for one hour, targeting 200 TPS (transactions per second), and listing the request rate every 2 seconds.

   pgbench-h$HOST_PGPOOL-Upostgrespostgres--client=10--jobs=4--rate=200--time=3600--progress=2--select-only 

   Where:

   • --client: Number of clients simulated, that is, number of concurrent database sessions.
   • --jobs: Number of worker threads within pgbench. Using more than one thread can be helpful on multi-CPU machines. Clients are distributed as evenly as possible among available threads. The default is 1.
   • --rate: The rate is given in transactions per second
   • --progress: Show progress report every sec seconds.

   The output is similar to the following:

   pgbench(14.5) startingvacuum...end. progress:5.0s,354.8tps,lat25.222msstddev15.038 progress:10.0s,393.8tps,lat25.396msstddev16.459 progress:15.0s,412.8tps,lat24.216msstddev14.548 progress:20.0s,405.0tps,lat24.656msstddev14.066 

6. In the Google Cloud console, navigate back to the PostgreSQL Overview dashboard in Cloud Monitoring. Notice the spike on the Connection per DB and Connection per Pod graphs.

7. Exit the client Pod.

   exit

8. Delete the client Pod.

   kubectldeletepod-npostgresqlpg-client 

Simulate a PostgreSQL service disruption

In this section, you'll simulate a service disruption in one of the PostgreSQL replicas by stopping the replication manager service. This will prevent the Pod from serving traffic to its peer replicas and its liveness probes to fail.

1. Open a new Cloud Shell session and configure kubectl command line access to the primary cluster.

   gcloudcontainerclustersget-credentials$SOURCE_CLUSTER\ --region=$REGION--project=$PROJECT_ID

2. View the PostgreSQL events emitted in Kubernetes.

   kubectlgetevents-npostgresql--field-selector=involvedObject.name=postgresql-postgresql-ha-postgresql-0--watch 

3. In the earlier Cloud Shell session, simulate a service failure by stopping PostgreSQL repmgr.

   1. Attach your session to the database container.

      kubectlexec-it-n$NAMESPACEpostgresql-postgresql-ha-postgresql-0-cpostgresql--/bin/bash 

   2. Stop the service using repmgr, and remove the checkpoint and the dry-run argument.

      exportENTRY='/opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh'exportRCONF='/opt/bitnami/repmgr/conf/repmgr.conf'$ENTRYrepmgr-f$RCONFnodeservice--action=stop--checkpoint 

The liveness probe configured for the PostgreSQL container will start to fail within five seconds. This repeats every ten seconds, until the failure threshold of six failures is reached. Once the failureThreshold value is reached, the container is restarted. You can configure these parameters to decrease the liveness probe tolerance to tune the SLO requirements of your deployment.

From the event stream, you will see the Pod's liveness and readiness probes fail, and a message that the container needs to be restarted. The output is similar to the following:

0s Normal Killing pod/postgresql-postgresql-ha-postgresql-0 Container postgresql failed liveness probe, will be restarted 0s Warning Unhealthy pod/postgresql-postgresql-ha-postgresql-0 Readiness probe failed: psql: error: connection to server at "127.0.0.1", port 5432 failed: Connection refused... 0s Normal Pulled pod/postgresql-postgresql-ha-postgresql-0 Container image "us-docker.pkg.dev/psch-gke-dev/main/bitnami/postgresql-repmgr:14.5.0-debian-11-r10" already present on machine 0s Normal Created pod/postgresql-postgresql-ha-postgresql-0 Created container postgresql 0s Normal Started pod/postgresql-postgresql-ha-postgresql-0 Started container postgresql 

Prepare for disaster recovery

To ensure that your production workloads remain available in the event of a service-interrupting event, you should prepare a disaster recovery (DR) plan. To learn more about DR planning, see the Disaster recovery planning guide.

Disaster recovery for Kubernetes can be implemented in two phases:

• Backup involves creating a point-in-time snapshot of your state or data before a service-interrupting event occurs.
• Recovery involves restoring your state or data from a backup copy after the occurrence of a disaster.

To backup and restore your workloads on GKE clusters, you can use Backup for GKE. You can enable this service on new and existing clusters. This deploys a Backup for GKE agent that runs in your clusters; the agent is responsible for capturing configuration and volume backup data and orchestrating recovery.

Backups and restores can be scoped to an entire cluster, a namespace, or an application (defined by selectors such as matchLabels).

Example PostgreSQL backup and restore scenario

The example in this section shows how you can perform a backup and restore operation at the application scope, using the ProtectedApplication Custom Resource.

The following diagram shows the component resources in the ProtectedApplication, namely a StatefulSet representing the postgresql-ha application and a deployment of pgpool, which use the same label (app.kubernetes.io/name: postgresql-ha).

To prepare to backup and restore your PostgreSQL workload, follow these steps:

1. Set up the environment variables. In this example you'll use a ProtectedApplication to restore the PostgreSQL workload and its volumes from the source GKE cluster (us-central1), then restore to another GKE cluster in a different region (us-west1).

   exportSOURCE_CLUSTER=cluster-db1 exportTARGET_CLUSTER=cluster-db2 exportREGION=us-central1 exportDR_REGION=us-west1 exportNAME_PREFIX=g-db-protected-app exportBACKUP_PLAN_NAME=$NAME_PREFIX-bkp-plan-01 exportBACKUP_NAME=bkp-$BACKUP_PLAN_NAMEexportRESTORE_PLAN_NAME=$NAME_PREFIX-rest-plan-01 exportRESTORE_NAME=rest-$RESTORE_PLAN_NAME

2. Verify that Backup for GKE is enabled on your clusters. It should already be enabled as part of the Terraform setup you performed earlier.

   gcloudcontainerclustersdescribe$SOURCE_CLUSTER\--project=$PROJECT_ID\--region=$REGION\--format='value(addonsConfig.gkeBackupAgentConfig)'

   If Backup for GKE is enabled, the output of the command shows enabled=True.

Set up a backup plan and perform a restore

Backup for GKE allows you to create a backup plan as a cron job. A backup plan contains a backup configuration including the source cluster, the selection of which workloads to back up, and the region in which backup artifacts produced under this plan are stored.

To perform a backup and restore, follow these steps:

1. Verify the status of ProtectedApplication on cluster-db1.

   kubectlgetProtectedApplication-A 

   The output looks similar to the following:

   NAMESPACE NAME READY TO BACKUP postgresql postgresql-ha true 

2. Create a backup plan for the ProtectedApplication.

   exportNAMESPACE=postgresql exportPROTECTED_APP=$(kubectlgetProtectedApplication-n$NAMESPACE|grep-v'NAME'|awk'{ print $1 }')

   gcloudbetacontainerbackup-restorebackup-planscreate$BACKUP_PLAN_NAME\ --project=$PROJECT_ID\ --location=$DR_REGION\ --cluster=projects/$PROJECT_ID/locations/$REGION/clusters/$SOURCE_CLUSTER\ --selected-applications=$NAMESPACE/$PROTECTED_APP\ --include-secrets\ --include-volume-data\ --cron-schedule="0 3 * * *"\ --backup-retain-days=7\ --backup-delete-lock-days=0

3. Manually create a backup.

   gcloudbetacontainerbackup-restorebackupscreate$BACKUP_NAME\ --project=$PROJECT_ID\ --location=$DR_REGION\ --backup-plan=$BACKUP_PLAN_NAME\ --wait-for-completion 

4. Set up a restore plan.

   gcloudbetacontainerbackup-restorerestore-planscreate$RESTORE_PLAN_NAME\--project=$PROJECT_ID\--location=$DR_REGION\--backup-plan=projects/$PROJECT_ID/locations/$DR_REGION/backupPlans/$BACKUP_PLAN_NAME\--cluster=projects/$PROJECT_ID/locations/$DR_REGION/clusters/$TARGET_CLUSTER\--cluster-resource-conflict-policy=use-existing-version\--namespaced-resource-restore-mode=delete-and-restore\--volume-data-restore-policy=restore-volume-data-from-backup\--selected-applications=$NAMESPACE/$PROTECTED_APP\--cluster-resource-scope-selected-group-kinds="storage.k8s.io/StorageClass","scheduling.k8s.io/PriorityClass"

5. Restore from the backup.

   gcloudbetacontainerbackup-restorerestorescreate$RESTORE_NAME\--project=$PROJECT_ID\--location=$DR_REGION\--restore-plan=$RESTORE_PLAN_NAME\--backup=projects/$PROJECT_ID/locations/$DR_REGION/backupPlans/$BACKUP_PLAN_NAME/backups/$BACKUP_NAME\--wait-for-completion 

Verify that your cluster is restored

To verify that the restored cluster has all the expected Pods, PersistentVolume, and StorageClass resources, follow these steps:

1. Configure kubectl command line access to the backup cluster cluster-db2.

   gcloudcontainerclustersget-credentials$TARGET_CLUSTER--region$DR_REGION--project$PROJECT_ID

2. Verify that the StatefulSet is ready with 3/3 Pods.

   kubectlgetall-n$NAMESPACE

   The output is similar to the following:

   NAME READY STATUS RESTARTS AGE pod/postgresql-postgresql-ha-pgpool-778798b5bd-k2q4b 1/1 Running 0 4m49s pod/postgresql-postgresql-ha-postgresql-0 2/2 Running 2 (4m13s ago) 4m49s pod/postgresql-postgresql-ha-postgresql-1 2/2 Running 0 4m49s pod/postgresql-postgresql-ha-postgresql-2 2/2 Running 0 4m49s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/postgresql-postgresql-ha-pgpool ClusterIP 192.168.241.46 <none> 5432/TCP 4m49s service/postgresql-postgresql-ha-postgresql ClusterIP 192.168.220.20 <none> 5432/TCP 4m49s service/postgresql-postgresql-ha-postgresql-headless ClusterIP None <none> 5432/TCP 4m49s service/postgresql-postgresql-ha-postgresql-metrics ClusterIP 192.168.226.235 <none> 9187/TCP 4m49s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/postgresql-postgresql-ha-pgpool 1/1 1 1 4m49s NAME DESIRED CURRENT READY AGE replicaset.apps/postgresql-postgresql-ha-pgpool-778798b5bd 1 1 1 4m49s NAME READY AGE statefulset.apps/postgresql-postgresql-ha-postgresql 3/3 4m49s 

3. Verify all Pods in the postgres namespace are running.

   kubectlgetpods-n$NAMESPACE

   The output is similar to the following:

   postgresql-postgresql-ha-pgpool-569d7b8dfc-2f9zx 1/1 Running 0 7m56s postgresql-postgresql-ha-postgresql-0 2/2 Running 0 7m56s postgresql-postgresql-ha-postgresql-1 2/2 Running 0 7m56s postgresql-postgresql-ha-postgresql-2 2/2 Running 0 7m56s 

4. Verify the PersistentVolumes and StorageClass. During the restore process, Backup for GKE creates a Proxy Class in the target workload to replace the StorageClass provisioned in the source workload (gce-pd-gkebackup-dn in the example output).

   kubectlgetpvc-n$NAMESPACE

   The output is similar to the following:

   NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-postgresql-postgresql-ha-postgresql-0 Bound pvc-be91c361e9303f96 8Gi RWO gce-pd-gkebackup-dn 10m data-postgresql-postgresql-ha-postgresql-1 Bound pvc-6523044f8ce927d3 8Gi RWO gce-pd-gkebackup-dn 10m data-postgresql-postgresql-ha-postgresql-2 Bound pvc-c9e71a99ccb99a4c 8Gi RWO gce-pd-gkebackup-dn 10m 

Validate that the expected data is restored

To validate that the expected data is restored, follow these steps:

1. Connect to your PostgreSQL instance.

   ./scripts/launch-client.sh kubectlexec-itpg-client-npostgresql--/bin/bash 

2. Verify the number of rows for each table.

   psql-h$HOST_PGPOOL-Upostgres-a-q-f/tmp/scripts/count-rows.sql selectCOUNT(*)fromtb01;

   You should see a similar result to the data you wrote earlier in the Create a test dataset. The output is similar to the following:

   300000(1row)

3. Exit the client Pod.

   exit