Simulate a zone failure in GKE regional clusters

A common regulatory requirement is that a company can demonstrate their disaster recovery (DR) capability. For applications that run in the cloud, this requirement includes the reliability and availability of services when servers hosted in one zone become unavailable for a period of time. This document is for Admins and architects, Operators, and Backup and disaster recovery (DR) administrators that want to learn how to simulate a zone failover when using a Google Kubernetes Engine (GKE) Standard regional cluster.

GKE regional clusters are created in a user-chosen region, and run the control plane on VMs situated in multiple zones within the chosen region. GKE Autopilot clusters are always regional, and GKE Standard clusters can be regional or zonal. This tutorial uses a GKE Standard regional cluster. Cluster nodes communicate with the control plane through a load balancer, which means that the node location and the control plane VM location don't always match. In the Google Cloud console, you can't disable a particular zone when you use a regional cluster. For more information, see GKE cluster architecture.

This tutorial provides three different methods for simulating zone failure. You can simulate a zone failure and verify the correct application response using whichever method is required for your own compliance purposes.

The methods in this document also apply to zonal clusters, including single-zone and multi-zonal. These methods only affect the nodes in targeted zones, and the GKE control plane isn't affected.

Objectives

  • Create a regional GKE Standard cluster using the default configuration.
  • Deploy a sample microservices application to the regional cluster.
  • Simulate a zone outage using one of the following three methods:
    • Reduce the node pool's zones in a regional cluster.
    • Use a single-zone node pool.
    • Cordon and drain the target failure-zone's nodes.
  • Verify the microservices availability.

Costs

This tutorial uses the following billable components of Google Cloud:

  • Compute Engine
  • GKE Standard mode cluster

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. Install the Google Cloud CLI.

  3. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  4. To initialize the gcloud CLI, run the following command: 

    gcloudinit

  5. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the Kubernetes Engine API, Compute Engine APIs: 

    gcloudservicesenablecontainer.googleapis.com compute.googleapis.com
  8. Install the Google Cloud CLI.

  9. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  10. To initialize the gcloud CLI, run the following command: 

    gcloudinit

  11. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  12. Make sure that billing is enabled for your Google Cloud project.

  13. Enable the Kubernetes Engine API, Compute Engine APIs: 

    gcloudservicesenablecontainer.googleapis.com compute.googleapis.com

Create a regional Standard cluster

Before simulating a zone failure, create a regional cluster with a multi-zone node pool. The cluster's control plane and nodes are replicated across multiple zones in the specified region.

Use the Google Cloud CLI to create the cluster:

  1. Create a new GKE Standard cluster using the default configuration:

    gcloudcontainerclusterscreateCLUSTER_NAME\--regionREGION\--num-nodes2

    Replace the following parameters:

    • CLUSTER_NAME: the name for your cluster.
    • REGION: the region for your cluster, such as us-central1.

    GKE takes a few minutes to create the cluster and verify everything works correctly. Two nodes are created in each zone of the region that you specify.

  2. Check the zones of each node created in the previous step:

    kubectlgetnode-o=custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,INT_IP:.status.addresses[0].address'

    The output looks like the following example:

    NAME ZONE INT_IP regional-cluster-1-default-pool-node1 asia-southeast1-c 10.128.0.37 regional-cluster-1-default-pool-node2 asia-southeast1-c 10.128.0.36 regional-cluster-1-default-pool-node3 asia-southeast1-b 10.128.0.38 regional-cluster-1-default-pool-node4 asia-southeast1-b 10.128.0.33 regional-cluster-1-default-pool-node5 asia-southeast1-a 10.128.0.35 regional-cluster-1-default-pool-node6 asia-southeast1-a 10.128.0.34

  3. Connect to the cluster:

    gcloudcontainerclustersget-credentialsCLUSTER_NAME\--regionREGION

Deploy a sample microservices application

To see the effect of the simulated failover in this document, deploy a sample microservices-based application to your cluster. In this document, you use the sample Cymbal Bank application:

  1. In your shell, clone the following GitHub repository and change into the directory:

    gitclonehttps://github.com/GoogleCloudPlatform/bank-of-anthos.git cdbank-of-anthos/

  2. Deploy the Cymbal Bank sample application to the GKE cluster you created in the previous section:

    kubectlapply-f./extras/jwt/jwt-secret.yaml kubectlapply-f./kubernetes-manifests

  3. Wait for the Pods to be ready:

    kubectlgetpods

  4. After a few minutes, you should see the Pods in a Running state:

    NAMEREADYSTATUSRESTARTSAGE accounts-db-01/1Running016s balancereader-7dc7d9ff57-sstm50/1Running015s contacts-7ddc76d94-rr28x0/1Running014s frontend-747b84bff4-2mtlv0/1Running013s ledger-db-01/1Running013s ledgerwriter-f6cc7889d-9qjfg0/1Running013s loadgenerator-57d4cb57cc-zqvqb1/1Running013s transactionhistory-5dd7c7fd77-lwkv80/1Running012s userservice-cd5ddb4bb-wwhml0/1Running012s

  5. When the Pods are all in a Running state, get the frontend Service external IP address:

    kubectlgetservicefrontend|awk'{print $4}'

  6. In a web browser window, open the IP address shown in the output of the kubectl get service command to access your instance of Cymbal Bank.

    The default credentials are automatically populated, so you can sign in to the app and explore some of the sample transactions and balances. There are no specific actions you need to take, other than to confirm that the Cymbal Bank runs successfully. It might take a minute or two for all the Services to correctly start and let you sign in. Wait until all the Pods are in a Running state and you can successfully sign in to the Cymbal Bank site before you move on to the next section and simulate a zone failure.

Simulate a zone failure

In this section, you simulate a failure with one of the zones. There are three different ways you can simulate this failover. You only need to choose one method. Simulate a zone failure and verify the correct application response using whichever method is required for your own compliance purposes.

Reduce node pool zones

By default, a node pool of a regional cluster has nodes that span across all zones of its region. In the following diagram, Cloud Load Balancing distributes traffic to a node pool that spans three zones. Each zone has two nodes, and your Pods can run in nodes in any of these zones.

A load balancer directs traffic to a regional cluster that runs across three zones. Each zone has two nodes in it.

In this section, you simulate a zone failure by updating the node pool to only run in two out of three zones. This approach verifies that your application can respond to the loss of a zone by correctly redistributing Pods and traffic across other zones.

To update the node pool to only run in certain zones and simulate failure, complete the following steps:

  1. Check the availability of regional cluster and Services:

    kubectlgetpo-owide\ kubectlgetnode-o=custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,INT_IP:.status.addresses[0].address'

    The result is similar to the following example output:

    NAME READY STATUS RESTARTS AGE IP NODE accounts-db-0 1/1 Running 0 6m30s 10.28.1.5 regional-cluster-1-default-pool-node3 balancereader-7dc7d9ff57-shwg5 1/1 Running 0 6m30s 10.28.5.6 regional-cluster-1-default-pool-node1 contacts-7ddc76d94-qv4x5 1/1 Running 0 6m29s 10.28.4.6 regional-cluster-1-default-pool-node2 frontend-747b84bff4-xvjxq 1/1 Running 0 6m29s 10.28.3.6 regional-cluster-1-default-pool-node6 ledger-db-0 1/1 Running 0 6m29s 10.28.5.7 regional-cluster-1-default-pool-node1 ledgerwriter-f6cc7889d-mttmb 1/1 Running 0 6m29s 10.28.1.6 regional-cluster-1-default-pool-node3 loadgenerator-57d4cb57cc-7fvrc 1/1 Running 0 6m29s 10.28.4.7 regional-cluster-1-default-pool-node2 transactionhistory-5dd7c7fd77-cmc2w 1/1 Running 0 6m29s 10.28.3.7 regional-cluster-1-default-pool-node6 userservice-cd5ddb4bb-zfr2g 1/1 Running 0 6m28s 10.28.5.8 regional-cluster-1-default-pool-node1 NAME ZONE INT_IP regional-cluster-1-default-pool-node5 asia-southeast1-c 10.148.0.6 regional-cluster-1-default-pool-node6 asia-southeast1-c 10.148.0.7 regional-cluster-1-default-pool-node2 asia-southeast1-a 10.148.0.8 regional-cluster-1-default-pool-node1 asia-southeast1-a 10.148.0.9 regional-cluster-1-default-pool-node3 asia-southeast1-b 10.148.0.5 regional-cluster-1-default-pool-node4 asia-southeast1-b 10.148.0.4

    In this example, all the Cymbal Bank workloads are deployed in all zones. To simulate a failure, you disable one of the zones, such as asia-southeast1-c, where the frontend service is deployed.

  2. Simulate a zone outage. Update the existing node pool (default-pool) to only specify two zones out of the three zones:

    gcloudcontainernode-poolsupdatedefault-pool\--cluster=CLUSTER_NAME\--node-locations=ZONE_A,ZONE_B\--region=REGION

    Replace ZONE_A, ZONE_B with the two zones where you want the node pool to continue to running.

  3. Verify microservices availability after you update the node pool:

    kubectlgetpo-owide kubectlgetnode-o=custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,INT_IP:.status.addresses[0].address'

    The output should look like the following example:

    NAME ZONE INT_IP regional-cluster-1-default-pool-node2 asia-southeast1-a 10.148.0.8 regional-cluster-1-default-pool-node1 asia-southeast1-a 10.148.0.9 regional-cluster-1-default-pool-node3 asia-southeast1-b 10.148.0.5 regional-cluster-1-default-pool-node4 asia-southeast1-b 10.148.0.4 NAME READY STATUS RESTARTS AGE IP NODE accounts-db-0 1/1 Running 0 28m 10.28.1.5 regional-cluster-1-default-pool-node3 balancereader-7dc7d9ff57-shwg5 1/1 Running 0 28m 10.28.5.6 regional-cluster-1-default-pool-node1 contacts-7ddc76d94-qv4x5 1/1 Running 0 28m 10.28.4.6 regional-cluster-1-default-pool-node2 frontend-747b84bff4-mdnkd 1/1 Running 0 9m21s 10.28.1.7 regional-cluster-1-default-pool-node3 ledger-db-0 1/1 Running 0 28m 10.28.5.7 regional-cluster-1-default-pool-node1 ledgerwriter-f6cc7889d-mttmb 1/1 Running 0 28m 10.28.1.6 regional-cluster-1-default-pool-node3 loadgenerator-57d4cb57cc-7fvrc 1/1 Running 0 28m 10.28.4.7 regional-cluster-1-default-pool-node2 transactionhistory-5dd7c7fd77-w2vqs 1/1 Running 0 9m20s 10.28.4.8 regional-cluster-1-default-pool-node2 userservice-cd5ddb4bb-zfr2g 1/1 Running 0 28m 10.28.5.8 regional-cluster-1-default-pool-node1

    In this example output, asia-southeast1-c is no longer in use. The frontend service you access from a browser with the URL http://EXTERNAL_IP is still accessible. A user would still be able to do deposit and payment actions, even though one of the zones is no longer available.

Use a single-zone node pool

In this section, you simulate a zone failure by deleting two of the node pools. This approach verifies that your application can respond to the loss of a node pool by correctly redistributing Pods and traffic across a node pool in another zone. To simulate a zone outage on a regional cluster, you expand the basic cluster previously created, running the Cymbal Bank application across multiple node pools. This method of simulating the zone interruption more closely reflects an actual zone failure than the first example of updating active zones in a node pool, as it's more common for multiple node pools to exist in a cluster:

A load balancer directs traffic to a regional cluster that runs across three node pools. The default node pool runs across all zones, and the other two node pools each run in a single zone.

The cluster that you build in this section to simulate a single-zone node pool failure includes the following components:

  • Default node pool — usually created when you create a regional GKE Standard cluster — that's a multi-zonal node pool (default-pool).

    This cluster with the single default-pool is what you created earlier in this document.

  • Additional node pools (zonal-node-pool-1 and zonal-node-pool-2) that also run services for the example Cymbal Bank application.

The dotted lines in the diagram show how traffic only serves zonal-node-pool-2 after you simulate a failure in default-pool and zonal-node-pool-1.

To create additional node pools and simulate failure, complete the following steps:

  1. Check availability of the regional cluster:

    gcloudcontainernode-poolslist\--cluster=CLUSTER_NAME\--regionREGION kubectlgetnode-o=custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,INT_IP:.status.addresses[0].address'

    The result is similar to the following example output:

    NAME: default-pool MACHINE_TYPE: e2-medium DISK_SIZE_GB: 100 NODE_VERSION: 1.27.8-gke.1067004 NAME ZONE. INT_IP regional-cluster-1-default-pool-node5-pzmc asia-southeast1-c 10.148.0.6 regional-cluster-1-default-pool-node6-qf1l asia-southeast1-c 10.148.0.7 regional-cluster-1-default-pool-node2-dlk2 asia-southeast1-a 10.148.0.8 regional-cluster-1-default-pool-node1-pkfd asia-southeast1-a 10.148.0.9 regional-cluster-1-default-pool-node3-6b6n asia-southeast1-b 10.148.0.5 regional-cluster-1-default-pool-node4-h0lc asia-southeast1-b 10.148.0.4

    In this example output, all Cymbal Bank Pods are deployed in all zones under the same cluster and run in the existing default-pool.

  2. Create two new single-zone node pools:

    gcloudbetacontainernode-poolscreatezonal-node-pool-1\--clusterCLUSTER_NAME\--regionREGION\--num-nodes4\--node-locationsZONE_A gcloudbetacontainernode-poolscreatezonal-node-pool-2\--clusterCLUSTER_NAME\--regionREGION\--num-nodes4\--node-locationsZONE_B

    Replace ZONE_A and ZONE_B with the two zones where you want the new single-zone node pools to run.

  3. To simulate a zone failure, delete the default-pool regional node pool and one of the new single-zone node pools:

    gcloudcontainernode-poolsdeletedefault-pool\--cluster=CLUSTER_NAME\--region=REGION gcloudcontainernode-poolsdeletezonal-node-pool-1\--cluster=CLUSTER_NAME\--region=REGION

    During the node-pool deletion process, workloads are shut down and rescheduled to another available node pool. When this process happens, the Services and Deployments aren't available. This behavior means that downtime windows need to be specified for DR reporting or documentation.

    Verify the continued availability of the microservices:

    kubectlgetpo-owide\ kubectlgetnode-o=custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,INT_IP:.status.addresses[0].address'

    The output should look similar like the following example:

    NAME ZONE INT_IP regional-cluster-1-node-pool3-node1 asia-southeast1-b 10.148.0.8 regional-cluster-1-node-pool3-node2 asia-southeast1-b 10.148.0.9 regional-cluster-1-node-pool3-node3 asia-southeast1-b 10.148.0.5 regional-cluster-1-node-pool3-node4 asia-southeast1-b 10.148.0.4 NAME READY STATUS RESTARTS AGE IP NODE accounts-db-0 1/1 Running 0 28m 10.28.1.5 regional-cluster-1-zonal-node-pool-2-node3 balancereader-7dc7d9ff57-shwg5 1/1 Running 0 28m 10.28.5.6 regional-cluster-1-zonal-node-pool-2-node1 contacts-7ddc76d94-qv4x5 1/1 Running 0 28m 10.28.4.6 regional-cluster-1-zonal-node-pool-2-node2 frontend-747b84bff4-mdnkd 1/1 Running 0 9m21s 10.28.1.7 regional-cluster-1-zonal-node-pool-2-node3 ledger-db-0 1/1 Running 0 28m 10.28.5.7 regional-cluster-1-zonal-node-pool-2-node4 ledgerwriter-f6cc7889d-mttmb 1/1 Running 0 28m 10.28.1.6 regional-cluster-1-zonal-node-pool-2-node3 loadgenerator-57d4cb57cc-7fvrc 1/1 Running 0 28m 10.28.4.7 regional-cluster-1-zonal-node-pool-2-node2 transactionhistory-5dd7c7fd77-w2vqs 1/1 Running 0 9m20s 10.28.4.8 regional-cluster-1-zonal-node-pool-2-node2 userservice-cd5ddb4bb-zfr2g 1/1 Running 0 28m 10.28.5.8 regional-cluster-1-zonal-node-pool-2-node1

    In this example output, as the default-pool and the zonal-node-pool-1 no longer exist, all Services run in zonal-node-pool-2.

Cordon and drain nodes in a zone

In this section, you cordon and drain specific nodes in your cluster. You cordon and drain all the nodes in a single zone, which simulates the loss of the Pods that run on those nodes across the zone:

A load balancer directs traffic to a regional cluster that runs across three zones. Each zone contains two nodes, and the Cymbal Bank sample application Pods run across all zones and nodes.

In this diagram, you cordon and drain the nodes in the first zone. The nodes in the other two zones continue to run. This approach verifies that your application can respond to the loss of all of the nodes in a zone by correctly redistributing Pods and traffic across nodes that run in other zones.

To cordon and drain the nodes in one of the zones, simulating failure, complete the following steps:

  1. Check availability of the regional cluster and Services. Look at the node names of the target failure zone. You want to specify a zone where the frontend Pods run:

    kubectlgetpods-owide

    The output should look like the following example:

    NAME READY STATUS RESTARTS AGE IP NODE accounts-db-0 1/1 Running 0 4m7s 10.96.4.4 regional-cluster-1-default-pool-node2 balancereader-7dc7d9ff57-lv4z7 1/1 Running 0 4m7s 10.96.1.5 regional-cluster-1-default-pool-node1 contacts-7ddc76d94-wxvg5 1/1 Running 0 4m7s 10.96.6.11 regional-cluster-1-default-pool-node3 frontend-747b84bff4-gvktl 1/1 Running 0 4m7s 10.96.1.4 regional-cluster-1-default-pool-node1 ledger-db-0 1/1 Running 0 4m7s 10.96.4.5 regional-cluster-1-default-pool-node2 ledger-db-1 1/1 Running 0 3m50s 10.96.0.13 regional-cluster-1-default-pool-node5 ledgerwriter-f6cc7889d-4hqbm 1/1 Running 0 4m6s 10.96.0.12 regional-cluster-1-default-pool-node5 loadgenerator-57d4cb57cc-fmq52 1/1 Running 0 4m6s 10.96.4.6 regional-cluster-1-default-pool-node2 transactionhistory-5dd7c7fd77-72zpx 1/1 Running 0 4m6s 10.96.6.12 regional-cluster-1-default-pool-node3 userservice-cd5ddb4bb-b7862 1/1 Running 0 4m6s 10.96.1.6 regional-cluster-1-default-pool-node1

  2. Associate the Pods listed in the previous output with the node's zone:

    kubectlgetnode-o=custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,INT_IP:.status.addresses[0].address'

    The output should look like the following example:

    NAME ZONE INT_IP regional-cluster-1-default-pool-node1 asia-southeast1-b 10.148.0.41 regional-cluster-1-default-pool-node2 asia-southeast1-b 10.148.0.42 regional-cluster-1-default-pool-node3 asia-southeast1-a 10.148.0.37 regional-cluster-1-default-pool-node4 asia-southeast1-a 10.148.0.38 regional-cluster-1-default-pool-node5 asia-southeast1-c 10.148.0.40 regional-cluster-1-default-pool-node6 asia-southeast1-c 10.148.0.39

    In the previous example output, the frontend Pods are located in regional-cluster-1-default-pool-node1 in zone asia-southeast1-b.

    In the next step, you trace all nodes in zone asia-southeast1-b, which in this example output are regional-cluster-1-default-pool-node1 and regional-cluster-1-default-pool-node2

  3. Cordon and drain target nodes in one of the zones. In this example, that's the two nodes in asia-southeast1-b:

    kubectldrainregional-cluster-1-default-pool-node1\--delete-emptydir-data--ignore-daemonsets kubectldrainregional-cluster-1-default-pool-node2\--delete-emptydir-data--ignore-daemonsets

    This command marks the nodes as unschedulable, and simulates node failures. Kubernetes reschedules Pods to other nodes in functioning zones.

  4. Look at where the new frontend Pods and other example Cymbal Bank Pods that were previously running on the node in the failure zone are now rescheduled:

    kubectlgetpo-owide kubectlgetnode-o=custom-columns='NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone,INT_IP:.status.addresses[0].address'

    The output should look like the following example:

    NAME READY STATUS RESTARTS AGE IP NODE accounts-db-0 1/1 Running 0 4m7s 10.96.4.4 regional-cluster-1-default-pool-node4 balancereader-7dc7d9ff57-lv4z7 1/1 Running 0 4m7s 10.96.1.5 regional-cluster-1-default-pool-node6 contacts-7ddc76d94-wxvg5 1/1 Running 0 4m7s 10.96.6.11 regional-cluster-1-default-pool-node3 frontend-747b84bff4-gvktl 1/1 Running 0 4m7s 10.96.1.4 regional-cluster-1-default-pool-node3 ledger-db-0 1/1 Running 0 4m7s 10.96.4.5 regional-cluster-1-default-pool-node6 ledger-db-1 1/1 Running 0 3m50s 10.96.0.13 regional-cluster-1-default-pool-node5 ledgerwriter-f6cc7889d-4hqbm 1/1 Running 0 4m6s 10.96.0.12 regional-cluster-1-default-pool-node5 loadgenerator-57d4cb57cc-fmq52 1/1 Running 0 4m6s 10.96.4.6 regional-cluster-1-default-pool-node4 transactionhistory-5dd7c7fd77-72zpx 1/1 Running 0 4m6s 10.96.6.12 regional-cluster-1-default-pool-node3 userservice-cd5ddb4bb-b7862 1/1 Running 0 4m6s 10.96.1.6 regional-cluster-1-default-pool-node3 NAME ZONE INT_IP regional-cluster-1-default-pool-node3 asia-southeast1-a 10.148.0.37 regional-cluster-1-default-pool-node4 asia-southeast1-a 10.148.0.38 regional-cluster-1-default-pool-node5 asia-southeast1-c 10.148.0.40 regional-cluster-1-default-pool-node6 asia-southeast1-c 10.148.0.39

    In this example output, there are no example Cymbal Bank Pods that run on cordoned nodes, and all Pods only now run in the other two zones.

    Pod Disruption Budgets (PDBs) on the nodes might block node draining. Assess PDB policies before the cordon and drain action. To understand more about PDB and its relation with managing disruption, see how to ensure reliability and uptime for your GKE cluster.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial:

Delete the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next