Deploy Redis to GKE using Spotahome


Redis is an open source in-memory NoSQL database primarily used for caching. It has built-in replication, Lua scripting, LRU eviction, transactions, on-disk persistence, and high availability.

This guide is intended for platform administrators, cloud architects, and operations professionals interested in deploying Redis clusters on Google Kubernetes Engine (GKE).

The guide shows you how to use the Spotahome Redis operator to deploy Redis clusters.

The operator is licensed under Apache License 2.0.

Spotahome offers the following benefits:

  • A Kubernetes-native Redis clusters management
  • High availability provided by Redis Sentinel
  • Seamless Prometheus integration for database observability
  • Support for setting custom Redis configurations

Objectives

  • Plan and deploy GKE infrastructure for Redis
  • Deploy and configure the Spotahome Redis operator
  • Configure Redis using the operator to ensure availability, security, observability, and performance

Deployment architecture

In this tutorial, you use the Spotahome Redis operator to deploy and configure a highly-available Redis cluster to GKE with a leader node and two read replicas, along with the Redis Sentinel cluster consisting of three replicas.

Redis Sentinel is a high-availability and monitoring system for open source Redis. It continuously monitors Redis instances, including the leader and its associated replicas. If the leader node fails, Sentinel can automatically promote one of the replicas to become the new leader, ensuring that there is always a functioning leader node available for data reads and writes. When significant events occur in a Redis cluster, such as a leader failure or a failover event, Sentinel can notify administrators or other systems through email or other notification mechanisms.

You also deploy a highly-available regional GKE cluster for Redis, with multiple Kubernetes nodes spread across several availability zones. This setup helps ensure fault tolerance, scalability, and geographic redundancy. It allows for rolling updates and maintenance while providing SLAs for uptime and availability. For more information, see Regional clusters.

The following diagram shows how a Redis cluster runs on multiple nodes and zones in a GKE cluster:

In the diagram, the Redis StatefulSet is deployed across three nodes in three different zones. You control how GKE deploys the StatefulSet to nodes and zones by setting Pod affinity and topology spread rules on the RedisFailover custom resource specification.

If one zone fails, using the recommended configuration, GKE reschedules Pods on new nodes.

The following diagram shows a Sentinel Deployment scheduled across three nodes in three different zones:

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. Install the Google Cloud CLI.
  3. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  4. To initialize the gcloud CLI, run the following command:

    gcloudinit
  5. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the Compute Engine, IAM, GKE, Backup for GKE, and Resource Manager APIs:

    gcloudservicesenablecompute.googleapis.com iam.googleapis.com container.googleapis.com gkebackup.googleapis.com cloudresourcemanager.googleapis.com
  8. Install the Google Cloud CLI.
  9. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  10. To initialize the gcloud CLI, run the following command:

    gcloudinit
  11. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  12. Make sure that billing is enabled for your Google Cloud project.

  13. Enable the Compute Engine, IAM, GKE, Backup for GKE, and Resource Manager APIs:

    gcloudservicesenablecompute.googleapis.com iam.googleapis.com container.googleapis.com gkebackup.googleapis.com cloudresourcemanager.googleapis.com
  14. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/storage.objectViewer, roles/container.admin, roles/iam.serviceAccountAdmin, roles/compute.admin, roles/gkebackup.admin, roles/monitoring.viewer

    gcloudprojectsadd-iam-policy-bindingPROJECT_ID--member="user:USER_IDENTIFIER"--role=ROLE
    • Replace PROJECT_ID with your project ID.
    • Replace USER_IDENTIFIER with the identifier for your user account. For example, user:myemail@example.com.

    • Replace ROLE with each individual role.

Set up your environment

In this tutorial, you use Cloud Shell to manage resources hosted on Google Cloud. Cloud Shell comes preinstalled with the software you'll need for this tutorial, including kubectl, the gcloud CLI, Helm, and Terraform.

To set up your environment with Cloud Shell, follow these steps:

  1. Launch a Cloud Shell session from the Google Cloud console, by clicking Cloud Shell activation iconActivate Cloud Shell in the Google Cloud console. This launches a session in the bottom pane of the Google Cloud console.

  2. Set environment variables:

    export PROJECT_ID=PROJECT_ID export KUBERNETES_CLUSTER_PREFIX=redis export REGION=us-central1 

    Replace PROJECT_ID with your Google Cloud project ID.

  3. Clone the GitHub repository:

    gitclonehttps://github.com/GoogleCloudPlatform/kubernetes-engine-samples 
  4. Change to the working directory:

    cdkubernetes-engine-samples/databases/redis-spotahome 

Create your cluster infrastructure

In this section, you run a Terraform script to create a private, highly-available, regional GKE cluster. The following steps allow public access to the control plane.

You can install the operator using a Standard or Autopilot cluster.

Standard

The following diagram shows a private regional Standard GKE cluster deployed across three different zones:

To deploy this infrastructure, run the following commands from the Cloud Shell:

exportGOOGLE_OAUTH_ACCESS_TOKEN=$(gcloudauthprint-access-token) terraform-chdir=terraform/gke-standardinit terraform-chdir=terraform/gke-standardapply-varproject_id=${PROJECT_ID}\-varregion=${REGION}\-varcluster_prefix=${KUBERNETES_CLUSTER_PREFIX}

When prompted, type yes. It might take several minutes for this command to complete and for the cluster to show a ready status.

Terraform creates the following resources:

  • A VPC network and private subnet for the Kubernetes nodes
  • A router to access the internet through NAT
  • A private GKE cluster in the us-central1 region
  • Two node pools with autoscaling enabled (One to two nodes per zone, one node per zone minimum)
  • A ServiceAccount with logging and monitoring permissions
  • Backup for GKE for disaster recovery
  • Google Cloud Managed Service for Prometheus for cluster monitoring

The output is similar to the following:

... Apply complete! Resources: 14 added, 0 changed, 0 destroyed. ... 

Autopilot

The following diagram shows a private regional Autopilot GKE cluster:

To deploy the infrastructure, run the following commands from the Cloud Shell:

exportGOOGLE_OAUTH_ACCESS_TOKEN=$(gcloudauthprint-access-token) terraform-chdir=terraform/gke-autopilotinit terraform-chdir=terraform/gke-autopilotapply-varproject_id=${PROJECT_ID}\-varregion=${REGION}\-varcluster_prefix=${KUBERNETES_CLUSTER_PREFIX}

When prompted, type yes. It might take several minutes for this command to complete and for the cluster to show a ready status.

Terraform creates the following resources:

  • VPC network and private subnet for the Kubernetes nodes
  • A router to access the internet through NAT
  • A private GKE cluster in the us-central1 region
  • A ServiceAccount with logging and monitoring permission
  • Google Cloud Managed Service for Prometheus for cluster monitoring

The output is similar to the following:

... Apply complete! Resources: 12 added, 0 changed, 0 destroyed. ... 

Connect to the cluster

Using Cloud Shell, configure kubectl to communicate with the cluster:

gcloudcontainerclustersget-credentials${KUBERNETES_CLUSTER_PREFIX}-cluster--region${REGION}

Deploy the Spotahome operator to your cluster

In this section, you deploy the Spotahome operator to your Kubernetes cluster using a Helm chart and then deploy a Redis cluster.

  1. Add the Spotahome Redis operator Helm Chart repository:

    helmrepoaddredis-operatorhttps://spotahome.github.io/redis-operator 
  2. Add a namespace for the Spotahome operator and the Redis cluster:

    kubectlcreatensredis 
  3. Deploy the Spotahome operator using the Helm command-line tool:

    helminstallredis-operatorredis-operator/redis-operator--version3.2.9-nredis 
  4. Check the deployment status of the Spotahome operator using Helm:

    helmls-nredis 

    The output is similar to the following:

    NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION redis-operator redis 1 2023-09-12 13:21:48.179503 +0200 CEST deployed redis-operator-3.2.9 1.2.4 

Deploy Redis

The basic configuration for the Redis cluster instance includes the following components:

  • Three replicas of Redis nodes: one leader and two read replicas.
  • Three replicas of Sentinel nodes, forming a quorum.
  • CPU resource allocation of one CPU request and two CPU limits, with 4 GB memory requests and limits for Redis, and 100 m/500 m CPU and 500 MB for Sentinel.
  • Tolerations, nodeAffinities, and topologySpreadConstraints configured for each workload, ensuring proper distribution across Kubernetes nodes, utilizing their respective node pools and different availability zones.

This configuration represents the minimal setup required to create a production-ready Redis cluster.

Create a basic Redis cluster

  1. Create a Secret with user credentials:

    exportPASSWORD=$(opensslrand-base6412) kubectlcreatesecretgenericmy-user-nredis\--from-literal=password="$PASSWORD"

    The operator doesn't have a feature to generate credentials, and the database password should be pre-generated.

  2. Create a new Redis cluster using the basic configuration:

    kubectlapply-nredis-fmanifests/01-basic-cluster/my-cluster.yaml 

    This command creates a RedisFailover custom resource of the Spotahome operator that specifies CPU, memory requests, and limits; and taints and affinities to distribute the provisioned Pod replicas across Kubernetes nodes.

  3. Wait a few minutes while Kubernetes starts the required workloads:

    kubectlwaitpods-lredisfailovers.databases.spotahome.com/name=my-cluster--forcondition=Ready--timeout=300s-nredis 
  4. Verify that the Redis workloads were created:

    kubectlgetpod,svc,statefulset,deploy,pdb-nredis 

    The output is similar to the following:

    NAME READY STATUS RESTARTS AGE pod/redis-operator-5dc65cb7cc-krlcs 1/1 Running 0 49m pod/rfr-my-cluster-0 2/2 Running 0 60s pod/rfr-my-cluster-1 2/2 Running 0 60s pod/rfr-my-cluster-2 2/2 Running 0 60s pod/rfs-my-cluster-8475dfd96c-h5zvw 1/1 Running 0 60s pod/rfs-my-cluster-8475dfd96c-rmh6f 1/1 Running 0 60s pod/rfs-my-cluster-8475dfd96c-shzxh 1/1 Running 0 60s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/redis-my-cluster ClusterIP 10.52.14.87 <none> 6389/TCP 55s service/redis-operator ClusterIP 10.52.13.217 <none> 9710/TCP 49m service/rfr-my-cluster ClusterIP None <none> 9121/TCP 61s service/rfs-my-cluster ClusterIP 10.52.15.197 <none> 26379/TCP 61s NAME READY AGE statefulset.apps/rfr-my-cluster 3/3 61s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/redis-operator 1/1 1 1 50m deployment.apps/rfs-my-cluster 3/3 3 3 62s NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE poddisruptionbudget.policy/rfr-my-cluster 2 N/A 1 64s poddisruptionbudget.policy/rfs-my-cluster 2 N/A 1 63s 

The operator creates the following resources:

  • A Redis StatefulSet and Sentinel Deployment
  • Three Pod replicas for Redis
  • Three Pod replicas for Sentinel
  • Two PodDisruptionBudgets, ensuring a minimum of two available replicas for cluster consistency
  • The rfr-my-cluster Service, which exposes Redis metrics
  • The redis-my-cluster Service, which targets the Redis cluster leader node
  • The rfs-my-cluster Service, which allows clients to connect to the cluster through the Sentinels. Sentinel support is required for client libraries.

Share Redis credentials

You can share Redis credentials with clients using the Spotahome operator legacy authentication method

You must use a database password using the requirepass setting. This password is then used by all clients. To manage additional users, use Redis CLI commands.

apiVersion:databases.spotahome.com/v1kind:RedisFailovermetadata:name:my-clusterspec:...auth:secretPath:my-user

Sharing Redis credentials with clients using this method has the following limitations:

  • Spotahome does not provide Custom Resources for user management. You can store credentials in Secrets and referred to them in auth specs.
  • There is no method to secure the connections with the TLS encryption using the custom resource.
  • Live updating of credentials is not supported.

Connect to Redis

You can deploy a Redis client and authenticate using a password stored in a Kubernetes Secret.

  1. Run the client Pod to interact with your Redis cluster:

    kubectlapply-nredis-fmanifests/02-auth/client-pod.yaml 

    The PASS environment variable takes the my-user Secret from the vault.

  2. Wait for the Pod to be ready, then connect to it:

    kubectlwaitpodredis-client--for=condition=Ready--timeout=300s-nredis kubectlexec-itredis-client-nredis--/bin/bash 
  3. Verify that the connection works:

    redis-cli-hredis-my-cluster-a$PASS--no-auth-warningSETmy-key"testvalue"

    The output is similar to the following:

    OK 
  4. Get the my-key value:

    redis-cli-hredis-my-cluster-a$PASS--no-auth-warningGETmy-key 

    The output is similar to the following:

    "testvalue" 
  5. Exit the Pod shell

    exit 

Understand how Prometheus collects metrics for your Redis cluster

The following diagram shows how Prometheus metrics collecting works:

In the diagram, a GKE private cluster contains:

  • A Redis Pod that gathers metrics on path / and port 9121
  • Prometheus-based collectors that process the metrics from the Redis Pod
  • A PodMonitoring resource that sends metrics to Cloud Monitoring

Google Cloud Managed Service for Prometheus supports metrics collection in the Prometheus format. Cloud Monitoring uses an integrated dashboard for Redis metrics.

The Spotahome operator exposes cluster metrics in Prometheus format using the redis_exporter as a sidecar.

  1. Create the PodMonitoring resource to scrape metrics by labelSelector:

    kubectlapply-nredis-fmanifests/03-prometheus-metrics/pod-monitoring.yaml 
  2. In the Google Cloud console, go to the GKE Clusters Dashboard page.

    Go to GKE Clusters Dashboard

    The dashboard shows non-zero metrics ingestion rate.

  3. In the Google Cloud console, go to the Dashboards page.

    Go to Dashboards

  4. Open the Redis Prometheus Overview dashboard. The dashboard shows the amount of connections and keys. It might take several minutes for the dashboard to auto-provision.

  5. Connect to the client Pod and prepare variables:

    kubectlexec-itredis-client-nredis--/bin/bash 
  6. Use the redis-cli tool to create new keys:

    foriin{1..50};do\redis-cli-hredis-my-cluster-a$PASS\--no-auth-warningSETmykey-$i"myvalue-$i";\done
  7. Refresh the page and observe that the Commands Per Second and Keys graphs have been updated to show the actual database state.

  8. Exit the Pod shell

    exit 

Clean up

Delete the project

    Delete a Google Cloud project:

    gcloud projects delete PROJECT_ID

Delete individual resources

  1. Set environment variables.

    exportPROJECT_ID=${PROJECT_ID}exportKUBERNETES_CLUSTER_PREFIX=redis exportREGION=us-central1 
  2. Run the terraform destroy command:

    exportGOOGLE_OAUTH_ACCESS_TOKEN=$(gcloudauthprint-access-token) terraform-chdir=terraform/FOLDERdestroy-varproject_id=${PROJECT_ID}\-varregion=${REGION}\-varcluster_prefix=${KUBERNETES_CLUSTER_PREFIX}

    Replace FOLDER with either gke-autopilot or gke-standard.

    When prompted, type yes.

  3. Find all unattached disks:

    exportdisk_list=$(gcloudcomputediskslist--filter="-users:* AND labels.name=${KUBERNETES_CLUSTER_PREFIX}-cluster"--format"value[separator=|](name,zone)")
  4. Delete the disks:

    foriin$disk_list;dodisk_name=$(echo$i|cut-d'|'-f1)disk_zone=$(echo$i|cut-d'|'-f2|sed's|.*/||')echo"Deleting $disk_name"gcloudcomputedisksdelete$disk_name--zone$disk_zone--quiet done

What's next

  • Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.