Customizing node system configuration

Stay organized with collections Save and categorize content based on your preferences.

This document shows you how to customize your Google Kubernetes Engine (GKE) node configuration using a configuration file called a node system configuration.

Overview

You can customize your node configuration by using various methods. For example, you can specify parameters such as the machine type and minimum CPU platform when you create a node pool.

A node system configuration is a configuration file that provides a way to adjust a limited set of system settings. You can use a node system configuration to specify custom settings for the Kubernetes node agent ( kubelet) and low-level Linux kernel configurations (sysctl) in your node pools.

You can also customize your containerd container runtime on your GKE nodes by using a different file called a runtime configuration file. For instructions, see Customize containerd configuration in GKE nodes.

You can also use DaemonSets to customize nodes, such as in Automatically bootstrapping GKE nodes with DaemonSets.

Using a node system configuration

You can customize your node system configuration by using any of the following methods:

• Configuration file: available in Standard mode. You use a YAML file that contains the kubelet and Linux kernel configuration parameters. The steps on this page show you how to create and use a configuration file.
• ComputeClass: available in Autopilot mode and Standard mode. You specify the node system configuration in your GKE ComputeClass specification. Compute classes let you define sets of node attributes for GKE to use when it scales your cluster up. Available in GKE version 1.32.1-gke.1729000 and later. For details, see About compute classes in GKE.

To use a node system configuration file, do the following:

1. Create a configuration file. This file contains your kubelet and sysctl configurations.
2. Add the configuration when you create a cluster, or when you create or update a node pool.

Creating a configuration file

Write your node system configuration file in YAML. The following example shows you how to add configurations for the kubelet and sysctl options:

kubeletConfig:cpuManagerPolicy:staticallowedUnsafeSysctls:-'kernel.shm*'-'kernel.msg*'-'kernel.sem'-'fs.mqueue*'-'net.*'linuxConfig:sysctl:net.core.somaxconn:'2048'net.ipv4.tcp_rmem:'4096873806291456'

In this example:

• cpuManagerPolicy: static configures the kubelet to use the static CPU management policy.
• net.core.somaxconn: '2048' limits the socket listen() backlog to 2,048 bytes.
• net.ipv4.tcp_rmem: '4096 87380 6291456' sets the minimum, default, and maximum value of the TCP socket receive buffer to 4,096 bytes, 87,380 bytes, and 6,291,456 bytes respectively.

If you want to add configurations solely for the kubelet or sysctl, only include that section in your configuration file. For example, to add a kubelet configuration, create the following file:

kubeletConfig:cpuManagerPolicy:static

For a complete list of the fields that you can add to your configuration file, see the Kubelet configuration options and Sysctl configuration options sections.

Adding the configuration to a node pool

After you have created the configuration file, add the --system-config-from-file flag by using the Google Cloud CLI. You can add this flag when you create a cluster, or when you create or update a node pool. You cannot add a node system configuration with the Google Cloud console.

Create a cluster with the node system configuration

You can add a node system configuration during cluster creation with the gcloud CLI or Terraform. The following instructions apply the node system configuration to the default node pool:

gcloudcontainerclusterscreateCLUSTER_NAME\--location=LOCATION\--system-config-from-file=SYSTEM_CONFIG_PATH

resource "google_container_cluster" "default" { name = "gke-standard-regional-cluster" location = "us-central1" initial_node_count = 1 node_config { # Kubelet configuration kubelet_config { cpu_manager_policy = "static" } linux_node_config { # Sysctl configuration sysctls = { "net.core.netdev_max_backlog" = "10000" } # Linux cgroup mode configuration cgroup_mode = "CGROUP_MODE_V2" # Linux huge page configuration hugepages_config { hugepage_size_2m = "1024" } } } }

Create a new node pool with the node system configuration

You can add a node system configuration when you use the gcloud CLI or Terraform to create a new node pool. You can also update the node system configuration of an existing node pool.

The following instructions apply the node system configuration to a new node pool:

gcloudcontainernode-poolscreatePOOL_NAME\--clusterCLUSTER_NAME\--location=LOCATION\--system-config-from-file=SYSTEM_CONFIG_PATH```Replacethefollowing: 

resource "google_container_node_pool" "default" { name = "gke-standard-regional-node-pool" cluster = google_container_cluster.default.name node_config { # Kubelet configuration kubelet_config { cpu_manager_policy = "static" } linux_node_config { # Sysctl configuration sysctls = { "net.core.netdev_max_backlog" = "10000" } # Linux cgroup mode configuration cgroup_mode = "CGROUP_MODE_V2" # Linux huge page configuration hugepages_config { hugepage_size_2m = "1024" } } } }

Update the node system configuration of an existing node pool

Run the following command:

gcloudcontainernode-poolsupdatePOOL_NAME\--cluster=CLUSTER_NAME\--location=LOCATION\--system-config-from-file=SYSTEM_CONFIG_PATH

Replace the following:

• POOL_NAME: the name of the node pool that you want to update
• CLUSTER_NAME: the name of the cluster that you want to update
• LOCATION: the compute zone or region of the cluster
• SYSTEM_CONFIG_PATH: the path to the file that contains your kubelet and sysctl configurations

This change requires recreating the nodes, which can cause disruption to your running workloads. For details about this specific change, find the corresponding row in the manual changes that recreate the nodes using a node upgrade strategy without respecting maintenance policies table. To learn more about node updates, see Planning for node update disruptions.

Editing a node system configuration

To edit a node system configuration, you can create a new node pool with the configuration that you want, or update the node system configuration of an existing node pool.

Editing by creating a node pool

To edit a node system configuration by creating a node pool:

1. Create a configuration file with the configuration that you want.
2. Add the configuration to a new node pool.
3. Migrate your workloads to the new node pool.
4. Delete the old node pool.

Editing by updating an existing node pool

To edit the node system configuration of an existing node pool, follow the instructions in the Update node pool tab for adding the configuration to a node pool. Updating a node system configuration overrides the node pool's system configuration with the new configuration, which requires recreating the nodes. If you omit any parameters during an update, they are set to their respective defaults.

If you want to reset the node system configuration back to the defaults, update your configuration file with empty values for the kubelet and sysctl. For example:

kubeletConfig:{}linuxConfig:sysctl:{}

Deleting a node system configuration

To remove a node system configuration:

1. Create a node pool.
2. Migrate your workloads to the new node pool.
3. Delete the node pool that has the old node system configuration.

Kubelet configuration options

The following table shows you the kubelet options that you can modify.

Sysctl configuration options

To tune the performance of your system, you can modify the following Kernel attributes:

• kernel.shmmni
• kernel.shmmax
• kernel.shmall
• net.core.busy_poll
• net.core.busy_read
• net.core.netdev_max_backlog
• net.core.rmem_max
• net.core.rmem_default
• net.core.wmem_default
• net.core.wmem_max
• net.core.optmem_max
• net.core.somaxconn
• net.ipv4.tcp_rmem
• net.ipv4.tcp_wmem
• net.ipv4.tcp_tw_reuse
• net.ipv6.conf.all.disable_ipv6
• net.ipv6.conf.default.disable_ipv6
• net.netfilter.nf_conntrack_acct - Available on GKE versions 1.32.0-gke.1448000 or later.
• net.netfilter.nf_conntrack_max - Available on GKE versions 1.32.0-gke.1448000 or later.
• net.netfilter.nf_conntrack_buckets - Available on GKE versions 1.32.0-gke.1448000 or later.
• net.netfilter.nf_conntrack_tcp_timeout_close_wait - Available on GKE versions 1.32.0-gke.1448000 or later.
• net.netfilter.nf_conntrack_tcp_timeout_established - Available on GKE versions 1.32.0-gke.1448000 or later.
• net.netfilter.nf_conntrack_tcp_timeout_time_wait - Available on GKE versions 1.32.0-gke.1448000 or later.
• vm.max_map_count

Different Linux namespaces might have unique values for a given sysctl, while others are global for the entire node. Updating sysctl options by using a node system configuration ensures that the sysctl is applied globally on the node and in each namespace, resulting in each Pod having identical sysctl values in each Linux namespace.

Linux cgroup mode configuration options

The kubelet and the container runtime use Linux kernel cgroups for resource management, such as limiting how much CPU or memory each container in a Pod can access. There are two versions of the cgroup subsystem in the kernel: cgroupv1 and cgroupv2. Kubernetes support for cgroupv2 was introduced as alpha in Kubernetes version 1.18, beta in 1.22, and GA in 1.25. For more details, refer to the Kubernetes cgroups v2 documentation.

Node system configuration lets you customize the cgroup configuration of your node pools. You can use cgroupv1 or cgroupv2. GKE uses cgroupv2 for new Standard node pools running version 1.26 and later, and cgroupv1 for versions earlier than 1.26. For node pools created with node auto-provisioning, the cgroup configuration depends on the initial cluster version, not the node pool version. cgroupv1 is not supported on Arm machines.

You can use node system configuration to change the setting for a node pool to use cgroupv1 or cgroupv2 explicitly. Just upgrading an existing node pool to 1.26 doesn't change the setting to cgroupv2, as existing node pools created running a version earlier than 1.26—without a customized cgroup configuration—continue to use cgroupv1 unless you explicitly specify otherwise.

For example, to configure your node pool to use cgroupv2, use a node system configuration file such as:

linuxConfig:cgroupMode:'CGROUP_MODE_V2'

The supported cgroupMode options are:

• CGROUP_MODE_V1: Use cgroupv1 on the node pool.
• CGROUP_MODE_V2: Use cgroupv2 on the node pool.
• CGROUP_MODE_UNSPECIFIED: Use the default GKE cgroup configuration.

To use cgroupv2, the following requirements and limitations apply:

• For a node pool running a version earlier than 1.26, you must use gcloud CLI version 408.0.0 or newer. Alternatively, use gcloud beta with version 395.0.0 or newer.
• Your cluster and node pools must run GKE version 1.24.2-gke.300 or later.
• You must use the Container-Optimized OS with containerd node image.
• If any of your workloads depend on reading the cgroup filesystem (/sys/fs/cgroup/...), ensure they are compatible with the cgroupv2 API.
  • Ensure any monitoring or third-party tools are compatible with cgroupv2.
• If you use JDK (Java workload), we recommend that you use versions which fully support cgroupv2, including JDK 8u372, JDK 11.0.16 or later, or JDK 15 or later.

Verify cgroup configuration

When you add a node system configuration, GKE must recreate the nodes to implement the changes. After you've added the configuration to a node pool and the nodes have been recreated, you can verify the new configuration.

You can verify the cgroup configuration for nodes in a node pool with gcloud CLI or the kubectl command-line tool:

gcloudcontainernode-poolsdescribePOOL_NAME\--format='value(Config.effectiveCgroupMode)'

Linux huge page configuration options

You can use the node system configuration file to use the Linux kernel feature huge pages.

Kubernetes supports huge pages on nodes as a type of resource, similar to CPU or memory. Use the following parameters to instruct your Kubernetes nodes to pre-allocate huge pages for consumption by Pods. To manage your Pods' consumption of huge pages, see Manage HugePages.

To pre-allocate huge pages for your nodes, specify the amounts and sizes. For example, to configure your nodes to allocate three 1-gigabyte-sized huge pages, and 1024 2-megabyte-sized huge pages, use a node system configuration such as the following:

linuxConfig:hugepageConfig:hugepage_size2m:1024hugepage_size1g:3

To use huge pages, the following limitations and requirements apply:

• To ensure that the node is not fully occupied by huge pages, the allocated huge pages overall size can't exceed 60% of the total memory on machines with less than 30 GB memory, and 80% on machines with more than 30 GB memory. For example, on e2-standard-2 machine with 8 GB of memory, you can't allocate more than 4.8 GB for huge pages. And on c4a-standard-8 with 32 GB of memory, huge pages cannot exceed 25.6 GB.
• 1 GB huge pages are only available on A3, C2D, C3, C3D, C4, C4A, CT5E, CT5LP, CT6E, H3, M2, M3, or Z3 machine types.

What's next

• Learn more about node pools.
• Learn how to create node pools.