Google Distributed Cloud (software only) for VMware known issues

Stay organized with collections Save and categorize content based on your preferences.

Generated user-cluster.yaml file is missing the hostgroups field

The user cluster configuration file, user-cluster.yaml, generated by the gkectl get-config command is missing the hostgroups field in the nodePools section. The gkectl get-config command generates the user-cluster.yaml file based on the contents of the OnPremUserCluster custom resource. The nodePools[i].vsphere.hostgroups field, however, exists in the OnPremNodePool custom resource and isn't copied to the user-cluster.yaml file when you run gkectl get-config.

Workaround

To resolve this issue, manually add the nodePools[i].vsphere.hostgroups field to the user-cluster.yaml file. The edited file should look similar to the following example:

apiVersion:v1kind:UserCluster...nodePools:-name:"worker-pool-1"enableLoadBalancer:truereplicas:3vsphere:hostgroups:-"hostgroup-1"...

You can use the edited user cluster configuration file to update your user cluster without triggering errors and the hostgroups field is persisted.

Bundled ingress is not compatible with gateway.networking.k8s.io resources

Istiod pods of bundled ingress cannot be ready if gateway.networking.k8s.io resources are installed into the user cluster. The following example error message can be found in pod logs:

 failed to list *v1beta1.Gateway: gateways.gateway.networking.k8s.io is forbidden: User \"system:serviceaccount:gke-system:istiod-service-account\" cannot list resource \"gateways\" in API group \"gateway.networking.k8s.io\" at the cluster scope" 

Workaround:

Apply the following ClusterRole and ClusterRoleBinding to your user cluster:

apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRolemetadata:name:istiod-gatewayrules:-apiGroups:-'gateway.networking.k8s.io'resources:-'*'verbs:-get-list-watch---apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRoleBindingmetadata:name:istiod-gateway-bindingroleRef:apiGroup:rbac.authorization.k8s.iokind:ClusterRolename:istiod-gatewaysubjects:-kind:ServiceAccountname:istiod-service-accountnamespace:gke-system

Admin cluster control plane nodes keep rebooting after running gkectl create admin

If hostnames in the ipblocks field contain uppercase letters, the admin cluster control plane nodes might reboot over and over.

Workaround:

Use lowercase hostnames only.

Runtime: out of memory "error" after running gkeadm create or upgrade

When creating or upgrading admin workstations with gkeadm, commands, you might get an OOM error when verifying the downloaded OS image.

For example,

 Downloading OS image "gs://gke-on-prem-release/admin-appliance/1.30.400-gke.133/gke-on-prem-admin-appliance-vsphere-1.30.400-gke.133.ova"... [==================================================>] 10.7GB/10.7GB Image saved to /anthos/gke-on-prem-admin-appliance-vsphere-1.30.400-gke.133.ova Verifying image gke-on-prem-admin-appliance-vsphere-1.30.400-gke.133.ova... | runtime: out of memory 

Workaround:

Non-HA admin cluster upgrade stuck at Creating or updating cluster control plane workloads

When upgrading a non-HA admin cluster, the upgrade might get stuck at the Creating or updating cluster control plane workloads.

This issue happens if in the admin master VM, ip a | grep cali returns a non-empty result.

For example,

 ubuntu@abf8a975479b-qual-342-0afd1d9c ~ $ ip a | grep cali 4: cali2251589245f@if3:  mtu 1500 qdisc noqueue state UP group default 

Workaround:

1. Repair the admin master:

    gkectl repair admin-master --kubeconfig=ADMIN_KUBECONFIG \ --config=ADMIN_CONFIG_FILE \ --skip-validation 

2. Select the 1.30 VM template if you see a prompt like the following example:

    Please select the control plane VM template to be used for re-creating the admin cluster's control plane VM. 

3. Resume the upgrade:

    gkectl upgrade admin --kubeconfig ADMIN_KUBECONFIG \ --config ADMIN_CONFIG_FILE

Redundant isControlPlane field in the cluster configuration file under network.controlPlaneIPBlock

Cluster configuration files generated by gkectl create-config in 1.31.0 contain a redundant isControlPlane field under network.controlPlaneIPBlock:

controlPlaneIPBlock: netmask:""gateway:""ips: -ip:""hostname:""isControlPlane:false-ip:""hostname:""isControlPlane:false-ip:""hostname:""isControlPlane:false

This field is not needed and can be safely removed from the configuration file.

Admin add-on nodes stuck at NotReady during non-HA to HA admin cluster migration

When migrating a non-HA admin cluster that uses MetalLB to HA, admin add-on nodes might get stuck at a NotReady status, preventing the migration from completing.

This issue only affects admin clusters configured with MetalLB, where auto-repair isn't enabled.

This issue is caused by a race condition during migration where MetalLB speakers are still using the old metallb-memberlist secret. As a result of the race condition, the old control plane VIP becomes inaccessible, which causes the migration to stall.

Workaround:

1. Delete the existing metallb-memberlist secret:

    kubectl --kubeconfig=ADMIN_KUBECONFIG -n kube-system \ delete secret metallb-memberlist 

2. Restart the metallb-controller Deployment so it can generate the new metallb-memberlist:

    kubectl --kubeconfig=ADMIN_KUBECONFIG -n kube-system \ rollout restart deployment metallb-controller 

3. Ensure the new metallb-memberlist is generated:

    kubectl --kubeconfig=ADMIN_KUBECONFIG -n kube-system \ get secret metallb-memberlist 

4. Update updateStrategy.rollingUpdate.maxUnavailable in the metallb-speaker DaemonSet from 1 to 100%.

   This step is required, because certain DaemonSet pods are running on the NotReady nodes.

    kubectl --kubeconfig=ADMIN_KUBECONFIG -n kube-system \ edit daemonset metallb-speaker 

5. Restart the metallb-speaker DaemonSet so it can pick up the new memberlist:

    kubectl --kubeconfig=ADMIN_KUBECONFIG -n kube-system \ rollout restart daemonset metallb-speaker 

   After a few minutes, the admin add-on nodes become Ready again, and the migration can continue.

6. If the initial gkectl command timed out after more than 3 hours, rerun gkectl update to resume the failed migration.

Cluster backup for non-HA admin cluster fails due to long datastore and datadisk names

When attempting to backup a non-HA admin cluster, the backup fails due to the combined length of the datastore and datadisk names exceeding the maximum character length.

The maximum character length for a datastore name is 80.
The backup path for a non-HA admin cluster has the naming syntax "__". So if the concatenated name exceeds the maximum length, backup folder creation will fail

Workaround:

Rename the datastore or datadisk to a shorter name.
Ensure that the combined length of the datastore and datadisk names does not exceed the maximum character length.

HA admin control plane node shows older version after running gkectl repair admin-master

After running the gkectl repair admin-master command, an admin control plane node might show an older version than the expected version.

This issue occurs because the backed up VM template used for the HA admin control plane node repair isn't refreshed in vCenter after an upgrade, because the backup VM template wasn't cloned during machine creation if the machine name remains unchanged.

Workaround:

1. Find out the machine name that is using the older Kubernetes version:

    kubectl get machine -o wide --kubeconfig=ADMIN_KUBECONFIG

2. Remove the onprem.cluster.gke.io/prevented-deletion annotation:

    kubectl edit machine MACHINE_NAME --kubeconfig=ADMIN_KUBECONFIG

   Save the edit.

3. Run the following command to delete the machine:

    kubectl delete machine MACHINE_NAME --kubeconfig=ADMIN_KUBECONFIG

   A new machine will be created with the correct version.

Terraform removes vCenter fields unexpectedly

When updating a user cluster or nodepool using Terraform, Terraform might attempt to set vCenter fields to empty values.

This issue can occur if the cluster wasn't originally created using Terraform.

Workaround:

To prevent the unexpected update, ensure that the update is safe before running terraform apply, as described in the following:

1. Run terraform plan
2. In the output, check whether the vCenter fields are set to nil.
3. If any vCenter field is set to an empty value, in the Terraform configuration, add vcenter to the ignore_changes list following the Terraform documentation. This prevents updates to these fields.
4. Run terraform plan again and check the output to confirm the update is as expected

User cluster control plane nodes always get rebooted during the first admin cluster update operation

After the kubeception user clusters' control plane nodes are created, updated or upgraded, they will be rebooted one by one, during the first admin cluster operation when the admin cluster is created at or upgraded to one of the affected versions. For kubeception clusters with 3 control-plane nodes, this shouldn't lead to control-plane downtime and the only impact is that the admin cluster operation takes longer.

Errors creating custom resources

In version 1.31 of Google Distributed Cloud, you might get errors when you try to create custom resources, such as clusters (all types) and workloads. The issue is caused by a breaking change introduced in Kubernetes 1.31 that prevents the caBundle field in a custom resource definition from transitioning from a valid to an invalid state. For more information about the change, see the Kubernetes 1.31 changelog.

Prior to Kubernetes 1.31, the caBundle field was often set to a makeshift value of \n, because in earlier Kubernetes versions the API server didn't allow empty CA bundle content. Using \n was a reasonable workaround to avoid confusion, as the cert-manager typically updates the caBundle later.

If the caBundle has been patched once from an invalid to a valid state, there shouldn't be issues. However, if the custom resource definition is reconciled back to \n (or another invalid value), you might encounter the following error:

 ...Invalid value: []byte{0x5c, 0x6e}: unable to load root certificates: unable to parse bytes as PEM block] 

Workaround

If you have a custom resource definition in which caBundle is set to an invalid value, you can safely remove the caBundle field entirely. This should resolve the issue.

cloud-init status always returns error

When upgrading a cluster that uses the Container Optimized OS (COS) OS image to 1.31, the cloud-init status command fails although cloud-init finished without errors.

Workaround:

Run the following command to check the status of cloud-init:

 systemctl show -p Result cloud-final.service 

If the output is similar to the following, then cloud-init finished successfully:

 Result=success 

Admin workstation preflight check fails when upgrading to 1.28 with disk size less than 100 GB

When upgrading a cluster to 1.28, the gkectl prepare command fails while running admin workstation preflight checks if the admin workstation disk size is less than 100 GB. In this case, the command displays an error message similar to the following:

 Workstation Hardware: Workstation hardware requirements are not satisfied 

In 1.28, the admin workstation disk size prerequisite was increased from 50 GB to 100 GB.

Workaround:

1. Roll back the admin workstation.
2. Update the admin workstation config file to increase the disk size to at least 100 GB.
3. Upgrade the admin workstation.

gkectl returns false error on netapp storageclass

The gkectl upgrade command returns an incorrect error about the netapp storageclass. The error message is similar to the following,

detectedunsupporteddrivers: csi.trident.netapp.io 

Workaround:

Run gkectl upgrade with `--skip-pre-upgrade-checks` flag.

Invalid CA certificate after cluster CA rotation in ClientConfig prevents cluster authentication

After you rotate the certificate authority (CA) certificates on a user cluster, the spec.certificateAuthorityData field in the ClientConfig contains an invalid CA certificate, which prevents authentication to the cluster.

Workaround:

Before the next gcloud CLI authentication, manually update the spec.certificateAuthorityData field in the ClientConfig with the correct CA certificate.

1. Copy the cluster CA certificate from the certificate-authority-data field in the admin cluster kubeconfig.
2. Edit the ClientConfig and paste the CA certificate in the spec.certificateAuthorityData field.

   kubectleditclientconfigdefault-nkube-public--kubeconfigUSER_CLUSTER_KUBECONFIG

Preflight check fails when disabling bundled ingress

When you disable bundled ingress by removing the loadBalancer.vips.ingressVIP field in the cluster configuration file, a bug in the MetalLB preflight check causes the cluster update to fail with the "invalid user ingress vip: invalid IP" error message.

Workaround:

Ignore the error message.
Skip the preflight check using one of the following methods:

• Add the --skip-validation-load-balancer flag to the gkectl update cluster command.
• Annotate the onpremusercluster object with onprem.cluster.gke.io/server-side-preflight-skip: skip-validation-load-balancer
.

Cluster upgrade fails due to missing anti-affinity group rule in vCenter

During a cluster upgrade, the machine objects may get stuck in the `Creating` phase and fail to link to the node objects due to a missing anti-affinity group (AAG) rule in vCenter.

If you describe the problematic machine objects, you can see recurring messages like "Reconfigure DRS rule task "task-xxxx" complete"

kubectl--kubeconfigKUBECONFIGdescribemachineMACHINE_OBJECT_NAME

Workaround:

Disable the anti-affinity group setting for in both the admin cluster config and the user cluster config and trigger force update command to unblock cluster upgrade:

gkectlupdateadmin--configADMIN_CLUSTER_CONFIG_FILE--kubeconfigADMIN_CLUSTER_KUBECONFIG--force 

gkectlupdatecluster--configUSER_CLUSTER_CONFIG_FILE--kubeconfigUSER_CLUSTER_KUBECONFIG--force 

Migrating a user cluster to Controlplane V2 fails if secrets encryption has ever been enabled

When migrating a user cluster to Controlplane V2, if always-on secrets encryption has ever been enabled, the migration process fails to properly handle the secret encryption key. Because of this issue, the new Controlplane V2 cluster is unable to decrypt secrets. If the output of the following command isn't empty, then always-on secrets encryption has been enabled at some point and the cluster is affected by this issue:

kubectl--kubeconfigADMIN_CLUSTER_KUBECONFIG\getonpremuserclusterUSER_CLUSTER_NAME\-nUSER_CLUSTER_NAME-gke-onprem-mgmt\-ojsonpath={.spec.secretsEncryption}

If you have already started the migration and the migration fails, contact Google for support. Otherwise, before the migration, disable always-on secrets encryption and decrypt secrets.

Migrating an admin cluster from non-HA to HA fails if secrets encryption is enabled

If the admin cluster has enabled always-on secrets encryption at 1.14 or earlier, and upgraded all the way from old versions to the affected 1.29 and 1.30 versions, when migrating admin cluster from non-HA to HA, the migration process fails to properly handle the secret encryption key, Because of this issue, the new HA admin cluster is unable to decrypt secrets.

To check if the cluster could be using the old formatted key:

kubectl--kubeconfigADMIN_CLUSTER_KUBECONFIGgetsecret-nkube-systemadmin-master-component-options-ojsonpath='{.data.data}'|base64-d|grep-oP'"GeneratedKeys":\[.*?\]'

If the output shows the empty key like the following, then it like means the cluster will be affected by this issue:

"GeneratedKeys":[{"KeyVersion":"1","Key":""}]

If you have already started the migration and the migration fails, contact Google for support.

Otherwise, before starting the migration, rotate the encryption key.

credential.yaml regenerated incorrectly during admin workstation upgrade

When upgrading the admin workstation using the gkeadm upgrade admin-workstation command, the credential.yaml file is regenerated incorrectly. The username and password fields are empty. Additionally, the privateRegistry key contains a typo.

The same misspelling of the privateRegistry key is also in the admin-cluster.yaml file.
Since the credential.yaml file is regenerated during the admin cluster upgrade process, the typo is present even if you corrected previously.

Workaround:

• Update the private registry key name in credential.yaml to match the privateRegistry.credentials.fileRef.entry in the admin-cluster.yaml.
• Update the private registry username and password in the credential.yaml.

User cluster upgrade fails due to pre-upgrade reconcile timeout

When upgrading a user cluster, the pre-upgrade reconcile operation might take longer than the defined timeout, resulting in an upgrade failure. The error message looks like the following:

 Failed to reconcile the user cluster before upgrade: the pre-upgrade reconcile failed, error message: failed to wait for reconcile to complete: error: timed out waiting for the condition, message: Cluster reconcile hasn't finished yet, please fix that before rerun the upgrade. 

The timeout for the pre-upgrade reconcile operation is 5 minutes plus 1 minute per node pool in the user cluster.

Workaround:

Ensure that the gkectl diagnose cluster command passes without errors.
Skip the pre-upgrade reconcile operation by adding the --skip-reconcile-before-preflight flag to the gkectl upgrade cluster command. For example:

gkectlupgradecluster--skip-reconcile-before-preflight--kubeconfigADMIN_CLUSTER_KUBECONFIG\ --configUSER_CLUSTER_CONFIG_FILE

Updating DataplaneV2 ForwardMode doesn't automatically trigger anetd DaemonSet restart

When you update the user cluster dataplaneV2.forwardMode field using gkectl update cluster, the change is only updated in the ConfigMap, the anetd DaemonSet won't pick up the config change until restarted and your changes aren't applied.

Workaround:

When the gkectl update cluster command is done, you see output of Done updating the user cluster. After you see that message, run the following command to restart the anetd DaemonSet to pick up the config change:

kubectl--kubeconfigUSER_CLUSTER_KUBECONFIGrolloutrestartdaemonsetanetd

Check the DaemonSet readiness after the restart:

kubectl--kubeconfigUSER_CLUSTER_KUBECONFIGgetdaemonsetanetd

In the output of the preceding command, verify that the number in the DESIRED column matches the number in the READY column.

etcdctl command not found during cluster upgrade at the admin cluster backup stage

During a 1.16 to 1.28 user cluster upgrade, the admin cluster is backed up. The admin cluster backup process displays the error message "etcdctl: command not found". The user cluster upgrade succeeds, and the admin cluster remains in a healthy state. The only issue is that the metadata file on the admin cluster isn't backed up.

The cause of the issue is that the etcdctl binary was added in 1.28, and isn't available on 1.16 nodes.

The admin cluster backup involve several steps, including taking an etcd snapshot and then writing the metadata file for the admin cluster. The etcd backup still succeeds because etcdctl can still be triggered after an exec into the etcd Pod. But writing the metadata file fails as it still relies on the etcdctl binary to be installed on the node. However, the metadata file backup isn't a blocker for taking a backup, so the backup process still succeeds, as does the user cluster upgrade.

Workaround:

If want to take a backup of the metadata file, follow Back up and restore an admin cluster with gkectl to trigger a separate admin cluster backup using the version of gkectl that matches the version of your admin cluster.

User cluster creation failure with manual load balancing

When creating a user cluster configured for manual load balancing, a gkectl check-config failure occurs indicating that the ingressHTTPNodePort value must be at least 30000, even when bundled ingress is disabled.

This issue occurs regardless of whether the ingressHTTPNodePort and ingressHTTPSNodePort fields are set or left blank.

Workaround:

To work around this issue, ignore the result returned by gkectl check-config. To disable bundled ingress, see Disable bundled ingress.

After migrating a user cluster to Controlplane V2, the admin cluster metrics-server PDB might become incorrectly configured

The issue with the PodDisruptionBudget (PDB) occurs when using high availability (HA) admin clusters, and there is 0 or 1 admin cluster node without a role after the migration. To check if there are node objects without a role, run the following command:

kubectl--kubeconfigADMIN_CLUSTER_KUBECONFIGgetnodes-owide

If there are two or more node objects without a role after the migration, then the PDB isn't misconfigured.

Symptoms:

The output of admin cluster diagnose includes the following information

 Checking all poddisruptionbudgets...FAILURE Reason: 1 pod disruption budget error(s). Unhealthy Resources: PodDisruptionBudget metrics-server: gke-managed-metrics-server/metrics-server might be configured incorrectly: the total replicas(1) should be larger than spec.MinAvailable(1). 

Workaround:

Run the following command to delete the PDB:

kubectl--kubeconfigADMIN_CLUSTER_KUBECONFIGdeletepdbmetrics-server-ngke-managed-metrics-server

Binary Authorization webook blocks CNI plugin to start causing one of nodepool failed to come up

Under rare race conditions, an incorrect installation sequence of the Binary Authorization webhook and the gke-connect pod may cause user cluster creation to stall due to a node failing to reach a ready state. In affected scenarios, user cluster creation may stall due to a node failing to reach a ready state. If this occurs, the following message will be displayed:

 Node pool is not ready: ready condition is not true: CreateOrUpdateNodePool: 2/3 replicas are ready 

Workaround:

1. Remove the Binary Authorization configuration from your config file. For setup instructions, please refer to the Binary Authorization day 2 installation guide for GKE on VMware.
2. To unblock an unhealthy node during the current cluster creation process, temporarily remove the Binary Authorization webhook configuration in user cluster using the following command.

   kubectl--kubeconfigUSER_KUBECONFIGdeleteValidatingWebhookConfigurationbinauthz-validating-webhook-configuration 

   Once the bootstrap process is complete, you can re-add the following webhook configuration.

   apiVersion:admissionregistration.k8s.io/v1kind:ValidatingWebhookConfigurationmetadata:name:binauthz-validating-webhook-configurationwebhooks:-name:"binaryauthorization.googleapis.com"namespaceSelector:matchExpressions:-key:control-planeoperator:DoesNotExistobjectSelector:matchExpressions:-key:"image-policy.k8s.io/break-glass"operator:NotInvalues:["true"]rules:-apiGroups:-""apiVersions:-v1operations:-CREATE-UPDATEresources:-pods-pods/ephemeralcontainersadmissionReviewVersions:-"v1beta1"clientConfig:service:name:binauthznamespace:binauthz-systempath:/binauthz# CA Bundle will be updated by the cert rotator.caBundle:Cg==timeoutSeconds:10# Fail OpenfailurePolicy:"Ignore"sideEffects:None

CPV2 user cluster upgrade stuck due to mirrored machine with deletionTimestamp

During a user cluster upgrade, the upgrade operation might get stuck if the mirrored machine object in the user cluster contains a deletionTimestamp. The following error message is displayed if the upgrade is stuck:

 machine is still in the process of being drained and subsequently removed 

This issue can occur if you previously attempted to repair the user control plane node by running gkectl delete machine against the mirrored machine in the user cluster.

Workaround:

1. Get the mirrored machine object and save it to a local file for backup purposes.
2. Run the following command to delete the finalizer from the mirrored machine and wait for it to be deleted from the user cluster.

   kubectl--kubeconfigADMIN_KUBECONFIGpatchmachine/MACHINE_OBJECT_NAME-nUSER_CLUSTER_NAME-gke-onprem-mgmt-p'{"metadata":{"finalizers":[]}}'--type=merge

3. Follow the steps in Controlplane V2 user cluster control plane node to trigger node repair on the control plane nodes, so that the correct source machine spec will be re-synced into the user cluster.
4. Rerun the gkectl upgrade cluster to resume the upgrade

Cluster creation failure due to control plane VIP in different subnet

For HA admin cluster or ControlPlane V2 user cluster, the control plane VIP needs to be in the same subnet as other cluster nodes. Otherwise, cluster creation fails because kubelet can't communicate with the API server using the control plane VIP.

Workaround:

Before cluster creation, ensure that the control plane VIP is configured in the same subnet as the other cluster nodes.

Cluster Creation/Upgrade Failure due to non-FQDN vCenter Username

Cluster creation/upgrade fails with an error in vsphere CSI pods indicating that the vCenter username is invalid. This occurs because the username used is not a fully qualified domain name. Error message in the vsphere-csi-controller pod as below:

 GetCnsconfig failed with err: username is invalid, make sure it is a fully qualified domain username 

This issue only occurs in version 1.29 and later, as a validation was added to the vSphere CSI driver to enforce the use of fully qualified domain usernames.

Workaround:

Use a fully qualified domain name for the vCenter username in the credentials configuration file. For example, instead of using "username1", use "username1@example.com".

Admin cluster upgrade fails for clusters created on versions 1.10 or earlier

When upgrading an admin cluster from 1.16 to 1.28, the bootstrap of the new admin master machine might fail to generate the control-plane certificate. The issue is caused by changes in how certificates are generated for the Kubernetes API server in version 1.28 and later. The issue reproduces for clusters created on versions 1.10 and earlier that have been upgraded all the way to 1.16 and the leaf certificate was not rotated before the upgrade.

To determine if the admin cluster upgrade failure is caused by this issue, do the following steps:

1. Connect to the failed admin master machine by using SSH.
2. Open /var/log/startup.log and search for an error like the following:

 Error adding extensions from section apiserver_ext 801B3213B57F0000:error:1100007B:X509 V3 routines:v2i_AUTHORITY_KEYID:unable to get issuer keyid:../crypto/x509/v3_akid.c:177: 801B3213B57F0000:error:11000080:X509 V3 routines:X509V3_EXT_nconf_int:error in extension:../crypto/x509/v3_conf.c:48:section=apiserver_ext, name=authorityKeyIdentifier, value=keyid>

Workaround:

1. Connect to the admin master machine by using SSH. For details, see Using SSH to connect to an admin cluster node.
2. Make a copy /etc/startup/pki-yaml.sh and name it /etc/startup/pki-yaml-copy.sh
3. Edit /etc/startup/pki-yaml-copy.sh. Find authorityKeyIdentifier=keyidset and change it to authorityKeyIdentifier=keyid,issuer in the sections for the following extensions: apiserver_ext, client_ext, etcd_server_ext, and kubelet_server_ext. For example:

    [ apiserver_ext ] keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage=serverAuth basicConstraints = critical,CA:false authorityKeyIdentifier = keyid,issuer subjectAltName = @apiserver_alt_names 

4. Save the changes to /etc/startup/pki-yaml-copy.sh.
5. Using a text editor, open /opt/bin/master.sh, find and replace all occurrences of /etc/startup/pki-yaml.sh with /etc/startup/pki-yaml-copy.sh, then save the changes.
6. Run /opt/bin/master.sh to generate the certificate and complete the machine startup.
7. Run the gkectl upgrade admin again to upgrade the admin cluster.
8. After the upgrade completes, rotate the leaf certificate for both admin and user clusters, as described in Start the rotation.
9. After the certificate rotation completes, make the same edits to /etc/startup/pki-yaml-copy.sh as you did previously, and run /opt/bin/master.sh.

Incorrect warning message for clusters with Dataplane V2 enabled

The following incorrect warning message is output when you run gkectl to create, update, or upgrade a cluster that already has Dataplane V2 enabled:

 WARNING: Your user cluster is currently running our original architecture with [DataPlaneV1(calico)]. To enable new and advanced features we strongly recommend to update it to the newer architecture with [DataPlaneV2] once our migration tool is available. 

There's a bug in gkectl which causes it to always show this warning as long as the dataplaneV2.forwardMode is not being used, even if you already have set enableDataplaneV2: true in your cluster configuration file.

Workaround:

You can safely ignore this warning.

HA admin cluster installation preflight check reports wrong number of required static IPs

When you create an HA admin cluster, the preflight check displays the following incorrect error message:

 - Validation Category: Network Configuration - [FAILURE] CIDR, VIP and static IP (availability and overlapping): needed at least X+1 IP addresses for admin cluster with X nodes 

The requirement is incorrect for 1.28 and higher HA admin clusters because they no longer have add-on nodes. Additionally, because the 3 admin cluster control plane node IPs are specified in the network.controlPlaneIPBlock section in the admin cluster configuration file, the IPs in IP block file are only needed for kubeception user cluster control plane nodes.

Workaround:

To skip the incorrect preflight check in a non-fixed release, add --skip-validation-net-config to the gkectl command.

Connect Agent loses connection to Google Cloud after non-HA to HA admin cluster migration

If you migrated from a non-HA admin cluster to an HA admin cluster, the Connect Agent in the admin cluster loses the connection to gkeconnect.googleapis.com with the error "Failed to verify JWT signature". This is because during the migration, the KSA signing key is changed, thus a re-registration is needed to refresh the OIDC JWKs.

Workaround:

To reconnect the admin cluster to Google Cloud, do the following steps to trigger a re-registration:

First get the gke-connect deployment name:

kubectl--kubeconfigKUBECONFIGgetdeployment-ngke-connect

Delete the gke-connect deployment:

kubectl--kubeconfigKUBECONFIGdeletedeploymentGKE_CONNECT_DEPLOYMENT-ngke-connect

Trigger a force reconcile for the onprem-admin-cluster-controller by adding a "force-reconcile" annotation to your onpremadmincluster CR:

kubectl--kubeconfigKUBECONFIGpatchonpremadminclusterADMIN_CLUSTER_NAME-nkube-system--typemerge-p'metadata: annotations: onprem.cluster.gke.io/force-reconcile: "true"'

The idea is that the onprem-admin-cluster-controller will always redeploy the gke-connect deployment and re-register the cluster if it finds no existing gke-connect deployment available.

After the workaround (it may take a few minutes for the controller to finish the reconcile), you can verify that the "Failed to verify JWT signature" 400 error is gone from the gke-connect-agent logs:

kubectl--kubeconfigKUBECONFIGlogsGKE_CONNECT_POD_NAME-ngke-connect

Docker bridge IP uses 172.17.0.1/16 for COS cluster control plane nodes

Google Distributed Cloud specifies a dedicated subnet, --bip=169.254.123.1/24, for the Docker bridge IP in the Docker configuration to prevent reserving the default 172.17.0.1/16 subnet. However, in version 1.28.0-1.28.500 and 1.29.0, the Docker service wasn't restarted after Google Distributed Cloud customized the Docker configuration because of a regression in the COS OS image. As a result, Docker picks the default 172.17.0.1/16 as its bridge IP address subnet. This might cause an IP address conflict if you already have a workload running within that IP address range.

Workaround:

To work around this issue, you must restart the docker service:

sudosystemctlrestartdocker

Verify that Docker picks the correct bridge IP address:

ipa|grepdocker0

This solution does not persist across VM re-creations. You must reapply this workaround whenever VMs are re-created.

Using multiple network interfaces with standard CNI does not work

The standard CNI binaries bridge, ipvlan, vlan, macvlan, dhcp, tuning, host-local, ptp, portmap are not included in the OS images in the affected versions. These CNI binaries are not used by data plane v2, but can be used for additional network interfaces in the multiple network interface feature.

Multiple network interface with these CNI plugins won't work.

Workaround:

Upgrade to the version with the fix if you are using this feature.

Netapp trident dependencies interfere with vSphere CSI driver

Installing multipathd on cluster nodes interferes with the vSphere CSI driver resulting in user workloads being unable to start.

Workaround:

• Disable multipathd

The admin cluster webhook might block updates

If some required configurations are empty in the admin cluster because validations were skipped, adding them might be blocked by the admin cluster webhook. For example, if the gkeConnect field wasn't set in an existing admin cluster, adding it with the gkectl update admin command might get the following error message:

 admission webhook "vonpremadmincluster.onprem.cluster.gke.io" denied the request: connect: Required value: GKE connect is required for user clusters 

 failed to apply OnPremAdminCluster 'kube-system/gke-admin-btncc': Internal error occurred: failed calling webhook "monpremadmincluster.onprem.cluster.gke.io": failed to call webhook: Post "https://onprem-admin-cluster-controller.kube-system.svc:443/mutate-onprem-cluster-gke-io-v1alpha1-onpremadmincluster?timeout=10s": dial tcp 10.96.55.208:443: connect: connection refused 

Workaround:

• For 1.15 admin clusters, run gkectl update admin command with --disable-admin-cluster-webhook flag. For example:

  gkectlupdateadmin--configADMIN_CLUSTER_CONFIG_FILE--kubeconfigADMIN_CLUSTER_KUBECONFIG--disable-admin-cluster-webhook 

• For 1.16 admin clusters, run gkectl update admin commands with --force flag. For example:

  gkectlupdateadmin--configADMIN_CLUSTER_CONFIG_FILE--kubeconfigADMIN_CLUSTER_KUBECONFIG--force 

controlPlaneNodePort field defaults to 30968 when manualLB spec is empty

If you will be using a manual load balancer (loadBalancer.kind is set to "ManualLB"), you shouldn't need to configure the loadBalancer.manualLB section in the configuration file for a high availability (HA) admin cluster in versions 1.16 and higher. But when this section is empty, Google Distributed Cloud assigns default values to all NodePorts including manualLB.controlPlaneNodePort, which causes cluster creation to fail with the following error message:

-ValidationCategory:ManualLB -[FAILURE]NodePortconfiguration:manualLB.controlPlaneNodePortmust notbesetwhenusingHAadmincluster,got:30968

Workaround:

Specify manualLB.controlPlaneNodePort: 0 in you admin cluster configuration for the HA admin cluster:

loadBalancer:...kind:ManualLBmanualLB:controlPlaneNodePort:0...

nfs-common is missing from Ubuntu OS image

nfs-common is missing from the Ubuntu OS image which may cause issues for customers using NFS-dependent drivers such as NetApp.

WarningFailedMount63s(x8over2m28s)kubeletMountVolume.SetUpfailedforvolume"pvc-xxx-yyy-zzz":rpcerror:code=Internaldesc=errormountingNFSvolume10.0.0.2:/trident_pvc_xxx-yyy-zzzonmountpoint/var/lib/kubelet/pods/aaa-bbb-ccc/volumes/kubernetes.io~csi/pvc-xxx-yyy-zzz/mount:exitstatus32".

Workaround:

Make sure your nodes can download packages from Canonical.

Next, apply the following DaemonSet to your cluster to install nfs-common:

apiVersion:apps/v1kind:DaemonSetmetadata:name:install-nfs-commonlabels:name:install-nfs-commonnamespace:kube-systemspec:selector:matchLabels:name:install-nfs-commonminReadySeconds:0updateStrategy:type:RollingUpdaterollingUpdate:maxUnavailable:100%template:metadata:labels:name:install-nfs-commonspec:hostPID:truehostIPC:truehostNetwork:trueinitContainers:-name:install-nfs-commonimage:ubuntuimagePullPolicy:IfNotPresentsecurityContext:privileged:truecommand:-chroot-/host-bash--cargs:-|apt install -y nfs-commonvolumeMounts:-name:hostmountPath:/hostcontainers:-name:pauseimage:gcr.io/gke-on-prem-release/pause-amd64:3.1-gke.5imagePullPolicy:IfNotPresentvolumes:-name:hosthostPath:path:/

Storage policy field is missing in the admin cluster configuration template

SPBM in admin clusters is supported in 1.28.0 and later versions. But the field vCenter.storagePolicyName is missing in the configuration file template.

Workaround:

Add the `vCenter.storagePolicyName` field in you admin cluster configuration file if you want to configure the storage policy for the admin cluster. Please refer to the instructions.

Kubernetes Metadata API does not support VPC-SC

The recently added API kubernetesmetadata.googleapis.com does not support VPC-SC. This will cause metadata collecting agent to fail to reach this API under VPC-SC. Subsequently, metric metadata labels will be missing.

Workaround:

Set in `kube-system` namespace the CR `stackdriver` set `featureGates.disableExperimentalMetadataAgent` field to `true` by running the command

`kubectl -n kube-system patch stackdriver stackdriver -p '{"spec":{"featureGates":{"disableExperimentalMetadataAgent":true}}}'`

then run

`kubectl -n kube-system patch deployment stackdriver-operator -p '{"spec":{"template":{"spec":{"containers":[{"name":"stackdriver-operator","env":[{"name":"ENABLE_LEGACY_METADATA_AGENT","value":"true"}]}]}}}}'`

The clusterapi-controller may crash when the admin cluster and any user cluster with ControlPlane V2 enabled use different vSphere credentials

When an admin cluster and any user cluster with ControlPlane V2 enabled use different vSphere credentials, e.g., after updating vSphere credentials for the admin cluster, the clusterapi-controller may fail to connect to the vCenter after restart. View the log of the clusterapi-controller running in the admin cluster's `kube-system` namespace,

kubectllogs-f-lcomponent=clusterapi-controllers-cvsphere-controller-manager\-nkube-system--kubeconfigKUBECONFIG

E121400:02:54.0956681machine_controller.go:165]Errorcheckingexistenceofmachineinstanceformachineobjectgke-admin-node-77f48c4f7f-s8wj2:Failedtocheckifmachinegke-admin-node-77f48c4f7f-s8wj2exists:failedtofinddatacenter"VSPHERE_DATACENTER":datacenter'VSPHERE_DATACENTER'notfound

Workaround:

Update vSphere credentials so that the admin cluster and all the user clusters with Controlplane V2 enabled are using the same vSphere credentials.

etcd high number of failed GRPC requests in Prometheus Alert Manager

Prometheus might report alerts similar to the following example:

AlertName:cluster:gke-admin-test1:Etcdclusterkube-system/kube-etcd:100%ofrequestsforWatchfailedonetcdinstanceetcd-test-xx-n001.

To check if this alert is a false positive that can be ignored, complete the following steps:

1. Check the raw grpc_server_handled_total metric against the grpc_method given in the alert message. In this example, check the grpc_code label for Watch.
   
   You can check this metric using Cloud Monitoring with the following MQL query:

   fetch k8s_container|metric'kubernetes.io/anthos/grpc_server_handled_total'|alignrate(1m)|every1m

2. An alert on all codes other than OK can be safely ignored if the code is not one of the following:

   Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded

Workaround:

To configure Prometheus to ignore these false positive alerts, review the following options:

1. Silence the alert from the Alert Manager UI.
2. If silencing the alert isn't an option, review the following steps to suppress the false positives:
   1. Scale down the monitoring operator to 0 replicas so that the modifications can persist.
   2. Modify the prometheus-config configmap, and add grpc_method!="Watch" to the etcdHighNumberOfFailedGRPCRequests alert config as shown in the following example:
      • Original:

        rate(grpc_server_handled_total{cluster="CLUSTER_NAME",grpc_code!="OK",job=~".*etcd.*"}[5m])

      • Modified:

        rate(grpc_server_handled_total{cluster="CLUSTER_NAME",grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded",grpc_method!="Watch",job=~".*etcd.*"}[5m])

        Replace the following CLUSTER_NAME with the name of your cluster.
   3. Restart the Prometheus and Alertmanager Statefulset to pick up the new configuration.
3. If the code falls into one of the problematic cases, then check etcd log and kube-apiserver log to debug more.

Egress NAT long lived connections are dropped

Egress NAT connections might be dropped after 5 to 10 minutes of a connection being established if there's no traffic.

As the conntrack only matters in the inbound direction (external connections to the cluster), this issue only happens if the connection doesn't transmit any information for a while and then the destination side transmits something. If the egress NAT'd Pod always instantiates the messaging, then this issue won't be seen.

This issue occurs because the anetd garbage collection inadvertently removes conntrack entries that the daemon thinks are unused. An upstream fix was recently integrated into anetd to correct the behavior.

Workaround:

There is no easy workaround, and we haven't seen issues in version 1.16 from this behavior. If you notice long lived connections dropped due to this issue, workarounds would be to use a workload on the same node as the egress IP address, or to consistently send messages on the TCP connection.

The CSR signer ignores spec.expirationSeconds when signing certificates

If you create a CertificateSigningRequest (CSR) with expirationSeconds set, the expirationSeconds is ignored.

Workaround:

If you're affected by this issue, you can update your user cluster by adding disableNodeIDVerificationCSRSigning: true in the user cluster configuration file and run the gkectl update cluster command to update the cluster with this configuration.

User cluster load balancer validation fails for disable_bundled_ingress

If you try to disable bundled ingress for an existing cluster, the gkectl update cluster command fails with an error similar to the following example:

 [FAILURE] Config: ingress IP is required in user cluster spec 

This error happens because gkectl checks for a load balancer ingress IP address during preflight checks. Although this check isn't required when disabling bundled ingress, the gkectl preflight check fails when disableBundledIngress is set to true.

Workaround:

Use the --skip-validation-load-balancer parameter when you update the cluster, as shown in the following example:

gkectlupdatecluster\--kubeconfigADMIN_CLUSTER_KUBECONFIG--configUSER_CLUSTER_CONFIG\--skip-validation-load-balancer

For more information, see how to disable bundled ingress for an existing cluster.

Admin cluster updates fail after CA rotation

If you rotate admin cluster certificate authority (CA) certificates, subsequent attempts to run the gkectl update admin command fail. The error returned is similar to the following:

 failed to get last CARotationStage: configmaps "ca-rotation-stage" not found 

Workaround:

If you're affected by this issue, you can update your admin cluster by using the --disable-update-from-checkpoint flag with the gkectl update admin command:

gkectlupdateadmin--configADMIN_CONFIG_file\--kubeconfigADMIN_CLUSTER_KUBECONFIG\--disable-update-from-checkpoint

When you use the --disable-update-from-checkpoint flag, the update command doesn't use the checkpoint file as the source of truth during the cluster update. The checkpoint file is still updated for future use.

CSI Workload preflight check fails due to Pod startup failure

During preflight checks, the CSI Workload validation check installs a Pod in the default namespace. The CSI Workload Pod validates that the vSphere CSI Driver is installed and can do dynamic volume provisioning. If this Pod doesn't start, the CSI Workload validation check fails.

There are a few known issues that can prevent this Pod from starting:

• If the Pod doesn't have resources limits specified, which is the case for some clusters with admissions webhooks installed, the Pod doesn't start.
• If Cloud Service Mesh is installed in the cluster with automatic Istio sidecar injection enabled in the default namespace, the CSI Workload Pod doesn't start.

If the CSI Workload Pod doesn't start, you see a timeout error like the following during preflight validations:

-[FAILURE]CSIWorkload:failureinCSIWorkloadvalidation:failedtocreatewriterJobtoverifythewritefunctionalityusingCSI:Jobdefault/anthos-csi-workload-writer-<run-id>replicasarenotinSucceededphase:timedoutwaitingforthecondition

To see if the failure is caused by lack of Pod resources set, run the following command to check the anthos-csi-workload-writer-<run-id> job status:

kubectldescribejobanthos-csi-workload-writer-<run-id>

If the resources limits aren't set properly for the CSI Workload Pod, the job status contains an error message like the following:

CPUandmemoryresourcelimitsisinvalid,asitarenotdefinedforcontainer:volume-tester

If the CSI Workload Pod doesn't start because of Istio sidecar injection, you can temporarily disable the automatic Istio sidecar injection in the default namespace. Check the labels of the namespace and use the following command to delete the label that starts with istio.io/rev:

kubectllabelnamespacedefaultistio.io/rev-

If the Pod is misconfigured, manually verify that dynamic volume provisioning with the vSphere CSI Driver works:

1. Create a PVC that uses the standard-rwo StorageClass.
2. Create a Pod that uses the PVC.
3. Verify that the Pod can read/write data to the volume.
4. Remove the Pod and the PVC after you've verified proper operation.

If dynamic volume provisioning with the vSphere CSI Driver works, run gkectl diagnose or gkectl upgrade with the --skip-validation-csi-workload flag to skip the CSI Workload check.

User cluster update timeouts when admin cluster version is 1.15

When you are logged on to a user-managed admin workstation, the gkectl update cluster command might timeout and fail to update the user cluster. This happens if the admin cluster version is 1.15 and you run gkectl update admin before you run the gkectl update cluster. When this failure happens, you see the following error when trying to update the cluster:

 Preflight check failed with failed to run server-side preflight checks: server-side preflight checks failed: timed out waiting for the condition Preflight check failed with failed to run server-side preflight checks: server-side preflight checks failed: timed out waiting for the condition 

During the update of a 1.15 admin cluster, the validation-controller that triggers the preflight checks is removed from the cluster. If you then try to update the user cluster, the preflight check hangs until the timeout is reached.

Workaround:

1. Run the following command to redeploy the validation-controller:

    gkectl prepare --kubeconfig ADMIN_KUBECONFIG --bundle-path BUNDLE_PATH --upgrade-platform 

2. After the prepare completes, run the gkectl update cluster again to update the user cluster

User cluster create timeouts when admin cluster version is 1.15

When you are logged on to a user-managed admin workstation, the gkectl create cluster command might timeout and fail to create the user cluster. This happens if the admin cluster version is 1.15. When this failure happens, you see the following error when trying to create the cluster:

 Preflight check failed with failed to run server-side preflight checks: server-side preflight checks failed: timed out waiting for the condition Preflight check failed with failed to run server-side preflight checks: server-side preflight checks failed: timed out waiting for the condition 

Since the validation-controller was added in 1.16 then when using 1.15 admin cluster the validation-controller that is responsible to trigger the preflight checks is missing. Therefore, when trying to create user cluster the preflight checks hang till timeout is reached.

Workaround:

1. Run the following command to deploy the validation-controller:

    gkectl prepare --kubeconfig ADMIN_KUBECONFIG --bundle-path BUNDLE_PATH --upgrade-platform 

2. After the prepare completes, run the gkectl create cluster again to create the user cluster

Admin cluster update or upgrade fails if the projects or locations of add-on services don't match each other

When you upgrade an admin cluster from version 1.15.x to 1.16.x, or add a connect, stackdriver, cloudAuditLogging, or gkeOnPremAPI configuration when you update an admin cluster, the operation might be rejected by admin cluster webhook. One of the following error messages might be displayed:

• "projects for connect, stackdriver and cloudAuditLogging must be the same when specified during cluster creation."
• "locations for connect, gkeOnPremAPI, stackdriver and cloudAuditLogging must be in the same region when specified during cluster creation."
• "locations for stackdriver and cloudAuditLogging must be the same when specified during cluster creation."

An admin cluster update or upgrade requires the onprem-admin-cluster-controller to reconcile the admin cluster in a kind cluster. When the admin cluster state is restored in the kind cluster, the admin cluster webhook can't distinguish if the OnPremAdminCluster object is for an admin cluster creation, or to resume operations for an update or upgrade. Some create-only validations are invoked on updating and upgrading unexpectedly.

Workaround:

Add the onprem.cluster.gke.io/skip-project-location-sameness-validation: true annotation to the OnPremAdminCluster object:

1. Edit the onpremadminclusters cluster resource:

   kubectleditonpremadminclustersADMIN_CLUSTER_NAME-nkube-system–kubeconfigADMIN_CLUSTER_KUBECONFIG

   Replace the following:
   • ADMIN_CLUSTER_NAME: the name of the admin cluster.
   • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
2. Add the onprem.cluster.gke.io/skip-project-location-sameness-validation: true annotation and save the custom resource.
3. Depending on the type of admin clusters, complete one of the following steps:
   • For non-HA admin clusters with a checkpoint file: add the parameter disable-update-from-checkpoint in the update command, or add the parameter `disable-upgrade-from-checkpoint` in the upgrade command. These parameters are only needed for the next time that you run the update or upgrade command:
     • gkectlupdateadmin--configADMIN_CONFIG_file--kubeconfigADMIN_CLUSTER_KUBECONFIG\--disable-update-from-checkpoint

     • gkectlupgradeadmin--configADMIN_CONFIG_file--kubeconfigADMIN_CLUSTER_KUBECONFIG\--disable-upgrade-from-checkpoint

   • For HA admin clusters or checkpoint file is disabled: update or upgrade admin cluster as normal. No additional parameters are needed on the update or upgrade commands.

User cluster deletion fails when using a user-managed admin workstation

When you are logged on to a user-managed admin workstation, the gkectl delete cluster command might timeout and fail to delete the user cluster. This happens if you have first run gkectl on the user-managed workstation to create, update, or upgrade the user cluster. When this failure happens, you see the following error when trying to delete the cluster:

 failed to wait for user cluster management namespace "USER_CLUSTER_NAME-gke-onprem-mgmt" to be deleted: timed out waiting for the condition 

During deletion, a cluster first deletes all of its objects. The deletion of the Validation objects (that were created during the create, update, or upgrade) are stuck at the deleting phase. This happens because a finalizer blocks the object's deletion, which causes cluster deletion to fail.

Workaround:

1. Get the names of all the Validation objects:

    kubectl --kubeconfig ADMIN_KUBECONFIG get validations \ -n USER_CLUSTER_NAME-gke-onprem-mgmt 

2. For each Validation object, run the following command to remove the finalizer from the Validation object:

    kubectl --kubeconfig ADMIN_KUBECONFIG patch validation/VALIDATION_OBJECT_NAME \ -n USER_CLUSTER_NAME-gke-onprem-mgmt -p '{"metadata":{"finalizers":[]}}' --type=merge 

   After removing the finalizer from all Validation objects, the objects are removed and the user cluster delete operation completes automatically. You don't need to take additional action.

Egress NAT gateway traffic to external server fails

If the source Pod and egress NAT gateway Pod are on two different worker nodes, traffic from the source Pod can't reach any external services. If the Pods are located on the same host, the connection to external service or application is successful.

This issue is caused by vSphere dropping VXLAN packets when tunnel aggregation is enabled. There's a known issue with NSX and VMware that only sends aggregated traffic on known VXLAN ports (4789).

Workaround:

Change the VXLAN port used by Cilium to 4789:

1. Edit the cilium-config ConfigMap:

   kubectleditcm-nkube-systemcilium-config--kubeconfigUSER_CLUSTER_KUBECONFIG

2. Add the following to the cilium-config ConfigMap:

   tunnel-port:4789

3. Restart the anetd DaemonSet:

   kubectlrolloutrestartdsanetd-nkube-system--kubeconfigUSER_CLUSTER_KUBECONFIG

This workaround reverts every time the cluster is upgraded. You must reconfigure after each upgrade. VMware must resolve their issue in vSphere for a permanent fix.

Upgrading an admin cluster with always-on secrets encryption enabled fails

The admin cluster upgrade from 1.14.x to 1.15.x with always-on secrets encryption enabled fails due to a mismatch between the controller-generated encryption key with the key that persists on the admin master data disk. The output of gkectl upgrade admin contains the following error message:

 E0926 14:42:21.796444 40110 console.go:93] Exit with error: E0926 14:42:21.796491 40110 console.go:93] Failed to upgrade the admin cluster: failed to create admin cluster: failed to wait for OnPremAdminCluster "admin-cluster-name" to become ready: failed to wait for OnPremAdminCluster "admin-cluster-name" to be ready: error: timed out waiting for the condition, message: failed to wait for OnPremAdminCluster "admin-cluster-name" to stay in ready status for duration "2m0s": OnPremAdminCluster "non-prod-admin" is not ready: ready condition is not true: CreateOrUpdateControlPlane: Creating or updating credentials for cluster control plane 

Running kubectl get secrets -A --kubeconfig KUBECONFIG` fails with the following error:

 Internal error occurred: unable to transform key "/registry/secrets/anthos-identity-service/ais-secret": rpc error: code = Internal desc = failed to decrypt: unknown jwk 

Workaround

If you have a backup of the admin cluster, do the following steps to workaround the upgrade failure:

1. Disable secretsEncryption in the admin cluster configuration file, and update the cluster using the following command:

   gkectlupdateadmin--configADMIN_CLUSTER_CONFIG_FILE--kubeconfigKUBECONFIG

2. When the new admin master VM is created, SSH to the admin master VM, replace the new key on the data disk with the old one from the backup. The key is located at /opt/data/gke-k8s-kms-plugin/generatedkeys on the admin master.
3. Update the kms-plugin.yaml static Pod manifest in /etc/kubernetes/manifests to update the --kek-id to match the kid field in the original encryption key.
4. Restart the kms-plugin static Pod by moving the /etc/kubernetes/manifests/kms-plugin.yaml to another directory then move it back.
5. Resume the admin upgrade by running gkectl upgrade admin again.

Preventing the upgrade failure

If you haven't already upgraded, we recommend that you don't upgrade to 1.15.0-1.15.4. If you must upgrade to an affected version, do the following steps before upgrading the admin cluster:

1. Backup the admin cluster.
2. Disable secretsEncryption in the admin cluster configuration file, and update the cluster using the following command:

   gkectlupdateadmin--configADMIN_CLUSTER_CONFIG_FILE--kubeconfigKUBECONFIG

3. Upgrade the admin cluster.
4. Renable always-on secrets encryption.

Disk errors and attach failures when using Changed Block Tracking (CBT)

Google Distributed Cloud does not support Changed Block Tracking (CBT) on disks. Some backup software uses the CBT feature to track disk state and perform backups, which causes the disk to be unable to connect to a VM that runs Google Distributed Cloud. For more information, see the VMware KB article.

Workaround:

Don't back up the Google Distributed Cloud VMs, as 3rd party backup software might cause CBT to be enabled on their disks. It's not necessary to back up these VMs.

Don't enable CBT on the node, as this change won't persist across updates or upgrades.

If you already have disks with CBT enabled, follow the Resolution steps in the VMware KB articleto disable CBT on the First Class Disk.

Data corruption on NFSv3 when parallel appends to a shared file are done from multiple hosts

If you use Nutanix storage arrays to provide NFSv3 shares to your hosts, you might experience data corruption or the inability for Pods to run successfully. This issue is caused by a known compatibility issue between certain versions of VMware and Nutanix versions. For more information, see the associated VMware KB article.

Workaround:

The VMware KB article is out of date in noting that there is no current resolution. To resolve this issue, update to the latest version of ESXi on your hosts and to the latest Nutanix version on your storage arrays.

Version mismatch between the kubelet and the Kubernetes control plane

For certain Google Distributed Cloud releases, the kubelet running on the nodes uses a different version than the Kubernetes control plane. There is a mismatch because the kubelet binary preloaded on the OS image is using a different version.

The following table lists the identified version mismatches:

Workaround:

No action is needed. The inconsistency is only between Kubernetes patch versions and no problems have been caused by this version skew.

Upgrading or updating an admin cluster with a CA version greater than 1 fails

When an admin cluster has a certificate authority (CA) version greater than 1, an update or upgrade fails due to the CA version validation in the webhook. The output of gkectl upgrade/update contains the following error message:

CAVersionmuststartfrom1

Workaround:

• Scale down the auto-resize-controller deployment in the admin cluster to disable node auto-resizing. This is necessary because a new field introduced to the admin cluster Custom Resource in 1.15 can cause a nil pointer error in the auto-resize-controller.

  kubectlscaledeploymentauto-resize-controller-nkube-system--replicas=0--kubeconfigKUBECONFIG

• Run gkectl commands with --disable-admin-cluster-webhook flag.For example:

  gkectlupgradeadmin--configADMIN_CLUSTER_CONFIG_FILE--kubeconfigKUBECONFIG--disable-admin-cluster-webhook 

Non-HA Controlplane V2 cluster deletion stuck until timeout

When a non-HA Controlplane V2 cluster is deleted, it is stuck at node deletion until it timesout.

Workaround:

If the cluster contains a StatefulSet with critical data, contact contact Cloud Customer Care to resolve this issue.

Otherwise, do the following steps:

• Delete all cluster VMs from vSphere. You can delete the VMs through the vSphere UI, or run the following command:

  govcvm.destroy

  .
• Force delete the cluster again:

  gkectldeletecluster--clusterUSER_CLUSTER_NAME--kubeconfigADMIN_KUBECONFIG--force 

Constant CNS attachvolume tasks appear every minute for in-tree PVC/PV after upgrading to version 1.15+

When a cluster contains in-tree vSphere persistent volumes (for example, PVCs created with the standard StorageClass), you will observe com.vmware.cns.tasks.attachvolume tasks triggered every minute from vCenter.

Workaround:

Edit the vSphere CSI feature configMap and set list-volumes to false:

kubectleditconfigmapinternal-feature-states.csi.vsphere.vmware.com-nkube-system--kubeconfigKUBECONFIG

Restart the vSphere CSI controller pods:

kubectlrolloutrestartdeploymentvsphere-csi-controller-nkube-system--kubeconfigKUBECONFIG

False warnings agaisnt PVCs

When a cluster contains intree vSphere persistent volumes, the commands gkectl diagnose and gkectl upgrade might raise false warnings against their persistent volume claims (PVCs) when validating the cluster storage settings. The warning message looks like the following

CSIPrerequisitespvc/pvc-name:PersistentVolumeClaimpvc-nameboundstoanin-treevSpherevolumecreatedbeforeCSImigrationenabled,butitdoesn'thavetheannotationpv.kubernetes.io/migrated-tosettocsi.vsphere.vmware.comafterCSImigrationisenabled 

Workaround:

Run the following command to check the annotations of a PVC with the above warning:

kubectlgetpvcPVC_NAME-nPVC_NAMESPACE-oyaml--kubeconfigKUBECONFIG

If the annotations field in the output contains the following, you can safely ignore the warning:

pv.kubernetes.io/bind-completed:"yes"pv.kubernetes.io/bound-by-controller:"yes"volume.beta.kubernetes.io/storage-provisioner:csi.vsphere.vmware.com 

Service account key rotation fails when multiple keys are expired

If your cluster is not using a private registry, and your component access service account key and Logging-monitoring (or Connect-register) service account keys are expired, when you rotate the service account keys, gkectl update credentials fails with an error similar to the following:

Error:reconciliationfailed:failedtoupdateplatform:...

Workaround:

First, rotate the component access service account key. Although the same error message is displayed, you should be able to rotate the other keys after the component access service account key rotation.

If the update is still not successful, contact Cloud Customer Care to resolve this issue.

1.15 User master machine encounters an unexpected recreation when the user cluster controller is upgraded to 1.16

During a user cluster upgrade, after the user cluster controller is upgraded to 1.16, if you have other 1.15 user clusters managed by the same admin cluster, their user master machine might be unexpectedly recreated.

There is a bug in the 1.16 user cluster controller which can trigger the 1.15 user master machine recreation.

The workaround that you do depends on how you encounter this issue.

Workaround when upgrading the user cluster using the Google Cloud console:

Option 1: Use a 1.16.6+ version of GKE on VMware with the fix.

Option 2: Do the following steps:

1. Manually add the rerun annotation by the following command:

   kubectleditonpremuserclustersUSER_CLUSTER_NAME-nUSER_CLUSTER_NAME-gke-onprem-mgmt--kubeconfigADMIN_KUBECONFIG

   The rerun annotation is:

   onprem.cluster.gke.io/server-side-preflight-rerun:true

2. Monitor the upgrade progress by checking the status field of the OnPremUserCluster.

Workaround when upgrading the user cluster using your own admin workstation:

Option 1: Use a 1.16.6+ version of GKE on VMware with the fix.

Option 2: Do the following steps:

1. Add the build info file /etc/cloud/build.info with the following content. This causes the preflight checks to run locally on your admin workstation rather than on the server.

   gke_on_prem_version:GKE_ON_PREM_VERSION

   For example:

   gke_on_prem_version:1.16.0-gke.669

2. Rerun the upgrade command.
3. After the upgrade completes, delete the build.info file.

Preflight check fails when the hostname isn't in the IP block file.

During cluster creation, if you don't specify a hostname for every IP address in the IP block file, the preflight check fails with the following error message:

multipleVMsfoundbyDNSnameinxxxdatacenter.AnthosOnpremdoesn'tsupportduplicatehostnameinthesamevCenterandyoumaywanttorename/deletetheexistingVM.`

There is a bug in the preflight check which assumes empty hostname as duplicate.

Workaround:

Option 1: Use a version with the fix.

Option 2: Bypass this preflight check by adding --skip-validation-net-config flag.

Option 3: Specify a unique hostname for each IP address in IP block file.

Volume mount failure when upgrade/update the admin cluster if using non-HA admin cluster and control plane v1 user cluster

For a non-HA admin cluster and a control plane v1 user cluster, when you upgrade or update the admin cluster, the admin cluster master machine recreation might happen at the same time as the user cluster master machine reboot, which can surface a race condition. This causes the user cluster control plane Pods to be unable to communicate to the admin cluster control plane, which causes volume attach issues for kube-etcd and kube-apiserver on the user cluster control plane.

To verify the issue, run the following commands for the impacted pod:

kubectl--kubeconfigADMIN_CLUSTER_KUBECONFIG--namespaceUSER_CLUSTER_NAMEdescribepodIMPACTED_POD_NAME

 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 101s kubelet Unable to attach or mount volumes: unmounted volumes=[kube-audit], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition Warning FailedMount 86s (x2 over 3m28s) kubelet MountVolume.SetUp failed for volume "pvc-77cd0635-57c2-4392-b191-463a45c503cb" : rpc error: code = FailedPrecondition desc = volume ID: "bd313c62-d29b-4ecd-aeda-216648b0f7dc" does not appear staged to "/var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/92435c96eca83817e70ceb8ab994707257059734826fedf0c0228db6a1929024/globalmount" 

Workaround:

1. SSH into user control plane node, since it is control plane v1 user cluster, the user control plane node is in admin cluster.
2. Restart the kubelet using the following command:

   sudosystemctlrestartkubelet 

   After restart, the kubelet can reconstruct stage global mount properly.

Control plane node fails to be created

During an upgrade or update of an admin cluster, a race condition might cause the vSphere cloud controller manager to unexpectedly delete a new control plane node. This causes the clusterapi-controller to be stuck waiting for the node to be created, and evenutally the upgrade/update times out. In this case, the output of the gkectl upgrade/update command is similar to the following:

controlplane'default/gke-admin-hfzdg'isnotready:condition"Ready":conditionisnotreadywithreason"MachineInitializing",message"Wait for the control plane machine "gke-admin-hfzdg-6598459f9zb647c8-0\"toberebooted"...

To identify the symptom, run the command below to get log in vSphere cloud controller manager in the admin cluster:

kubectlgetpods--kubeconfigADMIN_KUBECONFIG-nkube-system|grepvsphere-cloud-controller-manager kubectllogs-fvsphere-cloud-controller-manager-POD_NAME_SUFFIX--kubeconfigADMIN_KUBECONFIG-nkube-system 

Here is a sample error message from the above command:

nodename:81ff17e25ec6-qual-335-1500f723hasadifferentuuid.Skipdeletingthisnodefromcache. 

Workaround:

1. Reboot the failed machine to recreate the deleted node object.
2. SSH into each control plane node and restart the vSphere cloud controller manager static pod:

   sudocrictlps|grepvsphere-cloud-controller-manager|awk'{print $1}'sudocrictlstopPREVIOUS_COMMAND_OUTPUT

3. Rerun upgrade/update command.

Duplicate hostname in the same data center causes cluster upgrade or creation failures

Upgrading a 1.15 cluster or creating a 1.16 cluster with static IPs fails if there are duplicate hostnames in the same data center. This failure happens because the vSphere cloud controller manager fails to add an external IP and provider ID in the node object. This causes the cluster upgrade/create to timeout.

To identify the issue, get the vSphere cloud controller manager pod logs for the cluster. The command that you use depends on the cluster type, as follows:

• Admin cluster:

  kubectlgetpods--kubeconfigADMIN_KUBECONFIG-nkube-system|grepvsphere-cloud-controller-manager kubectllogs-fvsphere-cloud-controller-manager-POD_NAME_SUFFIX--kubeconfigADMIN_KUBECONFIG-nkube-system 

• User cluster (kubeception):

  kubectlgetpods--kubeconfigADMIN_KUBECONFIG-nUSER_CLUSTER_NAME|grepvsphere-cloud-controller-manager kubectllogs-fvsphere-cloud-controller-manager-POD_NAME_SUFFIX--kubeconfigADMIN_KUBECONFIG-nUSER_CLUSTER_NAME

• User cluster: (Controlplane V2):

  kubectlgetpods--kubeconfigUSER_KUBECONFIG-nkube-system|grepvsphere-cloud-controller-manager kubectllogs-fvsphere-cloud-controller-manager-POD_NAME_SUFFIX--kubeconfigUSER_KUBECONFIG-nkube-system 

Here is a sample error message:

I100317:17:46.7696761search.go:152]Findingnodeadmin-vm-2invc=vcsa-53598.e5c235a1.asia-northeast1.gve.googanddatacenter=Datacenter E100317:17:46.7717171datacenter.go:111]MultiplevmsfoundVMbyDNSName.DNSName:admin-vm-2 

Check if the hostname is duplicated in the data center:

exportGOVC_DATACENTER=GOVC_DATACENTERexportGOVC_URL=GOVC_URLexportGOVC_USERNAME=GOVC_USERNAMEexportGOVC_PASSWORD=GOVC_PASSWORDexportGOVC_INSECURE=truegovcfind.-typem-guest.hostNameHOSTNAME

exportGOVC_DATACENTER=mtv-lifecycle-vc01 exportGOVC_URL=https://mtv-lifecycle-vc01.anthos/sdk exportGOVC_USERNAME=xxx exportGOVC_PASSWORD=yyy exportGOVC_INSECURE=truegovcfind.-typem-guest.hostNamef8c3cd333432-lifecycle-337-xxxxxxxz ./vm/gke-admin-node-6b7788cd76-wkt8g ./vm/gke-admin-node-6b7788cd76-99sg2 ./vm/gke-admin-master-5m2jb 

The workaround that you do depends on the operation that failed.

Workaround for upgrades:

Do the workaround for the applicable cluster type.

• User cluster:
  1. Update the hostname of the affected machine in user-ip-block.yaml to a unique name and trigger a forced update:
     

     gkectlupdatecluster--kubeconfigADMIN_KUBECONFIG--configNEW_USER_CLUSTER_CONFIG--force 

  2. Rerun gkectl upgrade cluster
• Admin cluster:
  1. Update the hostname of the affected machine in admin-ip-block.yaml to a unique name and trigger a forced update:
     

     gkectlupdateadmin--kubeconfigADMIN_KUBECONFIG--configNEW_ADMIN_CLUSTER_CONFIG--force--skip-cluster-ready-check 

  2. If it is a non-HA admin cluster, and you find admin master vm is using duplicate hostname, you also need to:
     Get admin master machine name

     kubectlgetmachine--kubeconfigADMIN_KUBECONFIG-owide-A 

     Update admin master machine object
     Note: The NEW_ADMIN_MASTER_HOSTNAME should be same to what you set in admin-ip-block.yaml in step 1.
     

     kubectlpatchmachineADMIN_MASTER_MACHINE_NAME--kubeconfigADMIN_KUBECONFIG--type='json'-p'[{"op": "replace", "path": "/spec/providerSpec/value/networkSpec/address/hostname", "value":"NEW_ADMIN_MASTER_HOSTNAME"}]'

     Verify hostname is updated in admin master machine object:
     

     kubectlgetmachineADMIN_MASTER_MACHINE_NAME--kubeconfigADMIN_KUBECONFIG-oyaml kubectlgetmachineADMIN_MASTER_MACHINE_NAME--kubeconfigADMIN_KUBECONFIG-ojsonpath='{.spec.providerSpec.value.networkSpec.address.hostname}'

  3. Rerun admin cluster upgrade with checkpoint disabled:
     

     gkectlupgradeadmin--kubeconfigADMIN_KUBECONFIG--configADMIN_CLUSTER_CONFIG--disable-upgrade-from-checkpoint 

Workaround for installations:

Do the workaround for the applicable cluster type.

• Admin cluster:
  1. Delete the admin node machine.
  2. Delete the data disk.
  3. Delete the admin cluster checkpoint file.
  4. Update the hostname of the affected machine in admin-ip-block.yaml to a unique name.
  5. Rerun gkectl create admin.
• User cluster:
  1. Clean up resources.
  2. Update the hostname of the affected machine in user-ip-block.yaml to a unique name.
  3. Rerun gkectl create cluster.

$ and ` are not supported in vSphere username or password

The following operations fail when the vSphere username or password contains $ or `:

• Upgrading a 1.15 user cluster with Controlplane V2 enabled to 1.16
• Upgrading a 1.15 high-availability (HA) admin cluster to 1.16
• Creating a 1.16 user cluster with Controlplane V2 enabled
• Creating a 1.16 HA admin cluster

Use a 1.16.4+ version of Google Distributed Cloud with the fix or perform the below workaround. The workaround that you do depends on the operation that failed.

Workaround for upgrades:

1. Change the vCenter username or password on the vCenter side to remove the $ and `.
2. Update the vCenter username or password in your credentials configuration file.
3. Trigger a forced update of the cluster.
• User cluster:

  gkectlupdatecluster--kubeconfigADMIN_KUBECONFIG--configUSER_CLUSTER_CONFIG--force 

• Admin cluster:

  gkectlupdateadmin--kubeconfigADMIN_KUBECONFIG--configADMIN_CLUSTER_CONFIG--force--skip-cluster-ready-check 

Workaround for installations:

1. Change the vCenter username or password on the vCenter side to remove the $ and `.
2. Update the vCenter username or password in your credentials configuration file.
3. Do the workaround for the applicable cluster type.
• Admin cluster:
  1. Delete the admin node machine.
  2. Delete the data disk.
  3. Delete the admin cluster checkpoint file.
  4. Rerun gkectl create admin.
• User cluster:
  1. Clean up resources.
  2. Rerun gkectl create cluster.

PVC creation failure after node is recreated with the same name

After a node is deleted and then recreated with the same node name, there is a slight chance that a subsequent PersistentVolumeClaim (PVC) creation fails with an error like the following:

Theobject'vim.VirtualMachine:vm-988369'hasalreadybeendeletedorhasnotbeencompletelycreated

This is caused by race condition where vSphere CSI controller does not delete a removed machine from its cache.

Workaround:

Restart the vSphere CSI controller pods:

kubectlrolloutrestartdeploymentvsphere-csi-controller-nkube-system--kubeconfigKUBECONFIG

gkectl repair admin-master returns kubeconfig unmarshall error

When you run the gkectl repair admin-master command on an HA admin cluster, gkectl returns the following error message:

Exitwitherror:Failedtorepair:failedtoselectthetemplate:failedtogetclusternamefromkubeconfig,pleasecontactGooglesupport.failedtodecodekubeconfigdata:yaml:unmarshalerrors: line3:cannotunmarshal!!seqintomap[string]*api.Cluster line8:cannotunmarshal!!seqintomap[string]*api.Context 

Workaround:

Add the --admin-master-vm-template= flag to the command and provide the VM template of the machine to repair:

gkectlrepairadmin-master--kubeconfig=ADMIN_CLUSTER_KUBECONFIG\--configADMIN_CLUSTER_CONFIG_FILE\--admin-master-vm-template=/DATA_CENTER/vm/VM_TEMPLATE_NAME

To find the VM template of the machine:

1. Go to the Hosts and Clusters page in the vSphere client.
2. Click VM Templates and filter by the admin cluster name.

   You should see the three VM templates for the admin cluster.

3. Copy the name VM template that matches the name of the machine you're repairing and use the template name in the repair command.

gkectlrepairadmin-master\--config=/home/ubuntu/admin-cluster.yaml\--kubeconfig=/home/ubuntu/kubeconfig\--admin-master-vm-template=/atl-qual-vc07/vm/gke-admin-98g94-zx...7vx-0-tmpl

Seesaw VM broken due to disk space low

If you use Seesaw as the load balancer type for your cluster and you see that a Seesaw VM is down or keeps failing to boot, you might see the following error message in the vSphere console:

GRUB_FORCE_PARTUUIDset,initrdlessbootfailed.Attemptingwithinitrd 

This error indicates that the disk space is low on the VM because the fluent-bit running on the Seesaw VM is not configured with correct log rotation.

Workaround:

Locate the log files that consume most of the disk space using du -sh -- /var/lib/docker/containers/* | sort -rh. Clean up the log file with largest size and reboot the VM.

Note: If the VM is completely inaccessible, attach the disk to a working VM (e.g. admin workstation), remove the file from the attached disk, then reattach the disk back to the original Seesaw VM.

To prevent the issue from happening again, connect to the VM and modify the /etc/systemd/system/docker.fluent-bit.service file. Add --log-opt max-size=10m --log-opt max-file=5 in the Docker command, then run systemctl restart docker.fluent-bit.service

Admin SSH public key error after admin cluster upgrade or update

When you try to upgrade (gkectl upgrade admin) or update (gkectl update admin) a non-High-Availability admin cluster with checkpoint enabled, the upgrade or update may fail with errors like the following:

Checkingadminclustercertificates...FAILURE Reason:20adminclustercertificateserror(s). UnhealthyResources: AdminMasterclusterCAbundle:failedtogetclusterCAbundleonadminmaster,command[ssh-oIdentitiesOnly=yes-iadmin-ssh-key-oStrictHostKeyChecking=no-oConnectTimeout=30ubuntu@AdminMasterIP--sudocat/etc/kubernetes/pki/ca-bundle.crt]failedwitherror:exitstatus255,stderr:Authorizedusesonly.Allactivitymaybemonitoredandreported. ubuntu@AdminMasterIP:Permissiondenied(publickey).

failedtosshAdminMasterIP,failedwitherror:exitstatus255,stderr:Authorizedusesonly.Allactivitymaybemonitoredandreported. ubuntu@AdminMasterIP:Permissiondenied(publickey)

errordialingubuntu@AdminMasterIP:failedtoestablishanauthenticatedSSHconnection:ssh:handshakefailed:ssh:unabletoauthenticate,attemptedmethods[nonepublickey]...

Workaround:

If you're unable to upgrade to a patch version of Google Distributed Cloud with the fix, contact Google Support for assistance.

Upgrading an admin cluster enrolled in the GKE On-Prem API could fail

When an admin cluster is enrolled in the GKE On-Prem API, upgrading the admin cluster to the affected versions could fail because the fleet membership couldn't be updated. When this failure happens, you see the following error when trying to upgrade the cluster:

failedtoregistercluster:failedtoapplyHubMembership:MembershipAPIrequestfailed:rpcerror:code=InvalidArgumentdesc=InvalidFieldErrorforfieldendpoint.on_prem_cluster.resource_link:fieldcannotbeupdated 

An admin cluster is enrolled in the API when you explicitly enroll the cluster, or when you upgrade a user cluster using a GKE On-Prem API client.

Workaround:

gcloudalphacontainervmwareadmin-clustersunenrollADMIN_CLUSTER_NAME--projectCLUSTER_PROJECT--location=CLUSTER_LOCATION--allow-missing 

Enrolled admin cluster's resource link annotation is not preserved

When an admin cluster is enrolled in the GKE On-Prem API, its resource link annotation is applied to the OnPremAdminCluster custom resource, which is not preserved during later admin cluster updates due to the wrong annotation key being used. This can cause the admin cluster to be enrolled in the GKE On-Prem API again by mistake.

An admin cluster is enrolled in the API when you explicitly enroll the cluster, or when you upgrade a user cluster using a GKE On-Prem API client.

Workaround:

gcloudalphacontainervmwareadmin-clustersunenrollADMIN_CLUSTER_NAME--projectCLUSTER_PROJECT--location=CLUSTER_LOCATION--allow-missing 

CoreDNS orderPolicy not recognized

OrderPolicy doesn't get recognized as a parameter and isn't used. Instead, Google Distributed Cloud always uses Random.

This issue occurs because the CoreDNS template was not updated, which causes orderPolicy to be ignored.

Workaround:

Update the CoreDNS template and apply the fix. This fix persists until an upgrade.

1. Edit the existing template:

   kubectleditcm-nkube-systemcoredns-template

   Replace the contents of the template with the following:

   coredns-template:|- .:53{errors health{lameduck5s }ready kubernetescluster.localin-addr.arpaip6.arpa{podsinsecure fallthroughin-addr.arpaip6.arpa } {{-if.PrivateGoogleAccess}}importzones/private.Corefile {{-end}} {{-if.RestrictedGoogleAccess}}importzones/restricted.Corefile {{-end}}prometheus:9153 forward.{{.UpstreamNameservers}}{max_concurrent1000{{-ifne.OrderPolicy""}}policy{{.OrderPolicy}}{{-end}}}cache30 {{-if.DefaultDomainQueryLogging}}log {{-end}}loop reload loadbalance }{{range$i,$stubdomain:=.StubDomains}} {{$stubdomain.Domain}}:53{errors {{-if$stubdomain.QueryLogging}}log {{-end}}cache30forward.{{$stubdomain.Nameservers}}{max_concurrent1000{{-ifne$.OrderPolicy""}}policy{{$.OrderPolicy}}{{-end}}}} {{-end}}

OnPremAdminCluster status inconsistent between checkpoint and actual CR

Certain race conditions could cause the OnPremAdminCluster status to be inconsistent between checkpoint and actual CR. When the issue happens, you could encounter the following error when update the admin cluster after you upgraded it:

Exitwitherror: E032110:20:53.515562961695console.go:93]Failedtoupdatetheadmincluster:OnPremAdminCluster"gke-admin-rj8jr"isinthemiddleofacreate/upgrade(""->"1.15.0-gke.123"),whichmustbecompletedbeforeitcanbeupdated Failedtoupdatetheadmincluster:OnPremAdminCluster"gke-admin-rj8jr"isinthemiddleofacreate/upgrade(""->"1.15.0-gke.123"),whichmustbecompletedbeforeitcanbeupdated

Reconciliation process changes admin certificates on admin clusters

Google Distributed Cloud changes the admin certificates on admin cluster control planes with every reconciliation process, such as during a cluster upgrade. This behavior increases the possibility of getting invalid certificates for your admin cluster, especially for version 1.15 clusters.

If you're affected by this issue, you may encounter problems like the following:

• Invalid certificates may cause the following commands to time out and return errors:
  • gkectl create admin
  • gkectl upgrade amdin
  • gkectl update admin

  These commands may return authorization errors like the following:

  Failedtoreconcileadmincluster:unabletopopulateadminclients:failedtogetadmincontrollerruntimeclient:Unauthorized

• The kube-apiserver logs for your admin cluster may contain errors like the following:

  Unabletoauthenticatetherequest" err="[x509:certificatehasexpiredorisnotyetvalid...

Workaround:

Upgrade to a version of Google Distributed Cloud with the fix: 1.13.10+, 1.14.6+, 1.15.2+. If upgrading isn't feasible for you, contact Cloud Customer Care to resolve this issue.

Anthos Network Gateway components evicted or pending due to missing priority class

Network gateway Pods in kube-system might show a status of Pending or Evicted, as shown in the following condensed example output:

$kubectl-nkube-systemgetpods|grepang-node ang-node-bjkkc2/2Running05d2h ang-node-mw8cq0/2Evicted06m5s ang-node-zsmq70/2Pending07h

These errors indicate eviction events or an inability to schedule Pods due to node resources. As Anthos Network Gateway Pods have no PriorityClass, they have the same default priority as other workloads. When nodes are resource-constrained, the network gateway Pods might be evicted. This behavior is particularly bad for the ang-node DaemonSet, as those Pods must be scheduled on a specific node and can't migrate.

Workaround:

Upgrade to 1.15 or later.

As a short-term fix, you can manually assign a PriorityClass to the Anthos Network Gateway components. The Google Distributed Cloud controller overwrites these manual changes during a reconciliation process, such as during a cluster upgrade.

• Assign the system-cluster-critical PriorityClass to the ang-controller-manager and autoscaler cluster controller Deployments.
• Assign the system-node-critical PriorityClass to the ang-daemon node DaemonSet.

admin cluster upgrade fails after registering the cluster with gcloud

After you use gcloud to register an admin cluster with non-empty gkeConnect section, you might see the following error when trying to upgrade the cluster:

failedtoregistercluster:failedtoapplyHubMem\ bership:MembershipAPIrequestfailed:rpcerror:code=InvalidArgumentdesc=InvalidFieldErrorforfieldendpoint.o\ n_prem_cluster.admin_cluster:fieldcannotbeupdated

Delete the gke-connect namespace:

kubectldeletensgke-connect--kubeconfig=ADMIN_KUBECONFIG

kubectlgetonpremadmincluster-nkube-system--kubeconfig=ADMIN_KUBECONFIG

gcloudcontainerfleetmembershipsdeleteADMIN_CLUSTER_NAME

gkectl diagnose snapshot --log-since fails to limit the time window for journalctl commands running on the cluster nodes

This does not affect the functionality of taking a snapshot of the cluster, as the snapshot still includes all logs that are collected by default by running journalctl on the cluster nodes. Therefore, no debugging information is missed.

gkectl prepare windows fails

gkectl prepare windows fails to install Docker on Google Distributed Cloud versions earlier than 1.13 because MicrosoftDockerProvider is deprecated.

Workaround:

The general idea to workaround this issue is to upgrade to Google Distributed Cloud 1.13 and use the 1.13 gkectl to create a Windows VM template and then create Windows node pools. There are two options to get to Google Distributed Cloud 1.13 from your current version as shown below.

Note: We do have options to workaround this issue in your current version without needing to upgrade all the way to 1.13, but it will need more manual steps, please reach out to our support team if you would like to consider this option.

Option 1: Blue/Green upgrade

You can create a new cluster using Google Distributed Cloud 1.13+ version with windows node pools, and migrate your workloads to the new cluster, then tear down the current cluster. It's recommended to use the latest Google Distributed Cloud minor version.

Note: This will require extra resources to provision the new cluster, but less downtime and disruption for existing workloads.

Option 2: Delete Windows node pools and add them back when upgrading to Google Distributed Cloud 1.13

Note: For this option, the Windows workloads will not be able to run until the cluster is upgraded to 1.13 and Windows node pools are added back.

1. Delete existing Windows node pools by removing the windows node pools config from user-cluster.yaml file, then run the command:

   gkectlupdatecluster--kubeconfig=ADMIN_KUBECONFIG--configUSER_CLUSTER_CONFIG_FILE

2. Upgrade the Linux-only admin+user clusters to 1.12 following the upgrade user guide for the corresponding target minor version.
3. (Make sure to perform this step before upgrading to 1.13) Ensure the enableWindowsDataplaneV2: true is configured in OnPremUserCluster CR, otherwise the cluster will keep using Docker for Windows node pools, which will not be compatible with the newly created 1.13 Windows VM template that not have Docker installed. If not configured or setting to false, update your cluster to set it to true in user-cluster.yaml, then run:

   gkectlupdatecluster--kubeconfig=ADMIN_KUBECONFIG--configUSER_CLUSTER_CONFIG_FILE

4. Upgrade the Linux-only admin+user clusters to 1.13 following the upgrade user guide.
5. Prepare Windows VM template using 1.13 gkectl:

   gkectlpreparewindows--base-vm-templateBASE_WINDOWS_VM_TEMPLATE_NAME--bundle-path1.13_BUNDLE_PATH--kubeconfig=ADMIN_KUBECONFIG

6. Add back the Windows node pool configuration to user-cluster.yaml with the OSImage field set to the newly created Windows VM template.
7. Update the cluster to add Windows node pools

   gkectlupdatecluster--kubeconfig=ADMIN_KUBECONFIG--configUSER_CLUSTER_CONFIG_FILE

RootDistanceMaxSec configuration not taking effect for ubuntu nodes

The 5 seconds default value for RootDistanceMaxSec will be used on the nodes, instead of 20 seconds which should be the expected configuration. If you check the node startup log by SSH'ing into the VM, which is located at `/var/log/startup.log`, you can find the following error:

+has_systemd_unitsystemd-timesyncd /opt/bin/master.sh:line635:has_systemd_unit:commandnotfound

Using a 5 seconds RootDistanceMaxSec might cause the system clock to be out of sync with NTP server when the clock drift is larger than 5 seconds.

Workaround:

Apply the following DaemonSet to your cluster to configure RootDistanceMaxSec:

apiVersion:apps/v1kind:DaemonSetmetadata:name:change-root-distancenamespace:kube-systemspec:selector:matchLabels:app:change-root-distancetemplate:metadata:labels:app:change-root-distancespec:hostIPC:truehostPID:truetolerations:# Make sure pods gets scheduled on all nodes.-effect:NoScheduleoperator:Exists-effect:NoExecuteoperator:Existscontainers:-name:change-root-distanceimage:ubuntucommand:["chroot","/host","bash","-c"]args:-|while true; doconf_file="/etc/systemd/timesyncd.conf.d/90-gke.conf"if [ -f $conf_file ] && $(grep -q "RootDistanceMaxSec=20" $conf_file); thenecho "timesyncd has the expected RootDistanceMaxSec, skip update"elseecho "updating timesyncd config to RootDistanceMaxSec=20"mkdir -p /etc/systemd/timesyncd.conf.dcat > $conf_file << EOF[Time]RootDistanceMaxSec=20EOFsystemctl restart systemd-timesyncdfisleep 600donevolumeMounts:-name:hostmountPath:/hostsecurityContext:privileged:truevolumes:-name:hosthostPath:path:/

gkectl update admin fails because of empty osImageType field

When you use version 1.13 gkectl to update a version 1.12 admin cluster, you might see the following error:

Failedtoupdatetheadmincluster:updatingOSimagetypeinadmincluster isnotsupportedin"1.12.x-gke.x"

When you use gkectl update admin for version 1.13 or 1.14 clusters, you might see the following message in the response:

Exitwitherror: Failedtoupdatethecluster:theupdatecontainsmultiplechanges.Please updateonlyonefeatureatatime

If you check the gkectl log, you might see that the multiple changes include setting osImageType from an empty string to ubuntu_containerd.

These update errors are due to improper backfilling of the osImageType field in the admin cluster config since it was introduced in version 1.9.

Workaround:

Upgrade to a version of Google Distributed Cloud with the fix. If upgrading isn't feasible for you, contact Cloud Customer Care to resolve this issue.

SNI doesn't work on user clusters with Controlplane V2

The ability to provide an additional serving certificate for the Kubernetes API server of a user cluster with authentication.sni doesn't work when the Controlplane V2 is enabled (enableControlplaneV2: true).

Workaround:

Until a Google Distributed Cloud patch is available with the fix, if you need to use SNI, disable Controlplane V2 (enableControlplaneV2: false).

$ in the private registry username causes admin control plane machine startup failure

The admin control plane machine fails to start up when the private registry username contains $. When checking the /var/log/startup.log on the admin control plane machine, you see the following error:

++REGISTRY_CA_CERT=xxx ++REGISTRY_SERVER=xxx /etc/startup/startup.conf:line7:anthos:unboundvariable

Workaround:

Use a private registry username without $, or use a version of Google Distributed Cloud with the fix.

False-positive warnings about unsupported changes during admin cluster update

When you update admin clusters, you will see the following false-positive warnings in the log, and you can ignore them.

console.go:47]detectedunsupportedchanges:&v1alpha1.OnPremAdminCluster{... -CARotation:&v1alpha1.CARotationConfig{Generated:&v1alpha1.CARotationGenerated{CAVersion:1}}, +CARotation:nil, ... }

Update user cluster failed after KSA signing key rotation

After you rotate KSA signing keys and subsequently update a user cluster, gkectl update might fail with the following error message:

FailedtoapplyOnPremUserCluster'USER_CLUSTER_NAME-gke-onprem-mgmt/USER_CLUSTER_NAME': admissionwebhook"vonpremusercluster.onprem.cluster.gke.io"deniedtherequest: requestsmustnotdecrement*v1alpha1.KSASigningKeyRotationConfigVersion,oldversion:2,newversion:1"

Workaround:

1. Check the secret in admin cluster under USER_CLUSTER_NAME namespace, and get the name of ksa-signing-key secret:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=USER_CLUSTER_NAMEgetsecrets|grepksa-signing-key

2. Copy the ksa-signing-key secret, and name the copied secret as service-account-cert:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=USER_CLUSTER_NAMEgetsecretKSA-KEY-SECRET-NAME-oyaml|\ sed's/ name: .*/ name: service-account-cert/'|\ kubectl--kubeconfig=ADMIN_KUBECONFIG-n=USER_CLUSTER_NAMEapply-f-

3. Delete the previous ksa-signing-key secret:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=USER_CLUSTER_NAMEdeletesecretKSA-KEY-SECRET-NAME

4. Update the data.data field in ksa-signing-key-rotation-stage configmap to '{"tokenVersion":1,"privateKeyVersion":1,"publicKeyVersions":[1]}':

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=USER_CLUSTER_NAME\ editconfigmapksa-signing-key-rotation-stage

5. Disable the validation webhook to edit the version information in the OnPremUserCluster custom resource:

   kubectl--kubeconfig=ADMIN_KUBECONFIGpatchvalidatingwebhookconfigurationonprem-user-cluster-controller-p'webhooks:- name: vonpremnodepool.onprem.cluster.gke.io rules: - apiGroups: - onprem.cluster.gke.io apiVersions: - v1alpha1 operations: - CREATE resources: - onpremnodepools- name: vonpremusercluster.onprem.cluster.gke.io rules: - apiGroups: - onprem.cluster.gke.io apiVersions: - v1alpha1 operations: - CREATE resources: - onpremuserclusters'

6. Update the spec.ksaSigningKeyRotation.generated.ksaSigningKeyRotation field to 1 in your OnPremUserCluster custom resource:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=USER_CLUSTER_NAME-gke-onprem-mgmt\ editonpremuserclusterUSER_CLUSTER_NAME

7. Wait until the target user cluster to be ready, you can check the status by:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=USER_CLUSTER_NAME-gke-onprem-mgmt\ getonpremusercluster

8. Restore the validation webhook for the user cluster:

   kubectl--kubeconfig=ADMIN_KUBECONFIGpatchvalidatingwebhookconfigurationonprem-user-cluster-controller-p'webhooks:- name: vonpremnodepool.onprem.cluster.gke.io rules: - apiGroups: - onprem.cluster.gke.io apiVersions: - v1alpha1 operations: - CREATE - UPDATE resources: - onpremnodepools- name: vonpremusercluster.onprem.cluster.gke.io rules: - apiGroups: - onprem.cluster.gke.io apiVersions: - v1alpha1 operations: - CREATE - UPDATE resources: - onpremuserclusters'

9. Avoid another KSA signing key rotation until the cluster is upgraded to the version with the fix.

F5 BIG-IP virtual servers aren't cleaned up when Terraform deletes user clusters

When you use Terraform to delete a user cluster with a F5 BIG-IP load balancer, the F5 BIG-IP virtual servers aren't removed after the cluster deletion.

Workaround:

To remove the F5 resources, follow the steps to clean up a user cluster F5 partition

kind cluster pulls container images from docker.io

If you create a version 1.13.8 or version 1.14.4 admin cluster, or upgrade an admin cluster to version 1.13.8 or 1.14.4, the kind cluster pulls the following container images from docker.io:

If docker.io isn't accessible from your admin workstation, the admin cluster creation or upgrade fails to bring up the kind cluster. Running the following command on the admin workstation shows the corresponding containers pending with ErrImagePull:

dockerexecgkectl-control-planekubectlgetpods-A

The response contains entries like the following:

... kube-systemkindnet-xlhmr0/1 ErrImagePull03m12s ... local-path-storagelocal-path-provisioner-86666ffff6-zzqtp0/1 Pending03m12s ...

These container images should be preloaded in the kind cluster container image. However, kind v0.18.0 has an issue with the preloaded container images, which causes them to be pulled from the internet by mistake.

Workaround:

Run the following commands on the admin workstation, while your admin cluster is pending on creation or upgrade:

dockerexecgkectl-control-planectr-nk8s.ioimagestagdocker.io/kindest/kindnetd:v20230330-48f316cd@sha256:c19d6362a6a928139820761475a38c24c0cf84d507b9ddf414a078cf627497afdocker.io/kindest/kindnetd:v20230330-48f316cd dockerexecgkectl-control-planectr-nk8s.ioimagestagdocker.io/kindest/kindnetd:v20230330-48f316cd@sha256:c19d6362a6a928139820761475a38c24c0cf84d507b9ddf414a078cf627497afdocker.io/kindest/kindnetd@sha256:c19d6362a6a928139820761475a38c24c0cf84d507b9ddf414a078cf627497af dockerexecgkectl-control-planectr-nk8s.ioimagestagdocker.io/kindest/local-path-helper:v20230330-48f316cd@sha256:135203f2441f916fb13dad1561d27f60a6f11f50ec288b01a7d2ee9947c36270docker.io/kindest/local-path-helper:v20230330-48f316cd dockerexecgkectl-control-planectr-nk8s.ioimagestagdocker.io/kindest/local-path-helper:v20230330-48f316cd@sha256:135203f2441f916fb13dad1561d27f60a6f11f50ec288b01a7d2ee9947c36270docker.io/kindest/local-path-helper@sha256:135203f2441f916fb13dad1561d27f60a6f11f50ec288b01a7d2ee9947c36270 dockerexecgkectl-control-planectr-nk8s.ioimagestagdocker.io/kindest/local-path-provisioner:v0.0.23-kind.0@sha256:f2d0a02831ff3a03cf51343226670d5060623b43a4cfc4808bd0875b2c4b9501docker.io/kindest/local-path-provisioner:v0.0.23-kind.0 dockerexecgkectl-control-planectr-nk8s.ioimagestagdocker.io/kindest/local-path-provisioner:v0.0.23-kind.0@sha256:f2d0a02831ff3a03cf51343226670d5060623b43a4cfc4808bd0875b2c4b9501docker.io/kindest/local-path-provisioner@sha256:f2d0a02831ff3a03cf51343226670d5060623b43a4cfc4808bd0875b2c4b9501

Unsuccessful failover on HA Controlplane V2 user cluster and admin cluster when the network filters out duplicate GARP requests

If your cluster VMs are connected with a switch that filters out duplicate GARP (gratuitous ARP) requests, the keepalived leader election might encounter a race condition, which causes some nodes to have incorrect ARP table entries.

The affected nodes can ping the control plane VIP, but a TCP connection to the control plane VIP will time out.

Workaround:

iptables-IFORWARD-iens192--destinationCONTROL_PLANE_VIP-jDROP 

vsphere-csi-controller needs be restarted after the vCenter certificate rotation

vsphere-csi-controller should refresh its vCenter secret after vCenter certificate rotation. However, the current system does not properly restart the pods of vsphere-csi-controller, causing vsphere-csi-controller to crash after the rotation.

Workaround:

For clusters created at 1.13 and later versions, follow the instructions below to restart vsphere-csi-controller

kubectl --kubeconfig=ADMIN_KUBECONFIG rollout restart deployment vsphere-csi-controller -n kube-system

Admin cluster creation does not fail on cluster registration errors

Even when cluster registration fails during admin cluster creation, the command gkectl create admin does not fail on the error and might succeed. In other words, the admin cluster creation could "succeed" without being registered to a fleet.

 Failed to register admin cluster

You can also check whether you can find the cluster among registered clusters on cloud console.

Workaround:

For clusters created at 1.12 and later versions, follow the instructions for re-attempting the admin cluster registration after cluster creation. For clusters created at earlier versions,

1. Append a fake key-value pair like "foo: bar" to your connect-register SA key file
2. Run gkectl update admin to re-register the admin cluster.

Admin cluster re-registration might be skipped during admin cluster upgrade

During admin cluster upgrade, if upgrading user control plane nodes times out, the admin cluster will not be re-registered with the updated connect agent version.

Workaround:

1. Append a fake key-value pair like "foo: bar" to your connect-register SA key file
2. Run gkectl update admin to re-register the admin cluster.

False error message about vCenter.dataDisk

For a high-availability admin cluster, gkectl prepare shows this false error message:

 vCenter.dataDisk must be present in the AdminCluster spec

Workaround:

You can safely ignore this error message.

Node pool creation fails because of redundant VM-Host affinity rules

During creation of a node pool that uses VM-Host affinity, a race condition might result in multiple VM-Host affinity rules being created with the same name. This can cause node pool creation to fail.

Workaround:

Remove the old redundant rules so that node pool creation can proceed. These rules are named [USER_CLUSTER_NAME]-[HASH].

gkectl repair admin-master may fail due to failed to delete the admin master node object and reboot the admin master VM

The gkectl repair admin-master command may fail due to a race condition with the following error.

Failedtorepair:failedtodeletetheadminmasternodeobjectandreboottheadminmasterVM

Workaround:

This command is idempotent. It can rerun safely until the command succeeds.

Pods remain in Failed state afer re-creation or update of a control-plane node

After you re-create or update a control-plane node, certain Pods might be left in the Failed state due to NodeAffinity predicate failure. These failed Pods don't affect normal cluster operations or health.

Workaround:

You can safely ignore the failed Pods or manually delete them.

OnPremUserCluster not ready because of private registry credentials

If you use prepared credentials and a private registry, but you haven't configured prepared credentials for your private registry, the OnPremUserCluster might not become ready, and you might see the following error message:

 failed to check secret reference for private registry …

Workaround:

Prepare the private registry credentials for the user cluster according to the instructions in Configure prepared credentials.

gkectl upgrade admin fails with StorageClass standard sets the parameter diskformat which is invalid for CSI Migration

During gkectl upgrade admin, the storage preflight check for CSI Migration verifies that the StorageClasses don't have parameters that are ignored after CSI Migration. For example, if there's a StorageClass with the parameter diskformat then gkectl upgrade admin flags the StorageClass and reports a failure in the preflight validation. Admin clusters created in Google Distributed Cloud 1.10 and before have a StorageClass with diskformat: thin which will fail this validation however this StorageClass still works fine after CSI Migration. These failures should be interpreted as warnings instead.

For more information, check the StorageClass parameter section in Migrating In-Tree vSphere Volumes to vSphere Container Storage Plug-in.

Workaround:

After confirming that your cluster has a StorageClass with parameters ignored after CSI Migration run gkectl upgrade admin with the flag --skip-validation-cluster-health.

Migrated in-tree vSphere volumes using the Windows file system can't be used with vSphere CSI driver

Under certain conditions disks can be attached as readonly to Windows nodes. This results in the corresponding volume being readonly inside a Pod. This problem is more likely to occur when a new set of nodes replaces an old set of nodes (for example, cluster upgrade or node pool update). Stateful workloads that previously worked fine might be unable to write to their volumes on the new set of nodes.

Workaround:

1. Get the UID of the Pod that is unable to write to its volume:

   kubectl--kubeconfigUSER_CLUSTER_KUBECONFIGgetpod\POD_NAME--namespacePOD_NAMESPACE\-o=jsonpath='{.metadata.uid}{"\n"}'

2. Use the PersistentVolumeClaim to get the name of the PersistentVolume:

   kubectl--kubeconfigUSER_CLUSTER_KUBECONFIGgetpvc\PVC_NAME--namespacePOD_NAMESPACE\-ojsonpath='{.spec.volumeName}{"\n"}'

3. Determine the name of the node where the Pod is running:

   kubectl--kubeconfigUSER_CLUSTER_KUBECONFIGgetpods\--namespacePOD_NAMESPACE\-ojsonpath='{.spec.nodeName}{"\n"}'

4. Obtain powershell access to the node, either through SSH or the vSphere web interface.
5. Set environment variables:

    PS C:\Users\administrator> pvname=PV_NAME PS C:\Users\administrator> podid=POD_UID

6. Identify the disk number for the disk associated with the PersistentVolume:

    PS C:\Users\administrator> disknum=(Get-Partition -Volume (Get-Volume -UniqueId ("\\?\"+(Get-Item (Get-Item "C:\var\lib\kubelet\pods\$podid\volumes\kubernetes.io~csi\$pvname\mount").Target).Target))).DiskNumber

7. Verify that the disk is readonly:

    PS C:\Users\administrator> (Get-Disk -Number $disknum).IsReadonly

   The result should be True.
8. Set readonly to false.

    PS C:\Users\administrator> Set-Disk -Number $disknum -IsReadonly $false PS C:\Users\administrator> (Get-Disk -Number $disknum).IsReadonly

9. Delete the Pod so that it will get restarted:

   kubectl--kubeconfigUSER_CLUSTER_KUBECONFIGdeletepodPOD_NAME\--namespacePOD_NAMESPACE

10. The Pod should get scheduled to the same node. But in case the Pod gets scheduled to a new node, you might need to repeat the preceding steps on the new node.

vsphere-csi-secret is not updated after gkectl update credentials vsphere --admin-cluster

If you update the vSphere credentials for an admin cluster following updating cluster credentials, you might find vsphere-csi-secret under kube-system namespace in the admin cluster still uses the old credential.

Workaround:

1. Get the vsphere-csi-secret secret name:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=kube-systemgetsecrets|grepvsphere-csi-secret

2. Update the data of the vsphere-csi-secret secret you got from the above step:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=kube-systempatchsecretCSI_SECRET_NAME-p\"{\"data\":{\"config\":\"$(\kubectl--kubeconfig=ADMIN_KUBECONFIG-n=kube-systemgetsecretsCSI_SECRET_NAME-ojsonpath='{.data.config}'\|base64-d\|sed-e'/user/c user = \"VSPHERE_USERNAME_TO_BE_UPDATED\"'\|sed-e'/password/c password = \"VSPHERE_PASSWORD_TO_BE_UPDATED\"'\|base64-w0\)\"}}"

3. Restart vsphere-csi-controller:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=kube-systemrolloutrestartdeploymentvsphere-csi-controller

4. You can track the rollout status with:

   kubectl--kubeconfig=ADMIN_KUBECONFIG-n=kube-systemrolloutstatusdeploymentvsphere-csi-controller

   After the deployment is successfully rolled out, the updated vsphere-csi-secret should be used by the controller.

audit-proxy crashloop when enabling Cloud Audit Logs with gkectl update cluster

audit-proxy might crashloop because of empty --cluster-name. This behavior is caused by a bug in the update logic, where the cluster name is not propagated to the audit-proxy pod / container manifest.

Workaround:

For a control plane v2 user cluster with enableControlplaneV2: true, connect to the user control plane machine using SSH, and update /etc/kubernetes/manifests/audit-proxy.yaml with --cluster_name=USER_CLUSTER_NAME.

For a control plane v1 user cluster, edit the audit-proxy container in the kube-apiserver statefulset to add --cluster_name=USER_CLUSTER_NAME:

kubectleditstatefulsetkube-apiserver-nUSER_CLUSTER_NAME--kubeconfig=ADMIN_CLUSTER_KUBECONFIG

An additional control plane redeployment right after gkectl upgrade cluster

Right after gkectl upgrade cluster, the control plane pods might be re-deployed again. The cluster state from gkectl list clusters change from RUNNING TO RECONCILING. Requests to the user cluster might timeout.

This behavior is because of the control plane certificate rotation happens automatically after gkectl upgrade cluster.

This issue only happens to user clusters that do NOT use control plane v2.

Workaround:

Wait for the cluster state to change back to RUNNING again in gkectl list clusters, or upgrade to versions with the fix: 1.13.6+, 1.14.2+ or 1.15+.

Bad release 1.12.7-gke.19 has been removed

Google Distributed Cloud 1.12.7-gke.19 is a bad release and you should not use it. The artifacts have been removed from the Cloud Storage bucket.

Workaround:

Use the 1.12.7-gke.20 release instead.

gke-connect-agent continues to use the older image after registry credential updated

If you update the registry credential using one of the following methods:

• gkectl update credentials componentaccess if not using private registry
• gkectl update credentials privateregistry if using private registry

you might find gke-connect-agent continues to use the older image or the gke-connect-agent pods cannot be pulled up due to ImagePullBackOff.

This issue will be fixed in Google Distributed Cloud releases 1.13.8, 1.14.4, and subsequent releases.

Workaround:

Option 1: Redeploy gke-connect-agent manually:

1. Delete the gke-connect namespace:

   kubectl--kubeconfig=KUBECONFIGdeletenamespacegke-connect

2. Redeploy gke-connect-agent with the original register service account key (no need to update the key):
   For admin cluster:

   gkectlupdatecredentialsregister--kubeconfig=ADMIN_CLUSTER_KUBECONFIG--configADMIN_CLUSTER_CONFIG_FILE--admin-cluster

   For user cluster:

   gkectlupdatecredentialsregister--kubeconfig=ADMIN_CLUSTER_KUBECONFIG--configUSER_CLUSTER_CONFIG_FILE

Option 2: You can manually change the data of the image pull secret regcred which is used by gke-connect-agent deployment:

kubectl--kubeconfig=KUBECONFIG-n=gke-connectpatchsecretsregcred-p"{\"data\":{\".dockerconfigjson\":\"$(kubectl--kubeconfig=KUBECONFIG-n=kube-systemgetsecretsprivate-registry-creds-ojsonpath='{.data.\.dockerconfigjson}')\"}}"

Option 3: You can add the default image pull secret for your cluster in the gke-connect-agent deployment by:

1. Copy the default secret to gke-connect namespace:

   kubectl--kubeconfig=KUBECONFIG-n=kube-systemgetsecretprivate-registry-creds-oyaml|sed's/ namespace: .*/ namespace: gke-connect/'|kubectl--kubeconfig=KUBECONFIG-n=gke-connectapply-f-

2. Get the gke-connect-agent deployment name:

   kubectl--kubeconfig=KUBECONFIG-n=gke-connectgetdeployment|grepgke-connect-agent

3. Add the default secret to gke-connect-agent deployment:

   kubectl--kubeconfig=KUBECONFIG-n=gke-connectpatchdeploymentDEPLOYMENT_NAME-p'{"spec":{"template":{"spec":{"imagePullSecrets": [{"name": "private-registry-creds"}, {"name": "regcred"}]}}}}'

Manual LB configuration check failure

When you validate the configuration before creating a cluster with Manual load balancer by running gkectl check-config, then the command will fail with the following error messages.

-ValidationCategory:ManualLBRunningvalidationcheckfor"Networkconfiguration"...panic:runtimeerror:invalidmemoryaddressornilpointer dereference

Workaround:

Option 1: You can use the patch version 1.13.7 and 1.14.4 that will include the fix.

Option 2: You can also run the same command to validate the configuration but skip the load balancer validation.

gkectlcheck-config--skip-validation-load-balancer

etcd watch starvation

Clusters running etcd version 3.4.13 or earlier may experience watch starvation and non-operational resource watches, which can lead to the following problems:

• Pod scheduling is disrupted
• Nodes are unable to register
• kubelet doesn't observe pod changes

These problems can make the cluster non-functional.

This issue is fixed in Google Distributed Cloud releases 1.12.7, 1.13.6, 1.14.3, and subsequent releases. These newer releases use etcd version 3.4.21. All prior versions of Google Distributed Cloud are affected by this issue.

Workaround

If you can't upgrade immediately, you can mitigate the risk of cluster failure by reducing the number of nodes in your cluster. Remove nodes until the etcd_network_client_grpc_sent_bytes_total metric is less than 300 MBps.

To view this metric in Metrics Explorer:

1. Go to the Metrics Explorer in the Google Cloud console:

   Go to Metrics Explorer

2. Select the Configuration tab.
3. Expand the Select a metric, enter Kubernetes Container in the filter bar, and then use the submenus to select the metric:
   1. In the Active resources menu, select Kubernetes Container.
   2. In the Active metric categories menu, select Anthos.
   3. In the Active metrics menu, select etcd_network_client_grpc_sent_bytes_total.
   4. Click Apply.

GKE Identity Service can cause control plane latencies

At cluster restarts or upgrades, GKE Identity Service can get overwhelmed with traffic consisting of expired JWT tokens forwarded from the kube-apiserver to GKE Identity Service over the authentication webhook. Although GKE Identity Service doesn't crashloop, it becomes unresponsive and ceases to serve further requests. This problem ultimately leads to higher control plane latencies.

This issue is fixed in the following Google Distributed Cloud releases:

• 1.12.6+
• 1.13.6+
• 1.14.2+

To determine if you're affected by this issue, perform the following steps:

1. Check whether the GKE Identity Service endpoint can be reached externally:

   curl-s-o/dev/null-w"%{http_code}"\-XPOSThttps://CLUSTER_ENDPOINT/api/v1/namespaces/anthos-identity-service/services/https:ais:https/proxy/authenticate-d'{}'

   Replace CLUSTER_ENDPOINT with the control plane VIP and control plane load balancer port for your cluster (for example, 172.16.20.50:443).

   If you're affected by this issue, the command returns a 400 status code. If the request times out, restart the ais Pod and rerun the curl command to see if that resolves the problem. If you get a status code of 000, the problem has been resolved and you are done. If you still get a 400 status code, the GKE Identity Service HTTP server isn't starting. In this case, continue.

2. Check the GKE Identity Service and kube-apiserver logs:
   1. Check the GKE Identity Service log:

      kubectllogs-f-lk8s-app=ais-nanthos-identity-service\--kubeconfigKUBECONFIG

      If the log contains an entry like the following, then you are affected by this issue:

      I081122:32:03.58344832authentication_plugin.cc:295]StoppingOIDCauthenticationfor???.UnabletoverifytheOIDCIDtoken:JWTverificationfailed:TheJWTdoesnotappeartobefromthisidentityprovider.Tomatchthisprovider,the'aud'claimmustcontainoneofthefollowingaudiences:

   2. Check the kube-apiserver logs for your clusters:

      In the following commands, KUBE_APISERVER_POD is the name of the kube-apiserver Pod on the given cluster.

      Admin cluster:

      kubectl--kubeconfigADMIN_KUBECONFIGlogs\-nkube-systemKUBE_APISERVER_PODkube-apiserver

      User cluster:

      kubectl--kubeconfigADMIN_KUBECONFIGlogs\-nUSER_CLUSTER_NAMEKUBE_APISERVER_PODkube-apiserver

      If the kube-apiserver logs contain entries like the following, then you are affected by this issue:

      E081122:30:22.6560851webhook.go:127]Failedtomakewebhookauthenticatorrequest:errortryingtoreachservice:net/http:TLShandshaketimeout E081122:30:22.6562661authentication.go:63]"Unable to authenticate the request"err="[invalid bearer token, error trying to reach service: net/http: TLS handshake timeout]"

Workaround

If you can't upgrade your clusters immediately to get the fix, you can identify and restart the offending pods as a workaround:

1. Increase the GKE Identity Service verbosity level to 9:

   kubectlpatchdeploymentais-nanthos-identity-service--type=json\-p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", \ "value":"--vmodule=cloud/identity/hybrid/charon/*=9"}]'\--kubeconfigKUBECONFIG

2. Check the GKE Identity Service log for the invalid token context:

   kubectllogs-f-lk8s-app=ais-nanthos-identity-service\--kubeconfigKUBECONFIG

3. To get the token payload associated with each invalid token context, parse each related service account secret with the following command:

   kubectl-nkube-systemgetsecretSA_SECRET\--kubeconfigKUBECONFIG\-ojsonpath='{.data.token}'|base64--decode

4. To decode the token and see the source pod name and namespace, copy the token to the debugger at jwt.io.
5. Restart the pods identified from the tokens.

The memory usage increase issue of etcd-maintenance pods

The etcd maintenance pods that use etcddefrag:gke_master_etcddefrag_20210211.00_p0 image are affected. The `etcddefrag` container opens a new connection to etcd server during each defrag cycle and the old connections are not cleaned up.

Workaround:

Option 1: Upgrade to the latest patch version from 1.8 to 1.11 which contain the fix.

Option 2: If you are using patch version earlier than 1.9.6 and 1.10.3, you need to scale down the etcd-maintenance pod for admin and user cluster:

kubectlscale--replicas0deployment/gke-master-etcd-maintenance-nUSER_CLUSTER_NAME--kubeconfigADMIN_CLUSTER_KUBECONFIG kubectlscale--replicas0deployment/gke-master-etcd-maintenance-nkube-system--kubeconfigADMIN_CLUSTER_KUBECONFIG

Miss the health checks of user cluster control plane pods

Both the cluster health controller and the gkectl diagnose cluster command perform a set of health checks including the pods health checks across namespaces. However, they start to skip the user control plane pods by mistake. If you use the control plane v2 mode, this won't affect your cluster.

Workaround:

This won't affect any workload or cluster management. If you want to check the control plane pods healthiness, you can run the following commands:

kubectlgetpods-owide-nUSER_CLUSTER_NAME--kubeconfigADMIN_CLUSTER_KUBECONFIG

1.6 and 1.7 admin cluster upgrades may be affected by the k8s.gcr.io -> registry.k8s.io redirect

Kubernetes redirected the traffic from k8s.gcr.io to registry.k8s.io on 3/20/2023. In Google Distributed Cloud 1.6.x and 1.7.x, the admin cluster upgrades use the container image k8s.gcr.io/pause:3.2. If you use a proxy for your admin workstation and the proxy doesn't allow registry.k8s.io and the container image k8s.gcr.io/pause:3.2 is not cached locally, the admin cluster upgrades will fail when pulling the container image.

Workaround:

Add registry.k8s.io to the allowlist of the proxy for your admin workstation.

Seesaw validation failure on load balancer creation

gkectl create loadbalancer fails with the following error message:

-ValidationCategory:SeesawLB-[FAILURE]Seesawvalidation:xxxclusterlbhealthcheckfailed:LB"xxx.xxx.xxx.xxx"isnothealthy:Get"http://xxx.xxx.xxx.xxx:xxx/healthz":dialtcpxxx.xxx.xxx.xxx:xxx:connect:noroutetohost

This is due to the seesaw group file already existing. And the preflight check tries to validate a non-existent seesaw load balancer.

Workaround:

Remove the existing seesaw group file for this cluster. The file name is seesaw-for-gke-admin.yaml for the admin cluster, and seesaw-for-{CLUSTER_NAME}.yaml for a user cluster.

Application timeouts caused by conntrack table insertion failures

Google Distributed Cloud version 1.14 is susceptible to netfilter connection tracking (conntrack) table insertion failures when using Ubuntu or COS operating system images. Insertion failures lead to random application timeouts and can occur even when the conntrack table has room for new entries. The failures are caused by changes in kernel 5.15 and higher that restrict table insertions based on chain length.

To see if you are affected by this issue, you can check the in-kernel connection tracking system statistics on each node with the following command:

sudoconntrack-S

The response looks like this:

cpu=0found=0invalid=4insert=0insert_failed=0drop=0early_drop=0error=0search_restart=0clash_resolve=0chaintoolong=0cpu=1found=0invalid=0insert=0insert_failed=0drop=0early_drop=0error=0search_restart=0clash_resolve=0chaintoolong=0cpu=2found=0invalid=16insert=0insert_failed=0drop=0early_drop=0error=0search_restart=0clash_resolve=0chaintoolong=0cpu=3found=0invalid=13insert=0insert_failed=0drop=0early_drop=0error=0search_restart=0clash_resolve=0chaintoolong=0cpu=4found=0invalid=9insert=0insert_failed=0drop=0early_drop=0error=0search_restart=0clash_resolve=0chaintoolong=0cpu=5found=0invalid=1insert=0insert_failed=0drop=0early_drop=0error=519search_restart=0clash_resolve=126chaintoolong=0 ...

If a chaintoolong value in the response is a non-zero number, you're affected by this issue.

Workaround

The short term mitigation is to increase the size of both the netfiler hash table (nf_conntrack_buckets) and the netfilter connection tracking table (nf_conntrack_max). Use the following commands on each cluster node to increase the size of the tables:

sysctl-wnet.netfilter.nf_conntrack_buckets=TABLE_SIZE sysctl-wnet.netfilter.nf_conntrack_max=TABLE_SIZE

Replace TABLE_SIZE with new size in bytes. The default table size value is 262144. We suggest that you set a value equal to 65,536 times the number of cores on the node. For example, if your node has eight cores, set the table size to 524288.

calico-typha or anetd-operator crash loop on Windows nodes with Controlplane V2

With Controlplane V2 enabled, calico-typha or anetd-operator might be scheduled to Windows nodes and get into crash loop.

The reason is that the two deployments tolerate all taints including Windows node taint.

Workaround:

Either upgrade to 1.13.3+, or run the following commands to edit the `calico-typha` or `anetd-operator` deployment:

# If dataplane v2 is not used.kubectleditdeployment-nkube-systemcalico-typha--kubeconfigUSER_CLUSTER_KUBECONFIG# If dataplane v2 is used.kubectleditdeployment-nkube-systemanetd-operator--kubeconfigUSER_CLUSTER_KUBECONFIG

Remove the following spec.template.spec.tolerations:

-effect:NoScheduleoperator:Exists-effect:NoExecuteoperator:Exists

And add the following toleration:

-key:node-role.kubernetes.io/masteroperator:Exists

User cluster private registry credential file cannot be loaded

You might not be able to create a user cluster if you specify the privateRegistry section with credential fileRef. Preflight might fail with the following message:

 [FAILURE] Docker registry access: Failed to login. 

Workaround:

• If you did not intend to specify the field or you want to use the same private registry credential as admin cluster, you can simply remove or comment the privateRegistry section in your user cluster config file.
• If you want to use a specific private registry credential for your user cluster, you may temporarily specify the privateRegistry section this way:

  privateRegistry:address:PRIVATE_REGISTRY_ADDRESScredentials:username:PRIVATE_REGISTRY_USERNAMEpassword:PRIVATE_REGISTRY_PASSWORDcaCertPath:PRIVATE_REGISTRY_CACERT_PATH

  (NOTE: This is only a temporarily fix and these fields are already deprecated, consider using the credential file when upgrading to 1.14.3+.)

Cloud Service Mesh and other service meshes not compatible with Dataplane v2

Dataplane V2 takes over load balancing and creates a kernel socket instead of a packet based DNAT. This means that Cloud Service Mesh cannot do packet inspection as the pod is bypassed and never uses IPTables.

This manifests in kube-proxy free mode by loss of connectivity or incorrect traffic routing for services with Cloud Service Mesh as the sidecar cannot do packet inspection.

This issue is present on all versions of Google Distributed Cloud 1.10, however some newer versions of 1.10 (1.10.2+) have a workaround.

Workaround:

Either upgrade to 1.11 for full compatibility or if running 1.10.2 or later, run:

kubectleditcm-nkube-systemcilium-config--kubeconfigUSER_CLUSTER_KUBECONFIG 

Add bpf-lb-sock-hostns-only: true to the configmap and then restart the anetd daemonset:

kubectlrolloutrestartdsanetd-nkube-system--kubeconfigUSER_CLUSTER_KUBECONFIG 

kube-controller-manager might detach persistent volumes forcefully after 6 minutes

kube-controller-manager might timeout when detaching PV/PVCs after 6 minutes, and forcefully detach the PV/PVCs. Detailed logs from kube-controller-manager show events similar to the following:

 $ cat kubectl_logs_kube-controller-manager-xxxx | grep "DetachVolume started" | grep expired kubectl_logs_kube-controller-manager-gke-admin-master-4mgvr_--container_kube-controller-manager_--kubeconfig_kubeconfig_--request-timeout_30s_--namespace_kube-system_--timestamps:2023-01-05T16:29:25.883577880Z W0105 16:29:25.883446 1 reconciler.go:224] attacherDetacher.DetachVolume started for volume "pvc-8bb4780b-ba8e-45f4-a95b-19397a66ce16" (UniqueName: "kubernetes.io/csi/csi.vsphere.vmware.com^126f913b-4029-4055-91f7-beee75d5d34a") on node "sandbox-corp-ant-antho-0223092-03-u-tm04-ml5m8-7d66645cf-t5q8f" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching 

To verify the issue, log into the node and run the following commands:

# See all the mounting points with disks lsblk-f # See some ext4 errors sudodmesg-T

In the kubelet log, errors like the following are displayed:

 Error: GetDeviceMountRefs check failed for volume "pvc-8bb4780b-ba8e-45f4-a95b-19397a66ce16" (UniqueName: "kubernetes.io/csi/csi.vsphere.vmware.com^126f913b-4029-4055-91f7-beee75d5d34a") on node "sandbox-corp-ant-antho-0223092-03-u-tm04-ml5m8-7d66645cf-t5q8f" : the device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8bb4780b-ba8e-45f4-a95b-19397a66ce16/globalmount" is still mounted by other references [/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8bb4780b-ba8e-45f4-a95b-19397a66ce16/globalmount 

Workaround:

Connect to the affected node using SSH and reboot the node.

Cluster upgrade is stuck if 3rd party CSI driver is used

You might not be able to upgrade a cluster if you use a 3rd party CSI driver. The gkectl cluster diagnose command might return the following error:

 "virtual disk "kubernetes.io/csi/csi.netapp.io^pvc-27a1625f-29e3-4e4f-9cd1-a45237cc472c" IS NOT attached to machine "cluster-pool-855f694cc-cjk5c" but IS listed in the Node.Status" 

Workaround:

Perform the upgrade using the --skip-validation-all option.

gkectl repair admin-master creates the admin master VM without upgrading its vm hardware version

The admin master node created via gkectl repair admin-master may use a lower VM hardware version than expected. When the issue happens, you will see the error from the gkectl diagnose cluster report.

CSIPrerequisites [VM Hardware]: The current VM hardware versions are lower than vmx-15 which is unexpected. Please contact Anthos support to resolve this issue.

Workaround:

Shutdown the admin master node, follow https://kb.vmware.com/s/article/1003746 to upgrade the node to the expected version described in the error message, and then start the node.

VM releases DHCP lease on shutdown/reboot unexpectedly, which may result in IP changes

In systemd v244, systemd-networkd has a default behavior change on the KeepConfiguration configuration. Before this change, VMs did not send a DHCP lease release message to the DHCP server on shutdown or reboot. After this change, VMs send such a message and return the IPs to the DHCP server. As a result, the released IP may be reallocated to a different VM and/or a different IP may be assigned to the VM, resulting in IP conflict (at Kubernetes level, not vSphere level) and/or IP change on the VMs, which can break the clusters in various ways.

For example, you may see the following symptoms.

• vCenter UI shows that no VMs use the same IP, but kubectl get nodes -o wide returns nodes with duplicate IPs.

   NAME STATUS AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready 28h v1.22.8-gke.204 10.180.85.130 10.180.85.130 Ubuntu 20.04.4 LTS 5.4.0-1049-gkeop containerd://1.5.13 node2 NotReady 71d v1.22.8-gke.204 10.180.85.130 10.180.85.130 Ubuntu 20.04.4 LTS 5.4.0-1049-gkeop containerd://1.5.13

• New nodes fail to start due to calico-node error

   2023-01-19T22:07:08.817410035Z 2023-01-19 22:07:08.817 [WARNING][9] startup/startup.go 1135: Calico node 'node1' is already using the IPv4 address 10.180.85.130. 2023-01-19T22:07:08.817514332Z 2023-01-19 22:07:08.817 [INFO][9] startup/startup.go 354: Clearing out-of-date IPv4 address from this node IP="10.180.85.130/24" 2023-01-19T22:07:08.825614667Z 2023-01-19 22:07:08.825 [WARNING][9] startup/startup.go 1347: Terminating 2023-01-19T22:07:08.828218856Z Calico node failed to start

Workaround:

Deploy the following DaemonSet on the cluster to revert the systemd-networkd default behavior change. The VMs that run this DaemonSet will not release the IPs to the DHCP server on shutdown/reboot. The IPs will be freed automatically by the DHCP server when the leases expire.

apiVersion:apps/v1kind:DaemonSetmetadata:name:set-dhcp-on-stopspec:selector:matchLabels:name:set-dhcp-on-stoptemplate:metadata:labels:name:set-dhcp-on-stopspec:hostIPC:truehostPID:truehostNetwork:truecontainers:-name:set-dhcp-on-stopimage:ubuntutty:truecommand:-/bin/bash--c-|set -xdatewhile true; doexport CONFIG=/host/run/systemd/network/10-netplan-ens192.network;grep KeepConfiguration=dhcp-on-stop "${CONFIG}" > /dev/nullif (( $? != 0 )) ; thenecho "Setting KeepConfiguration=dhcp-on-stop"sed -i '/\[Network\]/a KeepConfiguration=dhcp-on-stop' "${CONFIG}"cat "${CONFIG}"chroot /host systemctl restart systemd-networkdelseecho "KeepConfiguration=dhcp-on-stop has already been set"fi;sleep 3600donevolumeMounts:-name:hostmountPath:/hostresources:requests:memory:"10Mi"cpu:"5m"securityContext:privileged:truevolumes:-name:hosthostPath:path:/tolerations:-operator:Existseffect:NoExecute-operator:Existseffect:NoSchedule

Component access service account key wiped out after admin cluster upgraded from 1.11.x

This issue will only affect admin clusters which are upgraded from 1.11.x, and won't affect admin clusters which are newly created after 1.12.

After upgrading a 1.11.x cluster to 1.12.x, the component-access-sa-key field in admin-cluster-creds secret will be wiped out to empty. This can be checked by running the following command:

kubectl--kubeconfigADMIN_KUBECONFIG-nkube-systemgetsecretadmin-cluster-creds-oyaml|grep'component-access-sa-key'

After the component access service account key been deleted, installing new user clusters or upgrading existing user clusters will fail. The following lists some error messages you might encounter:

• Slow validation preflight failure with error message: "Failed to create the test VMs: failed to get service account key: service account is not configured."
• Prepare by gkectl prepare failed with error message: "Failed to prepare OS images: dialing: unexpected end of JSON input"
• If you are upgrading a 1.13 user cluster using the Google Cloud Console or the gcloud CLI, when you run gkectl update admin --enable-preview-user-cluster-central-upgrade to deploy the upgrade platform controller, the command fails with the message: "failed to download bundle to disk: dialing: unexpected end of JSON input" (You can see this message in the status field in the output of kubectl --kubeconfig ADMIN_KUBECONFIG -n kube-system get onprembundle -oyaml).

Workaround:

Add the component access service account key back into the secret manually by running the following command:

kubectl--kubeconfigADMIN_KUBECONFIG-nkube-systemgetsecretadmin-cluster-creds-ojson|jq--argcasa"$(catCOMPONENT_ACESS_SERVICE_ACOOUNT_KEY_PATH|base64-w0)"'.data["component-access-sa-key"]=$casa'|kubectl--kubeconfigADMIN_KUBECONFIGapply-f-

Cluster autoscaler does not work when Controlplane V2 is enabled

For user clusters created with Controlplane V2 enabled, node pools with autoscaling enabled always use their autoscaling.minReplicas in the user-cluster.yaml. The log of the cluster-autoscaler pod shows an error similar to the following:

>kubectl--kubeconfig$USER_CLUSTER_KUBECONFIG-nkube-system\logs$CLUSTER_AUTOSCALER_POD--container_cluster-autoscaler TIMESTAMP1gkeonprem_provider.go:73]errorgettingonpremuserclusterreadystatus:Expectedtogetaonpremuserclusterwithidfoo-user-cluster-gke-onprem-mgmt/foo-user-cluster TIMESTAMP1static_autoscaler.go:298]Failedtogetnodeinfosforgroups:Expectedtogetaonpremuserclusterwithidfoo-user-cluster-gke-onprem-mgmt/foo-user-cluster 

>kubectl--kubeconfig$USER_CLUSTER_KUBECONFIG-nkube-system\getpods|grepcluster-autoscaler cluster-autoscaler-5857c74586-txx2c4648017n48076Ki30s 

Workaround:

Disable autoscaling in all the node pools with `gkectl update cluster` until upgrading to a version with the fix

CIDR is not allowed in the IP block file

When users use CIDR in the IP block file, the config validation will fail with the following error:

-ValidationCategory:ConfigCheck -[FAILURE]Config:AddressBlockforadminclusterspecisinvalid:invalidIP: 172.16.20.12/30 

Workaround:

Include individual IPs in the IP block file until upgrading to a version with the fix: 1.12.5, 1.13.4, 1.14.1+.

OS image type update in the admin-cluster.yaml doesn't wait for user control plane machines to be re-created

When Updating control plane OS image type in the admin-cluster.yaml, and if its corresponding user cluster was created via Controlplane V2, the user control plane machines may not finish their re-creation when the gkectl command finishes.

Workaround:

After the update is finished, keep waiting for the user control plane machines to also finish their re-creation by monitoring their node os image types using kubectl --kubeconfig USER_KUBECONFIG get nodes -owide. e.g. If updating from Ubuntu to COS, we should wait for all the control plane machines to completely change from Ubuntu to COS even after the update command is complete.

Pod create or delete errors due to Calico CNI service account auth token issue

An issue with Calico in Google Distributed Cloud 1.14.0 causes Pod creation and deletion to fail with the following error message in the output of kubectl describe pods:

 error getting ClusterInformation: connection is unauthorized: Unauthorized 

This issue is only observed 24 hours after the cluster is created or upgraded to 1.14 using Calico.

Admin clusters are always using Calico, while for user cluster there is a config field `enableDataPlaneV2` in user-cluster.yaml, if that field is set to `false`, or not specified, that means you are using Calico in user cluster.

The nodes' install-cni container creates a kubeconfig with a token that is valid for 24 hours. This token needs to be periodically renewed by the calico-node Pod. The calico-node Pod is unable to renew the token as it doesn't have access to the directory that contains the kubeconfig file on the node.

Workaround:

This issue was fixed in Google Distributed Cloud version 1.14.1. Upgrade to this or a later version.

If you can't upgrade right away, apply the following patch on the calico-node DaemonSet in your admin and user cluster:

kubectl-nkube-systemgetdaemonsetcalico-node\--kubeconfigADMIN_CLUSTER_KUBECONFIG-ojson\|jq'.spec.template.spec.containers[0].volumeMounts += [{"name":"cni-net-dir","mountPath":"/host/etc/cni/net.d"}]'\|kubectlapply--kubeconfigADMIN_CLUSTER_KUBECONFIG-f- kubectl-nkube-systemgetdaemonsetcalico-node\--kubeconfigUSER_CLUSTER_KUBECONFIG-ojson\|jq'.spec.template.spec.containers[0].volumeMounts += [{"name":"cni-net-dir","mountPath":"/host/etc/cni/net.d"}]'\|kubectlapply--kubeconfigUSER_CLUSTER_KUBECONFIG-f- 

• ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file.
• USER_CLUSTER_CONFIG_FILE: the path of your user cluster configuration file.

IP block validation fails when using CIDR

Cluster creation fails despite the user having the proper configuration. User sees creation failing due to the cluster not having enough IPs.

Workaround:

Split CIDR's into several smaller CIDR blocks, such as 10.0.0.0/30 becomes 10.0.0.0/31, 10.0.0.2/31. As long as there are N+1 CIDR's, where N is the number of nodes in the cluster, this should suffice.

Admin cluster backup does not include the always-on secrets encryption keys and configuration

When the always-on secrets encryption feature is enabled along with cluster backup, the admin cluster backup fails to include the encryption keys and configuration required by always-on secrets encryption feature. As a result, repairing the admin master with this backup using gkectl repair admin-master --restore-from-backup causes the following error:

ValidatingadminmasterVMxxx... Waitingforkube-apiservertobeaccessibleviaLBVIP(timeout"8m0s")...ERROR Failedtoaccesskube-apiserverviaLBVIP.Tryingtofixtheproblembyrebootingtheadminmaster Waitingforkube-apiservertobeaccessibleviaLBVIP(timeout"13m0s")...ERROR Failedtoaccesskube-apiserverviaLBVIP.Tryingtofixtheproblembyrebootingtheadminmaster Waitingforkube-apiservertobeaccessibleviaLBVIP(timeout"18m0s")...ERROR Failedtoaccesskube-apiserverviaLBVIP.Tryingtofixtheproblembyrebootingtheadminmaster

Recreating the admin master VM with a new boot disk (e.g., gkectl repair admin-master) will fail if the always-on secrets encryption feature is enabled using `gkectl update` command.

If the always-on secrets encryption feature is not enabled at cluster creation, but enabled later using gkectl update operation then the gkectl repair admin-master fails to repair the admin cluster control plane node. It is recommend that always-on secrets encryption feature is enabled at cluster creation. There is no current mitigation.

Upgrading the first user cluster from 1.9 to 1.10 recreates nodes in other user clusters

Upgrading the first user cluster from 1.9 to 1.10 could recreate nodes in other user clusters under the same admin cluster. The recreation is performed in a rolling fashion.

The disk_label was removed from MachineTemplate.spec.template.spec.providerSpec.machineVariables, which triggered an update on all MachineDeployments unexpectedly.

Workaround:

Docker restarts frequently after cluster upgrade

Upgrade user cluster to 1.10.0 might cause docker restart frequently.

You can detect this issue by running kubectl describe node NODE_NAME --kubeconfig USER_CLUSTER_KUBECONFIG

A node condition will show whether the docker restart frequently. Here is an example output:

NormalFrequentDockerRestart41m(x2over141m)systemd-monitorNodeconditionFrequentDockerRestartisnow:True,reason:FrequentDockerRestart

To understand the root cause, you need to ssh to the node that has the symptom and run commands like sudo journalctl --utc -u docker or sudo journalctl -x

Workaround:

• Use containerd for the container runtime
• Upgrade the cluster to the latest version of 1.10

Self-deployed GMP components not preserved after upgrading to version 1.12

If you are using a Google Distributed Cloud version below 1.12, and have manually set up Google-managed Prometheus (GMP) components in the gmp-system namespace for your cluster, the components are not preserved when you upgrade to version 1.12.x.

From version 1.12, GMP components in the gmp-system namespace and CRDs are managed by stackdriver object, with the enableGMPForApplications flag set to false by default. If you manually deploy GMP components in the namespace prior to upgrading to 1.12, the resources will be deleted by stackdriver.

Workaround:

• Back up all existing PodMonitoring custom resources (CRs).
• Upgrade all clusters to version 1.12, and enable Managed Service for Prometheus.
• Redeploy PodMonitoring CRs.

Missing ClusterAPI objects in cluster snapshot system scenario

In the system scenario, the cluster snapshot doesn't include any resources under the default namespace.

However, some Kubernetes resources like Cluster API objects that are under this namespace contain useful debugging information. The cluster snapshot should include them.

Workaround:

You can manually run the following commands to collect the debugging information.

exportKUBECONFIG=USER_CLUSTER_KUBECONFIG kubectlgetclusters.cluster.k8s.io-oyaml kubectlgetcontrolplanes.cluster.k8s.io-oyaml kubectlgetmachineclasses.cluster.k8s.io-oyaml kubectlgetmachinedeployments.cluster.k8s.io-oyaml kubectlgetmachines.cluster.k8s.io-oyaml kubectlgetmachinesets.cluster.k8s.io-oyaml kubectlgetservices-oyaml kubectldescribeclusters.cluster.k8s.io kubectldescribecontrolplanes.cluster.k8s.io kubectldescribemachineclasses.cluster.k8s.io kubectldescribemachinedeployments.cluster.k8s.io kubectldescribemachines.cluster.k8s.io kubectldescribemachinesets.cluster.k8s.io kubectldescribeservices

USER_CLUSTER_KUBECONFIG is the user cluster's kubeconfig file.

User cluster deletion stuck at node drain for vSAN setup

When deleting, updating or upgrading a user cluster, node drain may be stuck in the following scenarios:

• The admin cluster has been using vSphere CSI driver on vSAN since version 1.12.x, and
• There are no PVC/PV objects created by in-tree vSphere plugins in the admin and user cluster.

To identify the symptom, run the command below:

kubectllogsclusterapi-controllers-POD_NAME_SUFFIX--kubeconfigADMIN_KUBECONFIG-nUSER_CLUSTER_NAMESPACE

Here is a sample error message from the above command:

E092020:27:43.0865671machine_controller.go:250]Errordeletingmachineobject[MACHINE];Failedtodeletemachine[MACHINE]:failedtodetachdisksfromVM"[MACHINE]":failedtoconvertdiskpath"kubevols"toUUIDpath:failedtoconvertfullpath"ds:///vmfs/volumes/vsan:[UUID]/kubevols":ServerFaultCode:Ageneralsystemerroroccurred:Invalidfault

kubevols is the default directory for vSphere in-tree driver. When there are no PVC/PV objects created, you may hit a bug that node drain will be stuck at finding kubevols, since the current implementation assumes that kubevols always exists.

Workaround:

Create the directory kubevols in the datastore where the node is created. This is defined in the vCenter.datastore field in the user-cluster.yaml or admin-cluster.yaml files.

Cluster Autoscaler clusterrolebinding and clusterrole are deleted after deleting a user cluster.

On user cluster deletion, the corresponding clusterrole and clusterrolebinding for cluster-autoscaler are also deleted. This affects all other user clusters on the same admin cluster with cluster autoscaler enabled. This is because the same clusterrole and clusterrolebinding are used for all cluster autoscaler pods within the same admin cluster.

The symptoms are the following:

• cluster-autoscaler logs

kubectllogs--kubeconfigADMIN_CLUSTER_KUBECONFIG-nkube-system\ cluster-autoscaler

where ADMIN_CLUSTER_KUBECONFIG is the admin cluster's kubeconfig file. Here is an example of error messages you might see:

2023-03-26T10:45:44.866600973ZW032610:45:44.8664631reflector.go:424]k8s.io/client-go/dynamic/dynamicinformer/informer.go:91:failedtolist*unstructured.Unstructured:onpremuserclusters.onprem.cluster.gke.ioisforbidden:User"..."cannotlistresource"onpremuserclusters"inAPIgroup"onprem.cluster.gke.io"attheclusterscope 2023-03-26T10:45:44.866646815ZE032610:45:44.8664941reflector.go:140]k8s.io/client-go/dynamic/dynamicinformer/informer.go:91:Failedtowatch*unstructured.Unstructured:failedtolist*unstructured.Unstructured:onpremuserclusters.onprem.cluster.gke.ioisforbidden:User"..."cannotlistresource"onpremuserclusters"inAPIgroup"onprem.cluster.gke.io"attheclusterscope

Workaround: