No network connectivity between runtime plane and control plane

You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent Apigee Edge documentation for this topic.

Symptoms

  • API products, Developers, Apps do not get populated on the Apigee UI.
  • API proxy deployments do not complete.
  • Management API requests executed for reading and writing API products, Developers, and Apps fail.

Error messages

This section describes the possible error messages displayed when there is no network connectivity.

ApigeeIssue

If this issue was reported as an ApigeeIssue, when the following command is run: 

kubectl -n APIGEE_NAMESPACE get apigeeissues

where APIGEE_NAMESPACE is the name for a grouping of Kubernetes resources.

the following error code is displayed: 

NAME SEVERITY AGE URL control-plane-connectivity-failure Error 1hr https://cloud.google.com/apigee/docs/api-platform/troubleshoot/playbooks/no-network-connectivity

API products, Developers, and Apps Apigee UI pages

The API products, Developers, and Apps Apigee UI pages display the following error: 

Error:noconnectionsavailablefromtheApigeeconnectagent(s):refertodocumentationtotriagefurther.

apigee-synchronizer logs

The following error message can be seen on apigee-synchronizer logs: 

{"level":"SEVERE","thread":"Apigee-Timer-3","mdc":{"action":"SYNC","env":"dev","org":"example-hybrid-dev"},"className":"com.apigee.httpclient.adaptor.RequestAdaptor","method":"doSend","severity":"SEVERE","message":"request failed[CONTEXTratelimit_period\u003d\"1 MINUTES [skipped: 1]\" ]","formattedDate":"2023-02-01T06:27:48.700Z","logger":"SERVICES.HTTPCLIENTSERVICE","exceptionStackTrace":"java.net.SocketTimeoutException: connect timed out...com.apigee.httpclient.adaptor.RequestAdaptor.send(RequestAdaptor.java:251)atcom.apigee.hybrid.runtime.contract.sync.context.HttpContractDownloader.download(HttpContractDownloader.java:84)atcom.apigee.hybrid.runtime.contract.sync.context.ControlPlaneReplicationContext.getContract(ControlPlaneReplicationContext.java:66)

apigee-watcher logs

The following error message can be seen on apigee-watcher logs: 

{"level":"error","ts":1675232549.5093117,"caller":"watcher/watcher.go:60","msg":"error during watch","name":"mpstatus","error":"INTERNAL: error sending mp status to mgmt: INTERNAL: failed to send runtime status Post \"https://apigee.googleapis.com/v1/organizations/example-hybrid-dev/instances/apigee-hybrid-australia-southeast1:reportStatus\": dial tcp 172.217.24.42:443: i/o timeout","stacktrace":"edge-internal.git.corp.google.com/apigee-watcher.git/watcher. (*Watcher).Start.func1.1\n\t/go/src/edge-internal/apigee-watcher/watcher/watcher.go:60"}

Possible Causes

PlatformCauseDescription
AllNo network route available to the internetIf no network route is available to the internet, the Apigee runtime plane components will not be able to communicate with Apigee control plane APIs.
AllApigee API endpoints are not allow-listedIf there is a firewall in place, Apigee API endpoints may have not been allow-listed.
GKEPrivate Google Access is not enabledIf the deployment is on GKE, Private Google Access may have not been enabled in the subnet.
AllUnknown network failure If there is an unknown failure in the cluster network on connecting to the internet, this issue could occur.

Cause: No network route available to the internet

Diagnosis

  1. Depending on the platform used, verify whether there is a network route available from the cluster network to the internet. For example, for more information on networking on GKE, see Network overview.
  2. Check with your infrastructure and networking team to see if the cluster network uses a forward proxy server for connecting to the internet.

Resolution

  1. Check with the network administrator and add a network route to the internet if possible.
  2. If there is a forward proxy server used for communicating between the cluster network and internet, configure forward proxy server settings in Apigee using the overrides.yaml file and apply that change to the runtime plane using the Helm command:

    Dry run:

     helm upgrade ENV_GROUP_RELEASE_NAME apigee-virtualhost/ \ --namespace APIGEE_NAMESPACE \ --atomic \ --set envgroup=ENV_GROUP \ -f OVERRIDES_FILE \ --dry-run=server

    Make sure to include all of the settings shown, including --atomic so that the action rolls back on failure.

    Your installation may or may not have been set up using an ENV_GROUP_RELEASE_NAME that differs from ENV_GROUP. For information on the settings, see Install Apigee hybrid using Helm.

    Install the chart:

     helm upgrade ENV_GROUP_RELEASE_NAME apigee-virtualhost/ \ --namespace APIGEE_NAMESPACE \ --atomic \ --set envgroup=ENV_GROUP \ -f OVERRIDES_FILE

Cause: Apigee API endpoints are not allow-listed

Diagnosis

Check with the network administrator and verify whether the list of Apigee API endpoints are allow-listed in the firewall on the platform where Apigee is installed. For GKE, this could be Cloud Next Generation Firewall.

Resolution

If the above list of Apigee API endpoints are not allow-listed, contact the network administrator and complete that requirement.

Cause: Private Google Access is not enabled

Diagnosis

  1. If Apigee is deployed on a private GKE cluster that does not have access to the internet, Private Google Access needs to be enabled to allow the Apigee runtime plane components access to Google APIs internally.

  2. In the Google Cloud console, go to the VPC networks page.

    Go to VPC networks

  3. Click the name of a network.
  4. On the Subnets tab, In the Private Google Access column, verify that the relevant subnet is enabled. If it is not, that's the reason for this failure.

Resolution

  1. In the Google Cloud console, go to the VPC networks page.

    Go to VPC networks

  2. Click the name of a network.
  3. Click the name of a subnet.
  4. Click Edit.
  5. Select Private Google Access On.
  6. Click Save. Once enabled, this issue will be resolved.

Cause: Unknown network failure

Diagnosis

Contact your network administrator and verify whether there are any unknown failures in the cluster network.

Resolution

Work with your network administrator and resolve the issues in the cluster network. Once the network issues are fixed this problem will get resolved.

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

  1. The Google Cloud Project ID.
  2. The name of the Apigee hybrid organization.
  3. The outputs of the following commands executed on all Kubernetes cluster nodes: 
    ping apigee.googleapis.com
    traceroute apigee.googleapis.com
    telnet apigee.googleapis.com 443
  4. The overrides.yaml file, masking any sensitive information.
  5. The Kubernetes pod status in all namespaces: 
    kubectl get pods -A > kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt
  6. The Kubernetes cluster-info dump:

    Generate Kubernetes cluster-info dump:

    kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump

    Zip Kubernetes cluster-info dump:

    zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*