Troubleshooting
Stay organized with collections Save and categorize content based on your preferences.

This page applies to Apigee and Apigee hybrid.

View Apigee Edge documentation.

Istio 404 (Not Found) error

Debugging a 404 (Not Found) error on Istio can be frustrating. Hopefully this will give you a place to start tracking down where things may be going wrong.

Wildcard Gateway conflict

There can be only one Gateway definition that uses a wildcard "*" hosts value. If you've deployed anything else that includes a wildcard Gateway, client calls will fail with a 404 status.

Example:

 $ istioctl get gateways GATEWAY NAME HOSTS NAMESPACE AGE bookinfo-gateway * default 20s httpbin-gateway * default 3s

If so, you'll need to delete or change one of the conflicting gateways.

Search for where the route is failing

Istio is like an onion (or, perhaps, an Ogre), it has layers. A systematic way to debug a 404 is to work outward from the target.

The backend workload

Verify you can access the workload from the sidecar:

kubectl exec $WORKLOAD_POD -c istio-proxy -- curl localhost:80/headers

The backend sidecar

Set your service address and get the IP address of the workload pod.

SERVICE=httpbin.default.svc.cluster.local:80 POD_IP=$(kubectl get pod $WORKLOAD_POD -o jsonpath='{.status.podIP}')

Access the workload through the sidecar:

kubectl exec $WORKLOAD_POD -c istio-proxy -- curl -v http://$SERVICE/headers --resolve "$SERVICE:$POD_IP"

Or, if Istio mTLS is enabled:

kubectl exec $WORKLOAD_POD -c istio-proxy -- curl -v https://$SERVICE/headers --resolve "$SERVICE:$POD_IP" --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem --insecure

The gateway (or a frontend sidecar)

Access the service from the gateway:

kubectl -n istio-system exec $GATEWAY_POD -- curl -v http://$SERVICE/header

Or, if Istio mTLS is enabled:

kubectl -n istio-system exec $GATEWAY_POD -- curl -v https://$SERVICE/headers --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem --insecure

Missing analytics

If you aren't seeing analytics in the Analytics UI, consider these possible causes:

Apigee intake can be delayed a few minutes
Envoy gRPC Access Log not configured correctly
Envoy cannot reach Remote Service
Remote Service is failing upload

Missing or bad API key not being rejected

If API key validation is not working properly, consider these possible causes:

Direct proxy

Check the ext-authz configuration.

Sidecar

Be sure listener is configured for intercept.
Check the ext-authz configuration.

Invalid requests are being checked and allowed

Remote Service configured for fail open
Envoy not configured for RBAC checks

For information on how to address these issues, refer to the following Envoy documentation topic: External Authorization, and refer to information about the failure_mode_allow property. This property allows you to change the filter's behavior on errors.

Missing or bad JWT not being rejected

The probable cause is that the Envoy JWT filter is not configured.

Valid API key fails

Probable causes

Envoy cannot reach the remote service
Your credentials are not valid
Apigee API Product not configured for target and env

Troubleshooting steps

Check your API Product on Apigee

Is it enabled for your environment (test vs prod)?
The product must be bound to the same environment as your Remote Service.
Is it bound to the target you're accessing?
Check the Apigee remote service targets section. Remember, the service name must be a fully qualified host name. If it's an Istio service, the name will be something like helloworld.default.svc.cluster.localcode> - which represents the helloworld service in the default namespace.
Does the Resource Path match your request?
Remember, a path like / or /** will match any path. You may also use '*' or '**' wildcards for matching.
Do you have a Developer App?
The API Product must be bound to a Developer App to check its keys.

Check your request

Are you passing the Consumer Key in the x-api-key header