Troubleshooting Networking (CKA)
A request fails somewhere between the name and the pod. Read the error, test one hop at a time, and find the hop.
An interactive Kubernetes lesson: 21 steps, about 32 minutes, on a live simulation in your browser.
The shop again, and today it is the network's turn. web calls http://api, and customers reach shop.example.com/api through an Ingress. Both streams are green. Follow one request, because every failure in this lesson is one of these hops going missing.
web asks CoreDNS for the name api and gets the Service's ClusterIP. It sends a packet to that IP. On web's node, rules written by kube-proxy rewrite the destination to the IP of one Ready pod from the Service's endpoints. The CNI plugin carries the packet to that pod, on whichever node it runs. A NetworkPolicy may drop it on the way out or on the way in. Finally the container must be listening on the port.
What you will learn
The path of a request
- One request, many hops: A request walks: DNS name → ClusterIP → endpoint pod IP → across the pod network → through policy → to a listening port. One broken hop, one distinct symptom.
- A debug pod and three errors: Could not resolve = DNS. Refused = it got there and nothing was listening. Timeout = something dropped it on the way.
The name
- Everything fails at once: If the name fails and the IP works, the fault is DNS. CoreDNS is an ordinary Deployment behind an ordinary Service called kube-dns: debug it like one.
- Break it: the right name, the wrong place: The same DNS error has two causes: nobody answered (CoreDNS down) or the answer was no (NXDOMAIN: wrong name or namespace). nslookup tells them apart.
The Service
- Refused, with every pod Ready: Refused plus an empty endpoint list means the selector matches no Ready pod. Compare the Service's selector with the pods' labels, character by character.
- Break it: endpoints exist, still refused: Endpoints say who receives traffic and on which port. If the list is full and you are still refused, the port after the colon is wrong.
- port-forward: skip the whole path: port-forward reaches a pod through the API server and kubelet, skipping DNS, Service, kube-proxy and policy. It tests the app and nothing else.
Under the Service
- Break it: a ClusterIP that leads nowhere: Name resolves, endpoints are correct, the pod IP answers, the ClusterIP times out: the missing piece is kube-proxy's rules on the node.
- Break it: only the other node is silent: Same-node pods answer and other-node pods time out: the pod network between nodes is broken, and that is the CNI plugin.
Policy
- A policy, and a probe that lies: A NetworkPolicy drops silently, so the symptom is a timeout. It sees your debug pod's labels, not the labels of the pod you are debugging for.
- Break it: a policy that breaks DNS: An egress policy that forgets port 53 looks like a DNS outage for the pods it selects, and for nobody else.
The front door
- 404 at the front door: 404 at the front door: the controller is running and no Ingress rule matched the host and path. The Services behind it were never asked.
- Break it: rules nobody is reading: An Ingress with an empty ADDRESS has not been picked up by any controller: check ingressClassName against kubectl get ingressclass.
- Break it: 503 Service Unavailable: 503 at the front door: a rule matched, and the Service behind it is missing or has no endpoints. From here it is an ordinary Service problem.
- Read the status at the door: At the door: refused = no controller, 404 = no matching rule, 503 = no backend, 504 = backend not answering.
Find the hop
- Find the hop: Start at the client and test one hop at a time: name, endpoints, pod IP, ClusterIP, policy. The first probe that fails names the component.
- Drill: a shell inside the cluster
- Drill: who is behind the Service
- Drill: straight to the app
Recap & playground
- Cheat sheet
- Playground