The 2 A.M. Debugging Toolkit
The shop is down and you are on call: work from the wire up with ping, traceroute, dig, curl and tcpdump until the broken layer names itself.
An interactive Networking lesson: 21 steps, about 32 minutes, on a live simulation in your browser.
Your phone wakes you at 2 a.m. Monitoring says the shop is down: https://shop.example/ does not load. You open your laptop at home and try it yourself. The command below is what you get.
curl: (7) with Connection refused. The name resolved — curl printed an address, not a naming complaint — and a machine at that address actively refused the connection. Nothing here tells you which layer broke, and guessing from this output alone is how outages get longer.
What you will learn
The call
- 2 a.m.: the shop is down: Find the highest layer that still works. The break sits above it.
- Work from the wire up: Test from the wire up. Each tool proves one layer, and a failure at the bottom looks like every layer above it.
Layer 1: the wire
- Is the cable plugged in: When the link is down, every higher-layer symptom is noise. Check the wire before you run anything else.
- Pull the plug
Your machine and its gateway
- Your address and your way out: An address says who you are. A default route says where the rest of the world is: through the gateway.
- Ping the gateway: Ping the gateway first. One answered hop clears your machine, your wire and your way out at once.
Naming: dig
- Ask for the name: dig asks your resolver one question and prints the verdict. NOERROR with an answer clears the naming layer.
- Whose memory answered: +trace skips every cache. When answers disagree, it shows which level of the walk changed.
- Dead server or wrong name: NXDOMAIN indicts the name. SERVFAIL indicts the zone's servers. The status tells you whom to call.
- Drill: print only the address
The path: traceroute
- Walk the path: traceroute names every router to the destination. The last hop that answers marks where the path still lives.
- Read the stars
- Drill: trace without names
The connection: curl
- Knock on the door: curl's error number names the broken layer: 6 is naming, 7 is refused at the far end, 28 is silence.
- The fix: start the listener: Refused means your packet arrived and no process owned the port. Fix the server, not your route.
- Silence means timeout: Fast failure is an answer. Slow failure is the absence of one — suspect the machine or the whole path.
- Your gateway can fake a remote outage: A dead gateway and a dead server both look like silence. The gateway ping is the test that tells them apart.
The truth: tcpdump
- Watch the wire: tcpdump shows what really crossed the wire. Filter to the conversation, and capture on both ends to find which side the packets never reach.
- Drill: filter the capture
Recap & playground
- Cheat sheet
- Playground