Performance: CPU, Memory & I/O
Load, saturation, iowait, page cache, swap, steal and queue depth: make each kind of slowness on purpose, learn its fingerprint, then triage a real one in sixty seconds.
An interactive Linux lesson: 25 steps, about 35 minutes, on a live simulation in your browser.
web01 runs the shop by day and batch jobs by night, and every few weeks someone reports "the server is slow". Slow has a handful of causes, each with its own fingerprint. This lesson makes each one on purpose, in a quiet hour, so you recognise it at 3 a.m.
The method is Brendan Gregg's USE: for every resource (CPUs, memory, each disk, the network) ask three questions. Utilisation: how much of the time was it busy? Saturation: is work queueing for it? Errors: is it failing? A resource at 100% that nobody waits for is fine. A queue is the problem.
What you will learn
The USE method
- A method before the tools: For every resource ask three questions: how busy is it, is anything queueing for it, is it failing. Busy is not a problem; queueing is.
CPU: busy versus saturated
- Three hogs, two CPUs: Load average is a damped average of tasks that are running or waiting. It lags: it tells you about the last minutes, not this second.
- Watch the load climb: The 1-minute load reaches about 63% of a new level after one minute and 95% after three. Compare 1 with 15 to tell starting from fading.
- Saturation: who is waiting: Utilisation is how busy a CPU is; saturation is how long tasks wait for one. PSI, vmstat's r column and pidstat's %wait all measure the waiting.
- Stop it and watch the load fall
Disk: load without CPU
- A load that uses no CPU: On Linux the load average counts tasks waiting for disk (state D) as well as tasks wanting a CPU. High load with idle CPUs points at storage.
- Read iostat -x column by column: %util is the share of time a disk had anything in flight, not how close it is to its limit. Saturation shows in aqu-sz and in await rising above the device's normal latency.
- Who is doing the I/O?
- Drill: turn on delay accounting
- IO>: time spent waiting: iostat tells you which disk is busy; iotop, pidstat -d and /proc/PID/io tell you which process. Ask both questions.
Three ways to write
- Same disk, three speeds: A write is fast when it only has to reach memory, slower when it must reach the device, and slowest when it must be durable before the next one starts.
- Fast writes are a promise: A buffered write is finished when it reaches memory and safe when it reaches the disk. Dirty pages are the gap; sync and fsync close it.
Memory, cache and swap
- Read free -h correctly: Free memory is wasted memory. Read "available", not "free": the page cache is lent, not spent.
- Drill: empty the page cache
- Read the same file twice: The first read of a file costs a disk read; later reads cost a memory copy while it stays cached. Cold and warm numbers can differ tenfold, so say which one you measured.
- Ask for more memory than there is: Without swap, running out of memory ends with the OOM killer sending SIGKILL to the biggest process. dmesg says who and how big.
- Add swap, and slow down instead: Swap turns an out-of-memory kill into slowness. Steady si and so together mean the working set does not fit in RAM.
Steal and queue depth
- The CPU time that never arrives: Steal is CPU time your VM was owed and did not get. It only shows when the VM is busy, and the fix is outside the VM: a bigger instance, another host, or dedicated cores.
- IOPS, latency and queue depth: IOPS × latency = queue depth. Below a device's limit, more parallel requests buy throughput; at the limit, they only buy latency.
- A benchmark that measures memory: A storage benchmark must bypass the page cache (--direct=1) or use a file much larger than RAM; otherwise it measures memory.
Sixty seconds, then the playground
- The page: sixty seconds of facts: Collect before you conclude: uptime, dmesg | tail, vmstat 1, mpstat -P ALL 1, pidstat 1, iostat -xz 1, free -m, sar. Two minutes, every time.
- Which resource, and who?: High load, idle CPUs, b above zero: find the D-state task, the disk it waits on, and the unit it belongs to.
- Drill: stop the export
- Cheat sheet
- Playground: the 3 a.m. page