Case Study: A Chat System
WhatsApp in an interview and then in production: requirements, numbers, a WebSocket protocol, per-conversation sequence numbers, receipts, offline inboxes and push, retries without duplicates, presence, a gateway crash and the 5,000-member group.
An interactive System Design lesson: 20 steps, about 35 minutes, on a live simulation in your browser.
Design WhatsApp. Functional: one-to-one and group messages; delivery and read receipts (the ticks); messages sent to someone offline reach them later, with a push notification; an online indicator; history on every device.
Non-functional: delivery in well under a second while both people are online; messages in a conversation appear in the same order for everyone; no message is lost and none is shown twice, even when phones lose signal mid-send; the service stays up when a server dies.
What you will learn
Requirements and numbers
- What are we building?: Chat is a write path that must order and never lose messages, plus a delivery path that has to find a moving target: the one socket where the recipient happens to be connected.
- The napkin: messages and sockets: A chat system is sized twice: by messages per second for the storage path, and by open connections for the gateways.
- Drill: how many gateways?
Protocol, data model, design
- One message, end to end: One tick means the server stored it; two ticks mean the recipient's device has it; the server is the only party both phones talk to.
- The data model: one writer per conversation: Order comes from a single writer per conversation handing out sequence numbers, never from timestamps on phones or servers.
- Blue ticks: A receipt is a cumulative watermark (read up to #n), so it costs one message however many it covers.
Groups and offline users
- A group message, one member offline: An offline user costs a durable inbox write and a push notification; nothing waits in a server's memory for a phone that might never come back.
- dave comes back and syncs: Sync is "give me everything after #n": a per-conversation sequence number makes catching up after any disconnect one exact query.
Order and duplicates
- The second message overtakes the first: The network delivers in any order; the client displays in sequence order, holding back anything that arrives after a gap.
- A slow store makes the phone retry: At-least-once delivery plus a client-generated id that the server de-duplicates on gives effectively exactly-once messages.
- Without de-duplication: Retries without de-duplication turn every slow write into a duplicate message, and nothing in the logs looks like an error.
- Drill: hand out a sequence number
Presence
- bob walks into a tunnel: You cannot detect a silent disconnect, only infer it from missing heartbeats: presence is always a guess that lags by the timeout.
- Presence costs per friend: Presence fans out to every online friend on every change, so its cost is changes × friends; ration it before it outgrows messaging.
A gateway dies
- gateway-2 crashes: A crashed gateway leaves stale routing behind: until its users reconnect, the registry points at a machine that is gone, and live copies sent there are lost.
- The reconnect: After a crash, reconnects spread by jittered backoff, and the first ones may hit the dead node until health checks catch up; sync, not the live path, recovers what was lost.
The 5,000-member group
- One message to 5,000 people: Fan-out on write costs one write per recipient: perfect for small chats, ruinous for big groups.
- Fan-out on read: Fan-out on read stores once and makes every reader do the work; choose per conversation size, not once for the whole system.
Recap & playground
- Cheat sheet
- Playground