Eight assumptions that are true on your laptop and false everywhere else. Nearly every distributed systems bug you will ever debug traces back to one of them.
Prerequisites: Networking 101, Scalability Time to read: ~20 minutes
On one machine, a function call is:
Split that function across two machines and every one of those properties disappears. The call might succeed but the response is lost. It might take 30 seconds. Two calls might arrive out of order. A timeout tells you nothing about whether the work happened.
The fallacies of distributed computing (Peter Deutsch and colleagues at Sun, 1994) name the eight assumptions engineers keep making anyway.
The assumption: if I send it, it arrives.
The reality: packets are dropped, switches fail, cables are unplugged by cleaners, cloud providers have partitions, and BGP misconfigurations blackhole traffic. At scale, some link is failing right now.
What it causes:
response = payment_service.charge(card, 100) # ← what if this never returns?
db.mark_order_paid(order_id) # ← never reached
The charge succeeded. Your database says unpaid. The customer was charged and got nothing.
What you do about it:
🚨 The deepest consequence: a timeout is ambiguous. You cannot distinguish “the request never arrived,” “it arrived and is still running,” and “it completed and the response was lost.” All three look identical, and this ambiguity is the root of most distributed systems difficulty.
The assumption: a remote call is like a local call.
The reality: local function call ~1 ns; same-datacenter round trip ~500 µs (500,000× slower); cross-continent ~150 ms (150,000,000× slower). → Latency Numbers
What it causes: chatty APIs, and the N+1 problem in distributed form.
📐 A loop calling a service 50 times:
Same datacenter: 50 × 0.5 ms = 25 ms (annoying)
Cross-region: 50 × 150 ms = 7.5 s (broken)
What you do about it:
max() instead of sum().🚨 This is why “we’ll just call the other service” is a load-bearing design decision, not an implementation detail. Every arrow on your diagram is a latency budget line item.
The assumption: payload size doesn’t matter.
The reality: 1 Gbps = 125 MB/s. Serving 2 MB responses caps one machine at ~62 requests/second — with an idle CPU.
What it causes: APIs returning entire objects when the client needs three fields. Uploads proxied through app servers. Chatty replication saturating inter-AZ links (which you also pay for per gigabyte).
What you do about it: field selection / GraphQL, pagination, compression, and CDNs for large assets. Push blobs to object storage, never through your API tier.
The assumption: internal traffic is safe because it’s “behind the firewall.”
The reality: perimeter security fails. An attacker who compromises any one service, container, or CI runner is now inside. Cloud misconfigurations expose internal endpoints. Insider threats exist.
What it causes: plaintext internal traffic, services that trust any caller, credentials in environment variables and config files, databases with no authentication because “only our services can reach it.”
What you do about it: zero trust. Assume the network is hostile.
The assumption: the servers I know about are the servers that exist.
The reality: autoscaling adds and removes instances constantly. Containers are rescheduled. IPs change on every deploy. Kubernetes pods are ephemeral by design.
What it causes: hardcoded IPs, connection pools holding dead endpoints, DNS caching that outlives a failover, load balancer configs listing machines that were terminated last week.
What you do about it:
The assumption: someone understands the whole system.
The reality: twelve teams own pieces. A third party owns your payment gateway. Your cloud provider owns the network. Nobody has the complete picture, and the person who wrote the critical service left last year.
What it causes: changes that break downstream consumers nobody knew about. Incidents where four teams debug in parallel without talking. Undocumented dependencies discovered during an outage.
What you do about it:
The assumption: moving data around is free.
The reality: serialization costs CPU (often the largest single consumer in a “simple” service). Cross-AZ traffic is billed per gigabyte. Cross-region egress is expensive. Internet egress is very expensive.
📐 A service doing 150 MB/s of internet egress: ~390 TB/month ≈ $25,000/month in bandwidth alone — usually more than the compute.
What you do about it: binary protocols for internal high-volume paths (protobuf/gRPC), compression, colocating chatty services in one AZ, and CDNs (whose egress is cheaper than origin egress).
🎙️ Bringing up cross-AZ data transfer cost unprompted is a genuine senior signal — almost nobody does.
The assumption: everything speaks the same protocols and versions.
The reality: three programming languages, two protocol versions, a legacy SOAP service, a partner’s API from 2016, mobile clients running builds from two years ago that will never update.
What it causes: subtle serialization mismatches, timezone and encoding bugs, features that work on one client and not another, and breaking changes that break customers you didn’t know you had.
What you do about it: explicit schemas with compatibility rules (serialization), a schema registry that rejects breaking changes in CI, versioned APIs, and designing for the oldest client you still support.
9. Failures are always detectable. They aren’t. A slow node and a dead node are indistinguishable, and a GC pause looks exactly like a network partition to everyone else. → Failure Modes
10. Time is consistent across machines. It isn’t. Clocks drift, NTP corrects them backwards, and “later timestamp” does not reliably mean “happened later.” → Time & Clocks
🚨 Almost every fallacy reduces to one thing: partial failure.
On one machine, an operation either succeeds or the whole process dies. In a distributed system, some parts succeed and others don’t, and you often can’t tell which.
Service A calls B, which calls C.
C succeeds. B crashes before responding to A. A times out.
A doesn't know: did C run? Did B commit? Should I retry?
Every technique in Part 4 exists to make partial failure survivable:
| Technique | What it addresses |
|---|---|
| Idempotency | Retrying an ambiguous outcome is safe |
| Timeouts & retries | Bounding the wait, recovering from loss |
| Circuit breakers | Not inheriting a dependency’s failure |
| Sagas | Undoing partially-completed work |
| Consensus | Agreeing despite failures |
| Quorums | Making progress with some nodes down |
| Tracing | Seeing what actually happened |
1. Experience the ambiguous timeout. Build service A calling service B. Have B write to a database and then sleep 10 seconds before responding. Set A’s timeout to 2 seconds. A times out; the write happened. Now make A retry. Watch the double write. Then add an idempotency key and watch it stop. This single exercise teaches more than the rest of the chapter.
2. Break the network deliberately. With tc on Linux (or Toxiproxy, which is easier):
sudo tc qdisc add dev eth0 root netem delay 200ms loss 5%
Run your app. Watch what fails, what retries, and what falls over. Most systems behave much worse than their authors expect.
3. Measure serialization cost. Profile a service under load. Find what fraction of CPU is JSON encoding and decoding. It’s usually a surprise.
4. Find your hardcoded assumptions. Grep your codebase for IP addresses, hostnames, and calls with no timeout parameter. Every one is a latent incident.