system-design

Anti-Patterns: The Hall of Shame

The designs that look reasonable and aren’t. Recognizing these — in your own work and in an interview — is worth more than knowing any single pattern.

Prerequisites: most of Parts 4 and 5 Time to read: ~22 minutes


Why a chapter on what not to do

Every pattern in this repo has a shadow — a way of applying it that looks like the real thing and produces the opposite result. 🚨 Interviewers frequently test whether you can spot these, either by proposing one to see if you notice, or by asking “what could go wrong with this design?”

Being able to name an anti-pattern, explain why it’s harmful, and offer the correct alternative is one of the clearest signals of experience. This chapter is a catalogue.


The Distributed Monolith

🚨 The worst outcome in architecture — all the costs of microservices, none of the benefits.

What it looks like: services that must be deployed together, share a database, and communicate synchronously and chattily. A change to one requires changing several.

How you get there: splitting a monolith by technical layer, or by database table, instead of by business capability. Or extracting “services” that still share the original database.

Why it’s the worst: you pay the full distributed-systems tax — network calls, timeouts, tracing, versioning, availability multiplication — and you still can’t deploy independently. You’ve made everything harder and gained nothing.

The tell: a typical feature requires coordinated deploys across multiple services.

Fix: re-draw boundaries around business capabilities and aggregates, give each service its own data, or — often the right call — merge the services back into one.Service Decomposition


The Chatty API / N+1 Over the Network

What it looks like: a client or service makes many small calls where one would do.

get_order(id)                    → 1 call
for each item in order:
    get_product(item.product_id) → N calls          ← N+1

Why it’s harmful: each call is a network round trip. In-datacenter it’s tolerable; cross-region it’s catastrophic (50 items × 150 ms = 7.5 seconds). → Latency Numbers

Fix: batch (get_products([ids])), aggregate at a BFF, use GraphQL, or denormalize.


Golden Hammer

What it looks like: “We use Kubernetes/Kafka/microservices/GraphQL for everything.”

Why it’s harmful: every tool is a trade-off, and applying one universally means applying its costs where they don’t belong. Kafka for a 100-message-a-day queue. Microservices for a five-person team. Kubernetes for a single container.

🚨 In interviews this shows up as reaching for the most complex tool by reflex. The strong move is the opposite: propose the simplest thing that meets the requirements, and justify complexity only when the numbers demand it.

Fix: estimate first; choose tools to fit the actual scale and constraints.


Premature Optimization / Over-Engineering

What it looks like: designing for a billion users when you have a thousand. Sharding on day one. Multi-region before product-market fit. A cache with no measured hit rate.

Why it’s harmful: every bit of complexity has an ongoing cost — to build, operate, debug, and onboard into. Complexity added for scale you don’t have is pure cost with no benefit, and it slows you down exactly when you need to move fast.

🚨 Counter-intuitively, proposing the simpler design and justifying why the complex one isn’t needed yet often scores higher in interviews than the elaborate design. It shows you understand cost.

Fix: design for the current scale plus one order of magnitude, and mark where you’d revisit. “One Postgres with a replica handles this for the next three years; I’d revisit sharding at 10× growth.”


Premature Pessimization (the opposite)

Worth naming because it’s the reverse trap: choosing an architecture you’ll obviously outgrow next quarter to avoid “over-engineering.” A single unindexed table for event data you know will hit a billion rows. The judgment is to design for plausible near-term scale — not the current load, and not a fantasy of Google scale.


The Big Ball of Mud

What it looks like: no discernible architecture. Everything depends on everything, global state everywhere, no boundaries.

Why it’s harmful: every change risks breaking something unrelated, nobody understands the whole, and it can’t be safely modified.

Fix: introduce boundaries incrementally — a modular monolith with enforced module rules, then extract where warranted. Don’t rewrite (big-bang rewrites fail); strangle.


Shared Mutable Database (across services)

🚨 The most common way to accidentally build a distributed monolith.

What it looks like: two or more services reading and writing the same tables.

Why it’s harmful: it’s the deepest possible coupling. Neither service can change its schema without breaking the other, there’s no encapsulation, invariants can be violated by either writer, and you can’t scale or migrate independently.

Fix: each service owns its data. If service B needs A’s data: an API call, an event-driven local read model, or the recognition that they should be one service. → Service Decomposition


God Service / God Object

What it looks like: one service (or class) that does everything and that every other service depends on. Often an “orchestrator” or “common” or “core” service that accreted responsibilities.

Why it’s harmful: it’s a single point of failure, a bottleneck for every change, and impossible for one team to own. Every feature touches it.

Fix: decompose by responsibility. Be especially wary of anything named “core,” “common,” “shared,” or “utils” — these attract unrelated code.


Retry Storms / Missing Backpressure

What it looks like: every client retries aggressively with no backoff, no jitter, and no budget; queues are unbounded.

Why it’s harmful: a degraded service receives more load exactly when it’s failing, turning a recoverable blip into a total collapse. Unbounded queues turn a slowdown into a memory-exhaustion crash. → Retries, Resilience

Fix: exponential backoff with jitter, retry budgets, circuit breakers, bounded queues, and load shedding.


No Timeouts

What it looks like: network calls that can wait forever.

Why it’s harmful: 🚨 the single most common cause of cascading outages. One slow dependency ties up every thread waiting on it, and the whole service stops — including endpoints that don’t use that dependency.

Fix: a timeout on every network call, decreasing as you go inward. → Resilience Patterns


Exactly-Once Delivery Claims

What it looks like: a design that assumes messages are delivered exactly once.

Why it’s harmful: it’s impossible, so the design is built on a false premise and will produce duplicates in production that nothing handles.

Fix: at-least-once delivery plus idempotent consumers. Say “effectively-once.”


Ignoring the Ambiguous Timeout

What it looks like: on a timeout, blindly retrying an operation with side effects — a payment, an order.

Why it’s harmful: a timeout doesn’t tell you whether the operation succeeded, so retrying risks doing it twice. This is how people get double-charged.

Fix: idempotency keys, so a retry is safe. → Idempotency


Last-Write-Wins Where Data Loss Matters

What it looks like: resolving concurrent writes by keeping the latest timestamp, for data where losing a write is a bug.

Why it’s harmful: it silently discards data, based on clocks that disagree. No error, no log.

Fix: vector clocks to detect conflicts, CRDTs, or route writes to a single owner per key. → Conflict Resolution


Caching Without an Invalidation Strategy

What it looks like: “we’ll add a cache” with no answer for how it’s invalidated, no expected hit rate, and no plan for when it’s down.

Why it’s harmful: stale data served indefinitely; a cache stampede that takes down the database; or a cache outage that sends 100% of traffic to a database sized for 5%.

Fix: TTLs, explicit or tag-based invalidation, stampede protection, and a circuit breaker so a cache outage degrades rather than cascades. → Caching


Distributed Transaction Across Services (2PC everywhere)

What it looks like: using two-phase commit to make several services commit atomically on every request.

Why it’s harmful: 2PC blocks on coordinator failure, and availability multiplies down with every participant.

Fix: design so operations stay within one transactional boundary, or use sagas. And question whether services that must commit together should be one service.


Floats for Money

What it looks like: total FLOAT.

Why it’s harmful: 0.1 + 0.2 ≠ 0.3 in binary floating point; your accounts won’t balance.

Fix: integer minor units (cents) or exact NUMERIC. → Relational Modeling


Sequential IDs in Public URLs

What it looks like: /orders/1001 in URLs.

Why it’s harmful: leaks business metrics (subtract two IDs to get daily volume) and enables enumeration attacks (IDOR). → Unique ID Generation

Fix: the two-ID pattern — sequential internal, opaque (UUID) external.


Synchronous Everything

What it looks like: every operation blocks on every downstream call, including work the user doesn’t need done before responding.

Why it’s harmful: the user waits for everything, availability multiplies, and a traffic spike hits every dependency at once.

Fix: move non-critical work off the request path onto a queue. Synchronous for what the response needs; asynchronous for the rest.


Ignoring Failure (the Happy-Path-Only Design)

What it looks like: a design that works when everything works, with no answer for “what happens when this fails?”

Why it’s harmful: at scale, everything is failing right now. A design that assumes success is a design that assumes a fleet of one.

Fix: for every dependency, decide: retry, fail fast, degrade, or queue. Design the failure path, not just the happy path.


Configuration in Code / Secrets in Repos

What it looks like: hardcoded IPs, connection strings, API keys, and passwords in the codebase.

Why it’s harmful: secrets in git are a breach (and git never forgets); hardcoded config breaks when topology changes. → Secrets Management

Fix: config from the environment, secrets from a manager, addresses from service discovery.


No Observability

What it looks like: a distributed system with no tracing, no useful metrics, and logs you can’t correlate.

Why it’s harmful: you cannot debug what you cannot see, and in a distributed system a single request touches many services. → Three Pillars

Fix: structured logs with correlation IDs, the four golden signals, and distributed tracing — built in from the start, because retrofitting is painful.


The pattern behind the anti-patterns

🚨 Most of these reduce to one of four root causes:

  1. Complexity that doesn’t match the problem — golden hammer, over-engineering, premature optimization. Fix: estimate, then choose.
  2. Coupling disguised as decoupling — distributed monolith, shared database, god service. Fix: boundaries around capabilities and data.
  3. Ignoring that the network is unreliable — no timeouts, retry storms, exactly-once claims, ambiguous-timeout retries, happy-path-only. Fix: Part 4.
  4. Ignoring that state is hard — LWW data loss, cache without invalidation, floats for money. Fix: understand the data’s actual requirements.

🎙️ In an interview, when asked “what could go wrong with this?”, scan those four categories. They cover most of what interviewers are looking for.


🎙️ Soundbites


🛠️ Try it

1. Audit a real design for anti-patterns. Take a system you’ve built or an architecture diagram you have access to. Go through this chapter’s list and check each one. Most real systems have several — and finding them is exactly the skill an interviewer is testing.

2. Build a distributed monolith deliberately. Split a small app into two services that share a database. Then try to change one service’s schema. Feel the coupling. That’s the anti-pattern, from the inside.

3. Trigger a cascade with no timeout. Two services, no timeout on the call between them. Make the downstream one slow. Watch the upstream one exhaust its threads and stop serving everything. Then add a timeout and watch it contain the damage.

4. Practice the interview move. Take any case study in Part 12 and, before reading the solution, ask yourself “what could go wrong with the obvious design?” Scan the four root causes. Compare your list to the writeup’s follow-up questions.


Check yourself

1. Why is a distributed monolith worse than a plain monolith? Because it has all the costs of a distributed system with none of the benefits. A plain monolith at least gives you in-process calls (nanoseconds, no failure), ACID transactions, one deployment, and a single stack trace. A distributed monolith — services that share a database, deploy together, and call each other synchronously and chattily — pays the full distributed tax (network latency, timeouts, partial failure, tracing, versioning, availability multiplication) while *still* requiring coordinated deploys, because the services aren't actually independent. You've made everything harder to build, operate, and debug, and gained nothing. The fix is usually to merge the services back together and re-split (if at all) along business capabilities with independent data.
2. Why does proposing a simpler design sometimes score higher in an interview? Because it demonstrates that you understand complexity has a cost, which is a more senior signal than knowing how to build something elaborate. Anyone can add sharding, microservices, and multi-region; the harder judgment is knowing when *not* to, and being able to say "one Postgres with a replica handles this scale for years, and here's where I'd revisit." Over-engineering is a real, expensive mistake — complexity you don't need slows you down, adds failure modes, and costs money to operate — so choosing the simplest design that meets the requirements, with an explicit trigger for when to add more, shows exactly the cost-awareness that distinguishes strong engineers.
3. What are the four root causes most anti-patterns reduce to? **(1) Complexity that doesn't match the problem** — golden hammer, premature optimization, over-engineering; fixed by estimating before choosing tools. **(2) Coupling disguised as decoupling** — distributed monolith, shared databases, god services; fixed by drawing boundaries around business capabilities and giving each service its own data. **(3) Ignoring that the network is unreliable** — missing timeouts, retry storms, exactly-once claims, blindly retrying ambiguous timeouts, happy-path-only designs; fixed by the distributed-systems patterns in Part 4. **(4) Ignoring that state is hard** — last-write-wins where data loss matters, caching without invalidation, floats for money; fixed by understanding the data's actual consistency and correctness requirements. Scanning these four categories is a fast way to answer "what could go wrong?"
4. How can you tell you've built a distributed monolith rather than real microservices? The clearest tell is that a typical feature requires coordinated deploys across multiple services — you can't ship one without shipping others. Supporting symptoms: services that share a database or read each other's tables; chatty synchronous call chains between the same two services; distributed transactions appearing everywhere; and services that only ever call one other service. Each of these indicates the boundaries don't match how the system actually changes. The diagnostic is concrete: build a co-change matrix from recent pull requests and see which services always change together — those should be one service. The fix is to merge and re-split along business capabilities and aggregate boundaries, with each service owning its data.
5. Why is "no timeout on a network call" singled out as especially dangerous? Because it's the single most common cause of cascading outages. A call without a timeout waits indefinitely, so when a downstream dependency becomes slow (not even down — just slow), every request to it occupies a thread, connection, and memory for the full duration. Under normal traffic, all available threads become blocked waiting on that one dependency within seconds — and because the thread pool is shared, requests to completely unrelated endpoints can't get a thread either. The service stops serving *everything*, even functionality that never touches the slow dependency, and the failure propagates to that service's callers. One slow component takes down a healthy service. Every network call needs a timeout, set from the dependency's p99.9, and decreasing as requests go inward.

Further reading