system-design

API Gateway

One front door for many services. It removes duplicated cross-cutting work — and quietly becomes the most dangerous single point of failure in your architecture.

Prerequisites: Proxies, Load Balancers Time to read: ~14 minutes

The problem

You split your monolith into eight services. Now every one of them needs:

To validate the auth token
To rate limit per customer
To log and trace requests
To handle CORS
To terminate TLS
To be discoverable at a stable public address

Eight implementations of the same six concerns, in four languages, maintained by five teams, with subtly different bugs. Meanwhile the mobile client now has to know eight hostnames, and a screen that needs data from three services makes three round trips over a cellular network.

An API gateway is where you put the shared concerns.

What it does

flowchart LR
    M[Mobile] --> G
    W[Web] --> G
    P[Partner API] --> G
    G[API Gateway<br/>auth · rate limit · routing · TLS · logging]
    G --> S1[Users service]
    G --> S2[Orders service]
    G --> S3[Payments service]
    G --> S4[Search service]

Responsibility	What it removes from your services
Routing	Path/host → service mapping, in one config instead of client-side knowledge
Authentication	Validate the JWT or session once; pass verified identity downstream
Rate limiting & quotas	Per-user, per-API-key, per-endpoint → Rate Limiting
TLS termination	Certificates managed in one place
Request/response transformation	Version translation, field renaming, protocol bridging (REST ↔ gRPC)
Aggregation	One client call → several backend calls → one merged response
Caching	Serve repeat responses without touching services
Observability	Every request logged, traced, and measured in one place
API keys & developer portal	Onboarding for third-party consumers
Circuit breaking	Contain a failing service before it poisons callers

🚨 The distinction from a plain reverse proxy is emphasis, not mechanism. Both are proxies. A gateway adds API-specific concerns: identity, quotas, per-consumer policy, aggregation, and documentation. Kong and Ambassador are literally built on Nginx and Envoy.

Authentication: the highest-value job

The gateway is where you should validate identity, because doing it once beats doing it eight times.

1. Client sends:  Authorization: Bearer eyJhbGci...
2. Gateway validates the signature, expiry, and audience
3. Gateway forwards to the service with verified identity:
       X-User-Id: 4821
       X-User-Roles: customer,beta
       X-Request-Id: 7f3a...
4. Service trusts those headers — it's on a private network behind the gateway

🚨 And there’s the trap. If a service trusts X-User-Id blindly, and anyone can reach that service directly — from another pod, a compromised container, a misconfigured security group — they can impersonate any user by setting a header. This is a genuine, common vulnerability.

Mitigations, in increasing order of rigour:

Network isolation: services only accept traffic from the gateway (security groups, network policies).
The gateway strips any incoming X-User-* headers before adding its own. Do this always — it’s one line of config and it closes the obvious hole.
Pass a signed internal token rather than plain headers, so services can verify provenance.
mTLS between the gateway and services, so services only accept connections from the gateway’s certificate. → Service Mesh

Authorization is a different question. Authentication (“who are you?”) belongs at the gateway. Authorization (“may you do this?”) usually belongs in the service, because only the service knows that user 4821 owns order 993. Coarse checks (role-based route access) can live at the gateway; fine- grained ownership checks cannot. → AuthN vs AuthZ

🎙️ “I’d authenticate at the gateway and authorize in the services — the gateway can tell you who the caller is, but only the orders service knows whether this user owns this order.”

Aggregation, and where it goes wrong

A mobile home screen needs the user profile, recent orders, and recommendations. Three services.

Without aggregation: 3 round trips from a phone on a cellular network. At 100 ms each that’s 300 ms of pure latency, three TLS-protected requests, three chances to fail.

With aggregation: 1 client call. The gateway fans out to three services in parallel (in- datacenter, ~1 ms each) and merges the response. 100 ms total.

⚖️ But be careful. Aggregation pulls business logic into the gateway, and the gateway is infrastructure owned by a platform team while the logic belongs to product teams. Do too much of it and you’ve recreated a monolith — one that every team must coordinate to change, with a deployment cadence set by the most cautious stakeholder.

The usual resolution: BFF (Backend for Frontend). Instead of one gateway doing aggregation for everyone, give each client type its own thin aggregation layer, owned by the team that owns that client. The gateway stays generic (auth, rate limiting, routing); the BFFs handle client-specific shaping. → BFF

Or: GraphQL. Let the client specify exactly what it needs and have a GraphQL layer resolve it across services. Solves over-fetching and round trips elegantly; introduces its own problems (N+1 resolvers, caching difficulty, query cost control). → GraphQL

The single-point-of-failure problem

🚨 Everything goes through the gateway. If it’s down, your entire platform is down — not one service, all of them.

This is the central trade-off of the pattern and the thing to raise unprompted in an interview.

How to manage it:

Concern	Mitigation
Availability	Multiple instances across AZs, behind a redundant load balancer. Never one box.
It becomes a bottleneck	Keep it stateless and horizontally scalable. Never put per-request state in it.
Added latency	It’s a hop: budget 1–5 ms. Fine — unless you’ve stacked three gateways, which happens.
Config errors take everything down	Treat gateway config as code: version control, review, staged rollout, fast rollback.
Rate-limit state	Use a shared store (Redis) or approximate local counters — don’t make the gateway stateful.
Deployment coupling	If every team must change gateway config to ship, you’ve built a bottleneck. Automate route registration.

⚖️ The organizational risk is as real as the technical one. A gateway owned by one team that every other team must file tickets against becomes the slowest part of your delivery pipeline. Good implementations let services declare their own routes (via annotations, CRDs, or a self-service config repo) so the platform team owns the mechanism, not each individual route.

When you don’t need one

Do not add a gateway to a monolith. You already have one front door. Adding a gateway in front of a single application buys you nothing and adds a hop, a failure mode, and a thing to operate.

Do not add one for two or three services. A load balancer with path-based routing does 90% of the job. Add the gateway when the duplicated cross-cutting concerns actually hurt.

You may not need one for internal traffic. Service-to-service calls inside a cluster are often better served by a service mesh, which gives you mTLS, retries, and telemetry without routing everything through a central chokepoint. A common mature setup: gateway for north-south (external) traffic, mesh for east-west (internal) traffic.

🎙️ “With three services I’d use path-based routing on the load balancer rather than a gateway — the gateway earns its keep once we have enough services that duplicating auth and rate limiting is a real cost.”

The options

Tool	Character
Kong	Nginx-based, plugin ecosystem, open source + enterprise. The common self-hosted choice.
AWS API Gateway	Fully managed, deep Lambda integration. Watch the per-request pricing at scale — it gets expensive fast.
Envoy / Contour / Ambassador	Envoy-based, cloud-native, excellent observability, dynamic config.
Nginx / HAProxy	Roll your own. Fine if your needs are routing + TLS + basic rate limiting.
Apigee, Azure APIM	Enterprise, strong on API products, monetization, and developer portals.
Traefik	Auto-discovers services from Docker/Kubernetes labels. Low-friction for smaller setups.

📐 A cost note worth having: AWS API Gateway is roughly $1–3.50 per million requests. At 10,000 QPS sustained that’s ~26 billion requests/month — well over $25,000/month for the gateway alone. At high volume, self-hosting Envoy or an ALB is dramatically cheaper. Bringing up gateway cost unprompted is a strong senior signal.

⚖️ Trade-offs

	Gain	Cost
Central gateway	One place for auth, rate limiting, TLS, observability	Single point of failure; a hop; organizational bottleneck
Aggregation at the gateway	Fewer client round trips	Business logic leaks into infrastructure
BFF per client	Client-specific shaping, owned by the right team	More services to run
Managed gateway	No ops burden	Per-request cost; less control; vendor lock-in
Gateway + service mesh	Right tool for north-south and east-west	Two systems to understand and operate

In the real world

Netflix built Zuul as its edge gateway and has written extensively about it — dynamic routing, per-device request shaping, and stress-testing by routing a percentage of live traffic to canary clusters. Their move toward per-device BFFs came from exactly the aggregation-complexity problem described above.
Kubernetes Ingress (and its successor, the Gateway API) is the gateway pattern standardized: services declare their own routing rules, and the platform team operates the controller. It’s a good model for avoiding the organizational bottleneck.

🚨 Interview traps

Adding a gateway to a monolith or to a three-service system. Justify it.
Not mentioning it’s a single point of failure. Raise it before the interviewer does, and say how you’d make it redundant.
Putting all authorization in the gateway. It can’t know that user 4821 owns order 993.
Trusting identity headers in services without network isolation or header stripping.
Forgetting cost for managed gateways at high volume.
Making the gateway stateful (in-memory rate-limit counters, session storage) and then wondering why you can’t scale it horizontally.

🎙️ Soundbites

“The gateway handles authentication, rate limiting, and TLS so eight services don’t each reimplement them. But it’s now the single point of failure for the whole platform, so it runs as a stateless multi-AZ fleet behind a redundant load balancer.”
“Authenticate at the gateway, authorize in the service. And the gateway must strip any inbound X-User-* headers, or a caller who reaches a service directly can impersonate anyone.”
“Rather than putting aggregation in the shared gateway, I’d use a BFF per client type — otherwise product logic ends up in infrastructure that a platform team owns.”
“Gateway for north-south traffic, service mesh for east-west. Routing internal calls through a central gateway adds a hop and a chokepoint we don’t need.”

🛠️ Try it

Run Kong (or Traefik) in Docker in front of two toy services. Configure:

Path routing: /users → service A, /orders → service B.
A rate limit of 5 requests/minute on one route. Exceed it and inspect the 429 response and its headers.
A JWT plugin. Send a request without a token (401), with a bad token (401), and with a valid one (200) — and log what headers service A actually receives.

That third step is the important one: seeing exactly what the gateway injects makes the “services trust headers” security issue concrete rather than theoretical. Then try calling service A directly with a forged X-User-Id and confirm it works — that’s the vulnerability you’re defending against.

Check yourself

1. Why is an API gateway a bigger availability risk than an individual service?

Because it's in the path of *every* request. A single service being down degrades one feature; the gateway being down takes the entire platform offline, including services that are perfectly healthy. It also concentrates blast radius for config errors — one bad routing rule affects everything. Mitigations: run it as a stateless, horizontally scaled, multi-AZ fleet behind a redundant load balancer; treat its configuration as code with staged rollout and instant rollback; and make sure a gateway failure doesn't also take down your ability to deploy a fix.

2. Where should authorization live — gateway or service?

Coarse-grained authorization (does this role have access to this route at all?) can live at the gateway. Fine-grained authorization must live in the service, because only the service knows the data relationships — that user 4821 is the owner of order 993, or a member of the team that owns document X. Putting ownership checks in the gateway would require it to query business data, which couples infrastructure to your domain model and defeats the point.

3. Your services trust an X-User-Id header from the gateway. What's the attack?

Anyone who can reach a service directly — bypassing the gateway — sets `X-User-Id` to any value and is instantly that user. Attack paths: a compromised pod in the same cluster, an overly permissive security group, an internal tool, SSRF from another service, or a misconfigured internal load balancer. Defences: network policies so services only accept traffic from the gateway; the gateway strips inbound `X-User-*` headers before setting its own; a signed internal token the service verifies; and mTLS so services only accept the gateway's certificate.

4. When is a load balancer with path routing enough, and when do you need a gateway?

A load balancer suffices when your needs are routing, TLS termination, and health checking — typically two to five services with a shared auth library. You want a gateway when the *duplication* becomes expensive: many services in different languages each reimplementing token validation, per-consumer rate limits and quotas, third-party API keys and a developer portal, per-route transformation, or aggregation. The trigger is duplicated cross-cutting concerns, not service count per se.

5. Why might a gateway become an organizational bottleneck, and how do you avoid it?

If shipping a new endpoint requires a change to centrally-owned gateway config, then every team's delivery speed is gated on one team's review queue — and that team becomes the target of everyone's frustration. Avoid it by making route configuration self-service and colocated with the service: Kubernetes Ingress/Gateway API resources in the service's own repo, annotation-based discovery, or a config repo where service teams own their own sections. The platform team owns the gateway's *behaviour and reliability*, not each individual route.