system-design

Serverless

Someone else runs the servers, you pay per request, and it scales to zero. Excellent for some workloads and quietly expensive for others.

Prerequisites: Scalability, Monolith vs Microservices Time to read: ~22 minutes

The problem

You have an image-thumbnailing service. Traffic is spiky — a few requests most of the day, thousands when a customer bulk-uploads.

Provisioned servers: you size for peak (expensive, mostly idle) or for average (fails at peak). Autoscaling helps but takes minutes to react and never scales below one instance.

📐 If the service is genuinely used for 30 minutes a day, you’re paying for 24 hours of capacity to serve 30 minutes of work — roughly 48× more than the work requires.

Serverless: no instances when there’s no traffic, thousands of concurrent executions within seconds, and you pay per 100 ms of execution.

What “serverless” actually means

Not “no servers” — no servers you manage, and no capacity you provision.

Property	Meaning
No server management	No patching, no capacity planning, no OS
Automatic scaling	Zero to thousands of concurrent executions, per-request
Scales to zero	No traffic, no instances, no cost
Pay per use	Billed per invocation and per GB-second, not per hour
Event-driven	Triggered by HTTP, a queue message, a file upload, a schedule
Stateless by design	Each invocation is independent; nothing persists in the process

The categories:

FaaS (Functions as a Service) — AWS Lambda, Google Cloud Functions, Azure Functions, Cloudflare Workers. The narrow definition.
Serverless containers — AWS Fargate, Google Cloud Run, Azure Container Apps. Your container, their capacity management. 🚨 Often the better choice, and frequently overlooked.
Serverless data — DynamoDB on-demand, Aurora Serverless, S3, SQS. Arguably where serverless delivers the most value with the fewest caveats.
Serverless everything else — Firebase, Supabase, Vercel, Netlify.

Where it genuinely wins

1. Spiky or unpredictable traffic. The canonical case. Idle costs nothing; a 100× spike is absorbed automatically.

2. Event processing. A file lands in S3 → resize it. A message arrives → process it. A row changes → update the search index. This is what FaaS is actually best at, and it’s a much better fit than HTTP APIs.

3. Scheduled jobs. A cron job that runs for 30 seconds a day, on an always-on instance, is absurd. → Background Jobs

4. Glue code. Small integrations between services. Webhook receivers. Format transformations.

5. Genuinely low volume. An internal tool used ten times a day costs cents rather than the price of an instance.

6. Small teams without operational capacity. No Kubernetes to run, no patching, no capacity planning. This is often the real reason, and it’s a legitimate one.

📐 The cost crossover, roughly:

Lambda: ~$0.20 per million requests + ~$0.0000167 per GB-second

100k requests/month, 200 ms, 512 MB:  ~$0.20/month
1M   requests/month, 200 ms, 512 MB:  ~$1.90/month
100M requests/month, 200 ms, 512 MB:  ~$190/month
1B   requests/month, 200 ms, 512 MB:  ~$1,900/month

An equivalent always-on fleet handling 1B/month: perhaps $300–800/month.

🚨 The crossover is somewhere around sustained high volume. Below it, serverless is dramatically cheaper. Above it, provisioned capacity wins — sometimes by a lot.

Cold starts

🚨 The most-discussed limitation, and the most misunderstood.

When no warm instance exists, the platform must provision one: download your code, start the runtime, initialize your application, then handle the request.

Runtime	Typical cold start
Cloudflare Workers (V8 isolates)	< 5 ms
Go, Rust	100–300 ms
Node.js, Python	200–600 ms
Java, .NET	1–10 seconds
Any of the above in a VPC (older AWS)	Historically +10 s; largely fixed since 2019

Mitigations:

Provisioned concurrency — keep N instances warm. ⚖️ Effective, and it removes the scale-to-zero benefit for those instances, so you’re paying for idle capacity again.
Smaller deployment packages and fewer dependencies.
Lazy initialization — don’t create every client at module load.
Choose a faster runtime where cold starts matter.
SnapStart (Java on Lambda) — snapshot the initialized state and restore it.

⚖️ The honest assessment: cold starts matter for user-facing synchronous APIs with tight latency budgets, and matter very little for asynchronous event processing, where an extra 300 ms is irrelevant. This is a good discriminator for whether serverless fits.

The real constraints

Cold starts get the attention; these cause more actual problems.

Execution time limits. Lambda caps at 15 minutes. Long jobs must be decomposed, or moved to Fargate/Batch. This kills a lot of otherwise-good candidates.

🚨 Statelessness is absolute. No in-memory cache between invocations that you can rely on, no local files that persist, no WebSocket connections held open. Everything goes to external state, which adds latency and cost.

🚨 The database connection problem — the most common serverless failure.

1,000 concurrent Lambda invocations
× 1 database connection each
= 1,000 connections to a Postgres instance that handles ~200 well
→ the database falls over

Traditional connection pooling doesn’t work, because there’s no long-lived process to pool in. Solutions: a proxy (RDS Proxy, PgBouncer), a serverless-native database (DynamoDB, Aurora Serverless v2 with Data API), or a database designed for HTTP-style access (Neon, PlanetScale). → Connection Pooling

Local development and testing are genuinely worse. Emulators (LocalStack, SAM) approximate the cloud imperfectly; many teams end up deploying to a personal cloud environment to test.

Debugging and observability are harder. No SSH, no process to attach a profiler to, and distributed tracing becomes mandatory. → Distributed Tracing

Vendor lock-in is real — not so much the function code as the surrounding ecosystem of triggers, IAM, and managed services it’s wired into.

Cost unpredictability. Per-request billing means a bug, a retry storm, or an attack translates directly into a bill. 🚨 Set budget alarms and concurrency limits from day one.

Concurrency limits. Accounts have limits (Lambda defaults to 1,000 concurrent executions per region). One runaway function can starve every other function in the account, which is a genuinely surprising blast radius.

Serverless vs containers

🚨 The comparison people skip, and often the more relevant one.

	FaaS (Lambda)	Serverless containers (Cloud Run, Fargate)	Managed containers (EKS/ECS)
Unit	A function	A container	A container
Scales to zero	✅	✅ (Cloud Run)	❌
Cold start	200 ms – 10 s	1–5 s	None (always warm)
Max duration	15 min	Hours	Unlimited
Concurrency per instance	1	Many (Cloud Run: up to 1,000)	Many
Local dev	Poor	✅ Just a container	✅
Lock-in	High	Low — it’s a container	Low
Long-lived connections	❌	✅	✅

🎙️ A strong, non-obvious answer: “I’d use Cloud Run rather than Lambda here. We keep scale-to-zero and per-request billing, but it’s an ordinary container — so local development is normal, there’s no 15-minute limit, one instance handles many concurrent requests so connection pooling works, and we’re not locked in.”

The multi-concurrency point matters more than it appears: Lambda handles one request per instance, so 100 concurrent requests means 100 instances and 100 database connections. Cloud Run’s single instance handling 80 concurrent requests means one connection pool — which quietly solves the database problem.

Architectural consequences

Serverless pushes you toward a particular shape, and it’s worth naming.

Event-driven by default. Functions are triggered, so systems become chains of events. → Event-Driven Architecture

Managed services for everything stateful. No connection pools means DynamoDB rather than Postgres; no in-process cache means ElastiCache or DynamoDB; no local files means S3.

🚨 The “Lambda pinball” anti-pattern: decomposing a workflow into fifteen tiny functions, each invoking the next through a queue. You get enormous latency (each hop is a cold-start risk), a distributed system that’s impossible to trace, and per-invocation costs multiplied by fifteen. Prefer fewer, coarser functions — a “Lambdalith” (one function serving a whole API via an internal router) is a legitimate and increasingly recommended pattern.

Step Functions / workflow engines for anything multi-step, rather than chaining functions manually. → Saga Pattern

When not to use it

❌ Sustained high-volume traffic. Provisioned capacity is cheaper past the crossover.
❌ Consistent, predictable load. The main benefit (scale to zero, absorb spikes) doesn’t apply.
❌ Long-running processes. Video encoding, large batch jobs, ML training.
❌ Ultra-low latency requirements. Cold starts and platform overhead.
❌ WebSockets or long-lived connections. (API Gateway supports WebSockets, but the connection is managed externally and the model is awkward.)
❌ Heavy database workloads without a connection proxy.
❌ Teams that need good local development ergonomics.

🎙️ “Traffic here is steady at a few thousand requests per second, so scale-to-zero buys us nothing and per-request billing would cost several times a provisioned fleet. I’d use containers. I would use Lambda for the image processing pipeline, which is spiky and event-driven.”

Mixing is normal and correct. Most real systems use serverless for some workloads and provisioned capacity for others.

⚖️ Trade-offs

	Gain	Cost
Serverless	No ops, automatic scaling, pay per use, scales to zero	Cold starts, time limits, statelessness, lock-in
Provisioned capacity	Predictable cost and latency, no limits, warm connections	Capacity planning, patching, paying for idle
Provisioned concurrency	No cold starts	You’re paying for idle again
Serverless containers	Scale-to-zero without lock-in; normal local dev	Slightly slower cold starts than V8 isolates
Managed database (DynamoDB)	No connection problem, scales with the functions	Access-pattern-first modelling; different cost model

In the real world

Amazon Prime Video’s 2023 post on moving a serverless pipeline to a monolithic service for a ~90% cost reduction was widely discussed. The honest reading: the workload was sustained, high-volume, and involved passing large amounts of data between steps — exactly the profile where per-invocation billing and inter-function data transfer are expensive. It’s evidence about workload fit, not a verdict on serverless.
Cloudflare Workers’ V8 isolate model achieves sub-5 ms cold starts by running many customers’ code in one process with V8’s isolation rather than a container per function. Different trade-offs (limited runtimes, no arbitrary binaries) but a genuinely different point in the design space.
The connection exhaustion problem is common enough that AWS built RDS Proxy specifically for it, and it’s why serverless architectures gravitate toward DynamoDB.

🚨 Interview traps

Proposing serverless for sustained high-volume traffic without acknowledging the cost crossover.
Ignoring the database connection problem. It’s the most common real failure.
Not knowing serverless containers exist. Cloud Run/Fargate is often the better answer.
Chaining many tiny functions. Latency, cost, and untraceable systems.
Treating cold starts as the only limitation. Time limits and statelessness matter more.
No cost controls. Per-request billing means a bug becomes a bill.
Forgetting the account concurrency limit — one runaway function starves everything.

🎙️ Soundbites

“Serverless fits the image pipeline — spiky, event-driven, and idle most of the day, so scale-to-zero is a real saving. The main API is steady at a few thousand RPS, where provisioned containers are cheaper and have no cold starts.”
“A thousand concurrent Lambdas means a thousand database connections, which will take Postgres down. I’d either put RDS Proxy in front or use DynamoDB, which is designed for this access pattern.”
“I’d use Cloud Run rather than Lambda — we keep scale-to-zero and per-request billing, but it’s an ordinary container, so local development is normal, there’s no fifteen-minute limit, and one instance serves many concurrent requests, which fixes connection pooling.”
“Cold starts matter for the synchronous API and are irrelevant for the async event processing, so I’d only pay for provisioned concurrency on the former.”
“I’d set concurrency limits and budget alarms from day one — per-request billing means a retry loop turns directly into a bill.”

🛠️ Try it

1. Measure cold starts yourself. Deploy the same trivial function in Python, Go, and Java. Invoke each after 15 minutes of idleness and measure. Then invoke repeatedly and compare warm latency. The runtime difference is much larger than people expect — and it tells you which languages suit latency-sensitive serverless.

2. Break a database with concurrency. Point a Lambda at a small Postgres instance. Load-test at 200 concurrent invocations. Watch connections exhaust and the database refuse connections. Then add RDS Proxy (or switch to Cloud Run with multi-concurrency) and re-run.

3. Do the cost arithmetic honestly. For a workload you know — requests/month, average duration, memory — price Lambda against an equivalently-sized EC2/Fargate deployment. Find the crossover point. Then re-run it assuming a bug causes 10× the invocations, and see what the bill does.

4. Compare FaaS and serverless containers. Deploy the same API as a Lambda and as a Cloud Run service. Compare cold start, local development experience, and the code you had to write. The difference in developer ergonomics is usually the deciding factor in practice.

Check yourself

1. When is serverless clearly the right choice?

Spiky or unpredictable traffic where you'd otherwise provision for a peak you rarely hit; event-driven processing (a file lands, a message arrives, a row changes) where an extra few hundred milliseconds of cold start is irrelevant; scheduled jobs that run briefly and infrequently; genuinely low-volume services where an always-on instance is absurd; and small teams without the capacity to operate infrastructure. The common thread is **low or irregular utilization plus asynchronous invocation** — exactly where paying per request beats paying per hour, and where latency variability doesn't hurt.

2. Why do serverless functions break traditional databases?

Because there's no long-lived process to hold a connection pool. Each concurrent invocation opens its own connection, so 1,000 concurrent executions means up to 1,000 connections to a database that performs well with a couple of hundred — and Postgres allocates a backend process per connection, so it thrashes and then refuses new ones. Fixes: put a connection proxy in front (RDS Proxy, PgBouncer) that multiplexes many client connections onto few real ones; use a database designed for per-request access (DynamoDB, or HTTP-based APIs like Aurora Data API, Neon, PlanetScale); or use serverless *containers* where one instance handles many concurrent requests and can pool normally.

3. What are serverless containers, and when are they better than FaaS?

Cloud Run, Fargate, and Azure Container Apps run your ordinary container image with managed capacity, per-request billing, and (for Cloud Run) scale-to-zero. They're better than FaaS when you want serverless economics without FaaS constraints: no 15-minute execution limit, one instance serving many concurrent requests (so connection pooling and in-process caching work), normal local development since it's just a container, and minimal lock-in because the artifact is portable. The trade-off is slightly slower cold starts than V8-isolate platforms, and you still manage the container image. For HTTP APIs specifically, they're frequently the better choice and are often not considered.

4. What's the "Lambda pinball" anti-pattern?

Decomposing a workflow into many tiny functions, each triggering the next through a queue or event — fifteen functions where one would do. The costs compound: each hop risks a cold start, so end-to-end latency balloons; you pay a per-invocation charge fifteen times; data is serialized and transferred between every step; and the resulting system is a distributed workflow that's genuinely hard to trace or reason about. The fix is coarser functions — including the "Lambdalith," one function serving an entire API through an internal router — and using a workflow engine (Step Functions) for genuinely multi-step processes rather than hand-chaining invocations.

5. Why can per-request billing be a risk rather than just a benefit?

Because cost becomes directly proportional to invocations, with no natural ceiling. A retry loop, a recursive trigger (a function writing to the bucket that triggers it), a bot, or a denial-of-wallet attack translates immediately into spend — and unlike a fixed fleet, which simply degrades under excess load, serverless scales up and bills you for it. There are well-known cases of accidental five-figure bills from a single misconfigured trigger. Mitigations: set account and per-function concurrency limits, configure budget alarms and automated notifications, put rate limiting in front of public endpoints, and be extremely careful with any trigger whose output could re-trigger it.