system-design

Cost Optimization

The bill is a design constraint, not an afterthought. Bringing up cost — unprompted — in a design interview is one of the clearest senior signals there is, because almost nobody does.

Prerequisites: Back-of-the-Envelope Estimation, Cloud Primitives Time to read: ~16 minutes

Why cost is a design concern

🚨 Most engineers treat cost as finance’s problem. It isn’t — architectural decisions determine cost, often by 10× or more. The same functional system can cost $5,000/month or $50,000/month depending on choices about data storage, egress, instance types, and where computation happens.

🎙️ In an interview, mentioning cost unprompted is a strong signal — it shows you think about systems as things that must be operated economically, not just work on a whiteboard. “Before we finalize this, let me flag the cost drivers — egress is going to dominate here, which is another argument for the CDN.” Almost no candidate does this, and it distinguishes you immediately.

Where the money actually goes

🚨 The surprises — where costs hide — matter more than the obvious line items:

1. Egress bandwidth — the #1 surprise. 🚨 Data out to the internet is expensive (~$0.05-0.09/GB), cross-AZ traffic is billed, cross-region more so. For data-heavy systems (media, video, large APIs), egress often dominates the bill — more than compute and storage combined. → Cloud Primitives, Object Storage

📐 Serving 100 TB/month of media: ~$2,300 to store, ~$7,000+ to serve. The egress is 3× the storage. This is why CDNs (cheaper egress + absorb most requests) are a cost decision as much as a performance one.

2. Idle capacity. Provisioning for peak but running at 20% average means paying for 80% idle. The autoscaling and reserved-capacity answer.

3. Managed service premiums. Managed databases/queues cost 2-3× raw compute — worth it for the ops savings, but a real cost. → Cloud Primitives

4. Storage that never gets deleted. Data accumulates forever without retention policies — old logs, old backups, orphaned volumes. → Object Storage tiering

5. Over-provisioned instances. Instances sized for a peak that rarely comes, or defaulted to large.

6. Observability data. 🚨 Logs, metrics, and traces can cost a significant fraction of infrastructure spend if unmanaged — high-cardinality metrics, verbose logs, 100% trace sampling. → Observability costs

7. Zombie / orphaned resources. Forgotten instances, unattached disks, idle load balancers, dev environments left running. Surprisingly large in aggregate.

The optimization levers

🚨 Concrete levers, roughly by impact:

Right-sizing — match instance sizes to actual usage. Most instances are over-provisioned; profiling and downsizing is often the biggest easy win.

Pricing models — 🚨 a major lever cloud providers offer:

On-demand — flexible, most expensive. For unpredictable/spiky load.
Reserved / committed use — commit to 1-3 years for ~30-70% off. For predictable baseline load.
Spot / preemptible — up to 90% off, but can be reclaimed anytime. For fault-tolerant batch work, CI, stateless workers that tolerate interruption.

🎙️ “I’d cover the steady baseline with reserved instances for the discount, handle spikes with on-demand, and run fault-tolerant batch work on spot instances for up to 90% off. Matching the pricing model to the workload’s predictability is a big lever.”

CDN and caching — reduce egress (CDN egress is cheaper) and reduce compute (serve from cache instead of recomputing). Cost and performance. → CDN, Caching

Storage tiering — 🚨 lifecycle policies moving old data to cheaper tiers (Standard → Infrequent Access → Glacier → Deep Archive), up to 20× cheaper for cold data. And delete what you don’t need. → Object Storage

Keep data local — avoid cross-AZ/cross-region transfer where possible (colocate chatty services, use the same AZ). → The 8 Fallacies

Serverless for spiky/low workloads — 🚨 scale-to-zero means you pay nothing when idle, cheaper than an always-on instance for intermittent work — but more expensive at sustained high volume (the crossover). Match the model to the load.

Autoscaling — pay for what you use, not for peak. With the caveats.

Sample observability data — keep 100% of errors, sample the rest; watch metric cardinality.

Kill zombies — automated detection and cleanup of orphaned resources; tagging for accountability.

FinOps: making cost visible

🚨 A practice worth knowing: FinOps brings financial accountability to cloud spend — giving teams visibility into what they cost and ownership of optimizing it.

Tagging / cost allocation — tag resources by team/service so you know who and what drives cost. You can’t optimize what you can’t attribute.
Cost monitoring and alerts — 🚨 budget alarms so a cost spike (a bug, a runaway autoscaler, a denial-of-wallet attack) is caught in hours, not on the monthly invoice.
Showback / chargeback — showing teams their costs creates the incentive to optimize.
Regular review — cost as an ongoing engineering concern, not a quarterly finance surprise.

The core FinOps insight: cost visibility drives cost optimization — engineers optimize what they can see and are accountable for.

The trade-off: don’t over-optimize

⚖️ Cost optimization has diminishing returns and a real cost of its own — engineering time.

🚨 The most expensive resource is usually engineer-time, not infrastructure. Spending a week to save $50/month is a loss. Premature cost optimization (like premature performance optimization) wastes effort that could build features.

🎙️ The balanced position: “I’d optimize the big drivers — egress, idle capacity, storage tiering, reserved instances for baseline — because those move the bill meaningfully. But I wouldn’t micro-optimize; engineer-time usually costs more than the infrastructure, so I’d focus on the 20% of levers that give 80% of the savings and not spend a week saving $50 a month.”

Optimize the significant costs; ignore the trivial ones. And design for cost-awareness from the start (so you don’t build something inherently expensive), but don’t prematurely micro-optimize.

⚖️ Trade-offs

Lever	Gain	Cost
Reserved instances	30-70% off baseline	Commitment; less flexibility
Spot instances	Up to 90% off	Can be reclaimed; only for tolerant workloads
Right-sizing	Cut over-provisioning	Profiling effort; risk of under-sizing
CDN/caching	Cheaper egress + less compute	Staleness; another layer
Storage tiering	Up to 20× cheaper cold storage	Retrieval cost/delay for cold tiers
Autoscaling	Pay for use, not peak	Lag, cold starts, runaway risk
Serverless	Nothing when idle	Expensive at sustained high volume
FinOps/visibility	Drives all other savings	Tooling and process overhead

In the real world

Egress cost surprises are the most common cloud-cost story — teams design data-heavy systems, then get a bill dominated by data transfer they didn’t anticipate. It’s why understanding egress is a genuine senior differentiator.
Dropbox’s move off S3 to their own storage (“Magic Pocket”) reportedly saved substantial cost at their scale — the canonical example of when scale justifies leaving managed services, and of cost as a first-order architectural driver.
Spot instances powering CI/CD and batch at scale — companies run huge fault-tolerant workloads (rendering, data processing, CI) almost entirely on spot/preemptible instances for the ~90% discount, accepting occasional interruption. It’s a large, real lever.
FinOps as a discipline grew precisely because cloud cost became a top-three expense for many companies and was going unmanaged — nobody owned it, so it ballooned. Visibility and accountability are the fix.

🚨 Interview traps

Never mentioning cost — the biggest missed opportunity to signal seniority.
Ignoring egress — the #1 hidden cost driver.
Not knowing pricing models (reserved/spot) as a major lever.
Over-provisioning for peak without autoscaling or reserved capacity.
No storage retention/tiering — data accumulates forever.
Micro-optimizing — spending engineer-time to save trivial amounts.
No cost visibility/alarms — a runaway cost caught on the invoice, not in hours.

🎙️ Soundbites

“Let me flag the cost drivers — egress is going to dominate here, more than compute and storage combined, which is another argument for the CDN. Data out to the internet and across regions is the hidden cost in data-heavy systems.”
“I’d match the pricing model to the workload: reserved instances for the predictable baseline for 30-70% off, on-demand for spikes, and spot instances for fault-tolerant batch work at up to 90% off.”
“Storage tiering with lifecycle policies — old data to Infrequent Access then Glacier, up to 20× cheaper — plus actually deleting what we don’t need. And retention policies so data doesn’t accumulate forever.”
“I’d optimize the big drivers but not micro-optimize — engineer-time usually costs more than the infrastructure, so I’d focus on the 20% of levers giving 80% of the savings, not spend a week saving $50 a month.”
“Cost visibility drives optimization — tag resources by team, set budget alarms so a runaway cost is caught in hours not on the invoice, and show teams their spend so they have the incentive to optimize.”

🛠️ Try it

1. Estimate a real bill. For any case study, estimate the monthly cost: compute, storage, egress, managed services. Break it down and find the biggest line item — it’s often egress, which surprises people. This is estimation applied to money, and it’s a genuine interview skill.

2. Compute the egress-vs-storage ratio. For a media-serving scenario (10M photos, viewed 20×/month each), compute storage cost and egress cost separately. Watch egress dwarf storage — that’s the CDN argument in numbers.

3. Price the pricing models. For a steady baseline load, compare on-demand vs 1-year reserved vs 3-year reserved pricing. Then price a batch workload on-demand vs spot. The discounts (30-70%, up-to-90%) are large enough to change architecture decisions.

4. Find the zombies. In any cloud account you have access to, audit for orphaned resources — unattached disks, idle load balancers, forgotten instances, old snapshots. The aggregate is usually surprising, and it’s free money.

Check yourself

1. Why is cost a design concern rather than a finance problem?

Because architectural decisions *determine* cost, often by an order of magnitude — the same functional system can cost $5,000 or $50,000 a month depending on choices engineers make about where data is stored, how it's served (CDN or origin), which instance types and pricing models are used, where computation happens, and how data flows (cross-region egress or local). Finance can't fix a design that inherently shuffles terabytes across regions or serves media from origin instead of a CDN — those are engineering decisions with financial consequences baked in. Treating cost as someone else's problem means building systems that are expensive by construction. This is why raising cost in a design discussion is a strong seniority signal: it shows you understand that a system must be *operated economically*, not just work — and almost no candidate does it, so it distinguishes you. The engineer who says "egress will dominate here, which strengthens the CDN case" is designing with the full picture; the one who ignores cost has designed something that works on a whiteboard and may be unaffordable in production.

2. Why is egress bandwidth the most common cost surprise?

Because cloud providers charge asymmetrically and unintuitively for data movement: data flowing *in* is typically free, but data flowing *out* to the internet is expensive (~$0.05-0.09/GB), data crossing between availability zones is billed, and data crossing between regions costs even more. Engineers naturally focus on the visible costs — compute (instances) and storage (GB stored) — and overlook that *moving* data has its own, often dominant, cost. For data-heavy systems (serving media, video streaming, large API responses, cross-region replication, feeding analytics pipelines), the volume of data transferred out can exceed compute and storage combined: serving 100 TB/month of media might cost ~$2,300 to store but ~$7,000+ to serve, making egress 3× the storage. It's a surprise because it doesn't appear until you're actually serving traffic at scale, the per-GB rate seems small until multiplied by the volume, and it's not a resource you "provision" and see on a dashboard the way you see instances. Understanding that egress often drives the bill — and that CDNs reduce it (cheaper CDN egress plus absorbing most requests so origin egress plummets) — is a genuine senior differentiator.

3. What are the three main cloud pricing models and when do you use each?

**On-demand** — you pay a flat hourly rate with no commitment, the most flexible and most expensive option. Use it for unpredictable, spiky, or short-lived workloads where you can't commit in advance, and to handle traffic spikes above your reserved baseline. **Reserved / committed-use** — you commit to a certain amount of capacity for 1-3 years in exchange for a large discount (~30-70% off on-demand). Use it for your *predictable baseline* load — the steady capacity you know you'll run continuously — where the commitment is safe and the discount is substantial. **Spot / preemptible** — deeply discounted (up to ~90% off) capacity from the cloud's spare pool, but it can be *reclaimed at any time* with little notice. Use it for fault-tolerant, interruptible workloads: batch processing, CI/CD, rendering, data pipelines, and stateless workers that can lose an instance and retry without harm. The lever is matching the model to the workload's predictability and interruption-tolerance: cover the predictable baseline with reserved for the discount, absorb spikes with on-demand for flexibility, and run tolerant batch work on spot for the deepest savings. Getting this mix right can cut a compute bill dramatically.

4. Why shouldn't you over-optimize cost, and how do you decide what to optimize?

Because cost optimization has diminishing returns and consumes the most expensive resource of all — engineer-time — which usually costs more than the infrastructure it saves. Spending a week of an engineer's time (worth thousands of dollars) to shave $50/month off a bill is a net loss, and micro-optimizing every line item scatters effort that could build features or address the *significant* costs. Like premature performance optimization, premature or excessive cost optimization is wasteful. You decide what to optimize by impact: focus on the big drivers — egress, idle capacity, storage tiering and retention, reserved instances for baseline, right-sizing over-provisioned instances — which move the bill meaningfully, and apply the 80/20 rule (the ~20% of levers that yield ~80% of the savings). Ignore the trivial costs. The right posture is to design for cost-awareness from the start (so you don't build something inherently expensive — choosing a CDN, sensible data flows, appropriate services) while not obsessively micro-optimizing afterward. Optimize the costs that are large enough to justify the engineering time; leave the small ones alone.

5. What is FinOps and why does cost visibility drive optimization?

FinOps is the practice of bringing financial accountability and engineering ownership to cloud spending — making teams aware of what they cost and responsible for optimizing it, rather than leaving cloud cost as an unowned expense that balloons and surprises finance quarterly. Its core mechanisms: **tagging and cost allocation** (labeling resources by team and service so spend can be attributed to who and what drives it); **cost monitoring and budget alarms** (so a cost spike from a bug, a runaway autoscaler, or a denial-of-wallet attack is caught in hours rather than discovered on the monthly invoice); **showback/chargeback** (showing or charging teams their costs); and **regular review** (treating cost as an ongoing engineering concern). Cost visibility drives optimization because you can't optimize what you can't see or aren't accountable for — when spend is a single opaque bill that belongs to nobody, no one has the information or incentive to reduce it, so it grows unchecked. When each team can see its own costs, is accountable for them, and gets alerted to anomalies, engineers naturally make cost-conscious decisions (right-sizing, deleting zombies, choosing efficient designs) because they can measure the impact and it's their responsibility. Visibility plus ownership creates the feedback loop that makes optimization happen, which is why FinOps grew into a discipline as cloud cost became a top expense for many companies.