Work that happens without a user waiting. Simple until you have more than one server, at which point “run this nightly” becomes a genuine distributed systems problem.
Prerequisites: Message Queues, Coordination Services Time to read: ~20 minutes
Two categories of work happen outside a request:
1. Triggered work — something happened, and follow-up is needed. A user signed up, so send a welcome email. A video was uploaded, so transcode it. This is what message queues handle.
2. Scheduled work — it’s time. Every night at 2 a.m., generate invoices. Every 5 minutes, refresh exchange rates. On the 1st of the month, charge subscriptions. Send this reminder at 9 a.m. on Thursday.
The second category is where things get interesting, because “run this at 2 a.m.” on a fleet of 20 identical servers means all 20 run it, and you’ve sent every customer 20 invoices.
# crontab on the app server
0 2 * * * /usr/bin/php /app/generate_invoices.php
Works perfectly with one server. Then:
| Problem | What happens |
|---|---|
| Multiple servers | 🚨 All 20 run the job. 20× the emails, 20× the charges. |
| Server dies | The job silently doesn’t run. Nobody notices until the finance team asks. |
| No visibility | Did it run? Did it succeed? How long did it take? cron emails root, which nobody reads. |
| No retries | It failed at 2 a.m. It’ll try again in 24 hours. |
| Overlapping runs | The job takes 90 minutes but runs hourly. Now two copies run concurrently. |
| Autoscaling | The cron server got scaled down. The job is gone. |
| Timezones and DST | 2 a.m. happens twice in autumn and not at all in spring. Ask anyone who’s run billing. |
That last one is not a joke — DST transitions have caused real duplicate-billing incidents.
Four approaches, in increasing order of robustness.
Designate one machine as the job runner.
✅ Trivially simple. ❌ Single point of failure. That machine dies, jobs stop. And it’s a snowflake server that doesn’t fit an autoscaled, immutable-infrastructure world.
Fine for small systems. Say so honestly rather than pretending it’s wrong — but pair it with an alert if the job doesn’t report completion.
Every instance tries the job; only the lock holder proceeds.
def run_nightly_invoices():
lock = redis.set("lock:invoices:2026-07-22", instance_id, nx=True, ex=7200)
if not lock:
return # someone else is doing it
try:
generate_invoices()
finally:
redis.delete("lock:invoices:2026-07-22")
✅ No special host; any instance can run it. ❌ 🚨 Distributed locks are unsafe on their own. A GC pause longer than the TTL means the lock expires while the holder is still working, a second instance starts, and you generate invoices twice. → Distributed Locking
The mitigation that actually matters: make the job idempotent. Then a duplicate run is harmless and the lock is an optimization rather than a correctness requirement.
# Idempotent: keyed on a stable business identifier
INSERT INTO invoices (customer_id, period, amount)
VALUES (42, '2026-07', 1500)
ON CONFLICT (customer_id, period) DO NOTHING;
🎙️ “I’d use a lock to avoid duplicate work, but I’d rely on idempotency for correctness — the lock can fail and I don’t want that to double-charge anyone.”
The pattern most production systems use, and the right answer in interviews.
flowchart LR
S[Scheduler<br/>leader-elected, HA] -->|enqueues one message| Q[[Queue]]
Q --> W1[Worker 1]
Q --> W2[Worker 2]
Q --> W3[Worker 3]
The scheduler is a small, highly-available service whose only job is deciding when. It enqueues one message. Workers — a normal autoscaled fleet — do the actual work.
✅ Exactly-once scheduling (one message enqueued), with the queue giving you retries, DLQ, and concurrency control for free. ✅ Workers scale independently and are stateless. ✅ Full visibility — queue depth, processing time, failures.
The scheduler itself is made HA via leader election: several instances run, one is leader, the others stand by.
Let someone else run it: AWS EventBridge Scheduler, Google Cloud Scheduler, Kubernetes CronJobs, Temporal, Airflow.
✅ Highly available, monitored, and someone else is on call for it. ❌ Cost, and less control.
🚨 Kubernetes CronJobs are the common choice and worth knowing precisely. Note the semantics:
they guarantee at-least-once, not exactly-once — a CronJob can create two Jobs under certain
failure conditions. concurrencyPolicy: Forbid prevents overlapping runs, and startingDeadlineSeconds
controls what happens after a missed schedule. Read these settings before relying on them.
Once a job is enqueued, the queue handles the hard parts.
Retries with exponential backoff and jitter. A failed job retries at 1s, 2s, 4s, 8s, then goes to a dead-letter queue. Jitter prevents synchronized retry storms. → Retries & Timeouts
Priority queues. Password reset emails should not wait behind a 100,000-item marketing batch. Either separate queues per priority (simple, and you control worker allocation), or a priority field (more flexible, risks starving low-priority work).
🚨 Separate queues by latency requirement, not just priority. The classic failure: one queue holds both “send OTP” (must be instant) and “generate monthly report” (takes 20 minutes). One long job blocks the urgent one. Separate queues, separate worker pools.
Concurrency limits. 500 workers all calling a third-party API that allows 10 requests/second will get you rate limited and possibly banned. Limit concurrency per job type, not just globally.
Scheduled/delayed jobs. “Run this in 3 days” — a reminder, a trial expiry, an abandoned-cart
email. Implemented with a sorted set (Redis ZADD with a timestamp score) or a database table
polled by due time.
Long-running jobs. A job taking 2 hours needs: heartbeating so the queue doesn’t consider it dead and redeliver it; checkpointing so a failure at 90 minutes doesn’t restart from zero; and progress reporting so someone can see it’s alive.
Tools: Sidekiq (Ruby), Celery (Python), BullMQ (Node), Que/GoodJob (Postgres-backed), Temporal (durable workflows), Quartz (Java), Hangfire (.NET).
🚨 A Postgres-backed job queue is underrated. SELECT ... FOR UPDATE SKIP LOCKED gives you a
correct, transactional job queue in the database you already run — and jobs can be enqueued in the
same transaction as your business data, which eliminates the
dual-write problem entirely. For anything under a
few thousand jobs/second, this is often the right choice.
-- Atomically claim a job without blocking other workers
SELECT * FROM jobs
WHERE status = 'pending' AND run_at <= now()
ORDER BY priority DESC, run_at
FOR UPDATE SKIP LOCKED
LIMIT 1;
Some work is a sequence with dependencies, retries at each stage, and compensation on failure:
Order placed
→ charge card (retry 3×; on permanent failure, cancel the order)
→ reserve inventory (if this fails, refund the card)
→ notify warehouse
→ wait for shipment confirmation (could be days)
→ send tracking email
You can build this with chained queue messages, but you’ll reinvent state tracking, timeouts, compensation, and visibility — badly.
Workflow engines (Temporal, AWS Step Functions, Airflow, Cadence) handle it properly: durable execution that survives process restarts, automatic retries per step, compensation on failure, and long waits measured in days.
🚨 Temporal’s model is worth understanding because it comes up: you write ordinary sequential code, and the engine records every step’s result. If the process dies, execution resumes from where it left off by replaying the recorded history. Waiting three days for a webhook is just a line of code.
Airflow vs Temporal — a distinction interviewers sometimes probe: Airflow is DAG-oriented and built for scheduled data pipelines (ETL, batch analytics). Temporal is code-oriented and built for long-running business workflows with complex state. Different problems.
→ Saga Pattern — the compensation model behind this
Make them idempotent. 🚨 The single most important rule. Jobs will run twice — retries, duplicate delivery, a lock that expired, a human re-running it manually. Key every side effect on a stable business identifier. → Idempotency
Make them resumable. A job processing 10 million rows that fails at row 9 million should not start over. Checkpoint progress:
last_id = load_checkpoint(job_id) or 0
for batch in fetch_batches(after=last_id):
process(batch)
save_checkpoint(job_id, batch.last_id) # commit progress as you go
Make them observable. Every job should emit: started, completed, duration, items processed, and errors. 🚨 And alert when a job doesn’t run. A silent failure is worse than a loud one — you find out from an angry customer three days later. A “dead man’s switch” (alert if no completion signal within N hours) catches this.
Bound them. Batch limits, timeouts, and memory ceilings. A job that loads 50 million rows into memory will find the OOM killer.
Handle overlap explicitly. If a job can take longer than its interval, decide: skip this run,
queue it, or run concurrently. Kubernetes calls this concurrencyPolicy; make the choice
deliberately rather than discovering it in production.
Timezone discipline. 🚨 Schedule in UTC and convert for display. “Every day at 2 a.m. local time” across DST transitions means the job runs twice one night and zero times another. If business logic genuinely requires local time (regulatory reporting deadlines), handle the DST cases explicitly and make the job idempotent so the duplicate run is harmless.
A related decision worth knowing:
| Batch | Streaming | |
|---|---|---|
| Runs | On a schedule | Continuously |
| Latency | Hours | Seconds |
| Efficiency | High (bulk operations) | Lower per item |
| Complexity | Low | Higher |
| Reprocessing | Easy — rerun it | Harder — need replay |
| Use for | Reports, billing, ETL, ML training | Fraud detection, live dashboards, alerts |
🎙️ The pragmatic answer: “I’d start with a batch job every 15 minutes. If the business needs sub-minute latency we can move to streaming, but batch is dramatically simpler to build, test, and reprocess when something goes wrong.”
Many “we need real-time” requirements are satisfied by a job running every minute. → Batch vs Stream
| Decision | Gain | Cost |
|---|---|---|
| Dedicated cron host | Simple | SPOF; doesn’t fit autoscaling |
| Distributed lock | Any instance can run it | Locks are unsafe alone; needs idempotency anyway |
| Scheduler + queue | HA, retries, visibility, independent scaling | More components |
| Managed scheduler | No ops burden | Cost, less control |
| DB-backed queue | One less system; transactional enqueue | Throughput ceiling; polling load |
| Workflow engine | Durable multi-step execution | Real learning curve; another platform |
| Batch over streaming | Simpler, cheaper, easy to rerun | Higher latency |
1. Cause the duplicate-run problem. Run three instances of a “job” that inserts a row. Watch three rows appear. Then add a Redis lock and watch it become one. Then set the lock TTL to 2 seconds with a job that takes 5, and watch duplicates return — that’s the GC-pause failure mode, reproduced deliberately.
2. Build a Postgres job queue. It’s about 50 lines and genuinely useful:
CREATE TABLE jobs (
id BIGSERIAL PRIMARY KEY,
type TEXT NOT NULL,
payload JSONB NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
attempts INT NOT NULL DEFAULT 0,
run_at TIMESTAMPTZ NOT NULL DEFAULT now(),
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_jobs_pending ON jobs (run_at) WHERE status = 'pending';
Note the partial index — it only covers pending jobs, so it stays tiny even as
the table grows. Then implement claiming with FOR UPDATE SKIP LOCKED and run several workers
concurrently. Confirm no job is processed twice.
3. Make it idempotent, then prove it. Add a unique constraint on a business key and run the same job twice. Confirm the second run is a no-op rather than a duplicate.
4. Try a Kubernetes CronJob with concurrencyPolicy: Forbid and a job that runs longer than its
schedule. Watch runs get skipped. Then switch to Allow and watch them overlap.