system-design

Practice Problem: Design an Online Judge (LeetCode / HackerRank)

Prompt: Design a system where users submit code solutions to programming problems; the system runs the code against test cases in a sandbox, and returns pass/fail with time/memory usage. Attempt cold for 45 minutes first.

Tests: running untrusted code safely at scale, async job execution, resource limiting. Shares its core with the CI/CD platform.

Solution outline

1. Requirements

Functional: submit code (language + source) for a problem; run it against hidden test cases; return verdict (Accepted / Wrong Answer / TLE / Runtime Error) + time/memory; show submission history.
Non-functional: 🚨 secure isolation — user code is arbitrary and hostile (infinite loops, fork bombs, filesystem/network abuse, attempts to read other users’ data). Resource-limited (CPU time, memory, no network). Scalable (contest spikes: thousands submit at once). Reasonably fast feedback (seconds).
Out of scope: the problem-authoring UI, plagiarism detection.

2. Estimation

Normal load modest, but contests cause huge spikes — thousands of submissions in the first minutes → bursty job load. Each submission runs code against many test cases (CPU-bound, seconds each).

3. High-level design — async sandboxed execution

🚨 Submission is asynchronous (don’t run code inline on the web request):

flowchart LR
    User -->|submit| API[Submission API]
    API --> Q[[Judge queue]]
    API --> DB[(Submission: PENDING)]
    Q --> Judge[Judge workers<br/>sandboxed executors]
    Judge -->|run vs test cases<br/>resource-limited| Sandbox[Isolated container/VM]
    Judge --> DB2[(Verdict + time/mem)]
    DB2 -.notify.-> User

API accepts the submission, stores it PENDING, enqueues a judge job, returns immediately. Judge workers pull jobs and execute the code in an isolated sandbox against the test cases, then record the verdict. Results push back to the user (poll/WebSocket).

4. Deep dives — the security core

🚨 Running untrusted code safely is the whole problem (same as CI/CD):

Isolation: each run in a fresh, ephemeral container or microVM (gVisor/Firecracker for stronger isolation), destroyed after — no state leaks, a compromised run is thrown away.
Resource limits: hard CPU-time limit (→ TLE), memory limit (cgroups), no network, read-only/empty filesystem, process/thread caps (stop fork bombs), wall-clock timeout. Kill and report if exceeded.
Least privilege: the sandbox runs as an unprivileged user with no access to the host or other submissions’ data. (Deployment & Infra)

Other deep dives:

Judging: compile (if needed, sandboxed too), run against each test case, compare output (exact, or a special checker for multiple valid answers), aggregate the verdict (first failing case, or all).
Scaling for contests: autoscale the judge worker fleet on queue depth; fair scheduling so one user’s many submissions don’t starve others; the queue absorbs the spike.
Consistency/fairness: deterministic time limits (pin CPU, avoid noisy-neighbor jitter) so verdicts are fair and reproducible — a borderline TLE shouldn’t depend on which busy machine ran it.

5. Trade-offs

6. What a strong answer includes

Async job execution with a queue, and — the crux — ephemeral sandboxed execution with hard resource limits and no network/least privilege for untrusted code, plus autoscaling and fair scheduling for contest spikes, and deterministic time limits for fair verdicts. A weak answer runs submitted code directly on a server with no isolation — a catastrophic security hole.