System Design

Job Scheduler

Design a distributed job scheduler like Quartz or Airflow's scheduler.

Functional requirements

• Tenants create jobs with either a cron expression in a chosen time zone (e.g., every day at 02:00 America/Chicago) or a one-off run time, plus a payload and a target to invoke (an HTTP endpoint or a worker queue).
• Each due run is executed exactly once: never skipped while the cluster is healthy, and never fired twice because two scheduler nodes both saw it come due.
• Runs carry a per-job timeout, a retry policy (attempts and backoff), and a priority; tenants can pause, resume, or delete a job and cancel an in-flight run.
• Tenants can query a job's run history (scheduled time, start, end, status, attempt count, error) and see its upcoming runs.
• After scheduler downtime, missed runs follow the job's catch-up policy: skip them, run once, or run every missed occurrence.

Non-functional requirements

• Timeliness: a due run is handed to a worker within 1 second of its scheduled time at p99 under normal load, and no run ever starts before its scheduled time.
• Claiming a run is the strongly consistent operation: the decision that a run is due and owned must be agreed across scheduler nodes, while run history, dashboards, and upcoming-run views may lag by a few seconds.
• Job definitions and run outcomes are durable: an acknowledged create, update, or recorded outcome is never lost, and a scheduler node crashing mid-cycle loses no runs.
• No single scheduler or worker node is a single point of failure; losing any node delays its runs by at most a lease length (seconds), never loses them.
• Tenant isolation: one tenant's 400,000 jobs or a runaway retry storm must not delay other tenants' runs, and tenants cannot see or cancel each other's jobs.

Scaling & constraints

• 5 million active job definitions across 20,000 tenants; the largest tenant owns 400,000 jobs, the median tenant owns 10.
• ~100 million runs per day (~1,200/sec average) with a top-of-the-hour spike of ~50,000 runs due in the same second, since most cron expressions align to :00.
• Job definitions are ~1 KB; run records are ~500 bytes; run history is kept for 90 days (~9 billion rows, ~4.5 TB).
• API read:write ratio is ~10:1 (history and upcoming-run reads vs. job writes); ~50,000 job creations and updates per day.
• Run durations: p50 of 2 seconds, p99 of 5 minutes, maximum timeout 1 hour; ~3% of runs fail on the first attempt and are retried.
• Scheduler nodes restart a few times per day; a full scheduler outage of up to 2 hours must be survivable with catch-up afterward.

Out of scope

• Job dependency graphs and DAG orchestration.
• The worker execution sandbox, container images, and resource limits.
• The underlying message queue itself (assume a durable queue like Kafka or SQS).
• Tenant billing, plan quotas, and authentication.