Design a Job Scheduler

#Problem Statement

Design a distributed job scheduler that executes tasks at scheduled times (cron-style) or on demand. Think AWS Lambda Scheduled Events, GitHub Actions cron, or Airflow for DAG workflows.

📖

Real-world: The deep lesson of a job scheduler is why "exactly-once" is a myth and what to do about it. Distributed systems can't truly guarantee exactly-once execution — a worker can run a job and then crash before recording that it finished, so the system can't tell "done" from "never ran." The honest design is at-least-once delivery + idempotent jobs: accept that a job may run twice and make running it twice harmless (via an idempotency key), because the alternative — at-most-once — risks silently dropping work. Workers claim jobs with a lease/heartbeat so a dead worker's job gets reclaimed and retried with backoff. This is also a textbook thundering-herd trap: if a million jobs are all scheduled for midnight, they stampede at once — you add jitter. Security relevance for this audience: a scheduler is a remote-code-execution engine by design, so it's a juicy target — the same class of system as a CI runner, which is why least-privilege execution, isolation between tenants' jobs, and audit logging of who scheduled what are core, not optional. (Cron abuse is also a classic Linux privesc vector — see ../linux/privilege-escalation.md.)

#Requirements

#Functional

• Schedule jobs: one-time (at timestamp T) and recurring (cron expression)
• Execute jobs by triggering a user-defined handler (HTTP callback, container run, function)
• DAG support: job B runs only after job A completes
• Job status tracking (pending, running, succeeded, failed)
• Retry on failure with configurable policy

#Non-Functional

• At-least-once execution (jobs must not be skipped)
• At-most-once for idempotent best-effort jobs (configurable)
• Horizontal scaling: many workers
• Fault tolerance: scheduler node failure should not drop jobs
• Scale: 100M scheduled jobs, 1M executions/day

#Core Design

#The Timing Problem

A scheduler must fire jobs "at the right time." Naively: poll DB for jobs where next_run_at <= NOW(). At 1M jobs, this query runs every N seconds — at scale, expensive.

Betteruse a priority queue (min-heap) ordered by next_run_at. Pop jobs whose time has arrived.

But in a distributed system, this heap must survive node failure. Solution: store it in a database or Redis sorted set.

Redis sorted set:
  ZADD scheduled_jobs <next_run_at_unix_epoch> <job_id>
  
  Scheduler loop (every 1s):
    jobs = ZRANGEBYSCORE scheduled_jobs 0 <now()> LIMIT 100
    for job in jobs:
      if ZREM scheduled_jobs job_id:  # atomic dequeue
        enqueue to execution queue

ZREM returning 1 means this scheduler instance claimed the job — no double-execution even with multiple scheduler nodes.

#Execution Queue

Dispatched jobs go into an execution queue (Kafka / SQS). Workers pull jobs and execute them.

Scheduler → Kafka (execution_queue) → Workers
Workers:
  - Pull job from queue
  - Acquire distributed lock (job_id → Redis lock, 5-min TTL)
  - Execute handler
  - Update job status in DB
  - Release lock
  - If recurring: compute next_run_at, re-add to scheduled_jobs sorted set

#At-Least-Once vs Exactly-Once

Design choice: at-least-once with idempotency key passed to every execution. User's handler is responsible for idempotency.

#DAG Jobs

Dependencies between jobs:

Job B depends on Job A:
  - Job B's status: WAITING_FOR_DEPENDENCIES
  - When Job A completes: event published to Kafka
  - Dependency resolver consumer: check if all deps of B are done
    - If yes: update B to PENDING, add to scheduled_jobs

#Architecture

API Service → Job DB (Postgres)
                  ↓
           ┌──────────────┐
           │   Scheduler  │ ← reads scheduled_jobs sorted set
           │   (leader)   │   dispatches to queue
           └──────────────┘
                  ↓
           Execution Queue (Kafka)
                  ↓
          Worker Pool (auto-scaling)
                  ↓
    ┌────────────────────────────────┐
    │ Execute handler (HTTP / Lambda)│
    │ Update job status in DB        │
    │ Publish completion event       │
    └────────────────────────────────┘
                  ↓
         Dependency Resolver (Kafka consumer)
         → re-enqueues dependent jobs

#Distributed Scheduler Leadership

Multiple scheduler nodes run. Only one should scan and dispatch at a time (to avoid duplicate dispatches). Use leader election:

• Redis SETNX scheduler_leader <node_id> with TTL 30s; renew every 10s
• If leader fails, TTL expires, another node becomes leader

#Key Design Decisions

#Security Considerations

#Interview Tips