Security Notes
System Design

Design a Job Scheduler

5 min read 7 sections version 3 ยท updated 2026-10-11

DifficultyHard | HelloInterview: problem breakdown


Problem Statement

Design a distributed job scheduler that executes tasks at scheduled times (cron-style) or on demand. Think AWS Lambda Scheduled Events, GitHub Actions cron, or Airflow for DAG workflows.

๐Ÿ“–

Real-world: The deep lesson of a job scheduler is why "exactly-once" is a myth and what to do about it. Distributed systems can't truly guarantee exactly-once execution โ€” a worker can run a job and then crash before recording that it finished, so the system can't tell "done" from "never ran." The honest design is at-least-once delivery + idempotent jobs: accept that a job may run twice and make running it twice harmless (via an idempotency key), because the alternative โ€” at-most-once โ€” risks silently dropping work. Workers claim jobs with a lease/heartbeat so a dead worker's job gets reclaimed and retried with backoff. This is also a textbook thundering-herd trap: if a million jobs are all scheduled for midnight, they stampede at once โ€” you add jitter. Security relevance for this audience: a scheduler is a remote-code-execution engine by design, so it's a juicy target โ€” the same class of system as a CI runner, which is why least-privilege execution, isolation between tenants' jobs, and audit logging of who scheduled what are core, not optional. (Cron abuse is also a classic Linux privesc vector โ€” see ../linux/privilege-escalation.md.)


Requirements

Functional

  • Schedule jobs: one-time (at timestamp T) and recurring (cron expression)
  • Execute jobs by triggering a user-defined handler (HTTP callback, container run, function)
  • DAG support: job B runs only after job A completes
  • Job status tracking (pending, running, succeeded, failed)
  • Retry on failure with configurable policy

Non-Functional

  • At-least-once execution (jobs must not be skipped)
  • At-most-once for idempotent best-effort jobs (configurable)
  • Horizontal scaling: many workers
  • Fault tolerance: scheduler node failure should not drop jobs
  • Scale: 100M scheduled jobs, 1M executions/day

Core Design

The Timing Problem

A scheduler must fire jobs "at the right time." Naively: poll DB for jobs where next_run_at <= NOW(). At 1M jobs, this query runs every N seconds โ€” at scale, expensive.

Betteruse a priority queue (min-heap) ordered by next_run_at. Pop jobs whose time has arrived.

But in a distributed system, this heap must survive node failure. Solution: store it in a database or Redis sorted set.

Redis sorted set:
  ZADD scheduled_jobs <next_run_at_unix_epoch> <job_id>
  
  Scheduler loop (every 1s):
    jobs = ZRANGEBYSCORE scheduled_jobs 0 <now()> LIMIT 100
    for job in jobs:
      if ZREM scheduled_jobs job_id:  # atomic dequeue
        enqueue to execution queue

ZREM returning 1 means this scheduler instance claimed the job โ€” no double-execution even with multiple scheduler nodes.

Execution Queue

Dispatched jobs go into an execution queue (Kafka / SQS). Workers pull jobs and execute them.

Scheduler โ†’ Kafka (execution_queue) โ†’ Workers
Workers:
  - Pull job from queue
  - Acquire distributed lock (job_id โ†’ Redis lock, 5-min TTL)
  - Execute handler
  - Update job status in DB
  - Release lock
  - If recurring: compute next_run_at, re-add to scheduled_jobs sorted set

At-Least-Once vs Exactly-Once

At-least-once

job may execute multiple times if worker crashes after execution but before ACK. Handler must be idempotent.

Exactly-once

distributed lock + idempotency key in handler. Much harder to guarantee end-to-end.

Design choice: at-least-once with idempotency key passed to every execution. User's handler is responsible for idempotency.

DAG Jobs

Dependencies between jobs:

Job B depends on Job A:
  - Job B's status: WAITING_FOR_DEPENDENCIES
  - When Job A completes: event published to Kafka
  - Dependency resolver consumer: check if all deps of B are done
    - If yes: update B to PENDING, add to scheduled_jobs

Architecture

API Service โ†’ Job DB (Postgres)
                  โ†“
           โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
           โ”‚   Scheduler  โ”‚ โ† reads scheduled_jobs sorted set
           โ”‚   (leader)   โ”‚   dispatches to queue
           โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                  โ†“
           Execution Queue (Kafka)
                  โ†“
          Worker Pool (auto-scaling)
                  โ†“
    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚ Execute handler (HTTP / Lambda)โ”‚
    โ”‚ Update job status in DB        โ”‚
    โ”‚ Publish completion event       โ”‚
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                  โ†“
         Dependency Resolver (Kafka consumer)
         โ†’ re-enqueues dependent jobs

Distributed Scheduler Leadership

Multiple scheduler nodes run. Only one should scan and dispatch at a time (to avoid duplicate dispatches). Use leader election:

  • Redis SETNX scheduler_leader <node_id> with TTL 30s; renew every 10s
  • If leader fails, TTL expires, another node becomes leader

Key Design Decisions

DecisionChoiceReason
Job storagePostgres + Redis sorted setDurable jobs in DB; fast time-based index in Redis
SchedulingRedis ZADD + ZRANGEBYSCOREO(log N) insert; O(k) range query
Concurrency controlRedis ZREM atomic popExactly-one scheduler claims the job
ExecutionKafka โ†’ worker poolDecoupled; auto-scaling; at-least-once
Leader electionRedis SETNX + TTLSimple; good enough for scheduler
RetryExponential backoff in DBWorker increments retry_count; scheduler re-enqueues

Security Considerations

ThreatMitigation
Privilege escalation via job contentJobs execute in sandboxed environments; no host-level access by default
SSRF via HTTP handler URLsValidate handler URLs against allowlist; block internal IP ranges
Secret injection in job argsStore secrets in Vault/Secrets Manager; pass reference, not value, in job definition
Scheduler poisoning (add malicious jobs)Auth on job creation API; job definitions signed and validated
Resource exhaustionJob resource limits (CPU/memory/timeout); quota per team
Log exfiltrationJob logs stored centrally, not on worker hosts; access controlled
๐Ÿ“ฐ

Real incident โ€” Kaseya VSA (2021). The REvil ransomware group exploited a zero-day in Kaseya VSA, a remote-management server that IT service providers use to run scripts and updates on their customers' computers. They used it to push ransomware out as if it were a routine update; about 60 providers and up to 1,500 downstream businesses were hit in a single afternoon. A system whose job is to run code on many machines is, in an attacker's hands, a deployment system for malware.

๐ŸŽฏ

On the job. Ask what identity each job runs as. Jobs should run with the permissions of the team that owns them, not one shared all-powerful scheduler account, and creating or editing a job definition should require the same review as a code change โ€” because it is one.


Interview Tips

Redis sorted set for the timing index

this is the canonical answer for "how do you efficiently find jobs due now?"

At-least-once delivery is the default

explain why exactly-once is hard and shift responsibility to idempotent handlers.

DAG support

describe the dependency resolver as a separate Kafka consumer. Don't conflate it with the main scheduler.

Scheduler leadership / single active scheduler

mention leader election to avoid duplicate dispatches without central coordination.