Reqflow
← All comparisons
Open Distributed Job Scheduler →Open Instagram Feed →

Distributed Job Scheduler vs Instagram Feed

Distributed Job Scheduler

Distributed Job Scheduler

Cron-at-scale: exactly-once execution, missed-run recovery, at-least-once delivery, priority queues.

Components (8)

  • Client / Admin
  • API Gateway
  • Scheduler Service
  • Job Queue
  • Worker Pool
  • Job Store
  • Run History
  • Dead Letter Queue

Headline numbers

  • Scheduler poll load~3,000 DB reads/sec
  • Workers needed at peak~830 concurrent workers
  • Run history write rate~5,000 writes/sec
Instagram Feed

Instagram Feed

Fan-out on write vs read, ranking, CDN.

Components (10)

  • Mobile Client
  • CDN
  • API Gateway
  • Feed Service
  • Post Service
  • Kafka
  • Fan-out Worker
  • Redis
  • Postgres
  • S3

Headline numbers

  • Posts / sec (avg)~3,000/sec
  • Feed read QPS~58,000/sec
  • Fan-out writes / sec~600,000/sec

Key differences

Only in Distributed Job Scheduler
None.
In both
  • Client
  • Api Gateway
  • Service
  • Queue
  • Database
Only in Instagram Feed
  • CDN
  • Redis
  • S3

Flow shape

Distributed Job Scheduler flows
  • Scheduled job fires5 steps
  • ⚠ Worker crashes mid-execution4 steps
  • ⚠ Scheduler leader fails3 steps
Instagram Feed flows
  • Post a photo (fan-out on write)7 steps
  • Open feed (precomputed timeline)5 steps
  • ⚠ Fan-out Worker is down7 steps
  • ⚠ Timeline cache flushed (Redis restart)4 steps