← All comparisons
Open Apache Kafka →Open Instagram Feed →

Apache Kafka vs Instagram Feed

Apache Kafka

Apache Kafka

Partitioned, replicated, append-only log. Producers, brokers, consumer groups, leader/follower replication.

Components (8)

  • Producer
  • Broker 1
  • Broker 2
  • Broker 3
  • Coordination Service
  • Consumer A1
  • Consumer A2
  • Consumer B1

Headline numbers

  • Cluster ingest~2.4 GB/sec
  • Cluster storage~1.5 PB
  • Max consumer parallelism per group100
Instagram Feed

Instagram Feed

Fan-out on write vs read, ranking, CDN.

Components (10)

  • Mobile Client
  • CDN
  • API Gateway
  • Feed Service
  • Post Service
  • Kafka
  • Fan-out Worker
  • Redis
  • Postgres
  • S3

Headline numbers

  • Posts / sec (avg)~3,000/sec
  • Feed read QPS~58,000/sec
  • Fan-out writes / sec~600,000/sec

Key differences

Only in Apache Kafka
None.
In both
  • Service
Only in Instagram Feed
  • Mobile Client
  • CDN
  • API Gateway
  • Kafka
  • Redis
  • Postgres
  • S3

Flow shape

Apache Kafka flows
  • Produce a message (acks=all)3 steps
  • Group A consumes both partitions in parallel3 steps
  • Group B reads independently of Group A2 steps
  • New consumer joins: group rebalance4 steps
  • ⚠ Partition leader fails (failover)4 steps
  • ⚠ Slow consumer triggers rebalance4 steps
Instagram Feed flows
  • Post a photo (fan-out on write)7 steps
  • Open feed (precomputed timeline)5 steps
  • ⚠ Fan-out Worker is down7 steps
  • ⚠ Timeline cache flushed (Redis restart)4 steps