Load Balancing

A component that sits in front of a pool of backend servers and distributes incoming requests across them. The pool looks like one big server to clients; behind the scenes, the load balancer picks which actual instance handles each request.

Any service with more traffic than a single box can handle needs to spread that traffic somehow. Without a load balancer, you'd hard-code multiple endpoints into clients (terrible for ops) or rely on DNS round-robin (poor failure handling, no real-time adjustment).

Common strategies: round-robin (next-in-line), least-connections (newest request to the box with fewest open connections), weighted (bigger boxes get more), consistent hashing (sticky routing for sessions or caches). The LB also health-checks each backend and routes around failures.

• Horizontal scaling: add boxes, traffic spreads automatically
• Failure tolerance: sick boxes get pulled from rotation
• Single endpoint for clients; backends are abstracted away

• LB itself is a SPOF unless run as a pair / fleet
• Sticky sessions complicate scaling and are best avoided when possible
• Layer 4 (TCP) vs Layer 7 (HTTP) is a real tradeoff in flexibility vs performance

• API Rate Limiter →

  Edge proxy is essentially a load balancer with a rate-limit hook

• Uber →

  Implicit in front of every service tier, where Dispatch is horizontally scaled

• Stateless backends are dramatically easier to load-balance. If your backend has per-instance state, you've imported all the problems sticky sessions cause.
• Layer 4 LB is faster but can't make routing decisions based on URL/headers. Layer 7 (Envoy, Nginx) is slower but gives you path-based routing, header rewrites, etc.
• Health checks should hit a real endpoint that exercises the dependencies. A '/health' that always returns 200 will happily route traffic to a box that can't reach the database.