Load Balancer

A load balancer is a component that sits in front of multiple backend server instances and distributes incoming requests across them, so no single server is overwhelmed and the system as a whole can handle more traffic than any one server could alone. Beyond raw traffic distribution, load balancers provide health checking (automatically routing traffic away from a server instance that's failing or unresponsive) and are a core enabler of horizontal scaling and zero-downtime deployments (blue-green and rolling deployments both rely on a load balancer's ability to shift traffic between server groups). Examples range from managed cloud offerings (AWS Application Load Balancer, Google Cloud Load Balancing) to open-source software load balancers (NGINX, HAProxy, Envoy). Why it matters for AI/SaaS builders: a load balancer is what makes horizontal scaling actually work in practice — instead of trying to make one increasingly large server handle more traffic (which has a hard physical ceiling), you run multiple smaller, identical server instances behind a load balancer and add more instances as traffic grows. It's also the mechanism that gives a system resilience against individual server failures: if one instance crashes or becomes unhealthy, the load balancer's health checks detect it and stop routing traffic there, so users experience no disruption as long as other healthy instances remain. How it works: a load balancer receives every incoming request first and decides which backend instance should handle it, using a distribution algorithm — round-robin (cycling evenly through instances), least-connections (routing to whichever instance currently has the fewest active requests), or IP-hash (consistently routing a given client to the same instance, useful for session affinity). It continuously runs health checks against each backend instance (typically a lightweight HTTP request to a `/health` endpoint) and automatically stops routing to any instance that fails those checks, resuming once it recovers. Worked example: a SaaS API runs on four identical backend server instances behind an AWS Application Load Balancer. During a traffic spike, the load balancer distributes incoming requests roughly evenly across all four instances using a round-robin algorithm, keeping each instance's load manageable rather than one server getting overwhelmed. Ten minutes later, one instance develops a memory leak and starts failing its health check (`GET /health` stops returning `200 OK`); the load balancer detects this within seconds, automatically stops sending it any new traffic, and continues serving all users seamlessly from the remaining three healthy instances — while the unhealthy instance is automatically replaced by the auto-scaling group, with zero manual intervention and no customer-visible downtime.