Worldwide Repls, part 2: Load balancing for fun (although not quite profit)

Replit developed a custom load balancer to address issues with Google Cloud Load Balancer, which caused uneven load distribution and suboptimal container placement across its fleet.

The new system uses subsetting to select a subset of healthy machines and then applies a policy (random or round-robin) to route requests, improving container count variance between median and 99th percentile.

The load balancer was built after a proof-of-concept and simulator validated the approach, and it handles connections with load differences up to 6 or 7 orders of magnitude.