AI Dictionary › AI Fundamentals
A load balancer is a component that distributes incoming traffic across multiple servers so that none is overloaded while others sit idle. It acts like a traffic officer at an intersection: it receives every request and routes it to the most suitable destination. It is what allows a service to scale horizontally by adding machines rather than beefing up a single one.
The balancer receives each request and picks a target server according to an algorithm: round-robin (taking turns), least connections (the least busy server), or weights proportional to capacity. Meanwhile it runs periodic health checks: if a server stops responding, it is removed from rotation until it recovers. Balancers can operate at the network level (L4) or understand the application protocol (L7), routing for example by URL.
In AI, load balancers spread inference requests across GPU fleets, and LLM gateways apply the same principle by routing calls among different providers and models based on cost, latency and available quota. Beyond AI, they hold up every major website, from e-commerce platforms to streaming services.
The term comes from the analogy with balancing physical loads. Dedicated balancers emerged with the growth of the commercial web in the mid-1990s: the Cisco LocalDirector (1996) and the early products of F5 Networks, founded in 1996, are among the category's forerunners. With the cloud, load balancing became a managed service, available with one click from every major provider.
Grace's weekly leagues spread the competition across users of similar level, a balancing principle you find in identical form in AI infrastructure. The professional scenarios on AI architectures prepare you to talk with technical teams using the right vocabulary.
From our network
AGORÀ Intelligence: Enterprise AI Governance Platform
Govern AI at scale: policies, adoption and measurable results on your data. Built for boards and C-suite.
Visit agora-intelligence.com →From the Agora Intelligence blog
📱 Download the Android app (beta) iOS coming soon
Say what you mean. Get what you need.
Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.