Foundations of Python Network Programming 978-1-4302-3004-5

CHAPTER 7 ■ SERVER ARCHITECTURE
Load Balancing and Proxies
Event-driven servers take a single process and thread of control and make it serve as many clients as it
possibly can; once every moment of its time is being spent on clients that are ready for data, a process
really can do no more. But what if one thread of control is simply not enough for the load your network
service needs to meet?
The answer, obviously, is to run several instances of your service and to distribute clients among
them. This requires a key piece of software: a load balancer that runs on the port to which all of the
clients will be connecting, and which then turns around and gives each of the running instances of your
service the data being sent by some fraction of the incoming clients. The load balancer thus serves as a
proxy: to network clients it looks like your server, but to your server it looks like a client, and often
neither side knows the proxy is even there.
Load balancers are such critical pieces of infrastructure that they are often built directly into
network hardware, like that sold by Cisco, Barracuda, and f5. On a normal Linux system, you can run
software like HAProxy or delve into the operating system’s firewall rules and construct quite efficient
load balancing using the Linux Virtual Server (LVS) subsystem.
In the old days, it was common to spread load by simply giving a single domain name several
different IP addresses; clients looking up the name would be spread randomly across the various server
machines. The problem with this, of course, is that clients suffer when the server to which they are
assigned goes down; modern load balancers, by contrast, can often recover when a back-end server goes
down by moving its live connections over to another server without the client even knowing.
The one area in which DNS has retained its foothold as a load-balancing mechanism is geography.
The largest service providers on the Internet often resolve hostnames to different IP addresses
depending on the continent, country, and region from which a particular client request originates. This
allows them to direct traffic to server rooms that are within a few hundred miles of each customer, rather
than requiring their connections to cross the long and busy data links between continents.
So why am I mentioning all of these possibilities before tackling the ways that you can move beyond
a single thread of control on a single machine with threads and processes?
The answer is that I believe load balancing should be considered up front in the design of any
network service because it is the only approach that really scales. True, you can buy servers these days of
more than a dozen cores, mounted in machines that support massive network channels; but if,
someday, your service finally outgrows a single box, then you will wind up doing load balancing. And if
load balancing can help you distribute load between entirely different machines, why not also use it to
help you keep several copies of your server active on the same machine?
Threading and forking, it turns out, are merely limited special cases of load balancing. They take
advantage of the fact that the operating system will load-balance incoming connections among all of the
threads or processes that are running accept() against a particular socket. But if you are going to have to
run a separate load balancer in front of your service anyway, then why go to the trouble of threading or
forking on each individual machine? Why not just run 20 copies of your simple single-threaded server on
20 different ports, and then list them in the load balancer’s configuration?
Of course, you might know ahead of time that your service will never expand to run on several
machines, and might want the simplicity of running a single piece of software that can by itself use
several processor cores effectively to answer client requests. But you should keep in mind that a multithreaded
or multi-process application is, within a single piece of software, doing what might more
cleanly be done by configuring a proxy standing outside your server code.
Threading and Multi-processing
The essential idea of a threaded or multi-process server is that we take the simple and straightforward
server that we started out with—the one way back in Listing 7–2, the one that waits repeatedly on a
117