Erlang and OTP in Action.pdf - Synrc

13
processes within a certain period of time before it ought to give up, and more. All you need
to do is to provide some parameters and hooks.
But a system should not be structured as just a single-level hierarchy of supervisors and
workers. In any complex system, you will want a supervision tree, with multiple layers, that
allows subsystems to be restarted at different levels.
1.2.3 – Layering processes for fault tolerance
Layering brings related subsystems together under a common supervisor. More importantly,
it defines different levels of working base states that we can revert to. In the diagram below,
you can see that there are two distinct groups of worker processes, A and B, supervised
separately from one another. These two groups and their supervisors together form a larger
group C, under yet another supervisor higher up in the tree.
Figure illustrating a layered system of supervisors and workers
Let’s assume that the processes in group A work together to produce a stream of data
that group B consumes. Group B is however not required for group A to function. Just to
make things concrete, let’s say group A is processing and encoding multimedia data, while
group B presents it. Let’s further suppose that a small percent of the data entering group A
is corrupt in some way not predicted at the time the application was written.
This malformed data causes a process within group A to malfunction. Following the “let it
crash” philosophy, that process dies immediately without so much as trying to untangle the
mess, and because processes are isolated, none of the other processes have been affected
by the bad input. The supervisor, detecting that a process has died, restores the base state
we prescribed for group A, and the system picks up from a known point. The beauty of this is
that group B, the presentation system, has no idea that this is going on, and really does not
care. So long as group A pushes enough good data to group B for the latter to display
something of acceptable quality to the user, we have a successful system.
By isolating independent parts of our system and organizing them into a supervision tree,
we can create little subsystems that can be individually restarted, in fractions of a second, to
keep our system chugging along even in the face of unpredicted errors. If group A fails to
restart properly, its supervisor might eventually give up and escalate the problem to the
supervisor of the entire group C, which might then in a case like this decide to shut down B
as well and call it a day. If you imagine that our system is in fact running hundreds of
©Manning Publications Co. Please post comments or corrections to the Author Online forum:
http://www.manning-sandbox.com/forum.jspaforumID=454