New Statistical Algorithms for the Analysis of Mass - FU Berlin, FB MI ...

5.4. QAD GRID WORKER 135
Checkpoints
Checkpoints are particular breakpoints in a program defined on the source code
level. At these points the execution of the worker’s core algorithms is interrupted
and control is given to a (implementation specific) checkpoint handler.
Obviously, it is the programmer’s responsibility to integrate enough checkpoints
at reasonable positions. Within this checkpoint method the following
actions should be performed:
1. Check the runtime of the process. Measurements have shown that it is
only advantageous with respect to the total execution time (“run slow”
vs. “migrate and run faster”) if a process runs at least 3-5 minutes.
2. Check the current health condition of the machine hosting this worker.
The key property used here is the workload of a client as described in
section 5.4.3. Tests have shown that a workload above 100ms, that is, it
takes 100ms for the worker process to get back control from the operating
system, means the worker process runs significantly slower that normal.
Therefore, we consider a worker’s host system as overloaded when the
workload exceeds 100ms.
3. If the health check shows no overload condition control is given back to
the core algorithm.
4. If the worker is overloaded it sends a signal to the server and requests to
be migrated. If the server does not answer within 10 seconds or refuses
the migration control is again given back to the core algorithm.
5. If the migration request is accepted migration is performed as described
above and the worker terminates itself.
With termination of the worker the checkpoint handling is finished.
Worker Persistence
To realize a worker’s migration the worker must be capable of saving its execution
state. This property is called persistence and is a two step procedure:
(1) getting the process’ state (variables, stack and the point of execution) and
(2) storing the state into a file that can be transmitted over a network. Sophisticate
(but operating system dependent) systems can interrupt a process
at arbitrary points. To retain OS independence we cannot use this technique
and require each worker to define its own checkpoints (see above).
The key design pattern in our QAD Grid workers is separation of concerns
(SoC) which means to break up a computer program into distinct features
that overlap in functionality as little as possible. A concern is any piece of
interest or focus in a program. In our case, what needs to be implemented is
the separation of data structures needed for the core algorithm (core data) and
data necessary for the peripheral program. Obviously, only the core data needs
to be stored and implanted into the target worker that is going to continue
from the point the source worker has suspended. Consequently, for the worker
to store (and eventually load) its core data each data element needs
a) to be registered in and accessible through a globally (but within this particular
worker instance) available list,