New Statistical Algorithms for the Analysis of Mass - FU Berlin, FB MI ...

5.3. QAD GRID PLATFORM SERVER 125
Figure 5.3.8: Example for a hierarchical clustering (HC): the nodes shown on the
left are clustered by HC (single linkage) using Euclidean distances. The resulting
dendrogram (right) shows the results.
Out of this list of all workers being reliable and fast enough a distance matrix
is created. These geographical distance between two workers is calculated
using the haversine formula (Sinnott, 1984). This computes the great-circle
distances between two points on a sphere given their longitudes and latitudes
which is particularly well-conditioned even at very small distances. On this
distance matrix a (single linkage) hierarchical clustering (Johnson, 1967) is
performed. Searching from the top in the resulting dendrogram (see Figure
5.8(b)) that level is sought for that maximizes the number of clusters but has
at most c clusters. c was set a-priori to 10% of the number of workers. What
also could have been done is to use a technique known as multi dimensional
scaling (MDS) (Shepard, 1962) to recover the original Euclidean coordinates
and find clusters on the resulting map.
Within each clusters found the most reliable and fastest node is then selected
for mirroring.
Selecting Data to be Mirrored
Not all data in the Grid system is used all the time: if analyses have been
finished on a particular dataset it might never be used again. On the other
hand a particular (presumably) quite recent dataset will be analyzed on many
machines at the same time and long delays can occur if data is copied from a
single source. Therefore, it does not make sense to mirror all datasets across
the Grid but for some data it is extremely interesting to make it highlyavailable
during load spikes. Thus, the system has to select datasets that are
to be distributed (mirrored) across the Grid. The actual selection happens
in rounds, that is, each time the distribution process is started datasets of a
maximum of 5GB are selected. The data selection algorithm is organized in
two stages and works as follows:
Server Stage: At this stage data is selected from the central server to be
copied to the workers using the last in, first out principle. This means,
first all datasets are selected that are not currently distributed within
the Grid more than three times. The resulting list is then sorted by time
and date they were added to the system. Starting from the most recent
item datasets are selected up to a (total) maximum filesize of 5GByte.
P2P Stage: At this stage data is distributed directly between workers. The
procedure is similar to the server stage, but this time five copies of
the most recent datasets are created and distributed to the workers.
Tests have shown that five seems to be the minimum number to avoid
bottlenecks in our testbed.